How do I scrape page content with a generator? - jekyll

Question

I have a blog where each post has a bunch of images and videos. I would like to be able to tag each of them with some keywords, and then populate a set of pages based on the tags. For example going to /photos/car/ would list all images tagged with car.
I include images and videos using a simple plugin right now, that essentially only has a render function. I figure I could add the tags there.
But how can I make Jekyll 'scrape' my pages and populate pages with images?

Answer 1

Short answer is : this is tricky.
But why ?
A custom imgWithTags-tag liquid tag like {% img cookie-monster.png tag2,tag3 %} can do two things :
render the html tag
store tag -> image associations in a site.taggedImage variable
As this tag can be used on any post, page or collection document, the site.taggedImage variable will only be complete after the render process.
Once the render process is finished, we can grab the :site, :post_render hook to process our datas and create Tags pages.
But here, we can no longer rely on things like {% for p in tagPages %}...{% endfor %} to automatically generate links to our pages : rendering is finished.
The trick can be to maintain a link data file by hand, in order to be able to generate links with loops like this {% for p in site.data.tagPages %}...{% endfor %}
Let's give it a try
NOTE : this works only with Jekyll version 3.1.x (not 3.0.x)
_plugins/imgWithTags-tag.rb
module Jekyll
class ImgWithTags < Liquid::Tag
# Custom liquid tag for images with tags
# syntax : {% img path/to/img.jpg coma, separated, tags %}
# tags are optionnal
#
# This plugin does two things :
# it renders html tag <a href='url'><img src='url' /></a>
# it stores tags -> images associations in site.config['images-tags']
Syntax = /^(?<image_src>[^\s]+\.[a-zA-Z0-9]{3,4})\s*(?<tags>[\s\S]+)?$/
def initialize(tag_name, markup, tokens)
super
if markup =~ Syntax then
#url = $1
#tags = $2.split(",").collect!{|tag| tag.strip } if !$2.nil?
else
raise "Image Tag can't read this tag. Try {% img path/to/img.jpg [coma, separated, tags] %}."
end
end
def render(context)
storeImgTags(context) if defined?(#tags) # store datas if we have tags
site = context.registers[:site]
imgTag = "<a href='#{ site.baseurl }/assets/#{#url}'><img src='#{ site.baseurl }/assets/#{#url}' /></a>"
end
def storeImgTags(context)
# store tagged images datas in site config under the key site.config['images-tags']
imagesTags = context.registers[:site].config['images-tags'] || {}
#tags.each{|tag|
slug = Utils.slugify(tag) # My tag -> my-tag
# create a tag.slug entry if it doesn't exist
imagesTags[slug] = imagesTags[slug] || {'name' => tag, 'images' => [] }
# add image url in the tag.images array if the url doesn't already exist
# this avoid duplicates
imagesTags[slug]['images'] |= [#url.to_s]
}
context.registers[:site].config['images-tags'] = imagesTags
end
end
end
Liquid::Template.register_tag('img', Jekyll::ImgWithTags)
At this point : all tagged images links are rendered and all tags->images associations are stored.
_plugins/hook-site-post-render-imagesTagsPagesGenerator.rb
Jekyll::Hooks.register :site, :post_render do |site, payload|
puts "++++++ site post_render hook : generating Images Tags pages"
imagesTags = site.config['images-tags'] # datas stored by img tag
linksDatas = site.data['imagesTagsLinks'] # tags pages link in data file
pagesFolder = 'tag'
imagesTags.each do |slug, datas|
tagName = datas['name']
tagImages = datas['images']
pageDir = File.join(pagesFolder, slug)
tagPage = Jekyll::ImageTagPage.new(site, site.source, pageDir, tagName, tagImages)
# as page rendering has already fired we do it again for our new pages
tagPage.output = Jekyll::Renderer.new(site, tagPage, payload).run
tagPage.trigger_hooks(:post_render)
site.pages << tagPage
# verbose check to see if we reference every tag url in out data file
if !linksDatas.key?(tagName) # check if tag is in imagesTagsLinks data file
puts "Warning ---------> #{tagName} not in data file"
puts "Add : #{tagName}: #{tagPage.url}"
puts "in data/imagesTagsLinks.yml"
puts
else
if tagPage.url != linksDatas[tagName] then # check if url is correct in imagesTagsLinks data file
puts "Warning ---------> incorrect url for '#{tagName}'"
puts "Replace : #{tagName}: #{linksDatas[tagName]}"
puts "by : #{tagName}: #{tagPage.url}"
puts "in data/imagesTagsLinks.yml"
puts
end
end
end
puts "++++++ END site post_render hook"
end
module Jekyll
class ImageTagPage < Page
def initialize(site, base, dir, tag, images)
#site = site
#base = base
#dir = dir
#name = 'index.html'
self.process(#name)
self.read_yaml(File.join(base, '_layouts'), 'tag_index.html')
self.data['tag'] = tag
self.data['images'] = images
self.data['title'] = "Images for tag : #{tag}"
end
end
end
And the Tag layout page
_layouts/tag_index.html
---
layout: default
---
<h2>{{ page.title }}</h2>
{% for i in page.images %}
<img src="{{ site.baseurl }}/assets/{{ i }}" alt="Image tagged {{ page.tag }}">
{% endfor %}
Here, everything is in place to generate tags page.
We can now do a jekyll build and see the verbose output just tell us what to add in _data/imagesTagsLinks.yml
_data/imagesTagsLinks.yml
tag1: /tag/tag1/
tag2: /tag/tag2/
...
We can now link to our tag page from everywhere with a simple
{% for t in site.data.imagesTagsLinks %}
<li>{{ t[0] }}</li>
{% endfor %}
I've told you : it's tricky. But It does the job.
Note: the img tag can be improved, and why not a figure tag ?