Build Jekyll pages recursively from directory tree - jekyll

I love Markdown + MathJax for note taking, and I've found the simplest way to do this is to use a small, local Jekyll server that I can backup using Git. It's simple, clean, private, and redundant; each document is a single human-readable file before processing, meaning I hope that in 20 years the notes aren't worthless.
My only problem is that I wish I could have a directory with subdirectories of Markdown files and to have Jekyll build everything recursively. For example, imagine I had something like this:
...
- research
- foo
- derivations
- derivation1.md
- derivation2.md
...
- meetings
- 20190912_meeting1.md
- 20190912_meeting2.md
...
- bar
- derivations
- meetings
- personal
- courses
- qux
- baz
...
I would love to automatically render and a locally host a web server in which each directory is an index page, and each .md file is a document.
Is it possible to do this relatively easily in Jekyll? I've seen some stuff using nested collections, but it's pretty messy and manual.

Related

How can I remove one collection in my Jekyll page during development?

In my Jekyll page I have huge collections that take 35s to render and sometimes I just delete the biggest folders to iterate faster but I figure it should also be possible to have a secondary _config.dev.yml that undoes the _config.yml's collection definition. But I fail to get it to work.
How do I get jekyll to disregard a folder that is configured for a collection in _config.yml?
To have Jekyll pretend folders to be empty, simply exclude these folders in your _config.dev.yml:
exclude:
- _bigCollection
- _otherBigCollection
You could do it the other way round:
Do not define the collection in _config.yml.
When you deploy your site to production (or when you do want to render the big collections locally), use a secondary _config_with_collections.yml which includes just the collection definitions.

Showing LAtex formulas in GitHub markdown.md [duplicate]

Is there any way to render LaTex in README.md in a GitHub repository? I've googled it and searched on stack overflow but none of the related answers seems feasible.
For short expresions and not so fancy math you could use the inline HTML to get your latex rendered math on codecogs and then embed the resulting image. Here an example:
- <img src="https://latex.codecogs.com/gif.latex?O_t=\text { Onset event at time bin } t " />
- <img src="https://latex.codecogs.com/gif.latex?s=\text { sensor reading } " />
- <img src="https://latex.codecogs.com/gif.latex?P(s | O_t )=\text { Probability of a sensor reading value when sleep onset is observed at a time bin } t " />
Which should result in something like the next
Update: This works great in eclipse but not in github unfortunately. The only work around is the next:
Take your latex equation and go to http://www.codecogs.com/latex/eqneditor.php, at the bottom of the area where your equation appears displayed there is a tiny dropdown menu, pick URL encoded and then paste that in your github markdown in the next way:
![equation](http://latex.codecogs.com/gif.latex?O_t%3D%5Ctext%20%7B%20Onset%20event%20at%20time%20bin%20%7D%20t)
![equation](http://latex.codecogs.com/gif.latex?s%3D%5Ctext%20%7B%20sensor%20reading%20%7D)
![equation](http://latex.codecogs.com/gif.latex?P%28s%20%7C%20O_t%20%29%3D%5Ctext%20%7B%20Probability%20of%20a%20sensor%20reading%20value%20when%20sleep%20onset%20is%20observed%20at%20a%20time%20bin%20%7D%20t)
I upload repositories with equations to Gitlab because it has native support for LaTeX in .md files:
```math
SE = \frac{\sigma}{\sqrt{n}}
```
The syntax for inline latex is $`\sqrt{2}`$.
Gitlab renders equations with JavaScript in the browser instead of showing images, which improves the quality of equations.
More info here.
Let's hope Github will implement this as well in the future.
My trick is to use the Jupyter Notebook.
GitHub has built-in support for rendering .ipynb files. You can write inline and display LaTeX code in the notebook and GitHub will render it for you.
Here's a sample notebook file: https://gist.github.com/cyhsutw/d5983d166fb70ff651f027b2aa56ee4e
Readme2Tex
I've been working on a script that automates most of the cruft out of getting LaTeX typeset nicely into Github-flavored markdown: https://github.com/leegao/readme2tex
There are a few challenges with rendering LaTeX for Github. First, Github-flavored markdown strips most tags and most attributes. This means no Javascript based libraries (like Mathjax) nor any CSS styling.
The natural solution then seems to be to embed images of precompiled equations. However, you'll soon realize that LaTeX does more than just turning dollar-sign enclosed formulas into images.
Simply embedding images from online compilers gives this really unnatural look to your document. In fact, I would argue that it's even more readable in your everyday x^2 mathematical slang than jumpy .
I believe that making sure that your documents are typeset in a natural and readable way is important. This is why I wrote a script that, beyond compiling formulas into images, also ensures that the resulting image is properly fitted and aligned to the rest of the text.
For example, here is an excerpt from a .md file regarding some enumerative properties of regular expressions typeset using readme2tex:
As you might expect, the set of equations at the top is specified by just starting the corresponding align* environment
**Theorem**: The translation $[\![e]\!]$ given by
\begin{align*}
...
\end{align*}
...
Notice that while inline equations ($...$) run with the text, display equations (those that are delimited by \begin{ENV}...\end{ENV} or $$...$$) are centered. This makes it easy for people who are already accustomed to LaTeX to keep being productive.
If this sounds like something that could help, make sure to check it out. https://github.com/leegao/readme2tex
Since May 2022, this has been officially supported:
Inline:
Where $x = 0$, evaluate $x + 1$
Blocks:
Where
$$x = 0$$
Evaluate
$$x + 1$$
One can also use this online editor: https://www.codecogs.com/latex/eqneditor.php which generates SVG files on the fly. You can put a link in your document like this:
![](https://latex.codecogs.com/svg.latex?y%3Dx%5E2) which results in:
.
I test some solution proposed by others and I would like to recommend TeXify created and proposed in comment by agurodriguez and further described by Tom Hale - I would like develop his answer and give some reason why this is very good solution:
TeXify is wrapper of Readme2Tex (mention in Lee answer). To use Readme2Tex you must install a lot of software in your local machine (python, latex, ...) - but TeXify is github plugin so you don't need to install anything in your local machine - you only need to online installation that plugin in you github account by pressing one button and choose repositories for which TeXify will have read/write access to parse your tex formulas and generate pictures.
When in your repository you create or update *.tex.md file, the TeXify will detect changes and generate *.md file where latex formulas will be exchanged by its pictures saved in tex directory in your repo. So if you create README.tex.md file then TeXify will generate README.md with pictures instead tex formulas. So parsing tex formulas and generate documentation is done automagically on each commit&push :)
Because all your formulas are changed into pictures in tex directory and README.md file use links to that pictures, you can even uninstall TeXify and all your old documentation will still works :). The tex directory and *.tex.md files will stay on repository so you have access to your original latex formulas and pictures (you can also safely store in tex directory your other documentation pictures "made by hand" - TeXify will not touch them).
You can use equations latex syntax directly in README.tex.md file (without loosing .md markdown syntax) which is very handy. Julii in his answer proposed to use special links (with formulas) to external service e.g . http://latex.codecogs.com/gif.latex?s%3D%5Ctext%20%7B%20sensor%20reading%20%7D which is good however has some drawbacks: the formulas in links are not easy (handy) to read and update, and if there will be some problem with that third-party service your old documentation will stop work... In TeXify your old documentation will works always even if you uninstall that plugin (because all your pictures generated from latex formulas are stay in repo in tex directory).
The Yuchao Jiang in his answer, proposed to use Jupyter Notebook which is also nice however have som drawbacks: you cannot use formulas directly in README.md file, you need to make link there to other file *.ipynb in your repo which contains latex (MathJax) formulas. The file *.ipynb format is JSON which is not handy to maintain (e.g. Gist don't show detailed error with line number in *.ipynb file when you forgot to put comma in proper place...).
Here is link to some of my repo where I use TeXify for which documentation was generated from README.tex.md file.
Update
Today 2020.12.13 I realised that TeXify plugin stop working - even after reinstallation :(
For automatic conversion upon push to GitHub, take a look at the TeXify app:
GitHub App that looks in your pushes for files with extension *.tex.md and renders it's TeX expressions as SVG images
How it works (from the source repository):
Whenever you push TeXify will run and seach for *.tex.md files in your last commit. For each one of those it'll run readme2tex which will take LaTeX expressions enclosed between dollar signs, convert it to plain SVG images, and then save the output into a .md extension file (That means that a file named README.tex.md will be processed and the output will be saved as README.md). After that, the output file and the new SVG images are then commited and pushed back to your repo.
I just published a new version of xhub, a browser extension that renders LaTeX (and other things) in GitHub pages.
Cons:
You have to install the extension once.
Pros:
No need to set up anything.
Just write Markdown with math
Display math:
```math
e^{i\pi} + 1 = 0
```
and line math $`a^2 + b^2 = c^2`$.
(Syntax like on GitLab.)
Works on light and dark background. (Math has text-color)
You can copy-and-paste the math just like text
As an example, check out this GitHub README:
You can get a continuous integration service (e.g. Travis CI) to render LaTeX and commit results to github. CI will deploy a "cloud" worker after each new commit. The worker compiles your document into pdf and either cuses ImageMagick to convert it to an image or uses PanDoc to attempt LaTeX->HTML conversion where success may vary depending on your document. Worker then commits image or html to your repository from where it can be shown in your readme.
Sample TravisCi config that builds a PDF, converts it to a PNG and commits it to a static location in your repo is pasted below. You would need to add a line that fetches pdfconverts PDF to an image
sudo: required
dist: trusty
os: linux
language: generic
services: docker
env:
global:
- GIT_NAME: Travis CI
- GIT_EMAIL: builds#travis-ci.org
- TRAVIS_REPO_SLUG: your-github-username/your-repo
- GIT_BRANCH: master
# I recommend storing your GitHub Access token as a secret key in a Travis CI environment variable, for example $GH_TOKEN.
- secure: ${GH_TOKEN}
script:
- wget https://raw.githubusercontent.com/blang/latex-docker/master/latexdockercmd.sh
- chmod +x latexdockercmd.sh
- "./latexdockercmd.sh latexmk -cd -f -interaction=batchmode -pdf yourdocument.tex -outdir=$TRAVIS_BUILD_DIR/"
- cd $TRAVIS_BUILD_DIR
- convert -density 300 -quality 90 yourdocument.pdf yourdocument.png
- git checkout --orphan $TRAVIS_BRANCH-pdf
- git rm -rf .
- git add -f yourdoc*.png
- git -c user.name='travis' -c user.email='travis' commit -m "updated PDF"
# note we are again using GitHub access key stored in the CI environment variable
- git push -q -f https://your-github-username:$GH_TOKEN#github.com/$TRAVIS_REPO_SLUG $TRAVIS_BRANCH-pdf
notifications:
email: false
This Travis Ci configuration launches a Ubuntu worker downloads a latex docker image, compiles your document to pdf and commits it to a branch called branchanme-pdf.
For more examples see this github repo and its accompanying sx discussion, PanDoc example,
https://dfm.io/posts/travis-latex/, and this post on Medium.
I have been looking around and found that this answer in another question works best for me. i.e. use githubcontent math renderer, e.g. to display:
Use this link
Beware of the latex needs to be url encoded, but otherwise work quite well for me.
If you are having issues with https://www.codecogs.com/latex/eqneditor.php, I found that https://alexanderrodin.com/github-latex-markdown/ worked for me. It generates the Markdown code you need, so you just cut and paste it into your README.md document.
You may also take a look on my tool latexMarkdown2Markdown which convert LaTeX to SVG and generate a table of content with chapter numbering.
Good news!
According to this blogpost, now GitHub supports Mathjax in readme files.
You can use in-line LaTeX inspired syntax using $ delimiters, or in-blocks using $$ delimiters.
Writing inline expressions:
This sentence uses $ delimiters to show math inline:
$\sqrt{3x-1}+(1+x)^2$
Writing expressions as blocks:
The Cauchy-Schwarz Inequality
$$\left( \sum_{k=1}^n a_k b_k \right)^2 \leq \left( \sum_{k=1}^n a_k^2
\right) \left( \sum_{k=1}^n b_k^2 \right)$$
Source: https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/writing-mathematical-expressions
You can use markdowns, e.g.
![equ](https://latex.codecogs.com/gif.latex?log(y)=\beta_0&space;&plus;&space;\beta_1&space;x&space;&plus;&space;u)
Code can be typed here: https://www.codecogs.com/latex/eqneditor.php.
Edit: As germanium pointed out, it does not work for README.md but other git pages though no explanation is available.
My quick solution is this
step 1. Add latex to your .md file
$$x=\sqrt{2}$$
Note: math eqns must be in $$...$$ or \\(... \\).
step 2. Add the following to your scripts.html or theme file (append this code at the end)
<script type="text/javascript" async
src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-MML-AM_CHTML">
Done!. See your eq. by loading the page.

Meson dependencies catch22 and lazy access to target output

We ‘d like to introduce Meson to build our existing C++ application. Our structure is as follows:
Wie have 8 main modules (mod_X)
Every module has 20-40 subdirs, each with 5 – 100 cpp files; separated in libs and executables.
mod_INFRA/apps/myparser has a target that creates a code generator executable
Which depends only on mod_INFRA/libs/A
The code generator must be applied to certain files (*.rules) in numerous subdirs in all modules and subdirs, including mod_INFRA itself.
The generated source code must be compiled and considered with the target in subdir_X
What I’d like to achieve:
In root/meson.builddefine a common and re-usable custom_target or generator that I can invoke/apply in every module and subdir as needed.
Problem:
In root/meson.build, we define common variables such as compiler flags etc., and we do subdir(‘mod_INFRA’) for each module. In mod_INFRA/meson.build I do subdir(‘apps/xyz’), subdir(‘libs/abc’), etc. for each subdir. That’s all fine
However I struggle to define the custom_target or generator in root/meson.build. The required executable is not yet available before subdir('mod_INFRA'). And after subdir(..) it is too late, as I need the generator already to build files in other subdirs in mod_INFRA.
A possible solution could be a “proxy” that lazily resolves the executable by target name. E.g. if I could do (pseudo code): generator(getTargetOutput(‘myparser’), …). But I could not find out if that is available.
Any other thoughts on how to resolve it, without completely restructuring the directory structure?
- meson.build
- mod_INFRA
- meson.build
- apps
- meson.build
- myparser
- meson.build
- libs
- subdir_INFRA_A (required to build myparser)
- meson.build
- subdir_INFRA_B
- meson.build
- subdir_INFRA_C (requires parser to generate code)
- meson.build
- mod_A
- meson.build
- subdir_A_A (requires parser to generate code)
- meson.build
- subdir_A_B (requires parser to generate code)
- meson.build
- mod_B
...
Somebody suggested: From the top level can you do a subdir straight into "mod_INFRA/libs/subdir_INFRA_A" and "apps/myparser" directories to build them, before returning to the top level and then subdir down a single level repeatedly thereafter? That did the trick,

How to generate a whole website in a variabilized sub-directory ? (Jekyll)

I've been playing around with Jekyll lately and recently discovered that S3 wouldn't allow me to have enough buckets to have one bucket per website.
So I'm struggling to figure out two things
1) How do I specify that I want a 'child' subdirectory to be created, and all the generated website's files to be in it ?
2) How do I use a variable as the 'child' subdirectory's name ? (since it will be different for each website)
The baseurl config option doesn't seem to do anything about that. Any idea will be welcome :-)
Thanks a lot, and have a great day !
I'm not sure about the variable option, however, for child sub-directories:
A) If you can put the content into "child" folders directly within the jekyll content, then use different configuration files to manage each site through build properties. See the examples under the example below.
B) If you just want to put the same content in different places, perhaps you could use different baseurl or destination properties in different config files? Use the same config file approach as below, but with different versions of those properties instead of/in combination with the exclude property (see https://jekyllrb.com/docs/configuration/#serve-command-options).
Example: Use different config files
Use the exclude config property to exclude folders (and all of that folder's contents) when running a build. You could also specify different output folder locations per site, base URLs, and so on. I'll focus on the exclude property, but you can find other useful properties in the documentation: https://jekyllrb.com/docs/configuration/#global-configuration
For example, you could have the following files/folders:
/siteA-folder/siteA-content/blah.html
/siteB-folder/siteB=content/meh.html
/siteC-folder/siteC=content/foo.html
config.yaml
config_buildA.yaml
config buildB.yaml
Use the build option --config FILE1[,FILE2...] (https://jekyllrb.com/docs/configuration/#build-command-options) to explicitly call your custom config file A or B, each of which has its own version of the exclude property.
Snippet from config_buildA.yaml:
exclude:
- /siteB-folder/ # While building A, you want to exclude the siteB folder and contents
- /siteC-folder/ # Similarly, exclude siteC contents
Snippet from config_buildB.yaml:
exclude:
- /siteA-folder/ # While building B, you want to exclude the siteA folder and contents
- /siteC-folder/ # Similarly, exclude siteC contents
Building A from the command line:
jekyll build --config config_buildA.yaml
Building B from the command line:
jekyll build --config config_buildB.yaml
By managing excludes in the config file, you can re-use the common templates and css, and keep everything together (using less disk space overall).
If there's shared content, you could also look into using a more advanced feature, includes (see https://jekyllrb.com/docs/includes/), to manage the shared content in the includes folder, referenced within the html or markdown files (but that's getting a little of the beaten track for your question, so I'll stop there...)
EDIT: I see you've edited the question specifying the baseurl property doesn't do what you're looking for.
I've used a combination of the url site property baseurl and destination similar to below. In my case, I have different destinations depending on whether it's an archive or latest content, but you could use this to build to different web folders if you wanted.
Ex: config for latest content
url: http://example.com
baseurl: "/latest/"
destination: /path/to/latest/output/
Ex: config for archived content
url: http://example.com
baseurl: "/archive/content/"
destination: /path/to/archive/output/
Then, using the multiple versions of the config file, I can just call whichever one applies.

How can I ignore unused partials in handlebars precompilation?

I would like to collect only the depending partials of a single template. Or otherwise ignore unused partials. Because it is a large project with lots of templates/partials and dependencies it is not possible to do that by hand.
Example file and folder structure would be something like:
/sites/foo.hbs - includes {> partial1}
/sites/bar.hbs - includes {> partial2}
/partials/partial1.hbs
/partials/partial2.hbs
Now I would like to precompile foo.hbs only. For precompilation i have to configure foo.hbs as root and /partials/*.hbs as partials.
The precompilation should end up with the content of foo.hbs and partial1.hbs in a single file.
Is that possible with handlebars at all? Maybe with an existing gulp plugin?
Your question contains a (somewhat) logical error.
Partials were initially intended to act as separate templates.
I guess your intentions are basically more syntactic sugar'ish,
- as you wish to write your code separately though merge it on deployment (some people call that OCD, since it has no real benefit in terms of performance).
Anyhow.
You can run a node scripts that runs through your HBS's, locates partials, extracts them into their parent elements and outputs these final templates.
Take a look at this following answer -
find files by extension, *.html under a folder in nodejs