To create one webpage with multiple Altair charts, all nicely formatted and easily deployable to GitHub Pages, I am trying to get several tools to work together. I am rendering the charts' json into a markdown document with jinja and then using Hugo to convert the markdown to HTML. I am using a Hugo partial (in extend_head.html, shown below) to include the scripts for Vega, Vega-Lite, and VegaEmbed, and I see them in the HTML of the webpage.
extend_head.html
{{- /* Head custom content area start */ -}}
{{- /* Insert any custom code (web-analytics, resources, etc.) - it will appear in the <head></head> section of every page. */ -}}
{{- /* Can be overwritten by partial with the same name in the global layouts. */ -}}
{{- /* Head custom content area end */ -}}
{{- /* To render LaTex math */ -}}
{{ if or .Params.math .Site.Params.math }}
{{ partial "math.html" . }}
{{ end }}
<script src="https://cdn.jsdelivr.net/npm/vega#5"></script>
<script src="https://cdn.jsdelivr.net/npm/vega-lite#4.8.1"></script>
<script src="https://cdn.jsdelivr.net/npm/vega-embed#6"></script>
I also have the following Hugo shortcode to call vegaEmbed. It includes the .Get method with a positional argument.
altairshortcode.html
<p>
<div id="vis"></div>
<script type="text/javascript">
var spec = {{.Get 0}}
vegaEmbed('#vis', spec);
</script>
</p>
The markdown template prior to rendering figure1's json looks like
---
title: "Aa1chart"
date: 2023-01-29T15:43:31-05:00
draft: false
---
Figure 1.
{{ '{{' }}< altairshortcode {{ figure1 }} >{{ '}}' }}
The Python code that's telling jinja to render the json into the template is (omitting the creation of the plot itself)
from pathlib import Path
from jinja2 import Template
# a scatter plot was generated with Altair and confirmed to display; omitted for brevity
p = Path.cwd()/'content/posts/aa1chart.md'
template = Template(p.read_text())
o = Path.cwd()/'content/posts/aa1chart.md'
o.write_text(template.render({
'figure1': figure1.to_json(),
}
)
)
After rendering the jinja template with the chart json, hugo runs, but hugo server -D, which launches a local server, gives the following error: unrecognized character in shortcode action: U+007B '{'. Note: Parameters with non-alphanumeric args must be quoted
After manually enclosing the json in double quotes, I got a different error unterminated quoted string in shortcode parameter-argument: '{
Some alternatives tried:
Using a named, instead of positional, parameter in the shortcode: Hugo runs, but hugo server -D errors: got quoted positional parameter. Cannot mix named and positional parameters. In the markdown, I had {{ '{{' }}< altairshortcode cjson={{ stage0 }} >{{ '}}' }}, and {{.Get "cjson"}} in the shortcode.
{{ .Inner }} converts the chart's json to a string, and so the chart does not appear in the webpage.
Placing Altair's figure1.to_html() instead of figure1.to_json() can display one chart in the webpage, but if multiple chart placeholders exist in the markdown, only one chart appears, and it is the last chart in the first position.
Looking at Hugo's built-in shortcodes, I hypothesize that if I utilized more logic or functions in my custom shortcode, such as toJSON, the json-to-string conversion can be prevented.
Related
I am working with Airflow 2.2.3 in GCP (Composer) and I am seeing inconsistent behavior which I can't explain when trying to use template values.
When I reference the templated value directly, it works without issue:
ts = '{{ ds }}' # results in 2022-05-09
When I reference the templated value in a function call, it doesn't work as expected:
ts_parts = '{{ ds }}'.split('-') # result ['2022-05-09']
The non-function call value is rendered without any issues, so it doesn't have any dependency on operator scope. There are examples here that show rendering outside of an operator, so I expect that not to be the issue. It's possible that Composer has setting configured so that Airflow will apply rendering to all python files.
Here's the full code for reference
dag.py
with DAG('rendering_test',
description='Testing template rendering',
schedule_interval=None, # only run on demand
start_date=datetime(2020, 11, 10), ) as rendering_dag:
ts = '{{ ds }}'
ts_parts = '{{ ds }}'.split('-')
literal_parts = '2022-05-09'.split('-')
print_gcs_info = BashOperator(
task_id='print_rendered_values',
bash_command=f'echo "ts: {ts}\nts_parts: {ts_parts}\nliteral_parts {literal_parts}"'
)
I thought that Airflow writes the files to some location with template values, then runs jinja against them with some supplied values, then runs the resulting python code. It looks like there is some logic applied if the line contains a function call? The documentation mentions none of these architectural principles and gives very limited examples.
Airflow does not render values outside of operator scope.
Rendering is a part of task execution which means that it's a step that happens only when task is in the worker (after being scheduled).
In your code the rendering is a top level code which is not part of operator templated fields thus Airflow consider it to be a regular string.
In your case the os.path.dirname() is executed on '{{ dag_run.conf.name }}' before it was rendered.
To fix your issue you need to set the Jinja string in templated fields of the operator.
bash_command=""" echo "path: {{ dag_run.conf.name }} path: os.path.dirname('{{ dag_run.conf.name }}')" """
Triggering DAG with {"name": "value"} will give:
Note that if you wish to use f-string with Jinja strings you must double the number of { }
source_file_path = '{{ dag_run.conf.name }}'
print_template_info = BashOperator(
task_id='print_template_info',
bash_command=f""" echo "path: { source_file_path } path: os.path.dirname('{{{{ dag_run.conf.name }}}}')" """
)
Edit:
Let me clarify - Airflow template fields as part of task execution.
You can see in the code base that Airflow invokes render_templates before it invokes pre_execute() and before it invokes execute(). This means that this step happens when the task is running on a worker. Trying to template outside of operator means the task doesn't even run - so the step of templating isn't running.
In eleventy, I'm using the 'eleventy-cache-assets' utility to retrieve API data from TMDB and place it in a cache. The JSON is successfully retrieved and stored in the cache. I can also use the Nunjucks dump filter to dump the full JSON to a page. However, I cannot run a for loop over the JSON content. It's behaving as if the data does not exist. I'm likely making a schoolboy error here but I can't see it.
Here's the JS that retrieves the data (successfully).
module.exports = async function () {
try {
// Grabs either the fresh remote data or cached data (will always be fresh live)
let json = await Cache(
`https://api.themoviedb.org/3/movie/upcoming?api_key=${TOKEN}&language=en-GB®ion=GB`,
{
duration: "1d", // 1 day
type: "json",
}
);
return {
films: json,
};
} catch (e) {
return {
films: 0,
};
}
};
Here is how I'm trying to loop through the content. The else condition is returning. When I remove the else condition, nothing is returned (just the empty ul). If I've targeted the node incorrectly, I should have x number of empty li tags, but I don't.
<ul>
{% for film in films %}
<li>{{ results.title }}</li>
{% else %}
<li>This displays if the 'films' collection were empty</li>
{% endfor %}
</ul>
The issue is likely just that you're wrapping the result from the API in an additional object, and eleventy adds an additional wrapper based on the file name of the data file, so you need to modify the way you access the results.
According to the TMDB documentation, the API will return something like this (JSON-encoded):
const json = {
page: 1,
results: [
{/** ... */},
{/** ... */},
{/** ... */},
]
}
Then you're wrapping that result in another object:
return {
films: json,
};
Finally, eleventy makes that data available to templates under a variable name based on the name of your data file — i.e. the file contains your API call. You didn't mention the name of the file, let's assume it lives in your data directory as tmdb.js. That means the variable accessible to your template will look something like this:
const tmdb = {
films: {
page: 1,
results: [
{/** ... */},
{/** ... */},
{/** ... */},
]
}
}
See the documentation on how eleventy parses data files for more information.
This means you can access the results in this way:
<ul>
{% for film in tmdb.films.results %}
<li>{{ film.title }}</li>
{% else %}
<li>This displays if the 'films' collection were empty</li>
{% endfor %}
</ul>
Note also that your original code used {{ results.title }} which isn't defined in your scope, I changed that to use the loop variable film. Also, adjust the tmdb variable to the name of your data file.
I would also recommend not wrapping the response of the API in an additional object (the key films also implies it's a list of films, not the complete API response). It doesn't really add anything and only increases complexity.
I have this json that I'm trying to pretty-print:
{
"error": "BAD_RESULT",
"status": 500,
"description": "Something bad happenned."
}
This object is stored in a variable errorMsg, and displayed in html using the pre tag like so:
<pre>
{{ errorMsg | json }}
</pre>
Unfortunately, when I inspect the element in the developers console, I see that the content inside the pre tag is padded with lots of spaces (right after the opening pre, and right before the closing pre), and that causes an indentation of the opening bracket. If I manually delete the spaces from the console, the json is displayed perfectly. How can it be solved? Is there a pure css solution or maybe with js?
Thanks.
EDIT:
My code was originally:
<pre>
{{ errorMsg | json }}
</pre>
When it was supposed to be:
<pre>{{ errorMsg | json }}</pre>
Now it works.
If you are trying to trim strings, you could use Javascript .trim() method.
var str = " Hello World! ";
alert(str.trim());
Output:
Hello world!
https://www.w3schools.com/jsref/jsref_trim_string.asp
I'm trying to fetch external JSON data into Jekyll and things are not working out.
I found this code (link to a gist), that is a fork of another gist... this just adds the external method ("from url").
I tried to morph that into my own tag plugin (or what ever they are called), in order to simplify it and possibly resolve some issues:
_plugins/externaljson.rb
require 'json'
require 'net/http'
module ExternalJSON
class ExternalJSON_tag < Liquid::Tag
def initialize(tag_name, text, tokens)
super
#text = text
end
def render(context)
if /(.+) from (.+)/.match(#text)
url = context[$2].strip
uri = URI( url )
response = Net::HTTP.get( uri )
data = JSON.parse( response )
context[$1] = JSON data
return ''
end
end
end
end
Liquid::Template.register_tag('externalJSON', ExternalJSON::ExternalJSON_tag)
...but I didn't really solve all my issues or learn much of anything from that. The only thing I think I learned is that the problem is likely somewhere between parsing the file in ruby and jekyll reading it.
I ran this test using the tag plugin code above(↑):
---
layout: default
---
<!-- Using the code I modified -->
<!-- This capture exists to combine a string and a variable, but it's just a static url for the purposes of this example -->
{% capture myUrl %}
https://api.guildwars2.com/v2/recipes/2889
{% endcapture %}
{% externalJSON jsonData from myUrl %}
{% for data in jsonData %}
{{ data}}
{% endfor %}
<!-- Jekyll's native way of handling local data files -->
<!-- I just saved that json data from the url above(↑) locally for this one -->
{% for data in site.data.example %}
{{ data }}
{% endfor %}
This test made me realize that both methods output the data slightly differently.
My external attempt:
{"type":"Meal","output_item_id":12210,"output_item_count":2,"min_rating":0,"time_to_craft_ms":1000,"disciplines":["Chef"],"flags":[],"ingredients":[{"item_id":24359,"count":1},{"item_id":12132,"count":1}],"id":2889,"chat_link":"[&CUkLAAA=]"}
Jekyll's native method (for local files)
{"type"=>"Meal", "output_item_id"=>12210, "output_item_count"=>2, "min_rating"=>0, "time_to_craft_ms"=>1000, "disciplines"=>["Chef"], "flags"=>[], "ingredients"=>[{"item_id"=>24359, "count"=>1}, {"item_id"=>12132, "count"=>1}], "id"=>2889, "chat_link"=>"[&CUkLAAA=]"}
And if I try to do for example {{ data.type }}, my external attempt returns nothing and the Jekyll method returns the value just like it should. I just can't figure out how to change the formatting or what ever the missing piece is there.
What am I doing wrong?
replace your render(context) with following:
def render(context)
if /(.+) from url (.+)/.match(#text)
resp = Net::HTTP.get_response(URI($2.strip))
data = resp.body
context[$1] = JSON data
nil
else
# syntax error
raise ArgumentError, 'ERROR:bad_syntax'
end
end
Then call the data like so:
{% externalJSON data from url http://foo.json %}
This will provide you with a data object that can be called to render individual keys.
If the data is an array, loop through the elements and call the desired key
{% for entry in data %}
{{ entry.type }}
{% endfor %}
If the data is an object (hash), call the key directly.
{{ data.type }}
In my application i'm using JSON Template for displaying content. I'm using conditional statements inside Template
{{#if document_id_1}} \
{{filename_1 }}
{{document_size_1}}
{{document_type_1}}
{{/if}}\
{{#if document_id_2}} \
{{filename_2 }}
{{document_size_2}}
{{document_type_2}}
{{/if}}\
................
i need to run loop (for or while) on this to avoid mentioning more if conditions in my code.
How can i run loop inside JSON Template???
Assuming Mustache and assuming all of the documents looks the same and assuming this structure
{
documents: [
{ filename: 'document_1', ... },
]
}
{{#documents}}
{{ filename }}
//other properties below
{{/documents}}