Facing issue while generating CF manifests through jinja2 templating. jinja2 is unable to output env property under single quotes in manifest.
The template.yml.j2 template file has this,
JBP_CONFIG_JAVA_MAIN:'{java_main_class: "com.test.example.GeneralService", arguments: "setup.yml"}'
The final manifest.yml has missing single quotes,
JBP_CONFIG_JAVA_MAIN: {java_main_class: "com.test.example.GeneralService", arguments: "setup.yml"}
Expected output in manifest,
'{java_main_class: "com.test.example.GeneralService", arguments: "setup.yml"}' with quotes
I am really not getting why single quote is getting omitted after rendering the output manifest.
With jinja, it's tricky but easy to do, try this in your template file -
JBP_CONFIG_JAVA_MAIN: "'{java_main_class: "com.test.example.GeneralService", arguments: "setup.yml"}'"
Works for me.
Related
I'm using SparkKubernetesOperator which has a template_field called application_file. Normally on giving this field a file's name, airflow reads that file and templates the jinja variable in it (just like script field in the BashOperator).
So this works and the file information is shown in the Rendered Template tab with the jinja variables replaced with the correct values.
start_streaming = SparkKubernetesOperator(
task_id='start_streaming',
namespace='spark',
application_file='user_profiles_streaming_dev.yaml',
...
dag=dag,
)
I want to use different files in the application_file field for different environments
So I used a jinja template in the field. But when I change the application_file with user_profiles_streaming_{{ var.value.env }}.yaml, the rendered output is just user_profiles_streaming_dev.yaml and not the file contents.
I know that recursive jinja variable replacement is not possible in airflow but I was wondering if there is any workaround for having different template files.
What I have tried -
I tried using a different operator and doing xcom push to read the file contents and sending it to SparkKubernetesOperator. While this was good for reading different files based on environment, it did not solve the issue of having the jinja variable replaced.
I also tried making a custom operator which inherits the SparkKubernetesOperator and has a template_field applicaton_file_name thinking that jinja replacement will take place 2 times, but this didn't work too.
I made an env file which had the environment details (dev/prod). Then I added this code to the start of my dag file
ENV = None
with open('/home/airflow/env', 'r') as env_file:
value = env_file.read()
if value == None or value == "":
raise Exception("ENV FILE NOT PRESENT")
ENV = value
and then accessed the environment in the code like this
submit_job = SparkKubernetesOperator(
task_id='submit_job',
namespace="spark",
application_file=f"adhoc_{ENV}.yaml",
do_xcom_push=True,
dag=dag,
)
This way I could have separate dev and prod files.
My question is similar to this, where I am running into issues with putting JSON into a file. The issue is, no matter how I've formatted my strings inside the userData section of the CloudFormation template, I can't seem to capture an env $variable while maintaining a valid JSON object (with double quotes around the keys and values)
Below are two different ways I've tried to get the object into a file (via echo and cat << EOF < env-config.json) with virtually every combination of string escaping (single quotes wrapped around double quotes escaped around object keys...etc..)
echo '{\"development\": {\"EnvironmentConfig\": {\"api\": \" 'http://$ip:8000/api' \"}}}' >> env-config.json\n"
cat << EOF > env-config.json
{\"development\": {\"EnvironmentConfig\": {\"api\": \" 'http://$ip:8000/api' \"}}}
EOF
How can I place my perfectly formatted JSON object into a file while capturing an env $variable in it from the userData section of CloudFormation?
Thank you!
edit
Tools involved: gulp-ng-config, bash, cloudformation, json
Using gulp-ng-config to create a module with constants with the env-config.json file
I found out the answer, I needed single quotes around the url portion (as well as the double quotes) of my JSON like the below. This is what the whole line would look like in Cloudformation, I hope this helps someone:
"echo '{\"development\": {\"EnvironmentConfig\": {\"api\": \"'http://$ip:8000/api'\"}}}' >> env-config.json\n",
I want to include the value of the "version" parameter in package.json as part of the Jenkins build name.
I'm using the Jenkins Build Name Setter plugin - https://wiki.jenkins-ci.org/display/JENKINS/Build+Name+Setter+Plugin
So far I've tried to use PROPFILE syntax in the "Build name macro template" step:
${PROPFILE,file="./mainline/projectDirectory/package.json",property="\"version\""}
This successfully creates a build, but includes the quotes and comma surrounding the value of the version property in package.json, for example:
"0.0.1",
I want just the value inside returned, so it reads
0.0.1
How can I do this? Is there a different plugin that would work better for parsing package.json and getting it into the template, or should I resort to some sort of regex for removing the characters I don't want?
UPDATE:
I tried using token transforms based on reading the Token Macro Plugin documentation, but it's not working:
${PROPFILE%\"\,#\",file="./mainline/projectDirectory/package.json",property="\"version\""}
still just returns
However, using only one escaped character and only one of # or % works. No other combinations I tried work.
${PROPFILE%\,,file="./mainline/projectDirectory/package.json",property="\"version\""}
which returns "0.0.1" (comma removed)
${PROPFILE#\"%\"\,,file="./mainline/projectDirectory/package.json",property="\"version\""}
which returns "0.0.1", (no characters removed)
UPDATE:
Tried to use the new Jenkins Token Macro plugin's JSON macro with no luck.
Jenkins Build Name Setter set to update the build name with Macro:
${JSON,file="./mainline/pathToFiles/package.json",path="version"}-${P4_CHANGELIST}
Jenkins build logs for this job show:
10:57:55 Evaluated macro: 'Error processing tokens: Error while parsing action 'Text/ZeroOrMore/FirstOf/Token/DelimitedToken/DelimitedToken_Action3' at input position (line 1, pos 74):
10:57:55 ${JSON,file="./mainline/pathToFiles/package.json",path="version"}-334319
10:57:55 ^
10:57:55
10:57:55 java.io.IOException: Unable to serialize org.jenkinsci.plugins.tokenmacro.impl.JsonFileMacro$ReadJSON#2707de37'
I implemented a new macro JSON, which takes a file and a path (which is the key hierarchy in the JSON for the value you want) in token-macro-2.1. You can only use a single transform per macro usage.
Try the token transformations # and % (see Token-Makro-Plugin):
${PROPFILE#"%",file="./mainline/projectDirectory/package.json",property="\"version\""}
(This will only help if you are using pipelines. But for what it's worth,..)
What works for me is a combination of readJSON from the Pipeline Utility Steps plugin and directly setting currentBuild.displayName, thusly:
script {
// readJSON from "Pipeline Utility Steps"
def packageJson = readJSON file: 'package.json'
def version = packageJson.version
echo "Setting build version: ${packageJson.version}"
currentBuild.displayName = env.BUILD_NUMBER + " - " + packageJson.version
// currentBuild.description = "other cool stuff"
}
Omitting error handling etc obvs.
I am trying to pass JSON string in environment.
- name: Start {{service_name}}
shell: "<<starting springboot jar>> --server.port={{service_port}}\""
environment:
- SPRING_APPLICATION_JSON: '{"test-host.1":"{{test_host_1}}","test-host.2":"{{test_host_2}}"}'
test_host_1 is 172.31.00.00
test_host_2 is 172.31.00.00
But in spring logs, I get JSON parse exception where it prints
Caused by: com.fasterxml.jackson.core.JsonParseException: Unexpected character (''' (code 39)): was expecting double-quote to start field name
at [Source: {'test-host.1': '172.31.00.00', 'test-host.2': '172.31.00.00'}; line: 1, column: 3]
As seen, double quotes are converted to single quotes !!!
I tried escaping double quotes but with no luck.
Any idea why it happens, or any work around?
There is a thing about Ansible template engine.
If a string seems like an object (starts with { or [) Ansible converts it into object. See code.
To prevent this, you may use one of STRING_TYPE_FILTERS:
- SPRING_APPLICATION_JSON: "{{ {'test-host.1':test_host_1,'test-host.2':test_host_2} | to_json }}"
P.S. this is why hack with space character from #techraf's answer works: Ansible misses startswith("{") comparison and don't convert string to object.
Quick hack: add a space to the variable definition (after the first single quote) - a single space doesn't influence the actual variable value (space will be ignored):
- name: Start {{service_name}}
shell: "<<starting springboot jar>> --server.port={{service_port}}\""
environment:
- SPRING_APPLICATION_JSON: ' {"test-host.1":"{{test_host_1}}","test-host.2":"{{test_host_2}}"}'
With the space Ansible passes to shell (test1, test2 are values I set):
SPRING_APPLICATION_JSON='"'"' {"test-host.1":"test1","test-host.2":"test2"}'"'"'
Without the space:
SPRING_APPLICATION_JSON='"'"'{'"'"'"'"'"'"'"'"'test-host.2'"'"'"'"'"'"'"'"': '"'"'"'"'"'"'"'"'test2'"'"'"'"'"'"'"'"', '"'"'"'"'"'"'"'"'test-host.1'"'"'"'"'"'"'"'"': '"'"'"'"'"'"'"'"'test1'"'"'"'"'"'"'"'"'}'"'"'
Order is reversed too. Seems like without a space it interprets the JSON, with the space as string.
I don't really get why it happens so...
I was going through some electron package.json examples where I found some interpolations like given below:
"updater": {
"urls": {
"darwin": "{{& SQUIRREL_UPDATES_URL }}/update/%CHANNEL%/darwin?version=%CURRENT_VERSION%",
"win32": "{{& SQUIRREL_UPDATES_URL }}/update/%CHANNEL%/win32",
"linux": "{{& SQUIRREL_UPDATES_URL }}/update/%CHANNEL%/linux"
}
}
"piwik": {
"serverUrl": "{{& PIWIK_SERVER_URL }}"
},
"sentry": {
"dsn": "{{& SENTRY_DSN_PRIVATE }}"
}
I do not really know the following:
what does this {{}} mean in json
where does these variable exist
what does & mean in {{}} "{{& SENTRY_DSN_PRIVATE }}"
If anyone can explain then it would be really kind. Many thank in advance.
I guess you are talking about Whatsie and it's package.json.
If you take a look at one of the Gulp tasks located in the file tasks/compile.coffee, you'll be able to see the lines (in CoffeeScript):
# Move package.json
gulp.task 'compile:' + dist + ':package', ['clean:build:' + dist], ->
gulp.src './src/package.json'
.pipe mustache process.env
.pipe gulp.dest dir
Here the actual package.json is being passed to a mustache template engine - it receives a template as a first argument (package.json here acts like a template) and a data to be inserted in the template as a second argument - process.env.
As package.json acts like a template for mustache, you can use mustache syntax in it.
Curly braces {{}} are the part of it, they are used as placeholders which will be replaced by the actual data, when templates are being compiled. In the mustache docs you can also find a line:
You can also use & to unescape a variable: {{& name}}
So {{& name}} is to prevent values from being escaped. Otherwise, if you don't use & and values for output have some dangerous characters , they will be replaced by more secure ones (originally to prevent XSS in templates), as a result it will transform initial value, which is not always what you want. In this case author wants to preserve original value.
Going back to process.env - it is an object which gives access to environment variables in Node.JS. There is a file in repository .env-example with an example of env variables developer has to set in order to have the application work differently in different environments (for example on local machine or CI server). Names of some of the variables in this file are the ones that are used in a package.json as template placeholders - I guess author of the app uses all of this to simplify a build process for different environments.