I am writing Ansible play, one of the tasks is to append entry to a JSON document. E.g.
JSON document staff.json:
{
"staff":[
{
"john":[
{
"position":"techwriter"
},
{
"sex":"male"
}
]
}
]
}
I need to append this entry to staff:
{
"staff":[
{
"john":[
{
"position":"techwriter"
},
{
"sex":"male"
}
]
},
{
"jane":[
{
"position":"admin"
},
{
"sex":"female"
}
]
}
]
}
The entry would be generated from Ansible template, something like this:
{
"{{ staff_name }}":[
{
"position":"{{ staff_position }}"
},
{
"sex":"{{ staff_sex }}"
}
]
}
I've learned to use jq to append entry to the JSON document, as seen in "Add json array element with jq (cmdline)". However, I do not know how I can implement this in Ansible, as template would output to a file.
I need a solution of something like this:
cat staff.json | jq '.staff |= .+ ["OUTPUT_FROM_TEMPLATE"]'
Any ideas welcome.
You may use template lookup plugin:
- shell: cat staff.json | jq '.staff |= . + [{{ item | to_json }}]' > staff.json
with_template: person.j2
vars:
staff_name: jane
staff_position: admin
staff_sex: female
Note that you need to use to_json filter with item, because Ansible template engine converts json strings that it can evaluate into dict.
Related
I have a following Json input in a text file json.txt:
{
"files":[
{
"id":49894894,
"list":[
{
"name":"one",
"animal_potato_carrot":{
"options":[
{
"id":4989,
"url":"https://example.com/text.txt"
},
{
"id":3994,
"url":"https://example.com/randomfile.json"
}
]
}
},
{
"name":"two",
"cat_dog_rabbit":[
{
"id":4989,
"url":"https://example.com/text2.txt"
},
{
"id":3994,
"url":"https://example.com/randomfile.json"
}
]
},
{
"name":"three",
"animal_potato_carrot":{
"options":[
{
"id":4989,
"url":"https://example.com/text3.txt"
},
{
"id":3994,
"url":"https://example.com/randomfile.json"
}
]
}
}
]
}
]
}
I want to get only the first url in the list of options for each animal_potato_carrot or cat_dog_rabbit nested tag only (note they have different structures)
So my output will be first three urls in those blocks:
["https://example.com/text.txt", "https://example.com/text2.txt, "https://example.com/text3.txt"]
I tried jq json.txt -c '.. |."animal_potato_carrot"? | select(. != null)' but that returns all the things inside the body, not just the FIRST url.
Edit:
these two commands return the urls for animal_potato_carrot and cat_dog_rabbit separately but is there a way to combine these commands?
jq -c '[..|.animal_potato_carrot?|select(. != null)|.options[0].url]' json.txt
jq -c '[..|.cat_dog_rabbit?|select(. != null)|.[0].url]' json.txt
If you want to concatenate two arrays you can use the + operator:
jq -c '[..|.animal_potato_carrot?|select(. != null)|.options[0].url] + [..|.cat_dog_rabbit?|select(. != null)|.[0].url]' json.txt
Please notice that the order of items in the result is not exactly as you requested, because first all animal_potato_carrot-urls are determined and then all cat_dog_rabbit-urls.
Combining two filters with , may come closest to your needs:
jq -c '[..|(.animal_potato_carrot?.options),(.cat_dog_rabbit?)|.[0].url|select(. != null)]' json.txt
I need to replace the value of "JaegerAgentHost" with a variable that I already have.
I have multiple formats of JSON on each app.
APP1 JSON file:
{
"Settings": {
"JaegerServiceSettings": {
"JaegerAgentHost": "jaeger.apps.internal",
"JaegerAgentPort": "6831"
} } }
APP2 JSON file:
{
"JaegerServiceSettings": {
"JaegerAgentHost": "jaeger.apps.internal",
"JaegerAgentPort": "6831",
} }
App3 JSON file:
{
"JaegerAgentHost": "jaeger.apps.internal",
"JaegerAgentPort": "6831"
}
irrespective of the path of key-value of JaegerAgentHost, I should be able to replace the value of it with my variable that ultimately should become as below
expected output::
APP1 JSON file:
{
"Settings": {
"JaegerServiceSettings": {
"JaegerAgentHost": "jaeger.app",
"JaegerAgentPort": "6831"
} } }
APP2 JSON file:
{
"JaegerServiceSettings": {
"JaegerAgentHost": "jaeger.app",
"JaegerAgentPort": "6831",
}}
App3 JSON file:
{
"JaegerAgentHost": "jaeger.app",
"JaegerAgentPort": "6831"
}
Please advice how we can do it, when I have multiple JSON files like to find and replace the perticular key-value with jq and bash
As of now I have multiple command for each json file to replace, which is not best practice.
This is blocking me from making a common script for all.
I can use sed but worried about the structure changes that may happen to any of the JSON file as they were not uniform and would like to prefer jq.
One way would be to use walk.
Assuming $host holds the desired value, the jq filter would be:
walk(if type == "object" and has("JaegerAgentHost")
then .JaegerAgentHost = $host else . end)
An alternative would be to use .. and |=:
(..|objects|select(.JaegerAgentHost).JaegerAgentHost) |= $host
You could pass in the value using the --arg command-line option, e.g.
jq --arg host jaeger.app .....
I'm trying to improve a bash script I wrote using jq (Python version), but can't quite get the conditional nature of the task at hand to work.
The task: insert array from one JSON object ("lookup") into another ("target") only if the key of the "lookup" matches a particular "higher-level" value in the "target". Assume that the two JSON objects are in lookup.json and target.json, respectively.
A minimal example to make this clearer:
"Lookup" JSON:
{
"table_one": [
"a_col_1",
"a_col_2"
],
"table_two": [
"b_col_1",
"b_col_2",
"b_col_3"
]
}
"Target" JSON:
{
"top_level": [
{
"name": "table_one",
"tests": [
{
"test_1": {
"param_1": "some_param"
}
},
{
"test_2": {
"param_1": "another_param"
}
}]
},
{
"name": "table_two",
"tests": [
{
"test_1": {
"param_1": "some_param"
}
},
{
"test_2": {
"param_1": "another_param"
}
}
]
}
]
}
I want the output to be:
{
"top_level": [{
"name": "table_one",
"tests": [{
"test_1": {
"param_1": "some_param"
}
},
{
"test_2": {
"param_1": "another_param",
"param_2": [
"a_col_1",
"a_col_2"
]
}
},
{
"name": "table_two",
"tests": [{
"test_1": {
"param_1": "some_param"
}
},
{
"test_2": {
"param_1": "another_param",
"param_2": [
"b_col_1",
"b_col_2",
"b_col_3"
]
}
}
]
}
]
}
]
}
Hopefully, that makes sense. Early attempts slurped both JSON blobs and assigned them to two variables. I'm trying to select for a match on [roughly] ($lookup | keys[]) == $target.top_level.name, but I can't quite get this match or the subsequent the array insert working.
Any advice is well-received!
Assuming the JSON samples have been corrected, and that the following program is in the file "target.jq", the invocation:
jq --argfile lookup lookup.json -f target.jq target.json
produces the expected result.
target.jq
.top_level |= map(
$lookup[.name] as $value
| .tests |= map(
if has("test_2")
then .test_2.param_2 = $value
else . end) )
Caveat
Since --argfile is officially deprecated, you might wish to choose an alternative method of passing in the contents of lookup.json, but --argfile is supported by all extant versions of jq as of this writing.
The jq answer is already given, but the ask itself is so fascinating - it requires a cross-lookup from a source file into the file being inserted, so I could not help providing also an alternative solution using jtc utility:
<target.json jtc -w'<name>l:<N>v[-1][tests][-1:][0]' \
-i file.json -i'<N>t:' -T'{"param_2":{{}}}'
A brief overlook of the used options:
-w'<name>l:<N>v[-1][tests][-1:][0]' - selects points of insertions in the source (target.json) by finding and memorizing into namespace N keys to be looked up in the inserted file, then rolling back 1 level up in the JSON tree, selecting tests label, then the last entry in it and finally addressing a 1st element of the last one
-i file.json make an insertion from the file
-i'<N>t:' - this walk over file.json finds recursively a tag (label) preserved in the namespace N from the respective walk -w (if not this insert option with the walk argument, then the whole file would get inserted into the insertion points -w..)
-T'{"param_2":{{}}}' - finally, a template operation is applied onto the insertion result transforming found entry (in file.json) into the one with the right label
PS. I'm the developer of the jtc - multithreading JSON processing utility for unix.
PPS. the disclaimer is required by SO.
I want to provision a JSON file with Ansible. The content of this file is a variable in my Ansible's playbook.
And very important for my usecase: I need the indentation & line breaks to be exactly the same as in my variable.
The variable looks like this :
my_ansible_var:
{
"foobar": {
"foo": "bar"
},
"barfoo": {
"bar": "foo"
}
}
And it's use like this in my playbook :
- name: drop the gitlab-secrets.json file
copy:
content: "{{ my_ansible_var }}"
dest: "/some/where/file.json"
Problem: when this tasks is played, my file is provisionned but as a "one-line" file:
{ "foobar": { "foo": "bar" }, "barfoo": { "bar": "foo" } }
I tried several other ways:
Retrieve the base64 value of my JSON content, and use content: "{{ my_ansible_var | b64decode }}" : same problem at the end
I tried playing with YAML block indicator : none of the block indicators helped me with that problem
I tried adding some filters like to_json, to_nice_json(indent=2) : no more luck here
Question:
How in Ansible can I provison a JSON file while keeping the exact indentation I want ?
In your example my_ansible_var is a dict. If you don't need to access its keys in your playbook (e.g. my_ansible_var.foobar.foo) and just want it as JSON string for your copy task, force it to be a string.
There is type-detection feature in Ansible template engine, so if you feed it with dict-like or list-like string, it will be evaluated into object. See some details here.
This construction will work ok for your case:
---
- hosts: localhost
gather_facts: no
vars:
my_ansible_var: |
{
"foobar": {
"foo": "bar"
},
"barfoo": {
"bar": "foo"
}
}
tasks:
- copy:
content: "{{ my_ansible_var | string }}"
dest: /tmp/out.json
Note vertical bar in my_ansible_var definition and | string filter in content expression.
I am trying to get the value of the 'url' name which sits underneath a name that I do not know up front. e.g. it's not 'name' or 'size' - just a string that another tool generates - example "x1234" is not known to me by name:
"foo": {
"bar": {
"x1234": {
"url": "http://example.com"
}
}
}
so jq ".foo.bar" returns the "x1234" fragment but what I need is the "url" value underneath it. I've tried many things after reading the docs but I wasn't able to figure out the right syntax.
Can anyone tell me where I'm going wrong?
One approach is to use ... For example, provided the input is valid JSON:
$ jq '.. | .url? | select(.)' input.json
"http://example.com"
Or equivalently (and easier to type):
$ jq '.. | .url? // empty' input.json
Assuming foo and bar are known, you could just do
.foo.bar[].url
Let's say if bar is also unknown, then do the following
.foo[][].url
Here is a solution which uses tostream. If filter.jq contains
tostream
| select(length==2) as [$p,$v]
| if $p[-1] == "url" and ($v|endswith(".zip")) then $v else empty end
and if data.json contains (note outer { } added to make the example legal JSON and a second entry added to demonstrate excluding values not ending in .zip as asked in follow-up comment to peak's answer)
{
"foo": {
"bar": {
"x1234": {
"url": "http://example.com"
},
"x1234xxx": {
"url": "http://example.com/file.zip"
}
}
}
}
then the command
jq -M -f filter.jq data.json
produces
"http://example.com/file.zip"