How to construct JSON in KSH/BASH script? [closed] - json

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 9 months ago.
Improve this question
Sample input JSON is given below:
I want the code to construct the JSON object static/dynamically in KSH/BASH
Example :
```{
"param1": "value1",
"param2": "value2",
"param3": "value3"
}```
Nested JSON:
```{
"param1": "value1",
"param2": "value2",
"param3": {
"param4": "value4",
"param5": "value5"
}
}```

Using jq to generate your JSON will save you when the input data contains things like quotation marks.
simple JSON object
json=$(
jq -n \
--arg "$param1" "$value1" \
--arg "$param2" "$value2" \
--arg "$param3" "$value3" \
'$ARGS.named'
)
nested
nestedJson=$(
jq -n \
--arg "$param1" "$value1" \
--arg "$param2" "$value2" \
--arg "$param3" "$value3" \
--argjson "nested" "$json" \
'$ARGS.named'
)
dynamic: use bash associative arrays
declare -A data=([param4]="value4" [param5]="value5")
jq_args=(
-n
--arg "$param1" "$value1"
--arg "$param2" "$value2"
)
for key in "${!data[#]}"; do
jq_args+=( --arg "$key" "${data[$key]}" )
done
dynamicJson=$( jq "${jq_args[#]}" '$ARGS.named'

Preparing the JSON in KSH/Bash
json=$(cat <<-END
{
"${param1}": "${value1}",
"${param2}": "${value2}",
"${param3}": "${value3}"
}
END
)
Preparing the Nested-JSON in KSH/Bash
nestedJson=$(cat <<-END
{
"${param1}": "${value1}",
"${param2}": "${value2}",
"${param3}": "${value3}",
"nested" : ${json}
}
END
)
Dynamic JSON:
array=("param4:value4", "param5:value5")
dynamicJSON=$(cat <<-END
{
"${param1}": "${value1}",
"${param2}": "${value2}"
}
END
)
for element in ${array[#]}
do
key="$(cut -d":" -f1 <<<"$element")"
val="$(cut -d":" -f2 <<<"$element")"
dynamicJSON=$( sed '$s/}/,\n'"\"$key\""':'"\"$val\""'}/' <<<"$dynamicJSON")
done

You can reference environment variables directory in your jq program
using env.varname
$ export fname="foo"
$ export lname="bar"
$ export location="baz"
$ SAMPLE_JSON='{
"first_name": "",
"last_lname": "",
"location": "",
"key": "value"
}'
$ jq '{
"first_name":env.fname,
"last_lname":env.lname,
"location":env.location,
"key": .key
}' <<<"$SAMPLE_JSON"
{
"first_name": "foo",
"last_lname": "bar",
"location": "baz",
"key": "value"
}
Nested
SAMPLE_JSON='{
"param1": "value1",
"param2": "value2",
"param3": {
"param4": "value4",
"param5": "value5"
}
}'
jq '{
"param1": env.fname,
"param2": env.lname,
"param3": {
"param4": .param3.param4,
"param5": env.location
}
}' <<<"$SAMPLE_JSON"
{
"param1": "foo",
"param2": "bar",
"param3": {
"param4": "value4",
"param5": "baz"
}
}

Related

Add JSON object to JSON file using JQ

Given the following JSON file (sample.json)
{
"api": "3.0.0",
"data": {
"description": "something",
"title": "hello",
"version": "1.0",
"app": {
"name": "abc",
"id": "xyz"
}
}
}
I wish to add the following JSON object at root level to the file above:
{
"heading": {
"user": ["$username"]
}
}
Where $username is a Bash variable.
Is there a better way to achieve this than the following?
blob=$(jq -n --arg foo API_NAME '{"heading": {"user": [env.username]}}')
jq --argjson obj "$(echo $blob)" '. + $obj' < sample.json
Just move what you create as blob directly into the other filter, ending up with just one jq call:
jq --arg username "$username" '. + {heading: {user: [$username]}}' sample.json

generate JSON out of command line arguments

I want to create JSON output with jq that looks like this:
{
"records": [
{
"id": "1234",
"song": "Yesterday",
"artist": "The Beatles"
}
]
}
I assumed I have to twiddle with the "filter" of jq whose concept I don't fully get after reading the doc.
This is what I got so far:
$ jq --arg id 1234 \
--arg song Yesterday \
--arg artist "The Beatles" \
'.' \
<<<'{ "records" : [{ "id":"$id", "song":"$song", "artist":"$artist" }] }'
which prints
{
"records": [
{
"id" : "$id",
"song" : "$song",
"artist" : "$artist"
}
]
}
Do I modify the filter? Do I change the input?
An alternate way to your original attempt, on jq-1.6 you can use the $ARGS.positional attribute to construct your JSON from scratch
jq -n '
$ARGS.positional | {
records: [
{
id: .[0],
song: .[1],
artist: .[2]
}
]
}' --args 1234 Yesterday "The Beatles"
As for why your original attempt didn't work, looks you are not modifying your json at all, with your filter '.' you are basically just reading in and printing out "untouched". The arguments set using --arg need to be set to the object inside the filter.
You are looking for something like this:
jq --null-input \
--arg id 1234 \
--arg song Yesterday \
--arg artist "The Beatles" \
'.records[0] = {$id, $song, $artist}'
Each variable reference between curly brackets is converted to a key-value pair where its name is the key, and its value is the value. And assigning the resulting object to .records[0] forces the creation of the surrounding structure.
jq --null-input\
--argjson id 1234\
--arg song Yesterday\
--arg artist "The Beatles"\
'{ "records" : [{ $id, $song, $artist }] }'
gives
{
"records": [
{
"id": 1234,
"song": "Yesterday",
"artist": "The Beatles"
}
]
}
I think you got the JSON/JQ the wrong way round:
This should be your JQ script:
rec.jq
{
records: [
{
id: $id,
song: $song,
artist: $artist
}
]
}
And this should be your JSON (empty):
rec.json
{}
Then:
jq --arg id 123 --arg song "Yesterday" --arg artist "The Beatles" -f rec.jq rec.json
Which produces:
{
"records": [
{
"id": "123",
"song": "Yesterday",
"artist": "The Beatles"
}
]
}
Start with an empty JSON and add the missing bits:
$ jq --arg id 1234 \
--arg song Yesterday \
--arg artist "The Beatles" \
'. | .records[0].id=$id | .records[0].song=$song | .records[0].artist=$artist' \
<<<'{}'
Outputs
{
"records": [
{
"id": "1234",
"song": "Yesterday",
"artist": "The Beatles"
}
]
}
Another, cleaner, approach based on the answer of #Inian could be
jq -n \
--arg id 1234
--arg song Yesterday
--arg artist "The Beatles"
'{records: [{id:$id, song:$song, artist:$artist}]}'

compare 2 json arrays and return the difference

We have a custom CD Pipeline Tool, which unfortunately does not version the deployment parameters. So I put these in a Bitbucket Repo as a json file and validate them against a REST API of this CD Tool.
So I have 2 json arrays, which are structurally the same, but may contain different objects or values in these objects. I want to compare them to see if they are different and what is different.
So far, I used the solution from here:
Using jq or alternative command line tools to diff JSON files
So I have put this in my code:
jq --argjson a "${bb_cfg}" --argjson b "${cd_tool_cfg}" -n 'def post_recurse(f): def r: (f | select(. != null) | r), .; r; def post_recurse: post_recurse(.[]?); ($a | (post_recurse | arrays) |= sort) as $a | ($b | (post_recurse | arrays) |= sort) as $b | $a == $b'
now I get a true if they are identical or false if 2 jsons have differences, but I do not know what is different.
I tried to do this with this if I get false back:
diff --suppress-common-lines -y <(jq . -S <<< "${bb_cfg}") <(jq . -S <<< "${cd_tool_cfg}")
Input $bb_cfg:
[{
"key": "IGNORE_VALIDATION_ERROR",
"value": "true",
"tags": []
},
{
"key": "BB_CFG_REPO_NAME",
"value": "cd-tool-cfg",
"tags": []
}]
Input $cd_tool_cfg
[{
"key": "IGNORE_VALIDATION_ERROR",
"value": "false",
"tags": []
},
{
"key": "BB_CFG_REPO_NAME",
"value": "cd-tool-cfg",
"tags": []
}]
which works partly, because if only the value is different, the output is like this:
"value": "true" | "value": "false"
so I do not get the whole json object here to quickly find out what parameter is different.
What I eventually want is to get something like this:
{
"key": "IGNORE_VALIDATION_ERROR",
"value": "true",
"tags": []
}
{
"key": "IGNORE_VALIDATION_ERROR",
"value": "false",
"tags": []
}
where I can store this in a variable in my bash script and transform this in an output I can use.
You could use jq's -c or --compact-output option:
diff <(jq -c .[] <<<"$bb_cfg") <(jq -c .[] <<<"$cd_tool_cfg")
1c1
< {"key":"IGNORE_VALIDATION_ERROR","value":"true","tags":[]}
---
> {"key":"IGNORE_VALIDATION_ERROR","value":"false","tags":[]}
The -c option will simply output a json with each array member on a separate line.
The following command will give you something like you requested:
diff --old-line-format="%L" --unchanged-line-format="" --new-line-format="%L" <(jq -c .[] <<<"$bb_cfg") <(jq -c .[] <<<"$cd_tool_cfg") | jq
will output:
{
"key": "IGNORE_VALIDATION_ERROR",
"value": "true",
"tags": []
}
{
"key": "IGNORE_VALIDATION_ERROR",
"value": "false",
"tags": []
}

json array into json stream with jq

This task is similar to this one but in my case I would like to go other way around.
So say we have input:
[
{
"name": "John",
"email": "john#company.com"
},
{
"name": "Brad",
"email": "brad#company.com"
}
]
and desired output is:
{
"name": "John",
"email": "john#company.com"
}
{
"name": "Brad",
"email": "brad#company.com"
}
I tried to write a bash function which will do it in loop:
#!/bin/bash
json=`cat $1`
length=`echo $json | jq '. | length'`
for (( i=0; i<$length ; i++ ))
do
echo $json | jq ".[$i]"
done
but it is obviously extremly slow...
Is there any way how to use jq better for this?
You can use this :
jq '.[]' file
If you use the .[index] syntax, but omit the index entirely, it will return all of the elements of an array.
Test:
$ jq '.[]' file
{
"email": "john#company.com",
"name": "John"
}
{
"email": "brad#company.com",
"name": "Brad"
}
you can apply ".[]" filter.
This tutorial is very informative
https://stedolan.github.io/jq/tutorial/

Update inner attribute of JSON with jq

Could somebody help me to deal with jq command line utility to update JSON object's inner value?
I want to alter object interpreterSettings.2B263G4Z1.properties by adding several key-values, like "spark.executor.instances": "16".
So far I only managed to fully replace this object, not add new properties with command:
cat test.json | jq ".interpreterSettings.\"2B188AQ5T\".properties |= { \"spark.executor.instances\": \"16\" }"
This is input JSON:
{
"interpreterSettings": {
"2B263G4Z1": {
"id": "2B263G4Z1",
"name": "sh",
"group": "sh",
"properties": {}
},
"2B188AQ5T": {
"id": "2B188AQ5T",
"name": "spark",
"group": "spark",
"properties": {
"spark.cores.max": "",
"spark.yarn.jar": "",
"master": "yarn-client",
"zeppelin.spark.maxResult": "1000",
"zeppelin.dep.localrepo": "local-repo",
"spark.app.name": "Zeppelin",
"spark.executor.memory": "2560M",
"zeppelin.spark.useHiveContext": "true",
"spark.home": "/usr/lib/spark",
"zeppelin.spark.concurrentSQL": "false",
"args": "",
"zeppelin.pyspark.python": "python"
}
}
},
"interpreterBindings": {
"2AXUMXYK4": [
"2B188AQ5T",
"2AY8SDMRU"
]
}
}
I also tried the following but this only prints contents of interpreterSettings.2B263G4Z1.properties, not full object.
cat test.json | jq ".interpreterSettings.\"2B188AQ5T\".properties + { \"spark.executor.instances\": \"16\" }"
The following works using jq 1.4 or jq 1.5 with a Mac/Linux shell:
jq '.interpreterSettings."2B188AQ5T".properties."spark.executor.instances" = "16" ' test.json
If you have trouble adapting the above for Windows, I'd suggest putting the jq program in a file, say my.jq, and invoking it like so:
jq -f my.jq test.json
Notice that there is no need to use "cat" in this case.
p.s. You were on the right track - try replacing |= with +=