I have a json file test.json with the content:
[
{
"name": "Akshay",
"id": "234"
},
{
"name": "Amit",
"id": "28"
}
]
I have a shell script with content:
#!/bin/bash
function display
{
echo "name is $1 and id is $2"
}
cat test.json | jq '.[].name,.[].id' | while read line; do display $line; done
I want name and id of a single item to be passed together as arguments to the function display but the output is something like this :
name is "Akshay" and id is
name is "Amit" and id is
name is "234" and id is
name is "28" and id is
What should be the correct way to implement the code?
PS: I specifically want to use jq so please base the answer in terms of jq
Two major issues, and some additional items that may not matter for your current example use case but can be important when you're dealing with real-world data from untrusted sources:
Your current code iterates over all names before writing any ids.
Your current code uses newline separators, but doesn't make any effort to read multiple lines into each while loop iteration.
Your code uses newline separators, but newlines can be present inside strings; consequently, this is constraining the input domain.
When you pipe into a while loop, that loop is run in a subshell; when the pipeline exits, the subshell does too, so any variables set by the loop are lost.
Starting up a copy of /bin/cat and making jq read a pipe from its output is silly and inefficient compared to letting jq read from test.json directly.
We can fix all of those:
To write names and ids in pairs, you'd want something more like jq '.[] | (.name, .id)'
To read both a name and an id for each element of the loop, you'd want while IFS= read -r name && IFS= read -r id; do ... to iterate over those pairs.
To switch from newlines to NULs (the NUL being the only character that can't exist in a C string, or thus a bash string), you'd want to use the -j argument to jq, and then add explicit "\u0000" elements to the content being written. To read this NUL-delimited content on the bash side, you'd need to add the -d '' argument to each read.
To move the while read loop out of the subshell, we can use process substitution, as described in BashFAQ #24.
To let jq read directly from test.json, use either <test.json to have the shell connect the file directly to jq's stdin, or pass the filename on jq's command line.
Doing everything described above in a manner robust against input data containing JSON-encoded NULs would look like the following:
#!/bin/bash
display() {
echo "name is $1 and id is $2"
}
cat >test.json <<'EOF'
[
{ "name": "Akshay", "id": "234" },
{ "name": "Amit", "id": "28" }
]
EOF
while IFS= read -r -d '' name && IFS= read -r -d '' id; do
display "$name" "$id"
done < <(jq -j '
def stripnuls: sub("\u0000"; "<NUL>");
.[] | ((.name | stripnuls), "\u0000", (.id | stripnuls), "\u0000")
' <test.json)
You can see the above running at https://replit.com/#CharlesDuffy2/BelovedForestgreenUnits#main.sh
You can use string interpolation.
jq '.[] | "The name is \(.name) and id \(.id)"'
Result:
"The name is Akshay and id 234"
"The name is Amit and id 28"
"The name is hi and id 28"
If you want to get rid of the double-quotes from each object, then:
jq --raw-output '.[] | "The name is \(.name) and is \(.id)"'
https://jqplay.org/s/-lkpHROTBk0
Related
I have a javascript file which prints a JSON array of objects:
// myfile.js output
[
{ "id": 1, "name": "blah blah", ... },
{ "id": 2, "name": "xxx", ... },
...
]
In my bash script, I want to iterate through each object.
I've tried following, but it doesn't work.
#!/bin/bash
output=$(myfile.js)
for row in $(echo ${output} | jq -c '.[]'); do
echo $row
done
You are trying to invoke myfile.js as a command. You need this:
output=$(cat myfile.js)
instead of this:
output=$(myfile.js)
But even then, your current approach isn't going to work well if the data has whitespace in it (which it does, based on the sample you posted). I suggest the following alternative:
jq -c '.[]' < myfile.js |
while read -r row
do
echo "$row"
done
Output:
{"id":1,"name":"blah blah"}
{"id":2,"name":"xxx"}
Edit:
If your data is arising from a previous process invocation, such as mongo in your case, you can pipe it directly to jq (to remain portable), like this:
mongo myfile.js |
jq -c '.[]' |
while read -r row
do
echo "$row"
done
How can I make jq -c '.[]' < (mongo myfile.js) work?
In a bash shell, you would write an expression along the following lines:
while read -r line ; do .... done < <(mongo myfile.js | jq -c .[])
Note that there are two occurrences of "<" in the above expression.
Also, the above assumes mongo is emitting valid JSON. If it emits //-style comments, those would have somehow to be removed.
Comparison with piping into while
If you use the idiom:
... | while read -r line ; do .... done
then the bindings of any variables in .... will be lost.
Summary
Starting from a JSON array, I'd like to replace an attribute of each item by the contents of a text file.
Example:
We have this initial JSON array:
[
{ "name": "step-a", "message": "step-a message placeholder" },
{ "name": "step-b", "message": "step-b message placeholder" }
]
And two text files matching the name attribute values (with an added .txt):
.
├── step-a.txt # contains the text: "Step A error logs ..."
└── step-b.txt # contains the text: "Step B error logs ..."
The goal is to perform the replacement and end up with this JSON array:
[
{ "name": "step-a", "message": "Step A error logs ..." },
{ "name": "step-b", "message": "Step B error logs ..." }
]
Attempt
I tried something like this:
# the variable $INITIAL contains the initial JSON array
echo $INITIAL | jq -c '.[] | .message = "$(cat .displayName+".txt")"'
Is there a way to perform an operation like this with jq or is using extra bash logic necessary?
Thank you.
You can use input_filename to achieve the goal very simply and efficiently.
For example, with the following in program.jq
# program.jq
(reduce inputs as $step ( {}; .[input_filename | rtrimstr(".txt")] = $step )) as $dict
| $a
| map(. + {message: $dict[.name] })
the following invocation would yield the expected result:
jq -n -f program.jq --argfile a array.json step-*.txt
An alternative
Depending on your requirements, you might like to replace the last line of program.jq as given above by:
| map(.message = ($dict[.name] // .message))
Alternatives to --argfile
If you prefer not to use --argfile, feel free to use --argjson or even --slurpfile, with appropriate adjustments to the above.
You can pass variables/arguments to jq using:
jq -r --arg first $1
$1 being your variable (which you can do: _fileContents="$( cat step-a.txt )" )
first is the variable name you want to create (which you can use inside the jq command like a bash variable ( $_fileContents );
An example from one of my scripts:
export techUser=$(jq -r --arg first $1 '.Environment[].Servers[] | select (.Address | contains($first)) | ."Technical User"' $serverlist)
I hope this has helped you finding a way to make it work ;)
I have the following file
[
{
"id": 1,
"name": "Arthur",
"age": "21"
},
{
"id": 2,
"name": "Richard",
"age": "32"
}
]
To display login and id together, I am using the following command
$ jq '.[] | .name' test
"Arthur"
"Richard"
But when I put it in a shell script and try to assign it to a variable then the whole output is displayed on a single line like below
#!/bin/bash
names=$(jq '.[] | .name' test)
echo $names
$ ./script.sh
"Arthur" "Richard"
I want to break at every iteration similar to how it works on the command line.
Couple of issues in the information you have provided. The jq filter .[] | .login, .id will not produce the output as you claimed on jq-1.5. For your original JSON
{
"login":"dmaxfield",
"id":7449977
}
{
"login":"stackfield",
"id":2342323
}
It will produce four lines of output as,
jq -r '.login, .id' < json
dmaxfield
7449977
stackfield
2342323
If you are interested in storing them side by side, you need to do variable interpolation as
jq -r '"\(.login), \(.id)"' < json
dmaxfield, 7449977
stackfield, 2342323
And if you feel your output stored in a variable is not working. It is probably because of lack of double-quotes when you tried to print the variable in the shell.
jqOutput=$(jq -r '"\(.login), \(.id)"' < json)
printf "%s\n" "$jqOutput"
dmaxfield, 7449977
stackfield, 2342323
This way the embedded new lines in the command output are not swallowed by the shell.
For you updated JSON (totally new one compared to old one), all you need to do is
jqOutput=$(jq -r '.[] | .name' < json)
printf "%s\n" "$jqOutput"
Arthur
Richard
In case the .login or .id contains embedded spaces or other characters that might cause problems, a more robust approach is to ensure each JSON value is on a separate line. Consider, for example:
jq -c .login,.id input.json | while read login ; do read id; echo login="$login" and id="$id" ; done
login="dmaxfield" and id=7449977
login="stackfield" and id=2342323
jq does my head in sometimes. Assume you have a json file called emails.json that looks like this;
[
{
"ParameterKey": "foo1",
"ParameterValue": "bar1"
},
{
"ParameterKey": "foo2",
"ParameterValue": "bar2"
}
]
If I run my bash script (let's call it script.sh) using the argument foo1, I want to have bar1 assigned to a variable called emailAdd. Likewise, if I use the argument foo2, I want bar2 assigned.
I thought my script would look like the following, but I'm getting an empty variable.
#!/usr/bin/env bash
EMAIL=$1
emailAdd=$(jq --arg email "$EMAIL" '.[] | select(.ParameterKey=="$email") | .ParameterValue' < emails.json)
echo "address is " $emailAdd
So, running sh script.sh foo1 I would expect address is bar1, etc
You pretty much have it correct. You don't need the quotes around $email, because unlike shell, jq actually treats that as a variable containing a value, rather than something to expand to arbitrary text. You probably also want to use the -r option so that the output is bar1, rather than "bar1".
#!/usr/bin/env bash
EMAIL=$1
emailAdd=$(jq -r --arg email "$EMAIL" '.[] | select(.ParameterKey==$email) | .ParameterValue' < emails.json)
echo "address is $emailAdd"
Suppose I have a file with this JSON:
[
{
"label" : "deploy",
"pk" : 2388175,
"key" : "gsfd45"
},
{
"label" : "jenkins",
"key" : "eQtIAwP",
"pk" : 2388165
}
]
I want to get the value for key "pk" if it is in the hash that has label = "deploy".
How can I do this? Do I need to write a script?
To parse JSON in Bash, use jq!
$ jq '.[] | select(.label=="deploy").pk' file
2388175
If you want to store deploy in a variable, use --arg. From jq manual → Invoking jq:
--arg name value
This option passes a value to the jq program as a predefined variable. If you run jq with --arg foo bar, then $foo is available in the program and has the value "bar". Note that value will be treated as a string, so --arg foo 123 will bind $foo to "123".
$ v="deploy"
$ jq --arg var "$v" '.[] | select(.label==$var).pk' file
2388175
$ v="blabla"
$ jq --arg var "$v" '.[] | select(.label==$var).pk' file
# empty!
$ v="jenkins"
$ jq --arg var "$v" '.[] | select(.label==$var).pk' file
2388165
By pieces:
Print everything:
$ jq '.[]' file
{
"key": "gsfd45",
"pk": 2388175,
"label": "deploy"
}
{
"pk": 2388165,
"key": "eQtIAwP",
"label": "jenkins"
}
Print those records where label equals "deploy":
$ jq '.[] | select(.label=="deploy")' file
{
"key": "gsfd45",
"pk": 2388175,
"label": "deploy"
}
Print just the field pk in such case:
$ jq '.[] | select(.label=="deploy").pk' file
2388175
If jq was not availale on your server, python should be there, right? ^_*
#!/bin/python
import json
with open('data.json') as data_file:
data = json.load(data_file)
for d in data:
if d['label'] == 'deploy':
print(d["pk"])
assume your file named as data.json save it as id.py, and run with:
python id.py
It needs python3 installed on your system.
change the line print (d["pk"]) into print d["pk"] if you only have python2 installed.
The output would be:
2388175
Edit
added the if check, didn't notice OP wanted to check the label.
In awk. It's a bit incomplete but as you didn't have anything to show, you can work on this one:
$ awk -F: '$1~/"label"/{l=$2} l~/deploy/ && $1 ~ /pk/ {sub(/,/,"",$2);print $2}' file
2388175
When awk meets a record with "label" on it, it stores the $2. Once the pk is found and flag l has deploy in it, remove comma and print.
If the elegant solution provided by James Brown does not work (e.g. different ordering of the key/value pairs) here is something that tries to get at least the string between the braces into one record (by setting RS), then the string is splittet at the key value pair with key "pk" (by setting FS).
After that setup the pattern looks for the label/deploy key/value pair in $0 and then, only if there are two fields (e.g. the pk was present and a field split took place) the string after the comma in $2 is deleted and the value of key pk remains and is printed:
script.awk
BEGIN {
RS="[{}[\\]]"
FS="\"pk\"[^:]*:"
}
/"label"[^:]*:[^\"]*"deploy"/ {
if( NF == 2 ) {
# "pk" is present in $0, remove everything after comma
sub(/,.*/, "", $2)
print $2
}
}
You use this script with awk like this: awk -f script.awk yourfile.
I have only tried it with GNU awk, but RS and FS should also work with awk, too.