I want to create a JSON from the list of servers that are stored inside a variable but I get a parse error.
How can I make this work? I think quoting each server might work but I can't do that since the data is retrieved from a CSV using curl.
server_list=$(curl http://localhost/api/server_list.csv | cut -d ',' -f2);
echo $server_list
host001 host002 host003 host004
echo $server_list | jq '.'
parse error: Invalid numeric literal at line 1, column 15
if that had worked then I could run the following command to create json.
echo $server_list | jq -r '.' | jq -s '{hosts:map({"hostid":.})}')
One invocation of jq is sufficient:
jq -R 'split(" ") | {hosts:map({"hostid":.})}' <<< "$server_list"
Figured it out.
echo $server_list | jq -R 'split (" ")' | jq '.[]' | jq -s '{hosts:map({"hostid":.})}'
If you use echo $server_list | jq -R '.' | jq -s '{hosts:map({"hostid":.})}' will produce:
{
"hosts": [
{
"hostid": "host001 host002 host003 host004"
}
]
}
Not sure if that is exactly what you are expecting for output, but hopefully helps.
With a little bit of additional manipulation within jq you can easily expand the output result.
echo $server_list | jq -s -R '{hosts:split(" ")|map(split("\n"))|map({"hostid": .[0]})}'
{
"hosts": [
{
"hostid": "host001"
},
{
"hostid": "host002"
},
{
"hostid": "host003"
},
{
"hostid": "host004"
}
]
}
I would like to below code to output
KNOCK_KNOCK="Who is there?"
TEST_FILE='{
"KNOCK_KNOCK": "Who is there?"
}'
for s in $(echo $TEST_FILE | jq -r "to_entries|map(\"\
(.key)=\(.value|tostring)\")|.[]" ); do
echo $s
done
I got the loop from this post: Exporting JSON to environment variables
and cannot figure out how to modify to get my expected output. The problem seems to be the spaces in the .value
For the following test case I get the expected results:
TEST_FILE='{
"KNOCK_KNOCK": "Whoisthere?",
"POSTGRES_URI": "postgress://user:pass#testdb.com",
"WILL": "I.AM"
}'
for s in $(echo $TEST_FILE | jq -r "to_entries|map(\"\(.key)=\
(.value|tostring)\")|.[]" ); do
echo $s
done
KNOCK_KNOCK=Whoisthere?
POSTGRES_URI=postgress://user:pass#testdb.com
WILL=I.AM
I went with the following solution, which works for me, but the accepted answer is ok.
TEST_FILE='{
"KNOCK_KNOCK": "Who is there?"
}'
echo $TEST_FILE | sed 's_[{}]__'| sed 's_: _=_' | sed 's_ _export _'
One of many possibilities:
jq -r 'to_entries[] | [.key,.value] | join("=")' <<< "$TEST_FILE"
It would probably be advisable to use #sh as well:
$ echo $'{"KNOCK_KNOCK": "Who \'is\' there?"}'
| jq -r 'to_entries[] | [.key,(.value|#sh)] | join("=")'
KNOCK_KNOCK='Who '\''is'\'' there?'
$ KNOCK_KNOCK='Who '\''is'\'' there?'
$ echo "$KNOCK_KNOCK"
Who 'is' there?
$
Note also that if you are going to use $TEST_FILE, the double-quotes might be necessary.
You almost got it but you can do this without a loop with this line:
$ echo '{"KNOCK_KNOCK": "Who is there?"}' | jq -r 'to_entries[] | .key + "=\"" + (.value|tostring) + "\""'
KNOCK_KNOCK="Who is there?"
Explanation
to_entries transforms your object to an array of objects like this:
$ echo '{"KNOCK_KNOCK": "Who is there?"}' | jq -r 'to_entries'
[
{
"key": "KNOCK_KNOCK",
"value": "Who is there?"
}
]
You can then take each array element with the [] and use the key and value field to do simple string concatenation using the + operator.
input.json:-
{
"menu": {
"id": "file",
"value": "File",
"user": {
"address": "USA",
"email": "user#gmail.com"
}
}
}
Command:-
result=$(cat input.json | jq -r '.menu | keys[]')
Result:-
id
value
user
Loop through result:-
for type in "${result[#]}"
do
echo "--$type--"
done
Output:-
--id
value
user--
I want to do process the keys values in a loop. When I do the above, It result as a single string.
How can I do a loop with json keys result in bash script?
The canonical way :
file='input.json'
cat "$file" | jq -r '.menu | keys[]' |
while IFS= read -r value; do
echo "$value"
done
bash faq #1
But you seems to want an array, so the syntax is (missing parentheses) :
file='input.json'
result=( $(cat "$file" | jq -r '.menu | keys[]') )
for type in "${result[#]}"; do
echo "--$type--"
done
Output:
--id--
--value--
--user--
Using bash to just print an object keys from JSON data is redundant.
Jq is able to handle it by itself. Use the following simple jq solution:
jq -r '.menu | keys_unsorted[] | "--"+ . +"--"' input.json
The output:
--id--
--value--
--user--
I'm looking for a solution where I'm building out a JSON record and need to generate some text in JQ but pipe this text to an MD5 sum function and use it as a value for a key.
echo '{"first": "John", "last": "Big"}' | jq '. | { id: (.first + .last) | md5 }'
From looking at the manual and the GH issues I can't figure out how to do this since a function can't call out to a shell and there is not built in that provides a unique hash like functionality.
Edit
A better example what I'm looking for is this:
echo '{"first": "John", "last": "Big"}' | jq '. | {first, last, id: (.first + .last | md5) }'
to output:
{
"first": "John",
"last": "Big",
"id": "cda5c2dd89a0ab28a598a6b22e5b88ce"
}
Edit2
and a little more context. I'm creating NDJson files for use with esbulk. I need to generate a unique key for each record. Initially, I thought piping out to the shell would be the simplest solution so I could either use sha1sum or some other hash function easily, but that is looking more challenging than I thought.
A better example what I'm looking for is this:
echo '[{"first": "John", "last": "Big"}, {"first": "Justin", "last": "Frozen"}]' | jq -c '.[] | {first, last, id: (.first + .last | md5) }'
to output:
{"first":"John","last":"Big","id":"cda5c2dd89a0ab28a598a6b22e5b88ce"}
{"first":"Justin","last":"Frozen","id":"af97f1bd8468e013c432208c32272668"}
Using tee allows a pipeline to be used, e.g.:
echo '{"first": "John", "last": "Big"}' |
tee >( jq -r '.first + .last' | md5 | jq -R '{id: .}') |
jq -s add
Output:
{
"first": "John",
"last": "Big",
"id": "cda5c2dd89a0ab28a598a6b22e5b88ce"
}
Edit2:
The following uses a while loop to iterate through the elements of the array, but it calls jq twice at each iteration. For a solution that does not call jq at all within the loop, see elsewhere on this page.
echo '[{"first": "John", "last": "Big"}, {"first": "Justin", "last": "Frozen"}]' |
jq -c .[] |
while read -r line ; do
jq -r '[.[]]|add' <<< "$line" | md5 |
jq --argjson line "$line" -R '$line + {id: .}'
done
Looking around a little farther I ended up finding this: jq json parser hash the field value which was helpful in getting to my answer of:
echo '[{"first": "John", "last": "Big"}, {"first": "Justin", "last": "Frozen"}]' > /tmp/testfile
jsonfile="/tmp/testfile"
jq -c .[] "$jsonfile" | while read -r jsonline ;
do
# quickly parse the JSON line and build the pre-ID out to get md5sum'd and then store that in a variable
id="$(jq -s -j -r '.[] | .first + .last' <<<"$jsonline" | md5sum | cut -d ' ' -f1)"
# using the stored md5sum'd ID we can use that as an argument for adding it to the existing jsonline
jq --arg id "$id" -s -c '.[] | .id = "\($id)"' <<<"$jsonline"
done
output
{"first":"John","last":"Big","id":"467ffeee8fea6aef01a6ffdcaf747782"}
{"first":"Justin","last":"Frozen","id":"fda76523d5259c0b586441dae7c2db85"}
jq + md5sum trick:
json_data='{"first": "John", "last": "Big"}'
jq -r '.first + .last| #sh' <<<"$json_data" | md5sum | cut -d' ' -f1 \
| jq -R --argjson data "$json_data" '$data + {id: .}'
Sample output:
{
"first": "John",
"last": "Big",
"id": "f9e1e448a766870605b863e23d3fdbd8"
}
Here is an efficient solution to the restated problem. There are altogether just two calls to jq, no matter the length of the array:
json='[{"first": "John", "last": "Big"}, {"first": "Justin", "last": "Frozen"}]'
echo "$json" |
jq -c '.[] | [.[]] | add' |
while read -r line ; do echo "$line" | md5 ; done |
jq -s -R --argjson json "$json" 'split("\n")
| map(select(length>0))
| . as $in
| reduce range(0;length) as $i ($json; .[$i].id = $in[$i])'
This produces an array. Just tack on |.[] at the end to produce a stream of the elements.
Or a bit more tersely, with the goal of emitting one object per line without calling jq within the loop:
jq -c --slurpfile md5 <(jq -c '.[] | [.[]] | add' <<< "$json" |
while read -r line ; do printf '"%s"' $(md5 <<< "$line" ) ; done) \
'[., $md5] | transpose[] | .[0] + {id: .[1]}' <<< "$json"
Distinct Digest for each Record
I need to generate a unique key for each record.
It would therefore make sense to compute the digest based on each entire JSON object (or more generally, the entire JSON value), i.e. use jq -c ‘.[]’
I adapted accepted answer's script to my case, posting it here, it could be useful to someone.
input.json:
{"date":100,"text":"some text","name":"april"}
{"date":200,"text":"a b c","name":"may"}
{"date":300,"text":"some text","name":"april"}
output.json:
{"date":100,"text":"some text","name":"april","id":"4d93d51945b88325c213640ef59fc50b"}
{"date":200,"text":"a b c","name":"may","id":"3da904d79fb03e6e3936ff2127039b1a"}
{"date":300,"text":"some text","name":"april","id":"4d93d51945b88325c213640ef59fc50b"}
The bash script to generate output.json:
cat input.json |
while read -r line ; do
jq -r '.text' <<< "$line" | md5 |
jq -c --argjson line "$line" -R '$line + {id: .}' \
>> output.json
done
I pulled some config variables from a json file with jq.
And once modified, i want to write the whole config array (which can contain keys that were not there at first) to the json file.
The "foreach" part seems quite obvious.
But how to express "change keyA by value A or add keyA=>valueA" to the conf file ?
I'm stuck with something like
for key in "${!conf[#]}"; do
value=${conf[$key]}
echo $key $value
jq --arg key $key --arg value $value '.$key = $value' $conf_file > $new_file
done
Thanks
Extended solution with single jq invocation:
Sample conf array:
declare -A conf=([status]="finished" [nextTo]="2018-01-24" [result]=true)
Sample file conf.json:
{
"status": "running",
"minFrom": "2018-01-23",
"maxTo": "2018-01-24",
"nextTo": "2018-01-23",
"nextFrom": "2018-01-22"
}
Processing:
jq --arg data "$(paste -d':' <(printf "%s\n" "${!conf[#]}") <(printf "%s\n" "${conf[#]}"))" \
'. as $conf | map($data | split("\n")[]
| split(":") | {(.[0]) : .[1]})
| add | $conf + .' conf.json > conf.tmp && mv conf.tmp conf.json
The resulting conf.json contents:
{
"status": "finished",
"minFrom": "2018-01-23",
"maxTo": "2018-01-24",
"nextTo": "2018-01-24",
"nextFrom": "2018-01-22",
"result": "true"
}
I finally run into the following solution : not as compact as RomanPerekhrest, but a bit more human-readable. Thanks.
function update_conf() {
echo update_conf
if [ -n "$1" ] && [ -n "$2" ]; then
conf[$1]=$2
fi
for key in "${!conf[#]}"; do
value=${conf[$key]}
jq --arg key "$key" --arg value "$value" '.[$key] = $value' $confFile > $confTempFile && mv $confTempFile $confF$
done
}