Given a json that looks something like:
[{"id":1,"firstName":"firstName1","lastName":"lastName1"},
{"id":2,"firstName":"firstName2","lastName":"lastName2"},
{"id":3,"firstName":"firstName3","lastName":"lastName3"}]
What would be the best way to convert the id value from an int to a string and then saving the file?
I have tried:
echo "$(jq -r '[.[] | .id = .id|tostring]' test.json)" > test.json
But that seems to put each entry into a string and adds the backslashes
[
"{\"id\":1,\"firstName\":\"firstName1\",\"lastName\":\"lastName1\"}",
"{\"id\":2,\"firstName\":\"firstName2\",\"lastName\":\"lastName2\"}",
"{\"id\":3,\"firstName\":\"firstName3\",\"lastName\":\"lastName3\"}"
]
| has a lower priority than the assignment (=). The expression .id = .id | tostring is interpreted as (.id = .id) | tostring.
The assignment does change anything and can be removed. The script becomes [ .[] | tostring ], that explains the output (each object is serialized as JSON into a string).
The solution is to use parentheses to enforce the desired order of execution.
The command is:
jq '[ .[] | .id = (.id | tostring) ]' test.json
Do not use process expansion ($(...)) to compose an echo command line. It is inefficient and not needed.
Redirect the output of jq directly to a file. Use a different file than the input file (or it ends up destroying your data).
jq '[ .[] | .id = (.id | tostring) ]' test.json > output.json
Related
I was currently learning the usage of jq. I have a json file and I am able to loop through and filter out the values I need from the json. However, I am running into issue when I try to combine the output into single string instead of having the output in multiple lines.
File svcs.json:
[
{
"name": "svc-A",
"run" : "True"
},
{
"name": "svc-B",
"run" : "False"
},
{
"name": "svc-C",
"run" : "True"
}
]
I was using the jq to filter to output the service names with run value as True
jq -r '.[] | select(.run=="True") | .name ' svcs.json
I was getting the output as follows:
svc-A
svc-C
I was looking to get the output as single string separated by commas.
Expected Output:
"svc-A,svc-C"
I tried to using join, but was unable to get it to work so far.
The .[] expression explodes the array into a stream of its elements. You'll need to collect the transformed stream (the names) back into an array. Then you can use the #csv filter for the final output
$ jq -r '[ .[] | select(.run=="True") | .name ] | #csv' svcs.json
"svc-A","svc-C"
But here's where map comes in handy to operate on an array's elements:
$ jq -r 'map(select(.run=="True") | .name) | #csv' svcs.json
"svc-A","svc-C"
Keep the array using map instead of decomposing it with .[], then join with a glue string:
jq -r 'map(select(.run=="True") | .name) | join(",")' svcs.json
svc-A,svc-C
Demo
If your goal is to create a CSV output, there is a special #csv command taking care of quoting, escaping etc.
jq -r 'map(select(.run=="True") | .name) | #csv' svcs.json
"svc-A","svc-C"
Demo
I have a JSON object that looks like this:
[{"name":"NAME_1"},"NAME_2"]
I would like an output of
["NAME_1", "NAME_2"]
Some of the entries in the array are an object with a key "name" and some are just a string of the name. I am trying to extract an array of the names. Using
jq -cr '.[].name // []'
throws an error as it is trying to index .name of the string object. Is there a way to check if it is a string, and if so just use its value instead of .name?
echo '[{"name":"NAME_1"},"NAME_2"]' \
| jq '[ .[] | if (.|type) == "object" then .name else . end ]'
[
"NAME_1"
"NAME_2"
]
Ref:
https://stedolan.github.io/jq/manual/#ConditionalsandComparisons
https://stedolan.github.io/jq/manual/#type
As #LéaGris comments, a simpler version
jq '[ .[] | .name? // . ]' file
https://stedolan.github.io/jq/manual/#ErrorSuppression/OptionalOperator:%3f
https://stedolan.github.io/jq/manual/#Alternativeoperator://
You can use the type function which returns "object" for objects.
jq '.[] | if type == "object" then .name else . end' file.json
To get the output as array, just wrap the whole expression into [ ... ].
Just use the error suppression operator with ?, map and scalars
jq 'map( .name?, scalars )'
Note that by using scalars, it is assumed that other than objects with name, all others are names of form NAME_*. If there are other strings as well, and you need to exclude some of them you might need to add some additional logic to do that. e.g. using startswith(..) with a string of your choice.
map( .name?, select( scalars | startswith("NAME") ) )
Demo
With your shown samples only, please try following jq code. Using tostream function here to get the required values from requirement.
jq -c '[.[] | tostream | if .[1] != null then .[1] else empty end]' Input_file
I have a json file I am parsing with jq. This is a sample of the file
[{
"key1":{...},
"key2":{...}
}]
[{
"key1":{...},
"key2":{...}
}]
...
each line is a list containing a json (which I know is not technically a json format but jq still works on such a file)
The below jq command works:
cat file.json | jq -r '.[] | [.key1,.key2]'
The above correctly shows:
[
<value_of_key1>,<value_of_key2>
]
[
<value_of_key1>,<value_of_key2>
]
However, I want .key1,.key2 to be dynamic since these keys can change. So I want to pass a variable to jq. Something like:
$KEYS=.key1,.key2
cat file.json | jq -r --arg var "$KEYS" '.[] | [$var]'
But the above is returning the keys themselves:
[
".key1,.key2"
]
[
".key1,.key2"
]
why is this happening? what is the correct command to make this happen?
This answer does not help me. I am not getting any errors as the OP in that question.
Fetching the value of a jq variable doesn't cause it to be executed as jq code.
Furthermore, jq lacks the facility to take a string, compile it as jq code, and evaluate the result. (This is commonly known as eval.)
So, short of a writing a jq parser and evaluator in jq, you will need to impose limits and/or accept a different format.
For example,
keys='[ [ "key1", "childkey" ], [ "key2", "childkey2" ] ]' # JSON
jq --argjson keys "$keys" '.[] | [ getpath( $keys[] ) ]' file.json
or
keys='key1.childkey,key2.childkey2'
jq --arg keys "$keys" '
( ( $keys / "," ) | map( . / "." ) ) as $keys |
.[] | [ getpath( $keys[] ) ]
' file.json
Suppose you have:
cat file
[{
"key1":1,
"key2":2
}]
[{
"key1":1,
"key2":2
}]
You can use a jq command like so:
jq '.[] | [.key1,.key2]' file
[
1,
2
]
[
1,
2
]
You can use -f to execute a filter from a file and nothing keeps you from creating the file separately from the shell variables.
Example:
keys=".key1"
echo ".[] | [${keys}]" >jqf
jq -f jqf file
[
1
]
[
1
]
Or just build the string directly into jq:
# note double " causing string interpolation
jq ".[] | [${keys}]" file
You can use --argjson option and destructuring.
file.json
[{"key1":{"a":1},"key2":{"b":2}}]
[{"key1":{"c":1},"key2":{"d":2}}]
$ in='["key1","key2"]' jq -c --argjson keys "$in" '$keys as [$key1,$key2] | .[] | [.[$key1,$key2]]' file.json
output:
[{"a":1},{"b":2}]
[{"c":1},{"d":2}]
Elaborating on ikegami's answer.
To start with here's my version of the answer:
$ in='key1.a,key2.b'; jq -c --arg keys "$in" '($keys/","|map(./".")) as $paths | .[] | [getpath($paths[])]' <<<$'[{"key1":{"a":1},"key2":{"b":2}}] [{"key1":{"a":3},"key2":{"b":4}}]'
This gives output
[1,2]
[3,4]
Let's try it.
We have input
[{"key1":{"a":1},"key2":{"b":2}}]
[{"key1":{"a":3},"key2":{"b":4}}]
And we want to construct array
[["key1","a"],["key2","b"]]
then use it on getpath(PATHS) builtin to extract values out of our input.
To start with we are given in shell variable with string value key1.a,key2.b. Let's call this $keys.
Then $keys/"," gives
["key1.a","key2.b"]
["key1.a","key2.b"]
After that $keys/","|map(./".") gives what we want.
[["key1","a"],["key2","b"]]
[["key1","a"],["key2","b"]]
Let's call this $paths.
Now if we do .[]|[getpath($paths[])] we get the values from our input equivalent to
[.[] | .key1.a, .key2.b]
which is
[1,2]
[3,4]
I have a Redis hash with keys and values like string key -- serialized JSON value.
Corresponding rediscli query (hgetall some_redis_hash) being dumped in a file:
redis_key1
{"value1__key1": "value1__value1", "value1__key2": "value1__value2" ...}
redis_key2
{"value2__key1": "value2__value1", "value2__key2": "value2__value2" ...}
...
and so on.
So the question is, how do I pretty-print these values enclosed in brackets? (note that key strings between are making the document invalid, if you'll try to parse the entire one)
The first thought is to get particular pairs from Redis, strip parasite keys, and use jq on the remaining valid JSON, as shown below:
rediscli hget some_redis_hash redis_key1 > file && tail -n +2 file
- file now contains valid JSON as value, the first string representing Redis key is stripped by tail -
cat file | jq
- produces pretty-printed value -
So the question is, how to pretty-print without such preprocessing?
Or (would be better in this particular case) how to merge keys and values in one big JSON, where Redis keys, accessible on the upper level, will be followed by dicts of their values?
Like that:
rediscli hgetall some_redis_hash > file
cat file | cool_parser
- prints { "redis_key1": {"value1__key1": "value1__value1", ...}, "redis_key2": ... }
A simple way for just pretty-printing would be the following:
cat file | jq --raw-input --raw-output '. as $raw | try fromjson catch $raw'
It tries to parse each line as json with fromjson, and just outputs the original line (with $raw) if it can't.
(The --raw-input is there so that we can invoke fromjson enclosed in a try instead of running it on every line directly, and --raw-output is there so that any non-json lines are not enclosed in quotes in the output.)
A solution for the second part of your questions using only jq:
cat file \
| jq --raw-input --null-input '[inputs] | _nwise(2) | {(.[0]): .[1] | fromjson}' \
| jq --null-input '[inputs] | add'
--null-input combined with [inputs] produces the whole input as an array
which _nwise(2) then chunks into groups of two (more info on _nwise)
which {(.[0]): .[1] | fromjson} then transforms into a list of jsons
which | jq --null-input '[inputs] | add' then combines into a single json
Or in a single jq invocation:
cat file | jq --raw-input --null-input \
'[ [inputs] | _nwise(2) | {(.[0]): .[1] | fromjson} ] | add'
...but by that point you might be better off writing an easier to understand python script.
I'm trying to create a json file from a string with the following format:
string="key1=value1,key2=value2"
Is there a way to create a json using jq by specifying the = and , symbols as separators for the keys and values?
The output I'm looking for would be:
{"key1": "value1", "key2” :”value2"}
I've tried to use this post as a reference:
Create JSON using jq from pipe-separated keys and values in bash -- however, it expects input that contains a line with only keys, before later lines with only values; here, the keys and values are all interspersed.
Here's a reduce-free solution that assumes string is the shell variable (not part of the string to be parsed), and that parsing of the string can be accomplished by first splitting on ",":
jq -R 'split(",")
| map( index("=") as $i | {(.[0:$i]) : .[$i+1:]})
| add' <<< "$string"
Notice that this allows "=" to appear within the values.
The only trickiness here is that when a key name is specified programmatically, it must be enclosed within parentheses.
Supplemental question
string="key1=value1|key2=value2,value3|key3=value4"
In this case, you would first split on "|", and then find the first occurrence of "=":
split("|")
| map( index("=") as $i | {(.[0:$i]) : .[$i+1:]})
| add
| map_values(if index(",") then split(",") else . end)
Output:
{
"key1": "value1",
"key2": [
"value2",
"value3"
],
"key3": "value4"
}
string="key1=value1,key2=value2"
jq -Rc '
split(",")
| [.[] | match( "([^=]*)=(.*)" )]
| reduce .[].captures as $item ({}; .[$item[0].string]=$item[1].string)
' <<<"$string"
echo -n "key1=value1,key2=value2" | \
jq -csR '[split(",")[]|split("=") | {(.[0]): .[1]}]|add'
this gives
{"key1":"value1","key2":"value2"}