How pack multiline key=value string to array of objects? - json

I have multiline string like
a=aValue
b=bValue
c=cValue
a=dValue
b=eValue
c=fValue
How using jq get json like this?
[
{"a": "aValue", "b": "bValue", "c": "cValue"},
{"a": "dValue", "b": "eValue", "c": "fValue"}
]

Here's an answer that is not tied to the number of distinct keys, and avoids slurping the lines (i.e., has minimal memory requirements):
jq -Rn 'foreach (inputs, null) as $in ({};
if $in == null then .emit = .object
else ($in | capture("(?<key>[^=]*)=(?<value>.*)") // null) as $kv
| if $kv == null
then .
elif (.object | . and has($kv.key))
then .emit = .object | .object = ([$kv]|from_entries)
else .emit = null | .object += ([$kv]|from_entries)
end
end ;
select(.emit).emit )'
The trick here is to use inputs,null so that the "end of file" condition is handled properly.
Note that the above produces a stream, so if you want all the objects in an array, simply enclose the entire jq program in square brackets:
jq -Rn '[ .... ]'

If you don't have any problem with slurping the input, this should do the trick:
jq -R -s '[ split("\n") | map(split("=") | {(.[0]): .[1]}) | _nwise(3) | add ]' file
Online demo
_nwise(n) is an undocumented internal function that given an array emits a stream of subarrays of length n. You can see how it's implemented here. For everything else, see the manual.
If values may contain equals signs, use .[1:] | join("=") instead of .[1].
For a generally-applicable, memory-friendly approach, see peak's answer.

Related

Get value if object or string if string in jq array

I have a JSON object that looks like this:
[{"name":"NAME_1"},"NAME_2"]
I would like an output of
["NAME_1", "NAME_2"]
Some of the entries in the array are an object with a key "name" and some are just a string of the name. I am trying to extract an array of the names. Using
jq -cr '.[].name // []'
throws an error as it is trying to index .name of the string object. Is there a way to check if it is a string, and if so just use its value instead of .name?
echo '[{"name":"NAME_1"},"NAME_2"]' \
| jq '[ .[] | if (.|type) == "object" then .name else . end ]'
[
"NAME_1"
"NAME_2"
]
Ref:
https://stedolan.github.io/jq/manual/#ConditionalsandComparisons
https://stedolan.github.io/jq/manual/#type
As #LéaGris comments, a simpler version
jq '[ .[] | .name? // . ]' file
https://stedolan.github.io/jq/manual/#ErrorSuppression/OptionalOperator:%3f
https://stedolan.github.io/jq/manual/#Alternativeoperator://
You can use the type function which returns "object" for objects.
jq '.[] | if type == "object" then .name else . end' file.json
To get the output as array, just wrap the whole expression into [ ... ].
Just use the error suppression operator with ?, map and scalars
jq 'map( .name?, scalars )'
Note that by using scalars, it is assumed that other than objects with name, all others are names of form NAME_*. If there are other strings as well, and you need to exclude some of them you might need to add some additional logic to do that. e.g. using startswith(..) with a string of your choice.
map( .name?, select( scalars | startswith("NAME") ) )
Demo
With your shown samples only, please try following jq code. Using tostream function here to get the required values from requirement.
jq -c '[.[] | tostream | if .[1] != null then .[1] else empty end]' Input_file

Convert value of json from int to string using jq

Given a json that looks something like:
[{"id":1,"firstName":"firstName1","lastName":"lastName1"},
{"id":2,"firstName":"firstName2","lastName":"lastName2"},
{"id":3,"firstName":"firstName3","lastName":"lastName3"}]
What would be the best way to convert the id value from an int to a string and then saving the file?
I have tried:
echo "$(jq -r '[.[] | .id = .id|tostring]' test.json)" > test.json
But that seems to put each entry into a string and adds the backslashes
[
"{\"id\":1,\"firstName\":\"firstName1\",\"lastName\":\"lastName1\"}",
"{\"id\":2,\"firstName\":\"firstName2\",\"lastName\":\"lastName2\"}",
"{\"id\":3,\"firstName\":\"firstName3\",\"lastName\":\"lastName3\"}"
]
| has a lower priority than the assignment (=). The expression .id = .id | tostring is interpreted as (.id = .id) | tostring.
The assignment does change anything and can be removed. The script becomes [ .[] | tostring ], that explains the output (each object is serialized as JSON into a string).
The solution is to use parentheses to enforce the desired order of execution.
The command is:
jq '[ .[] | .id = (.id | tostring) ]' test.json
Do not use process expansion ($(...)) to compose an echo command line. It is inefficient and not needed.
Redirect the output of jq directly to a file. Use a different file than the input file (or it ends up destroying your data).
jq '[ .[] | .id = (.id | tostring) ]' test.json > output.json

How can I use jq to remove quotes from key names and remove braces?

I'm interested in transforming from json format to tfvars using jq, i.e.:
Input:
{
"foo": "aaa",
"bar": "bbb",
}
Desired output:
foo = "aaa"
bar = "bbb"
I tried
echo "{\"foo\": \"aaa\",\"bar\": \"bbb\"}" | jq '.[]'
"aaa"
"bbb"
The tfvars specification is hard to come by, but numbers should not be quoted, null is a special case, and arrays are also allowed as values, e.g. https://learn.hashicorp.com/tutorials/terraform/google-cloud-platform-variables?in=terraform/gcp-get-started gives the following as an example:
cidrs = [ "10.0.0.0/16", "10.1.0.0/16" ]
So the following should be closer to a general solution:
jq -r '
def q:
if type | IN("string", "boolean") then "\"\(tostring)\""
else .
end;
to_entries[] | "\(.key) = \(.value|q)"
'
A modification of the earlier answer might work.
jq -r 'to_entries[] | "\(.key) = \"\(.value)\""'

Map arrays to objects with no common fields

How might one use jq-1.5-1-a5b5cbe to join a filtered set of arrays from STDIN to a set of objects which contains no common fields, assuming that all elements will be in predictable order?
Standard Input (pre-slurpfile; generated by multiple GETs):
{"ref":"objA","arr":["alpha"]}
{"ref":"objB","arr":["bravo"]}
Existing File:
[{"name":"foo"},{"name":"bar"}]
Desired Output:
[{"name":"foo","arr":["alpha"]},{"name":"bar","arr":["bravo"]}]
Current Bash:
$ multiGET | jq --slurpfile stdin /dev/stdin '.[].arr = $stdin[].arr' file
[
{
"name": "foo",
"arr": [
"alpha"
]
},
{
"name": "bar",
"arr": [
"alpha"
]
}
]
[
{
"name": "foo",
"arr": [
"bravo"
]
},
{
"name": "bar",
"arr": [
"bravo"
]
}
]
Sidenote: I wasn't sure when to use pretty/compact JSON in this question; please comment with your opinion on best practice.
Get jq to read file before stdin, so that the first entity in file will be . and you can get everything else using inputs.
$ multiGET | jq -c '. as $objects
| [ foreach (inputs | {arr}) as $x (-1; .+1;
. as $i | $objects[$i] + $x
) ]' file -
[{"name":"foo","arr":["alpha"]},{"name":"bar","arr":["bravo"]}]
"Slurping" (whether using -s or --slurpfile) is sometimes necessary but rarely desirable, because of the memory requirements. So here's a solution that takes advantage of the fact that your multiGET produces a stream:
multiGET | jq -n --argjson objects '[{"name":"foo"},{"name":"bar"}]' '
$objects
| [foreach inputs as $in (-1; .+1;
. as $ix
| $objects[$ix] + ($in | del(.ref)))]
'
Here's a functional approach that might be appropriate if your stream was in fact already packaged as an array:
multiGET | jq -s --argjson objects '[{"name":"foo"},{"name":"bar"}]' '
[$objects, map(del(.ref))]
| transpose
| map(add)
'
If the $objects array is in a file or too big for the command line, I'd suggest using --argfile, even though it is technically deprecated.
If the $objects array is in a file, and if you want to avoid --argfile, you could still avoid slurping, e.g. by using the fact that unless -n is used, jq will automatically read one JSON entity from stdin:
(echo '[{"name":"foo"},{"name":"bar"}]';
multiGET) | jq '
. as $objects
| [foreach inputs as $in (-1; .+1;
. as $ix | $objects[$ix] + $in | del(.ref))]
'

Create JSON from string with format "key1=value1,key2=value2" using jq

I'm trying to create a json file from a string with the following format:
string="key1=value1,key2=value2"
Is there a way to create a json using jq by specifying the = and , symbols as separators for the keys and values?
The output I'm looking for would be:
{"key1": "value1", "key2” :”value2"}
I've tried to use this post as a reference:
Create JSON using jq from pipe-separated keys and values in bash -- however, it expects input that contains a line with only keys, before later lines with only values; here, the keys and values are all interspersed.
Here's a reduce-free solution that assumes string is the shell variable (not part of the string to be parsed), and that parsing of the string can be accomplished by first splitting on ",":
jq -R 'split(",")
| map( index("=") as $i | {(.[0:$i]) : .[$i+1:]})
| add' <<< "$string"
Notice that this allows "=" to appear within the values.
The only trickiness here is that when a key name is specified programmatically, it must be enclosed within parentheses.
Supplemental question
string="key1=value1|key2=value2,value3|key3=value4"
In this case, you would first split on "|", and then find the first occurrence of "=":
split("|")
| map( index("=") as $i | {(.[0:$i]) : .[$i+1:]})
| add
| map_values(if index(",") then split(",") else . end)
Output:
{
"key1": "value1",
"key2": [
"value2",
"value3"
],
"key3": "value4"
}
string="key1=value1,key2=value2"
jq -Rc '
split(",")
| [.[] | match( "([^=]*)=(.*)" )]
| reduce .[].captures as $item ({}; .[$item[0].string]=$item[1].string)
' <<<"$string"
echo -n "key1=value1,key2=value2" | \
jq -csR '[split(",")[]|split("=") | {(.[0]): .[1]}]|add'
this gives
{"key1":"value1","key2":"value2"}