I have 2 json objects comming from a rest api. I want to compare if they are the same object.
objectA:
{
"type": {
"S": "equal"
},
"preFilter": {
"BOOL": true
}
}
objectB:
{
"preFilter": {
"BOOL": true
},
"type": {
"S": "equal"
}
}
They are the same, but an md5sum will see them as different. I tried inserting them in 2 different files, and compare the files using something proposed
here: but I would like to know if it's possible to use jq on the fly to compare variables.
I've been trying to change
--argfile a a.json
for
--arg a $a
(being $a a json string) with no luck. Any idea how to approach strings, not files?
It would probably be simplest to use the --argjson command-line option, e.g.
jq -n --argjson a "$a" --argjson b "$b" '$a == $b'
Of course there are alternatives, e.g. using jq -s ...
Related
disclaimer: indeed, there are already different answers (like JQ Join JSON files by key or denormalizing JSON with jq) for but none of them helped me yet or did have different circumstances I was unable to derive a solution from ;/
I have 2 files, both are lists of objects where one of them ha field references to object ids of the other one
given
[
{
"id": "5b9f50ccdcdf200283f29052",
"reference": {
"id": "5de82d5072f4a72ad5d5dcc1"
}
}
]
and
[
{
"id": "5de82d5072f4a72ad5d5dcc1",
"name": "FooBar"
}
]
my goal would be to get a denormalized object list:
expected
[
{
"id": "5b9f50ccdcdf200283f29052",
"reference": {
"id": "5de82d5072f4a72ad5d5dcc1",
"name": "FooBar"
}
}
]
while I'm able to do the main parts, I didn't challenged to bring both together yet:
with
example 1
jq -s '(.[1][] | select(.id == "5de82d5072f4a72ad5d5dcc1"))' objects.json referredObjects.json
I get
{
"id": "5de82d5072f4a72ad5d5dcc1",
"name": "FooBar"
}
and with
example 2
jq -s '.[0][] | .reference = {}' objects.json referredObjects.json
I can manipulate any .reference getting
{
"id": "5b9f50ccdcdf200283f29052",
"reference": {}
}
(even I loose the list structure)
But: I can't do s.th. like
execpted "join"
jq -s '.[0][] as $obj | $obj.reference = (.[1][] | select(.id == $obj.reference.id))' objects.json referredObjects.json
even approaches with foreach or reduce looks promising
jq -s '[foreach .[0][] as $obj ({}; .reference.id = ""; . + $obj )]' objects.json referredObjects.json
=>
[
{
"reference": {
"id": "5de82d5072f4a72ad5d5dcc1"
},
"id": "5b9f50ccdcdf200283f29052"
}
]
where I expected to get the same as in second example
I end up in headaches and looking forward to write a ineffective while routine in any language ... hopefully I would appreciate any help on this
~Marcel
Transform the second file into an object where ids and names are paired and use it as a reference while updating the first file.
$ jq '(map({(.id): .}) | add) as $idx
| input
| map_values(.reference = $idx[.reference.id])' file2 file1
[
{
"id": "5b9f50ccdcdf200283f29052",
"reference": {
"id": "5de82d5072f4a72ad5d5dcc1",
"name": "FooBar"
}
}
]
The following solution uses the same strategy as used in the solution by #OguzIsmail but uses the built-in function INDEX/2 to construct the dictionary from the second file.
The important point is that this strategy allows the arrays in both files to be of arbitrary size.
Invocation
jq --argfile file2 file2.json -f program.jq file1.json
program.jq
INDEX($file2[]; .id) as $dict
| map(.reference.id as $id | .reference = $dict[$id])
Using jq how can I convert an array into object indexed by filename, or read multiple files into one object indexed by their filename?
e.g.
jq -s 'map(select(.roles[]? | contains ("mysql")))' -C dir/file1.json dir/file2.json
This gives me the data I want, but I need to know which file they came from.
So instead of
[
{ "roles": ["mysql"] },
{ "roles": ["mysql", "php"] }
]
for output, I want:
{
"file1": { "roles": ["mysql"] },
"file2": { "roles": ["mysql", "php"] }
}
I do want the ".json" file extension stripped too if possible, and just the basename (dir excluded).
Example
file1.json
{ "roles": ["mysql"] }
file2.json
{ "roles": ["mysql", "php"] }
file3.json
{ }
My real files obviously have other stuff in them too, but that should be enough for this example. file3 is simply to demonstrate "roles" is sometimes missing.
In other words: I'm trying to find files that contain "mysql" in their list of "roles". I need the filename and contents combined into one JSON object.
To simplify the problem further:
jq 'input_filename' f1 f2
Gives me all the filenames like I want, but I don't know how to combine them into one object or array.
Whereas,
jq -s 'map(input_filename)' f1 f2
Gives me the same filename repeated once for each file. e.g. [ "f1", "f1" ] instead of [ "f1", "f2" ]
If your jq has inputs (as does jq 1.5) then the task can be accomplished with just one invocation of jq.
Also, it might be more efficient to use any than iterating over all the elements of .roles.
The trick is to invoke jq with the -n option, e.g.
jq -n '
[inputs
| select(.roles and any(.roles[]; contains("mysql")))
| {(input_filename | gsub(".*/|\\.json$";"")): .}]
| add' file*.json
jq approach:
jq 'if (.roles[] | contains("mysql")) then {(input_filename | gsub(".*/|\\.json$";"")): .}
else empty end' ./file1.json ./file2.json | jq -s 'add'
The expected output:
{
"file1": {
"roles": [
"mysql"
]
},
"file2": {
"roles": [
"mysql",
"php"
]
}
}
Looks like it's not actual for jq 1.4, could you provide any other ways to join JSON files by key?
e.g
{
"key": "874102296-1",
"que_lat": "40"
}
{
"key": "874102296-2",
"que_lat": "406790"
}
and
{
"key": "874102296-1",
"in_time": "1530874104733",
"latency": "12864258288242"
}
{
"key": "874102296-2",
"in_time": "1530874104746"
}
As a result, i'd like to have something like this:
{
"key": "874102296-1",
"in_time": "1530874104733",
"full_latency": "12864258288242",
"que_lat": "40"
}
{
"key": "874102296-2",
"in_time": "1530874104746",
"que_lat": "406790"
}
Thanks!
The problem can easily be solved using the def of hashJoin given in the SO page that you cite.
Solution using jq 1.5 or higher
If you have jq 1.5 or higher, you could use this invocation:
jq -n --slurpfile f1 file1.json --slurpfile f2 file2.json -f join.jq
where join.jq contains the second def of hashJoin, together with:
hashJoin($f1; $f2; .key)[]
Solution using jq 1.4
If you have jq 1.4, the trickiness is to read each of the two files separately as an array. Here's one approach that assumes bash:
jq -n --argfile f1 <(jq -s . file1.json) --argfile f2 <(jq -s . file2.json) -f join.jq
where join.jq is as above.
If you cannot use bash, then it might be simplest to create temporary files.
I'm trying to create a JSON file by executing the following command:
jq --arg greeting world '{"hello":"$greeting"}' > file.json
This command stuck without any input. While
jq -n --arg greeting world '{"hello":"$greeting"}' > file.json
doesn't parse correctly. I'm just wondering is really possible to create a JSON file.
So your code doesn't work because included the variable inside double quotes which gets treated as string. That is why it is not working
As #Jeff Mercado, pointed out the solution is
jq -n --arg greeting world '{"hello":$greeting}' > file.json
About the - in a name. This is actually possible. But as of now this is not available in released version of jq. If you compile the master branch of jq on your system. There is a new variable called $ARGS.named which can be used to access the information.
I just compiled and check the below command and it works like a charm
./jq -n --arg name-is tarun '{"name": $ARGS.named["name-is"]}'
{
"name": "tarun"
}
$ARGS provides access to named (--arg name value) and positional (--args one two three) arguments from the jq command line, and allows you to build up objects easily & safely.
Named arguments:
$ jq -n '{christmas: $ARGS.named}' \
--arg one 'partridge in a "pear" tree' \
--arg two 'turtle doves'
{
"christmas": {
"one": "partridge in a \"pear\" tree",
"two": "turtle doves"
}
}
Positional arguments:
$ jq -n '{numbers: $ARGS.positional}' --args 1 2 3
{
"numbers": [
"1",
"2",
"3"
]
}
Note you can access individual items of the positional array, and that the named arguments are directly available as variables:
jq -n '{first: {name: $one, count: $ARGS.positional[0]}, all: $ARGS}' \
--arg one 'partridge in a "pear" tree' \
--arg two 'turtle doves' \
--args 1 2 3
{
"first": {
"name": "partridge in a \"pear\" tree",
"count": "1"
},
"all": {
"positional": [
"1",
"2",
"3"
],
"named": {
"one": "partridge in a \"pear\" tree",
"two": "turtle doves"
}
}
}
To add to what Jeff and Tarun have already said, you might want to use the \() string interpolation syntax in your command. eg.
jq -n --arg greeting world '{"hello":"\($greeting)"}'
for me this produces
{
"hello": "world"
}
Regarding your reply to Jeff's comment, the argument name you choose has to be a valid jq variable name so an arg like greeting-for-you won't work but you could use underscores so greeting_for_you would be ok. Or you could use the version Tarun described.
I've got JSON that looks like this
{
"keyword1": {
"identifier1": 16
},
"keyword2": {
"identifier2": 16
}
}
and I need to loop through the keywords to get the identifiers (not sure if I'm using the right terminology here). Seems pretty simple, but because the keywords are all named different, I don't know how to handle that.
The original tag for this question was jq so here is a jq solution:
.[] | keys[]
For example, with the input as shown in the question:
$ jq '.[] | keys[]' input.json
"identifier1"
"identifier2"
To retrieve the key names in the order they appear in the JSON object, use keys_unsorted.
I'd think something along these lines would work well:
jq '. | to_entries | .[].key'
see https://stedolan.github.io/jq/manual/#to_entries,from_entries,with_entries
or if you wanted to get the values from a variable:
JSON_DATA={main:{k1:v1,k2:v2}}
result=$(jq -n "$JSON_DATA" | jq '.main | to_entries | .[].value' --raw-output)
echo $result
##outputs: v1 v2
I came here hoping to sort out a bunch of keys from my JSON, I found two features handy. There are three functions "to_entries", "from_entries", and "with_entries". You can filter the values by key or value, like so:
JSON_DATA='
{
"fields": {
"first": null,
"second": "two",
"third": "three"
}
}
'
echo "$JSON_DATA" | jq '{fields: .fields | with_entries(select(.value != null and .key != "third")) }'
Output:
{
"fields": {
"second": "two"
}
}
simpler solution - just treat internal hash as a new hash and add one more filter. The query that helped me:
$ docker network inspect bridge|jq '.[].Containers'
{
"35c9e1273c43db01c45b5f43f6999d04c18beff3996ea09fb8b87a8b635c38ff": {
"Name": "nginx",
"EndpointID": "a6e788d6f90eb14df2321a2eb02517f0862c1fe7fe50c02f2b8c103c0c79cb6b",
"MacAddress": "02:42:ac:11:00:02",
"IPv4Address": "172.17.0.2/16",
"IPv6Address": ""
},
"b46c157cec243969f9227251dfd6fa65b7a904e145df80a63f79d4dc8b281355": {
"Name": "sweet_gates",
"EndpointID": "a600d9c1ee35b9f7db31249ae8f589c202e0b260e10a394757a88bfd66b5b42f",
"MacAddress": "02:42:ac:11:00:03",
"IPv4Address": "172.17.0.3/16",
"IPv6Address": ""
}
}
As I needed only couple of fields, add to above .json one more query:
$ docker network inspect bridge|jq -jr '.[].Containers[]|.IPv4Address, "\t", .Name, "\n"'
172.17.0.2/16 nginx
172.17.0.3/16 sweet_gates