Processing string representation of json within jq - json

I have a function in bash psql_json() that hits a postgres database and returns a json. I can not edit this bash function but it injects a postgres statement into this this query which is then sent to the db:
"select row_to_json(t)::json from ($arg) t;"
Where arg is some statement ie:
select * from table
However, this function returns a weirdly formatted json string as seen below:
{\"id\":1,\"firstName\":\"firstName1\",\"lastName\":\"lastName1\"}\n
{\"id\":2,\"firstName\":\"firstName2\",\"lastName\":\"lastName2\"}
The above output is what happens after running these statements:
local_fn="$(mktemp)"
psql_json 'select * from table' > "$local_fn"
cat "$local_fn"
Now when I try to put this json as is into jq, I get the following error:
cat json1.json | jq '.'
jq: error: syntax error, unexpected $end, expecting ';' or ')' (Unix shell quoting issues?)
I found this thread which seems to indicate the issue is that I am passing a string into jq which it doesnt like and is unable to parse, so I tried both:
cat json1.json | jq 'fromjson]'
cat json1.json | jq '[.[]|fromjson]'
and they both return
parse error: Invalid numeric literal at line 1, column 3
Is there any way I can get string representation of the json above into jq in a clean way to process it or would I need to clean/edit the string in bash?

You could fix the input using jq or a text-processing tool such as sed:
< weird.json jq -R 'sub("^";"\"") | sub("$";"\"") | fromjson | fromjson'
or
< weird.json sed -e 's/^/"/' -e 's/$/"/' | jq -R 'fromjson|fromjson'
With your input, the result in both cases is:
{
"id": 1,
"firstName": "firstName1",
"lastName": "lastName1"
}
{
"id": 2,
"firstName": "firstName2",
"lastName": "lastName2"
}

Related

How to dynamically parse a JSON object with shell jq

I got a question about shell's jq. So my JSON object is:
{"1543446000": {"name": "John", "company": "foo"}, "1543359600": {"name": "Kate", "company": "bar"}}
The numers 1543446000 and 1543359600 are UNIX timestamps. How can I parse one of the JSON objects by the timestamp with a shell variable?
My shell script so far:
#!/bin/sh
URL="https://pastebin.com/raw/w7awz7kZ"
export DATE=$(date -d "$today 0" +%s)
JSON=$(curl -H "Accept: application/json" $API_URL)
JSON=$(echo $JSON | jq --arg date $DATE '.$date')
echo $JSON
Doesn't seem to work. My intention is to select the inner JSON object described by one of the timestamps, which are basically midnight of today. So I want to select today's data set.
Any suggestions?
Greets,
Innoberger
You need to use the full syntax for key access, as the dollar sign preclude you using the shorter form. The error message should provide this suggestion.
$ jq --arg date 1543359600 '.$date' tmp.json
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.$date
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.$date
jq: 2 compile errors
Note the error message
'try .["field"] instead of .field'.
You won't need the quotes, though, as that would be how you specify a literal key $date.
$ jq --arg date 1543359600 '.[$date]' tmp.json
{
"name": "Kate",
"company": "bar"
}

Json invalid for perl but ok for jq

Why does the following string passes as JSON with jq but not with perl?
Example:
$ cat dummy.json | jq '.'
{
"field": {
"customer_id": "abc"
},
"result": "processed"
}
But with perl fails:
$ cat dummy.json | perl -MData::Dumper -MJSON=decode_json -ne'print decode_json($_)'
, or } expected while parsing object/hash, at character offset 1 (before "\n") at -e line 1.
What am I messing up here?
The decoder works fine, but you're only reading one line.
Slurp the file with perl -0777…

Looping over a list of keys to extract from a JSON file with jq

I'm trying to extract a series of properties (named in an input file) in jq and getting error when I feed those from bash via a loop:
while read line; do echo $line; cat big.json | jq ".$line"; sleep 1; done < big.properties.service
cfg.keyload.service.count
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
When i try to do it manually it works
$ line=cfg.keyload.service.count
$ echo $line
cfg.keyload.service.count
$ cat big.json | jq ".$line"
1
Is there any way to get it work in loop?
Here is example
cat >big.json <<EOF
{
"cfg": {
"keyload": {
"backend": {
"app": {
"shutdown": {
"timeout": "5s"
},
"jmx": {
"enable": true
}
}
}
}
}
}
EOF
cat >big.properties.service <<EOF
cfg.keyload.backend.app.shutdown.timeout
cfg.keyload.backend.app.jmx.enable
cfg.keyload.backend.app.jmx.nonexistent
cfg.nonexistent
EOF
...output should be:
cfg.keyload.backend.app.shutdown.timeout
"5s"
cfg.keyload.backend.app.jmx.enable
true
cfg.keyload.backend.app.jmx.nonexistent
null
cfg.nonexistent
null
Immediate Issue - Invalid Input
The "invalid character" at hand here is almost certainly a carriage return. Use dos2unix to convert your input file to a proper UNIX text file, and your original code will work (albeit very inefficiently, rereading your whole big.json every time it wants to extract a single property).
Performant Implementation - Loop In JQ, Not Bash
Don't use a bash loop for this at all -- it's much more efficient to have jq do the looping.
Note the sub("\r$"; "") used in this code to remove trailing carriage returns so it can accept input in DOS format.
jq -rR --argfile infile big.json '
sub("\r$"; "") as $keyname
| ($keyname | split(".")) as $pieces
| (reduce $pieces[] as $piece ($infile; .[$piece]?)) as $value
| ($keyname, ($value | tojson))
' <big.properties.service
properly emits as output, when given the inputs in the question:
cfg.keyload.backend.app.shutdown.timeout
"5s"
cfg.keyload.backend.app.jmx.enable
true
cfg.keyload.backend.app.jmx.nonexistent
null
cfg.nonexistent
null
Your properties file is effectively paths in the json that you want to retrieve values from. Convert them to paths that jq recognizes so you can get those values. Just make an array of keys that would need to be traversed. Be sure to read your properties file as raw input (-R) since it's not json, and use raw output (-r) to be able to output the paths as you want.
$ jq --argfile big big.json '
., (split(".") as $p | $big | getpath($p) | tojson)
' -Rr big.properties.service
cfg.keyload.backend.app.shutdown.timeout
"5s"
cfg.keyload.backend.app.jmx.enable
true
cfg.keyload.backend.app.jmx.nonexistent
null
cfg.nonexistent
null

How to properly chain multiple jq statements together when processing json in the shell such as with curl?

I am new to jq so if this is not a jq question or a json question please point me in the right direction. I am not sure of the correct terminology so it is making it hard for me to properly articulate the problem.
I am using to curl to pull some json that I want to filter out keys with specific values. Here is some of the sample json:
{
"id": "593f468c81aaa30001960e16",
"name": "Name 1",
"channels": [
"593f38398481bc00019632e5"
],
"geofenceProfileId": null
}
{
"id": "58e464585180ac000a748b57",
"name": "Name 2",
"channels": [
"58b480097f04f20007f3cdca",
"580ea26616de060006000001"
],
"geofenceProfileId": null
}
{
"id": "58b4d6db7f04f20007f3cdd2",
"name": "Name 3",
"channels": [
"58b8a25cf9f6e19cf671872f"
],
"geofenceProfileId": "57f53018271c810006000001"
}
When I run the following command:
curl -X GET -H 'authorization: Basic somestring=' "https://myserver/myjson" |
jq '.[] | {id: .id, name: .name, channels: .channels, geofenceProfileId: .geofenceProfileId}' |
jq '.[] | select(.channels == 58b8a25cf9f6e19cf671872f)'
I get the following error:
jq: error: syntax error, unexpected IDENT, expecting ';' or ')' (Unix shell quoting issues?) at , line 1:
.[] | select(.channels == 58b8a25cf9f6e19cf671872f)
jq: 1 compile error
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 351k 0 351k 0 0 1109k 0 --:--:-- --:--:-- --:--:-- 1110k
Is this error because jq pretty prints the output of the first statement and the second statement is expecting it to be in one code block? If so, how do I convert it back to non pretty print format or how can I use jq to run a new filter on the output?
Basically I am trying to parse hundreds of records and filter out all of the records that are in a specific channel number or have a specific geofenceProfileId.
I'd suggest you start with:
jq 'select(.channels | index("58b8a25cf9f6e19cf671872f"))'
In fact, this might even be exactly the filter you want. If you want to remove the "channels" once you've made the selection, you could augment the filter above as follows:
select(.channels | index("58b8a25cf9f6e19cf671872f")) | del(.channels)
The main thing to note is that one can create a pipeline WITHIN a single invocation of jq. So most likely you'll end up with: curl ... | jq ...
Btw
The jq expression {"id": .id} can be abbreviated to {id}, so instead of:
{id.id, name: .name, channels: .channels, geofenceProfileId: .geofenceProfileId}
you could write:
{id, name, channels, geofenceProfileId}
Probably not related to your case but I managed to transform my command
npm pkg get version -ws | jq "select(to_entries | min_by(.value) | .value)"
to
npm pkg get version -ws | jq "to_entries | min_by(.value) | .value"
and result is same. May be it helps. SO the idea is to pipe inside jq statement

Parsing errors of JSON using jq

I have a curl command which results in the following example json:
json={"id":"12345","key":"ABC-DEF","url":"https://google.com"}
Now, I want to parse this, and get the key out of it and store it in a variable. What I did was the following:
json={"id":"12345","key":"ABC-DEF","url":"https://google.com"}
ID=$(echo $json | jq '.key' )
But the above gives me a error as: parse error: Invalid numeric literal at line 1, column 4. Can someone help me with this? Thanks!
You will need to quote the input string so the shell doesn't do anything with your string
json='{"id":"12345","key":"ABC-DEF","url":"https://google.com"}'
ID=$(echo "$json" | jq '.key' )