Why does the following string passes as JSON with jq but not with perl?
Example:
$ cat dummy.json | jq '.'
{
"field": {
"customer_id": "abc"
},
"result": "processed"
}
But with perl fails:
$ cat dummy.json | perl -MData::Dumper -MJSON=decode_json -ne'print decode_json($_)'
, or } expected while parsing object/hash, at character offset 1 (before "\n") at -e line 1.
What am I messing up here?
The decoder works fine, but you're only reading one line.
Slurp the file with perl -0777…
Related
I have a function in bash psql_json() that hits a postgres database and returns a json. I can not edit this bash function but it injects a postgres statement into this this query which is then sent to the db:
"select row_to_json(t)::json from ($arg) t;"
Where arg is some statement ie:
select * from table
However, this function returns a weirdly formatted json string as seen below:
{\"id\":1,\"firstName\":\"firstName1\",\"lastName\":\"lastName1\"}\n
{\"id\":2,\"firstName\":\"firstName2\",\"lastName\":\"lastName2\"}
The above output is what happens after running these statements:
local_fn="$(mktemp)"
psql_json 'select * from table' > "$local_fn"
cat "$local_fn"
Now when I try to put this json as is into jq, I get the following error:
cat json1.json | jq '.'
jq: error: syntax error, unexpected $end, expecting ';' or ')' (Unix shell quoting issues?)
I found this thread which seems to indicate the issue is that I am passing a string into jq which it doesnt like and is unable to parse, so I tried both:
cat json1.json | jq 'fromjson]'
cat json1.json | jq '[.[]|fromjson]'
and they both return
parse error: Invalid numeric literal at line 1, column 3
Is there any way I can get string representation of the json above into jq in a clean way to process it or would I need to clean/edit the string in bash?
You could fix the input using jq or a text-processing tool such as sed:
< weird.json jq -R 'sub("^";"\"") | sub("$";"\"") | fromjson | fromjson'
or
< weird.json sed -e 's/^/"/' -e 's/$/"/' | jq -R 'fromjson|fromjson'
With your input, the result in both cases is:
{
"id": 1,
"firstName": "firstName1",
"lastName": "lastName1"
}
{
"id": 2,
"firstName": "firstName2",
"lastName": "lastName2"
}
Let's say 123.json with below content:
{
"LINE" : {
"A_serial" : "1234",
"B_serial" : "2345",
"C_serial" : "3456",
"X_serial" : "76"
}
}
If I want to use a shell script to change the parameter of X_serial by the original number +1 which is 77 in this example.
I have tried the below script to take out the parameter of X_serial:
grep "X_serial" 123.json | awk {print"$3"}
which outputs 76. But then I don't know how to make it into 77 and then put it back to the parameter of X_serial.
It's not a good idea to use line-oriented tools for parsing/manipulating JSON data. Use jq instead, for example:
$ jq '.LINE.X_serial |= "\(tonumber + 1)"' 123.json
{
"LINE": {
"A_serial": "1234",
"B_serial": "2345",
"C_serial": "3456",
"X_serial": "77"
}
}
This simply updates .LINE.X_serial by converting its value to a number, increasing the result by one, and converting it back to a string.
You need to install powerful JSON querying processor like jq processor. you can can easily install from here
once you install jq processor, try following command to extract the variable from JSON key value
value=($(jq -r '.X_serial' yourJsonFile.json))
you can modify the $value as you preferred operations
With pure Javascript: nodejs and bash :
node <<EOF
var o=$(</tmp/file);
o["LINE"]["X_serial"] = parseInt(o["LINE"]["X_serial"]) + 1;
console.log(o);
EOF
Output
{ LINE:
{ A_serial: '1234',
B_serial: '2345',
C_serial: '3456',
X_serial: 78 }
}
sed or perl, depending on whether you just need string substitution or something more sophisticated, like arithmetic.
Since you tried grep and awk, let's start with sed:
In all lines that contain TEXT, replace foo with bar
sed -n '/TEXT/ s/foo/bar/ p'
So in your case, something like:
sed -n '/X_serial/ s/\"76\"/\"77\"/ p'
or
$ cat 123.json | sed '/X_serial/ s/\"76\"/\"77\"/' > new.json
This performs a literal substiution: "76" -> "77"
If you would like to perform arithmetic, like "+1" or "+10" then use perl not sed:
$ cat 123.json | perl -pe 's/\d+/$&+10/e if /X_serial/'
{
"LINE" : {
"A_serial" : "1234",
"B_serial" : "2345",
"C_serial" : "3456",
"X_serial" : "86"
}
}
This operates on all lines containing X_serial (whether under "LINE" or under something else), as it is not a json parser.
I have file that looks like this:
$ cat sample-test.json |jq .
{
"logRef": "c4fa4367-23f6-462f-b5fd-f972d0916a30",
"timestamp": 1563268297545,
"someOtherField": "nonImportantValue"
}
{
"logRef": "c4fa4367-23f6-462f-b5fd-f972d0916a31",
"timestamp": 1563268297595,
"someOtherField2": "nonImportantValue3"
}
And I would like to convert it to csv like this:
logRef;timestamp
c4fa4367-23f6-462f-b5fd-f972d0916a30;1563268297545
c4fa4367-23f6-462f-b5fd-f972d0916a31;1563268297595
I was trying
$ cat sample-test.json |jq '.logRef, .timestamp |#csv'
jq: error (at <stdin>:1): string ("c4fa4367-2...) cannot be csv-formatted, only array
jq: error (at <stdin>:2): string ("c4fa4367-2...) cannot be csv-formatted, only array
Your input is fine (it's a JSON stream).
The problem with your filter is that #csv expects an array. So this will work:
[.logRef,.timestamp] | #csv
However it quotes strings, so if you want your strings unquoted (which might mean the result won't be CSV), then you could use:
"\(.logRef),\(.timestamp)"
In all cases, you'll need to use jq's-r command-line option.
The problem in your json file. Looks like it has incorrect format (without root array element [] and commas between documents). If you fix it, jq will work as expected.
> cat sample-test.json
[{
"logRef": "c4fa4367-23f6-462f-b5fd-f972d0916a30",
"timestamp": 1563268297545,
"someOtherField": "nonImportantValue"
},
{
"logRef": "c4fa4367-23f6-462f-b5fd-f972d0916a31",
"timestamp": 1563268297595,
"someOtherField2": "nonImportantValue3"
}]
cat sample-test.json |jq -r 'map(.logRef), map(.timestamp) | #csv'
"c4fa4367-23f6-462f-b5fd-f972d0916a30","c4fa4367-23f6-462f-b5fd-f972d0916a31"
1563268297545,1563268297595
I've also fixed the command with map() function.
I'm trying to extract a series of properties (named in an input file) in jq and getting error when I feed those from bash via a loop:
while read line; do echo $line; cat big.json | jq ".$line"; sleep 1; done < big.properties.service
cfg.keyload.service.count
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
When i try to do it manually it works
$ line=cfg.keyload.service.count
$ echo $line
cfg.keyload.service.count
$ cat big.json | jq ".$line"
1
Is there any way to get it work in loop?
Here is example
cat >big.json <<EOF
{
"cfg": {
"keyload": {
"backend": {
"app": {
"shutdown": {
"timeout": "5s"
},
"jmx": {
"enable": true
}
}
}
}
}
}
EOF
cat >big.properties.service <<EOF
cfg.keyload.backend.app.shutdown.timeout
cfg.keyload.backend.app.jmx.enable
cfg.keyload.backend.app.jmx.nonexistent
cfg.nonexistent
EOF
...output should be:
cfg.keyload.backend.app.shutdown.timeout
"5s"
cfg.keyload.backend.app.jmx.enable
true
cfg.keyload.backend.app.jmx.nonexistent
null
cfg.nonexistent
null
Immediate Issue - Invalid Input
The "invalid character" at hand here is almost certainly a carriage return. Use dos2unix to convert your input file to a proper UNIX text file, and your original code will work (albeit very inefficiently, rereading your whole big.json every time it wants to extract a single property).
Performant Implementation - Loop In JQ, Not Bash
Don't use a bash loop for this at all -- it's much more efficient to have jq do the looping.
Note the sub("\r$"; "") used in this code to remove trailing carriage returns so it can accept input in DOS format.
jq -rR --argfile infile big.json '
sub("\r$"; "") as $keyname
| ($keyname | split(".")) as $pieces
| (reduce $pieces[] as $piece ($infile; .[$piece]?)) as $value
| ($keyname, ($value | tojson))
' <big.properties.service
properly emits as output, when given the inputs in the question:
cfg.keyload.backend.app.shutdown.timeout
"5s"
cfg.keyload.backend.app.jmx.enable
true
cfg.keyload.backend.app.jmx.nonexistent
null
cfg.nonexistent
null
Your properties file is effectively paths in the json that you want to retrieve values from. Convert them to paths that jq recognizes so you can get those values. Just make an array of keys that would need to be traversed. Be sure to read your properties file as raw input (-R) since it's not json, and use raw output (-r) to be able to output the paths as you want.
$ jq --argfile big big.json '
., (split(".") as $p | $big | getpath($p) | tojson)
' -Rr big.properties.service
cfg.keyload.backend.app.shutdown.timeout
"5s"
cfg.keyload.backend.app.jmx.enable
true
cfg.keyload.backend.app.jmx.nonexistent
null
cfg.nonexistent
null
I have the following file
[
{
"id": 1,
"name": "Arthur",
"age": "21"
},
{
"id": 2,
"name": "Richard",
"age": "32"
}
]
To display login and id together, I am using the following command
$ jq '.[] | .name' test
"Arthur"
"Richard"
But when I put it in a shell script and try to assign it to a variable then the whole output is displayed on a single line like below
#!/bin/bash
names=$(jq '.[] | .name' test)
echo $names
$ ./script.sh
"Arthur" "Richard"
I want to break at every iteration similar to how it works on the command line.
Couple of issues in the information you have provided. The jq filter .[] | .login, .id will not produce the output as you claimed on jq-1.5. For your original JSON
{
"login":"dmaxfield",
"id":7449977
}
{
"login":"stackfield",
"id":2342323
}
It will produce four lines of output as,
jq -r '.login, .id' < json
dmaxfield
7449977
stackfield
2342323
If you are interested in storing them side by side, you need to do variable interpolation as
jq -r '"\(.login), \(.id)"' < json
dmaxfield, 7449977
stackfield, 2342323
And if you feel your output stored in a variable is not working. It is probably because of lack of double-quotes when you tried to print the variable in the shell.
jqOutput=$(jq -r '"\(.login), \(.id)"' < json)
printf "%s\n" "$jqOutput"
dmaxfield, 7449977
stackfield, 2342323
This way the embedded new lines in the command output are not swallowed by the shell.
For you updated JSON (totally new one compared to old one), all you need to do is
jqOutput=$(jq -r '.[] | .name' < json)
printf "%s\n" "$jqOutput"
Arthur
Richard
In case the .login or .id contains embedded spaces or other characters that might cause problems, a more robust approach is to ensure each JSON value is on a separate line. Consider, for example:
jq -c .login,.id input.json | while read login ; do read id; echo login="$login" and id="$id" ; done
login="dmaxfield" and id=7449977
login="stackfield" and id=2342323