Invalid numeric literal with jq - json

I have a large amount of JSON from a 3rd party system which I would like to pre-process with jq, but I am having difficulty composing the query, test case follows:
$ cat test.json
{
"a": "b",
"c": "d",
"e": {
"1": {
"f": "g",
"h": "i"
}
}
}
$ cat test.json|jq .e.1.f
jq: error: Invalid numeric literal at EOF at line 1, column 3 (while parsing '.1.') at <top-level>, line 1:
.e.1.f
How would I get "g" as my output here? Or how do I cast that 1 to a "1" so it is handled correctly?

From jq manual :
You can also look up fields of an object using syntax like .["foo"]
(.foo above is a shorthand version of this, but only for
identifier-like strings).
You also need quotes and use -r if you want raw output :
jq -r '.e["1"].f' test.json

I wrote a shell script function that calls the curl command, and pipes it into the jq command.
function getName {
curl http://localhost:123/getname/$1 | jq;
}
export -f getName
When I ran this from the CLI,
getName jarvis
I was getting this response:
parse error: Invalid numeric literal at line 1, column 72
I tried removing the | jq from the curl command, and I got back the result without jq parsing:
<Map><timestamp>1234567890</timestamp><status>404</status><error>Not Found</error><message>....
I first thought that I had a bad character in the curl command, or that I was using the function param $1 wrong.
Then I counted the number of chars in the result string, and I noticed that the 72nd char in that string was the empty space between "Not Found".
The underlying issue was that I didn't have a method named getname yet in my spring REST controller, so the response was coming back 404 Not Found. But in addition, jq wasn't handling the empty space in the response except by outputting the error message.
I'm new to jq so maybe there is a way to get around the empty space issue, but that's for another day.

Related

Use variables in JQ queries

I want to use the value of a variable USER_PROXY in the JQ query statement.
export USER_PROXY= "proxy.zyz.com:122"
BY refering the SO answer no:1 from HERE , and also the LINK, I made the following shell script.
jq -r --arg UPROXY ${USER_PROXY} '.proxies = {
"default": {
"httpProxy": "http://$UPROXY\",
"httpsProxy": "http://$UPROXY\",
"noProxy": "127.0.0.1,localhost"
}
}' ~/.docker/config.json > tmp && mv tmp ~/.docker/config.json
However, I see I get the bash error as below. What is it that is missing here. Why is JQ variable UPROXY not getting the value from USER_PROXY bash variable.
export USER_PROXY= "proxy.zyz.com:122"
You can't have a space here. This sets USER_PROXY to an empty string and tries to export a non-existant variable 'proxy.zyz.com:122'. You probably want
export USER_PROXY="proxy.zyz.com:122"
jq -r --arg UPROXY ${USER_PROXY} '.proxies = {
"default": {
"httpProxy": "http://$UPROXY\",
"httpsProxy": "http://$UPROXY\",
"noProxy": "127.0.0.1,localhost"
}
}' ~/.docker/config.json > tmp && mv tmp ~/.docker/config.json
You need quotes around ${USER_PROXY} otherwise any whitespace in it will break it. Instead use --arg UPROXY "${USER_PROXY}".
This isn't the syntax for using variables inside a string in jq. Instead of "...$UPROXY..." you need "...\($UPROXY)..."
You are escaping the " at the end of the string by putting a \ before it. I am not sure what you mean here. I think you perhaps meant to use a forward slash instead?
This last issue is the immediate cause of the error message you're saying. It says "syntax error, unexpected IDENT, expecting '}' at line 4" and then shows you what it found on line 4: "httpsProxy": .... It parsed the string from line 3, which looks like: "http://$UPROXY\"\n " because the escaped double quote doesn't end the string. After finding the end of the string on line 4, jq expects to find a } to close the object, or a , for the next key-value-pair, but it finds httpsProxy, which looks like an identifier. So that's what the error message is saying. It found an IDENTifier when it was expecting a } (or a , but it doesn't mention that).

How to use jq to give true or false when uri field is present in my output json

I have a JSON which goes like this:
{
"results":[
{
"uri":"www.xxx.com"
}
]
}
EDIT
When uri is not present, JSON looks like this:
{
"results":[
]
}
In some cases, uri is present and in some cases, it is not.
Now, I want to use jq to return boolean value if uri is present or not.
This is what I wrote so far but despite uri being present, it gives null.
${search_query_response} contains the JSON
file_status=$(jq -r '.uri' <<< ${search_query_response})
Can anyone guide me?
Since you use jq, it means you are working within a shell script context.
If the boolean result is to be handled by the shell script, you can make jq set its EXIT_CODE depending on the JSON request success or failure status, with jq -e
Example shell script using the EXIT_CODE from jq:
if uri=$(jq -je '.results[].uri') <<<"$search_query_response"
then
printf 'Search results contains an URI: %s.\n' "$uri"
else
echo 'No URI in search results.'
fi
See man jq:
-e / --exit-status:
Sets the exit status of jq to 0 if the last output values was neither false nor null, 1 if the last output value was either false or null, or 4 if no valid result was ever produced. Normally jq exits with 2 if there was any usage problem or system error, 3 if there was a jq program compile error, or 0 if the jq program ran.
Another way to set the exit status is with the halt_error builtin function.
The has function does the job:
jq '.results|map(has("uri"))|.[]'
map the has function on .results.

How can I prettyprint JSON on the command line, but allow invalid JSON objects to pass though?

I'm currently tailing some logs in bash that are half JSON, half text like below:
{"response":{"message":"asdfasdf"}}
{"log":{"example":"asdfasdf"}}
here is some text
{"another":{"example":"asdfasdf"}}
more text
Each line is either a full valid JSON object or some text that would fail a JSON parser.
I've looked at jq and underscore-cli to see if they have options to return the invalid object in the case of failure, but I'm not seeing any.
I've also tried to use a || operator to cat the piped input, but I'm losing the value somehow. Maybe I should read up on pipes more? Example: getLogs -t | (underscore print || cat)
I think I could write a script that stores the input. Format it, and return the output if successful. If it fails returned the stored value. I feel like there should be a simpler way though. Any thoughts?
You can use this node library
install with
$ npm install -g js-beautify
Here is what I did:
$ js-beautify -r test.js
beautified test.js
I tested it with an incomplete json file and it worked
jq can check for invalid json
#!/bin/bash
while read p; do
if jq -e . >/dev/null 2>&1 <<<"$p"; then
echo $p | jq
else
echo 'Skipping invalid json'
fi
done < /tmp/tst.txt
{
"response": {
"message": "asdfasdf"
}
}
{
"log": {
"example": "asdfasdf"
}
}
Skipping invalid json
{
"another": {
"example": "asdfasdf"
}
}
Skipping invalid json

Convert json filtered into csv with jq

I have file that looks like this:
$ cat sample-test.json |jq .
{
"logRef": "c4fa4367-23f6-462f-b5fd-f972d0916a30",
"timestamp": 1563268297545,
"someOtherField": "nonImportantValue"
}
{
"logRef": "c4fa4367-23f6-462f-b5fd-f972d0916a31",
"timestamp": 1563268297595,
"someOtherField2": "nonImportantValue3"
}
And I would like to convert it to csv like this:
logRef;timestamp
c4fa4367-23f6-462f-b5fd-f972d0916a30;1563268297545
c4fa4367-23f6-462f-b5fd-f972d0916a31;1563268297595
I was trying
$ cat sample-test.json |jq '.logRef, .timestamp |#csv'
jq: error (at <stdin>:1): string ("c4fa4367-2...) cannot be csv-formatted, only array
jq: error (at <stdin>:2): string ("c4fa4367-2...) cannot be csv-formatted, only array
Your input is fine (it's a JSON stream).
The problem with your filter is that #csv expects an array. So this will work:
[.logRef,.timestamp] | #csv
However it quotes strings, so if you want your strings unquoted (which might mean the result won't be CSV), then you could use:
"\(.logRef),\(.timestamp)"
In all cases, you'll need to use jq's-r command-line option.
The problem in your json file. Looks like it has incorrect format (without root array element [] and commas between documents). If you fix it, jq will work as expected.
> cat sample-test.json
[{
"logRef": "c4fa4367-23f6-462f-b5fd-f972d0916a30",
"timestamp": 1563268297545,
"someOtherField": "nonImportantValue"
},
{
"logRef": "c4fa4367-23f6-462f-b5fd-f972d0916a31",
"timestamp": 1563268297595,
"someOtherField2": "nonImportantValue3"
}]
cat sample-test.json |jq -r 'map(.logRef), map(.timestamp) | #csv'
"c4fa4367-23f6-462f-b5fd-f972d0916a30","c4fa4367-23f6-462f-b5fd-f972d0916a31"
1563268297545,1563268297595
I've also fixed the command with map() function.

parse error: Invalid numeric literal at line 1, column 9

I understand that jq search needs to be blocked by {} and the key needs to be encased with ", for example:
{
"id": 36815684
}
But if I have something like this:
X-RateLimit-Reset: 1452786798
I get this error:
parse error: Invalid numeric literal at line 1, column 9
Do I need to fall back to sed/awk/perl .. or is there a more elegant way of using jq?
Apart from not using jq at all, you have two main options:
(1) pre-processing the non-JSON to make it JSON
(2) using the -R command-line option, e.g.
echo "X-RateLimit-Reset: 1452786798" | jq -R 'split(":")'
[
"X-RateLimit-Reset",
" 1452786798"
]
Thus, if you know the value is going to be numeric:
echo "X-RateLimit-Reset: 1452786798" |
jq -Rc 'split(":") | {(.[0]) : (.[1]|tonumber)}'
{"X-RateLimit-Reset":1452786798}
Note that although the "j" in jq is for JSON, jq (with the -R option) does just fine for text-processing.