Good Afternoon all,
I am presenting the following problem.
new_ecdsa_config.json looks something like:
{
"certificate": "value long string
multine"
}
When I run the github action for reading the value
- name: Add new ECDSA to Organization
run: |
cat new_ecdsa_config.json | jq '.'
I get the following error:
parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at line 26, column 53
Error: Process completed with exit code 4.
Any ideas??
When you described the problem on github, you gave this as the example:
{
"certificate": "value long string
multine"
}
As the error message says, that is not valid JSON. (You can double-check this at jsonlint.com if you like.)
If you want the JSON representation of a multiline string, you'd have to escape the newline, e.g. along the lines of:
{
"certificate": "value long string\nmultiline"
}
Related
I want to use the value of a variable USER_PROXY in the JQ query statement.
export USER_PROXY= "proxy.zyz.com:122"
BY refering the SO answer no:1 from HERE , and also the LINK, I made the following shell script.
jq -r --arg UPROXY ${USER_PROXY} '.proxies = {
"default": {
"httpProxy": "http://$UPROXY\",
"httpsProxy": "http://$UPROXY\",
"noProxy": "127.0.0.1,localhost"
}
}' ~/.docker/config.json > tmp && mv tmp ~/.docker/config.json
However, I see I get the bash error as below. What is it that is missing here. Why is JQ variable UPROXY not getting the value from USER_PROXY bash variable.
export USER_PROXY= "proxy.zyz.com:122"
You can't have a space here. This sets USER_PROXY to an empty string and tries to export a non-existant variable 'proxy.zyz.com:122'. You probably want
export USER_PROXY="proxy.zyz.com:122"
jq -r --arg UPROXY ${USER_PROXY} '.proxies = {
"default": {
"httpProxy": "http://$UPROXY\",
"httpsProxy": "http://$UPROXY\",
"noProxy": "127.0.0.1,localhost"
}
}' ~/.docker/config.json > tmp && mv tmp ~/.docker/config.json
You need quotes around ${USER_PROXY} otherwise any whitespace in it will break it. Instead use --arg UPROXY "${USER_PROXY}".
This isn't the syntax for using variables inside a string in jq. Instead of "...$UPROXY..." you need "...\($UPROXY)..."
You are escaping the " at the end of the string by putting a \ before it. I am not sure what you mean here. I think you perhaps meant to use a forward slash instead?
This last issue is the immediate cause of the error message you're saying. It says "syntax error, unexpected IDENT, expecting '}' at line 4" and then shows you what it found on line 4: "httpsProxy": .... It parsed the string from line 3, which looks like: "http://$UPROXY\"\n " because the escaped double quote doesn't end the string. After finding the end of the string on line 4, jq expects to find a } to close the object, or a , for the next key-value-pair, but it finds httpsProxy, which looks like an identifier. So that's what the error message is saying. It found an IDENTifier when it was expecting a } (or a , but it doesn't mention that).
I use JQ 1.5 in a Windows10 enviroment (PowerShell).
I built a jq statement that works on the example data on jqplay but throws a error on my enviroment:
Sample: Code share
Code:
. | { last_update: .starbase_detailed_scan.last_update_time, user_name: .starbase_detailed_scan.owner_name, alliance_id: .starbase_detailed_scan.owner_alliance_id, drydocks: .starbase_detailed_scan.num_drydocks, tier: .starbase_detailed_scan.owner_level, defence_plattform: .starbase_detailed_scan.num_defence_platforms, shield_triggered: .starbase_detailed_scan.player_shield.triggered_on, shield_end: .starbase_detailed_scan.player_shield.expiry_time, parsteel: .resources."325683920".current_amount, tritanium: .resources."743985951".current_amount, dilithium: .resources."2614028847".current_amount, user_id: .starbase_detailed_scan.owner_user_id, defence_rating: .starbase_detailed_scan.defense_rating }
The problem are the JSON objects with a numeric identifier. On jqplay I got the correct values. On PowerShell jq I get an error. I expected that this is a PowerShell problem so I tried to move the filter into a filter file. The error is then gone but I get only NULL as value for the three objects.
Numbers in the json path need to be marked Oldschool like:
.starbase_detailed_scan.resources["2614028847"]
BR
Timo
This question already has answers here:
Parse or view JSON data fields using JQ tool utility where field names have a "-" dash in the key name
(4 answers)
Closed 4 years ago.
I use jq 1.5 in a Windows Environment to modify an json object that i receiving from Amazon s3. I have there a funny Problem. I use jq to extract single keys of the object:
{
"s3_direct_url": "https://fanzo-photos.s3.amazonaws.com/photos/images/034/005/322/screen1.jpg",
"url": "https://fanzo-photos.s3.amazonaws.com",
"fields": {
"key": "photos/images/034/005/322/screen1.jpg",
"success_action_status": "200",
"Content-Type": "image/jpeg",
"acl": "public-read",
"policy": "eyJleHBpcmF0aW9uIjoiMjAxOC0xMS0wMlQxMzo0NzoxNVoiLCJjb25kaXRpb25zIjpbeyJidWNrZXQiOiJmYW56by1waG90b3MifSx7ImtleSI6InBob3Rvcy9pbWFnZXMvMDM0LzAwNS8zMjIvc2NyZWVuMS5qcGcifSx7InN1Y2Nlc3NfYWN0aW9uX3N0YXR1cyI6IjIwMCJ9LHsiQ29udGVudC1UeXBlIjoiaW1hZ2UvanBlZyJ9LHsiYWNsIjoicHVibGljLXJlYWQifSx7IngtYW16LWNyZWRlbnRpYWwiOiJBS0lBSlkzWVRCV1NMQzQ2SFdCQS8yMDE4MTEwMi91cy1lYXN0LTEvczMvYXdzNF9yZXF1ZXN0In0seyJ4LWFtei1hbGdvcml0aG0iOiJBV1M0LUhNQUMtU0hBMjU2In0seyJ4LWFtei1kYXRlIjoiMjAxODExMDJUMTI0NzE1WiJ9XX0=",
"x-amz-credential": "AKIAJY3YTBWSLC46HWBA/20181102/us-east-1/s3/aws4_request",
"x-amz-algorithm": "AWS4-HMAC-SHA256",
"x-amz-date": "20181102T124715Z",
"x-amz-signature": "52d8246536e8743fba8e7668cb65a08a1142221d54a58676b6ab14e3835482a3"
},
"id": 34005322,
"media_type": "InputMedia"
}
If i extract informations from the 'fields' object without a '-' in the key name that works fine. If i try to extract a key with '-' in the name i got following error:
jq: error: amz/0 is not defined at <top-level>, line 1:
.fields.x-amz-credential
jq: error: credential/0 is not defined at <top-level>, line 1:
.fields.x-amz-credential
jq: 2 compile errors
exit status 3
Update:
After the hint with the FAQ and the " i rebuild the jq command and tested it in jqplay: .fields."Content-Type" where it works as expected. Under powershell that variant didn't working since the powershell didn't excepting the quotating.
.\jq .fields."Content-Type" jq: error: Type/0 is not defined at <top-level>, line 1: .fields.Content-Type jq: 1 compile error
BR
Timo
Any tips?
Yes! If you cannot find the answer in the onine jq manual, check the jq FAQ:
𝑸: How can I access the value of a key with hyphens or $ or other
special characters in it? Why does .a.["$"] produce a syntax error?
A: The basic form for accessing the value of a key is .["KEYNAME"]
where "KEYNAME" is any valid JSON string, but recent versions of jq
also allow ."KEYNAME".
Using the basic form might require explicit use of the pipe symbol, as
in .["a-b"]|.["x-y"], but this can be abbreviated to .["a-b"]["x-y"].
I have the following grok pattern that works in Logstash and in the Grok debugger in Kibana.
\[%{TIMESTAMP_ISO8601:req_time}\] %{IP:client_ip} (?:%{IP:forwarded_for}|\(-\)) (?:%{QS:request}|-) %{NUMBER:response_code:int} %{WORD}:%{NUMBER:request_length:int} %{WORD}:%{NUMBER:body_bytes_sent:int} %{WORD}:(?:%{QS:http_referer}|-) %{WORD}:(?:%{QS:http_user_agent}|-) (%{WORD}:(\")?(%{NUMBER:request_time:float})(\")?)?"
I am trying to create a new ingest pipeline via the PUT method, but I get an error that contains:
"type": "parse_exception",
"reason": "Failed to parse content to map",
"caused_by": {
"type": "i_o_exception",
"reason": "Unrecognized character escape '[' (code 91)\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper#61326735; line: 7, column: 25]"
}
Elasticsearch requires that grok patterns used in pipelines submitted using the PUT method are properly escaped JSON, while Logstash patterns use different escaping.
That includes preceding brackets with double backslashes (\\[) and double quotes with triple backslashes (\\\"). The working pattern (after running through a JSON escaping tool) is:
\\[%{TIMESTAMP_ISO8601:req_time}\\] %{IP:client_ip} (?:%{IP:forwarded_for}|\\(-\\)) (?:%{QS:request}|-) %{NUMBER:response_code:int} %{WORD}:%{NUMBER:request_length:int} %{WORD}:%{NUMBER:body_bytes_sent:int} %{WORD}:(?:%{QS:http_referer}|-) %{WORD}:(?:%{QS:http_user_agent}|-) (%{WORD}:(\\\")?(%{NUMBER:request_time:float})(\\\")?)?
I have a large amount of JSON from a 3rd party system which I would like to pre-process with jq, but I am having difficulty composing the query, test case follows:
$ cat test.json
{
"a": "b",
"c": "d",
"e": {
"1": {
"f": "g",
"h": "i"
}
}
}
$ cat test.json|jq .e.1.f
jq: error: Invalid numeric literal at EOF at line 1, column 3 (while parsing '.1.') at <top-level>, line 1:
.e.1.f
How would I get "g" as my output here? Or how do I cast that 1 to a "1" so it is handled correctly?
From jq manual :
You can also look up fields of an object using syntax like .["foo"]
(.foo above is a shorthand version of this, but only for
identifier-like strings).
You also need quotes and use -r if you want raw output :
jq -r '.e["1"].f' test.json
I wrote a shell script function that calls the curl command, and pipes it into the jq command.
function getName {
curl http://localhost:123/getname/$1 | jq;
}
export -f getName
When I ran this from the CLI,
getName jarvis
I was getting this response:
parse error: Invalid numeric literal at line 1, column 72
I tried removing the | jq from the curl command, and I got back the result without jq parsing:
<Map><timestamp>1234567890</timestamp><status>404</status><error>Not Found</error><message>....
I first thought that I had a bad character in the curl command, or that I was using the function param $1 wrong.
Then I counted the number of chars in the result string, and I noticed that the 72nd char in that string was the empty space between "Not Found".
The underlying issue was that I didn't have a method named getname yet in my spring REST controller, so the response was coming back 404 Not Found. But in addition, jq wasn't handling the empty space in the response except by outputting the error message.
I'm new to jq so maybe there is a way to get around the empty space issue, but that's for another day.