I have a first json file like this:
{
"env_vars": {
"TERRAFORM_CFG_TLS_CERT": "-----BEGIN CERTIFICATE----\\nMIIIqzCCB5O"
}
}
If I use the command:
echo <file> | jq -r '.env_vars'
The result is as expected (the backslash are still there):
{
"TERRAFORM_CFG_TLS_CERT": "-----BEGIN CERTIFICATE----\\nMIIIqzCCB5O"
}
But if i execute this command:
cat <file> | jq -r '.env_vars' | jq -r 'keys[] as $k | "\($k)=\"\(.[$k])\""'
The result is:
TERRAFORM_CFG_TLS_CERT: "-----BEGIN CERTIFICATE----\nMIIIqzCCB5O"
=> One backslash has been removed... why ?
How to avoid this ?
Thanks.
Using the -r option tells jq to "translate" the JSON string into a "raw" string by interpreting the characters that are special to JSON (see e.g. http://json.org). Thus, following the [mcve] guidelines a bit more closely, we could start with:
$ jq . <<< '"X\\nY"'
"X\\nY"
$ jq -r . <<< '"X\\nY"'
X\nY
If you check the json.org specification of strings, you'll see this is exactly correct.
So if for some reason you want each occurrence of \\ in the JSON string to be replaced by two backslash characters (i.e. JSON: "\\\\"), you could use sub or gsub. That's a bit tricky, because the first argument of these functions is a regex. Behold:
$ jq -r 'gsub("\\\\"; "\\\\")' <<< '"X\\nY"'
X\\nY
You should output the string as json to preserve the escapes. By taking a string and outputting it raw, you're getting exactly what that string was, a literal backslash followed by an n.
$ ... | jq -r '.env_vars | to_entries[] | "\(.key): \(.value | tojson)"'
If any of the values are non-strings, add a tostring to the filter.
Related
I'm tying to use jq to make to exportable variables in bash. I ran in to an issue with strings that contain spaces. I tried modifying my jq but all I get are invalid character errors. I have tried both single quotes or double quotes but neither have worked for me. What am I missing here?
input:
{
"PD_TOKEN":"Token y_NbAkKc66ryYTWUXYEu",
"API":"http://cool.api/",
"HELP_URL":"https://help.com"
}
jq:
jq -r 'to_entries|map("\(.key)=\(.value|tostring)")' /json
current result:
PD_TOKEN=Token y_NbAkKc66ryYTWUXYEu
wanted result (note the quotation marks):
PD_TOKEN="Token y_NbAkKc66ryYTWUXYEu"
You can use #sh instead of tostring to escape shell-safely:
$ jq -r 'to_entries | map("\(.key)=\(.value|#sh)")[]' infile.json
PD_TOKEN='Token y_NbAkKc66ryYTWUXYEu'
API='http://cool.api/'
HELP_URL='https://help.com'
Additionally, without map:
Using the array iterator in the first step (h/tip oguz ismail):
jq -r 'to_entries[] | "\(.key)=\(.value|#sh)"' infile.json
Iterate over keys (h/tip ikegami):
jq -r 'keys[] as $key | "\($key)=\(.[$key]|#sh)"' infile.json
I parsed a json file with jq like this :
# cat test.json | jq '.logs' | jq '.[]' | jq '._id' | jq -s
It returns an array like this : [34,235,436,546,.....]
Using bash script i described an array :
# declare -a msgIds = ...
This array uses () instead of [] so when I pass the array given above to this array it won't work.
([324,32,45..]) this causes problem. If i remove the jq -s, an array forms with only 1 member in it.
Is there a way to solve this issue?
We can solve this problem by two ways. They are:
Input string:
// test.json
{
"keys": ["key1","key2","key3"]
}
Approach 1:
1) Use jq -r (output raw strings, not JSON texts) .
KEYS=$(jq -r '.keys' test.json)
echo $KEYS
# Output: [ "key1", "key2", "key3" ]
2) Use #sh (Converts input string to a series of space-separated strings). It removes square brackets[], comma(,) from the string.
KEYS=$(<test.json jq -r '.keys | #sh')
echo $KEYS
# Output: 'key1' 'key2' 'key3'
3) Using tr to remove single quotes from the string output. To delete specific characters use the -d option in tr.
KEYS=$((<test.json jq -r '.keys | #sh')| tr -d \')
echo $KEYS
# Output: key1 key2 key3
4) We can convert the comma-separated string to the array by placing our string output in a round bracket().
It also called compound Assignment, where we declare the array with a bunch of values.
ARRAYNAME=(value1 value2 .... valueN)
#!/bin/bash
KEYS=($((<test.json jq -r '.keys | #sh') | tr -d \'\"))
echo "Array size: " ${#KEYS[#]}
echo "Array elements: "${KEYS[#]}
# Output:
# Array size: 3
# Array elements: key1 key2 key3
Approach 2:
1) Use jq -r to get the string output, then use tr to delete characters like square brackets, double quotes and comma.
#!/bin/bash
KEYS=$(jq -r '.keys' test.json | tr -d '[],"')
echo $KEYS
# Output: key1 key2 key3
2) Then we can convert the comma-separated string to the array by placing our string output in a round bracket().
#!/bin/bash
KEYS=($(jq -r '.keys' test.json | tr -d '[]," '))
echo "Array size: " ${#KEYS[#]}
echo "Array elements: "${KEYS[#]}
# Output:
# Array size: 3
# Array elements: key1 key2 key3
To correctly parse values that have spaces, newlines (or any other arbitrary characters) just use jq's #sh filter and bash's declare -a. (No need for a while read loop or any other pre-processing)
// foo.json
{"data": ["A B", "C'D", ""]}
str=$(jq -r '.data | #sh' foo.json)
declare -a arr="($str)" # must be quoted like this
$ declare -p arr
declare -a arr=([0]="A B" [1]="C'D" [2]="")
The reason that this works correctly is that #sh will produce a space-separated list of shell-quoted words:
$ echo "$str"
'A B' 'C'\''D' ''
and this is exactly the format that declare expects for an array definition.
Use jq -r to output a string "raw", without JSON formatting, and use the #sh formatter to format your results as a string for shell consumption. Per the jq docs:
#sh:
The input is escaped suitable for use in a command-line for a POSIX shell. If the input is an array, the output will be a series of space-separated strings.
So can do e.g.
msgids=($(<test.json jq -r '.logs[]._id | #sh'))
and get the result you want.
From the jq FAQ (https://github.com/stedolan/jq/wiki/FAQ):
𝑸: How can a stream of JSON texts produced by jq be converted into a bash array of corresponding values?
A: One option would be to use mapfile (aka readarray), for example:
mapfile -t array <<< $(jq -c '.[]' input.json)
An alternative that might be indicative of what to do in other shells is to use read -r within a while loop. The following bash script populates an array, x, with JSON texts. The key points are the use of the -c option, and the use of the bash idiom while read -r value; do ... done < <(jq .......):
#!/bin/bash
x=()
while read -r value
do
x+=("$value")
done < <(jq -c '.[]' input.json)
++ To resolve this, we can use a very simple approach:
++ Since I am not aware of you input file, I am creating a file input.json with the following contents:
input.json:
{
"keys": ["key1","key2","key3"]
}
++ Use jq to get the value from the above file input.json:
Command: cat input.json | jq -r '.keys | #sh'
Output: 'key1' 'key2' 'key3'
Explanation: | #sh removes [ and "
++ To remove ' ' as well we use tr
command: cat input.json | jq -r '.keys | #sh' | tr -d \'
Explanation: use tr delete -d to remove '
++ To store this in a bash array we use () with `` and print it:
command:
KEYS=(`cat input.json | jq -r '.keys | #sh' | tr -d \'`)
To print all the entries of the array: echo "${KEYS[*]}"
Feel free to edit the title; not sure how to word it. I'm trying to turn shell output into JSON data for a reporting system I'm writing for work. Quick question, no matter what i do, when I take raw input in slurp mode and output the JSON, the last item in the array is blank (""). I feel like this is some sort of rookie jq issue I'm running into, but can't figure out how to word the issue. This seems to happen no matter what command I run on the shell and pipe to jq:
# rpm -qa | grep kernel | jq -R -s 'split("\n")'
[
"kernel-2.6.32-504.8.1.el6.x86_64",
"kernel-firmware-2.6.32-696.20.1.el6.noarch",
"kernel-headers-2.6.32-696.20.1.el6.x86_64",
"dracut-kernel-004-409.el6_8.2.noarch",
"abrt-addon-kerneloops-2.0.8-43.el6.x86_64",
"kernel-devel-2.6.32-358.11.1.el6.x86_64",
"kernel-2.6.32-131.4.1.el6.x86_64",
"kernel-devel-2.6.32-696.20.1.el6.x86_64",
"kernel-2.6.32-696.20.1.el6.x86_64",
"kernel-devel-2.6.32-504.8.1.el6.x86_64",
"libreport-plugin-kerneloops-2.0.9-33.el6.x86_64",
""
]
Any help is appreciated.
Every line ends with a newline. Either remove the final newline, or omit the empty element at the end of the array.
vnix$ printf 'foo\nbar\n' |
> jq -R -s '.[:-1] | split("\n")'
[
"foo",
"bar"
]
vnix$ printf 'foo\nbar\n' |
> jq -R -s 'split("\n")[:-1]'
[
"foo",
"bar"
]
The notation x[:-1] retrieves the value of a string or array x with the last element removed. This is called "slice notation".
Just to spell this out, if you take the string "foo\n" and split on newline, you get "foo" from before the newline and "" after it.
To make this really robust, maybe trim the last character only if it really is a newline.
vnix$ printf 'foo\nbar\n' |
> jq -R -s 'sub("\n$";"") | split("\n")'
[
"foo",
"bar"
]
vnix$ printf 'foo\nbar' |
> # notice, no final ^ newine
> jq -R -s 'sub("\n$";"") | split("\n")'
[
"foo",
"bar"
]
Assuming you have access to jq 1.5 or later, you can circumvent the problem entirely and economically using inputs:
jq -nR '[inputs]'
Just be sure to include the -n option, otherwise the first line will go missing.
You can also use
rpm -qa | grep kernel | jq -R . | jq -s .
to get the desired result.
Please see https://github.com/stedolan/jq/issues/563
I have a sample json response as shown below which i am trying to parse using jq in shell script.[{"id":1,"notes":"Demo1\nDemo2"}]
This is the command through which I am trying to access notes in the shell script.
value=($(curl $URL | jq -r '.[].notes'))
When I echo "$value" I only get Demo1. How to get the exact value: Demo1\nDemo2 ?
To clarify, there is no backslash or n in the notes field. \n is JSON's way of encoding a literal linefeed, so the value you should be expecting is:
Demo1
Demo2
The issue you're seeing is because you have split the value on whitespace and created an array. Each value can be accessed by index:
$ cat myscript
data='[{"id":1,"notes":"Demo1\nDemo2"}]'
value=($(printf '%s' "$data" | jq -r '.[].notes'))
echo "The first value was ${value[0]} and the second ${value[1]}"
$ bash myscript
The first value was Demo1 and the second Demo2
To instead get it as a simple string, remove the parens from value=(..):
$ cat myscript2
data='[{"id":1,"notes":"Demo1\nDemo2"}]'
value=$(printf '%s' "$data" | jq -r '.[].notes')
echo "$value"
$ bash myscript2
Demo1
Demo2
I'm trying to get jq to parse a JSON structure like:
{
"a" : 1,
"b" : 2,
"c" : "{\"id\":\"9ee ...\",\"parent\":\"abc...\"}\n"
}
That is, an element in the JSON is a string with escaped json.
So, I have something along the lines of
$ jq [.c] myFile.json | jq [.id]
But that crashes with jq: error: Cannot index string with string
This is because the output of .c is a string, not more JSON.
How do I get jq to parse this string?
My initial solution is to use sed to replace all the escape chars (\":\", \",\" and \") but that's messy, I assume there's a way built into jq to do this?
Thanks!
edit:
Also, the jq version available here is:
$ jq --version
jq version 1.3
I guess I could update it if required.
jq has the fromjson builtin for this:
jq '.c | fromjson | .id' myFile.json
fromjson was added in version 1.4.
You can use the raw output (-r) that will unescape characters:
jq -r .c myfile.json | jq .id
ADDENDUM: This has the advantage that it works in jq 1.3 and up; indeed, it should work in every version of jq that has the -r option.
Motivation: you want to parse JSON string - you want to escape a JSON object that's wrapped with quotes and represented as a String buffer, and convert it to a valid JSON object. For example:
some JSON unescaped string :
"{\"name\":\"John Doe\",\"position\":\"developer\"}"
the expected result ( a JSON object ):
{"name":"John Doe","position":"developer"}
Solution: In order to escape a JSON string and convert it into a valid JSON object use the sed tool in command line and use regex expressions to remove/replace specific characters:
cat current_json.txt | sed -e 's/\\\"/\"/g' -e 's/^.//g' -e 's/.$//g'
s/\\\"/\"/g replacing all backslashes and quotes ( \" ) into quotes only (")
s/^.//g replacing the first character in the stream to none character
s/.$//g replacing the last character in the stream to none character