I'm using groovy script and write this line
Version_List=$(jq -r '.items[] .version' "${projectContent}")
the ${projectContent} is another variable which is result of curl command
when i run the pipeline this is the error /bin/jq: Argument list too long
Use jq -Rs . <<< "${projectContent}" to convert the variable content into a JSON string (you may even use curl directly, like so: curl … | jq -Rs), then import that string into a jq variable using --slurpfile with Process Substitution
jq --slurpfile projectContent <(
jq -Rs . <<< "${projectContent}"
) '...' # your jq code here using variable $projectContent
Don't forget to also add either -n or an input file.
Related
I'm looking for a solution to add a new attribute with a JSON object value into an existing JSON file.
My current script:
if [ ! -f "$src_file" ]; then
echo "Source file $src_file does not exists"
exit 1
fi
if [ ! -f "$dst_file" ]; then
echo "Destination file $dst_file does not exists"
exit 1
fi
if ! jq '.devDependencies' "$src_file" >/dev/null 2>&1; then
echo "The key "devDependencies" does not exists into source file $src_file"
exit 1
fi
dev_dependencies=$(jq '.devDependencies' "$src_file" | xargs )
# Extract data from source file
data=$(cat $src_file)
# Add new key-value
data=$(echo $data | jq --arg key "devDependencies" --arg value "$dev_dependencies" '. + {($key): ($value)}')
# Write data into destination file
echo $data > $dst_file
It's working but the devDependencies value from $dev_dependencies is wrote as string:
"devDependencies": "{ #nrwl/esbuild: 15.6.3, #nrwl/eslint-pl[...]".
How can I write it as raw JSON ?
I think you want the --argjson option instead of --arg. Compare
$ jq --arg k '{"foo": "bar"}' -n '{x: $k}'
{
"x": "{\"foo\": \"bar\"}"
}
with
$ jq --argjson k '{"foo": "bar"}' -n '{x: $k}'
{
"x": {
"foo": "bar"
}
}
--arg will create a string variable. Use --argjson to parse the value as JSON (can be object, array or number).
From the docs:
--arg name value:
This option passes a value to the jq program as a predefined variable.
If you run jq with --arg foo bar, then $foo is available in the
program and has the value "bar". Note that value will be treated as a
string, so --arg foo 123 will bind $foo to "123".
Named arguments are also available to the jq program as $ARGS.named.
--argjson name JSON-text:
This option passes a JSON-encoded value to the jq program as a
predefined variable. If you run jq with --argjson foo 123, then $foo
is available in the program and has the value 123.
Note that you don't need multiple invocations of jq, xargs, command substitution or variables (don't forget to quote all your variables when expanding).
To "merge" the contents of two files, read both files with jq and let jq do the work. This avoids all the complications that arise from jumping between jq and shell context. A single line is all that's needed:
jq --slurpfile deps "$dep_file" '. + { devDependencies: $deps[0].devDependencies }' "$source_file" > "$dest_file"
or
jq --slurpfile deps "$dep_file" '. + ($deps[0]|{devDependencies})' "$source_file" > "$dest_file"
alternatively (still a one-liner):
jq --slurpfile deps "$dev_file" '.devDependencies = $deps[0].devDependencies' "$source_file" > "$dest_file"
peak's answer here reminded me of the very useful input filter, which can make the program even shorter as it avoids the variable:
jq '. + (input|{devDependencies})' "$source_file" "$dep_file" > "$dest_file"
I have a file with this kind of text with pattern
[{"foo":"bar:baz:foo*","bar*":"baz*","etc":"etc"},
{"foo2":"bar2:baz2:foo2*","bar2*":"baz2*","etc":"etc"},
{"foo3":"bar3:baz3:foo3*","bar3*":"baz3*","etc":"etc"},
{"foo4":"bar4:baz4:foo4*","bar4*":"baz4*","etc":"etc"}]
I need to take every string like this
{"foo":"bar:baz:foo*","bar*":"baz*","etc":"etc"} and send each of them to some url via curl
for i in text.txt
do (awk,sed,grep etc)
then curl $string
I can't figure out how to get the desired lines properly from the file without unnecessary symbols
I suggest that you can use jq to process your json file. jq is capable of reading json, and formatting output. Here's an example jq script to process your json file (which I unimaginatively call 'jsonfile'):
jq -r '.[] | "curl -d '\'' \(.) '\'' http://restful.com/api " ' jsonfile
Here's the output:
curl -d ' {"foo":"bar:baz:foo*","bar*":"baz*","etc":"etc"} ' http://restful.com/api
curl -d ' {"foo2":"bar2:baz2:foo2*","bar2*":"baz2*","etc":"etc"} ' http://restful.com/api
curl -d ' {"foo3":"bar3:baz3:foo3*","bar3*":"baz3*","etc":"etc"} ' http://restful.com/api
curl -d ' {"foo4":"bar4:baz4:foo4*","bar4*":"baz4*","etc":"etc"} ' http://restful.com/api
Here's what's going on:
We pass three arguments to the jq program: jq -r <script> <inputfile>.
The -r tells jq to output the results in raw format (that is, please don't escape quotes and stuff).
The script looks like this:
.[] | "some string \(.)"
The first . means take the whole json structure and the [] means iterate through each array element in the structure. The | is a filter that processes each element in the array. The filter is to output a string. We are using \(.) to interpolate the whole element passed into the | filter.
Wow... I've never really explained a jq script before (and it shows). But the crux of it is, we are using jq to find each element in the json array and insert it into a string. Our string is this:
curl -d '<the json dictionary array element>' http://restful.com/api
Ok. And you see the output. It works. But wait a second, we only have output. Let's tell the shell to run each line like this:
jq -r '.[] | "curl -d '\'' \(.) '\'' http://restful.com/api " ' jsonfile | bash
By piping the output to bash, we execute each line that we output. Essentially, we are writing a bash script with jq to curl http://restful.com/api passing the json element as the -d data parameter to POST the json element.
Revisiting for single quote issue
#oguz ismail pointed out that bash will explode if there is a single quote in the json input file. This is true. We can avoid the quote by escaping, but we gain more complexity - making this a non-ideal approach.
Here's the problem input (I just inserted a single quote):
[{"foo":"bar:'baz:foo*","bar*":"baz*","etc":"etc"},
{"foo2":"bar2:baz2:foo2*","bar2*":"baz2*","etc":"etc"},
{"foo3":"bar3:baz3:foo3*","bar3*":"baz3*","etc":"etc"},
{"foo4":"bar4:baz4:foo4*","bar4*":"baz4*","etc":"etc"}]
Notice above that baz is now 'baz. The problem is that a single single quote makes the bash shell complain about unmatched quotes:
$ jq -r '.[] | "curl -d '\'' \(.) '\'' http://restful.com/api " ' jsonfile | bash
bash: line 4: unexpected EOF while looking for matching `"'
bash: line 5: syntax error: unexpected end of file
Here's the solution:
$ jq -r $'.[] | "\(.)" | gsub( "\'" ; "\\\\\'" ) | "echo $\'\(.)\'" ' jsonfile | bash
{"foo":"bar'baz:foo*","bar*":"baz*","etc":"etc"}
{"foo2":"bar2:baz2:foo2*","bar2*":"baz2*","etc":"etc"}
{"foo3":"bar3:baz3:foo3*","bar3*":"baz3*","etc":"etc"}
{"foo4":"bar4:baz4:foo4*","bar4*":"baz4*","etc":"etc"}
Above I am using $'' to quote the jq script. This allows me to escape single quotes using '. I've also changed the curl command to echo so I can test the bash script without bothering the folks at http://restful.com/api.
The 'trick' is to make sure that the bash script we generate also escapes all single quotes with a backslash . So, we have to change ' to \'. That's what gsub is doing.
gsub( "\'" ; "\\\\\'" )
After making that substitution ( ' --> \' ) we pipe the entire string to this:
"echo $\'\(.)\'"
which surrounds the output of gsub with echo $''. Now we are using $' again so the \' is properly understood by bash.
So we wind up with this when we put the curl back in:
jq -r $'.[] | "\(.)" | gsub( "\'" ; "\\\\\'" ) | "curl -d $\'\(.)\' http://restful.com/api " ' jsonfile | bash
Use jq command. This is just example parsing.
for k in $(jq -c '.[]' a.txt); do
echo "hello-" $k
done
Output:
hello- {"foo":"bar:baz:foo*","bar*":"baz*","etc":"etc"}
hello- {"foo2":"bar2:baz2:foo2*","bar2*":"baz2*","etc":"etc"}
hello- {"foo3":"bar3:baz3:foo3*","bar3*":"baz3*","etc":"etc"}
hello- {"foo4":"bar4:baz4:foo4*","bar4*":"baz4*","etc":"etc"}
You can use the $k anywhere inside the loop you want.
for k in $(jq -c '.[]' a.txt); do
curl -d "$k" <url>
done
I have the date in the file which looks like
test,test
test1,test1
I want to convert it into like:
{"test":"test","test1":"test1"}
I have tried jq for this purpose jq -R -s -c 'split("\n")'
But its oupting in the format ["test,test","test1,test1",""]
jq 1.5 has inputs, which allows a simple and efficient solution:
jq -R -n -c '[inputs|split(",")|{(.[0]):.[1]}] | add' input.txt
Important: don't forget the -n (--null-input) option, otherwise you'll lose the first line.
Alternative
If your jq does not have inputs, then it's time to upgrade if at all possible. Otherwise:
jq -R -s '
split("\n")
| map(if index(",") then split(",")|{(.[0]):.[1]}
else empty end)
| add' input.txt
As #peak indicates, use the inputs with the split function. But to merge the key/values into one single object, use the reduce method:
jq -Rn '[inputs|split(",")| {(.[0]): .[1]}] | reduce .[] as $obj ({}; . + $obj) ' input.csv
The reduce method reduces each item in the array into a single item. In this case, we indicate that each item should be assigned to the $obj variable, and that we start out with the empty {} object. The second argument to the reduce method indicates how to "reduce" things down to a single item. In this case, we are adding/merging the $obj we assigned with the {} object we started out with and then returning the resulting object to be used in the next iteration. After all the iterations have completed, the final item (in this case, the combined object) is returned.
What you ask is possible to achieve with just standar unix shell utilities (assuming your input in file.txt):
bash $ echo { \"$(<file.txt sed 's/,/":"/g' | paste -s -d, - | sed 's/,/","/g')\" }
{ "test":"test","test1":"test1" }
bash $
resulting output is a valid json
I have a sample json response as shown below which i am trying to parse using jq in shell script.[{"id":1,"notes":"Demo1\nDemo2"}]
This is the command through which I am trying to access notes in the shell script.
value=($(curl $URL | jq -r '.[].notes'))
When I echo "$value" I only get Demo1. How to get the exact value: Demo1\nDemo2 ?
To clarify, there is no backslash or n in the notes field. \n is JSON's way of encoding a literal linefeed, so the value you should be expecting is:
Demo1
Demo2
The issue you're seeing is because you have split the value on whitespace and created an array. Each value can be accessed by index:
$ cat myscript
data='[{"id":1,"notes":"Demo1\nDemo2"}]'
value=($(printf '%s' "$data" | jq -r '.[].notes'))
echo "The first value was ${value[0]} and the second ${value[1]}"
$ bash myscript
The first value was Demo1 and the second Demo2
To instead get it as a simple string, remove the parens from value=(..):
$ cat myscript2
data='[{"id":1,"notes":"Demo1\nDemo2"}]'
value=$(printf '%s' "$data" | jq -r '.[].notes')
echo "$value"
$ bash myscript2
Demo1
Demo2
If I have a JSON like this,
{
"hello1": "world1",
"testk": "testv"
}
And I want to export each of these key-value pairs as environment variables, how to do it via shell script? So for example, when I write on the terminal, echo $hello1, world1 should be printed and similarly for other key-value pairs?
Note: The above JSON is present in a variable called $values and not in a file.
I know it will be done via jq and written a shell script for this, but it doesn't work.
for row in $(echo "${values}" | jq -r '.[]'); do
-jq() {
echo ${row} | jq -r ${1}
}
echo $(_jq '.samplekey')
done
Borrowing from this answer which does all of the hard work of turning the JSON into key=value pairs, you could get these into the environment by looping over the jq output and exporting them:
for s in $(echo $values | jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]" ); do
export $s
done
If the variables being loaded contain embedded whitespace, this is also reasonable, if slightly more complex:
while read -rd $'' line
do
export "$line"
done < <(jq -r <<<"$values" \
'to_entries|map("\(.key)=\(.value)\u0000")[]')
Using command substitution $() :
# $(jq -r 'keys[] as $k | "export \($k)=\(.[$k])"' file.json)
# echo $testk
testv
Edit : Responding to this comment
You should do
$( echo "$values" | jq -r 'keys[] as $k | "export \($k)=\(.[$k])"' )
Just mind the double quotes around $values
Note: Couldn't confirm if there is security implication to this approach, that is if the user could manipulate the json to wreak havoc.
Another way, without using jq, is to parse the json with grep & sed:
for keyval in $(grep -E '": [^\{]' my.json | sed -e 's/: /=/' -e "s/\(\,\)$//"); do
echo "export $keyval"
eval export $keyval
done
Explanation:
First, grep will filter all "key" : value pairs (value can be
"string", number, or boolean).
Then, sed will replace : with =, and remove trailing ,.
Lastly, exporting the "key"=value with eval
Here's an output example, exporting json keys, from an AWS record-set:
export "Name"="\052.apps.nmanos-cluster-a.devcluster.openshift.com."
export "Type"="A"
export "HostedZoneId"="Z67SXBLZRQ7X7T"
export "DNSName"="a24070461d50270e-1391692.us-east-1.elb.amazonaws.com."
export "EvaluateTargetHealth"=false
None of the existing answers preserve whitespace in the values in a POSIX shell. The following line will use jq to take each key:value of some JSON and export them as environment variables, properly escaping whitespace and special characters.
2023-01-28: BUGFIX UPDATE:
My previous answer did not work for all possible values and could cause errors. Please instead use the following line, which uses jq's #sh format string to properly escape values for the shell. You must also enclose everything after eval in quotes to preserve newlines. I've updated the sample JSON file to include more characters to test with.
This answer now appears to be the only one that handles all cases. There are no loops and it's one line to export all values. The downside is that it uses eval, which is theoretically dangerous... but because the entire key=value is now being escaped for the shell, this should be safe to use.
New answer (use this one):
eval "export $(echo "$values" \
| jq -r 'to_entries | map("\(.key)=\(.value)") | #sh')"
Old answer (don't use this one):
eval export $(echo "$values" \
| jq -r 'to_entries|map("\"\(.key)=\(.value|tostring)\"")|.[]' )
edit thanks #Delthas for pointing out a missing 'export'
Sample JSON file:
bash-5.2$ cat <<'EOJSON' > foo.json
{
"foo_1": "bar 1",
"foo_2": "This ! is ' some # weird $text { to ( escape \" here",
"foo_3": "this is some \nsample new line\n text to\ntry and escape"
}
EOJSON
Sample script:
bash-5.2$ cat <<'EOSH' > foo.sh
values="`cat foo.json`"
eval "export $(echo "$values" | jq -r 'to_entries | map("\(.key)=\(.value)") | #sh')"
export
echo "foo_2: $foo_2"
echo "foo_3: $foo_3"
EOSH
Running the sample script:
bash-5.2$ env -i sh foo.sh
export PWD='/path/to/my/home'
export SHLVL='1'
export foo_1='bar 1'
export foo_2='This ! is '"'"' some # weird $text { to ( escape " here'
export foo_3='this is some
sample new line
text to
try and escape'
foo_2: This ! is ' some # weird $text { to ( escape " here
foo_3: this is some
sample new line
text to
try and escape
Pros:
no need for Bash
preserves whitespace in values
no loops
(update) properly escapes all values for use in the shell
Cons:
uses eval, which is considered "unsafe". however, because jq is escaping all input, this is unlikely to cause a security issue (unless jq is found to have a bug which does not properly escape data using the #sh filter).
The approach illustrated by the following shell script avoids most (but not all) problems with special characters:
#!/bin/bash
function json2keyvalue {
cat<<EOF | jq -r 'to_entries|map("\(.key)\t\(.value|tostring)")[]'
{
"hello1": "world1",
"testk": "testv"
}
EOF
}
while IFS=$'\t' read -r key value
do
export "$key"="$value"
done < <(json2keyvalue)
echo hello1="$hello1"
echo testk="$testk"
Note that the above assumes that there are no tabs in the keys themselves.
jtc solution:
export $(<file.json jtc -w'[:]<>a:<L>k' -qqT'"{L}={}"')
I've come up with a solution (here in bash):
function source_json_as_environ() {
eval "$(jq -r '
def replace_dot:
. | gsub("\\."; "_");
def trim_spaces:
. | gsub("^[ \t]+|[ \t]+$"; "");
to_entries|map(
"export \(.key|trim_spaces|replace_dot)="
+ "\(.value|tostring|trim_spaces|#sh)"
)|.[]' $#)"
}
And you can use it like this:
$ source_json_as_environ values.json