Replace the placeholder in the String with actual value using jq - json

It is not working. How to check the part of the String (not the whole String)?
test.json
{
"url": "https://<part1>.test/hai/<part1>"
}
jq --arg input "$arg" \
'if .url == "<part1>"
then . + {"url" : ("https://" + $input + ".test/hai/" + $input) }
else . end' test.json > test123.json

In your test.json, the value of .url is "https://<part1>.test/hai/<part1>" so evidently you don't want to check that its value is "<part1>".
Perhaps you meant to test the condition: .url | contains("<part1>") or maybe .url | endswith("<part1>") -- the problem description is unclear.
If your jq has support for regular expressions, you could use test/1, e.g.
.url | test("//<part1>.*<part1>$")
Another possibility would be to use gsub.

Related

Is there a fix for this Expression to get the key value pairs right

I have this python script which prints this output in raw string format.
{"A":"ab3241c","B":"d12e31234f","c":"[g$x>Q)M&.N+v8"}
I am using jq to set the values of A,B and c
Something like this
expression='. | to_entries | .[] | .key + "=\"" + .value + "\""'
eval "$(python script.py | jq -r "$expression")"
This works fine A and B. And when I do something like
echo $A
ab3241c
But the problem is with c where i get the output as
[g\u003eQ)M\u0026.N+v8
so $x> and & are getting converted to unicode.
Can I fix the expression to avoid this?
I fixed it using
expression as
expression='to_entries | map("\(.key)=\(.value | #sh)") | .[]'
Using eval here is like shooting yourself in the foot.
Why not just pipe the output of your python command to the jq command?
Consider:
function mypython {
cat <<"EOF"
{"A":"ab3241c","B":"d12e31234f","c":"[g$x>Q)M&.N+v8"}
EOF
}
expression='to_entries[] | .key + "=\"" + .value + "\""'
mypython | jq -r "$expression"
Note that using expression here seems pointless as well. In general, it would be better either to "inline" it, or put it in a file and use jq's -f command-line option.
(Notice also that the first "." in your expression is not needed.)
If you really must use eval, then consider:
function mypython {
cat <<"EOF"
{"A":"ab3241c","B":"d12e31234f","c":"[g$x>Q)M&.N+v8"}
EOF
}
expression='to_entries[] | .key + \"=\\\"\" + .value + \"\\\"\"'
eval "mypython | jq -r \"$expression\""
Output
A="ab3241c"
B="d12e31234f"
c="[g$x>Q)M&.N+v8"

Does `jq`'s `walk` have an issue with sorting while walking?

I have a small jq use where I want to reverse sort by a tree's size. I know it's possible with du -h and sort -h etc. But I'm trying it out with tree's json output and jq.
It looks like:
$ tree -h -pug --du -nA -J perl5 | \
awk -v RS= '{ gsub(/,[[:space:]]*]/, "\n]", $0) }1' | \
jq '
walk(if type == "object" and has("contents")
then (.contents|sort_by(.size)|reverse)
else . end)'
jq: error (at <stdin>:65): Cannot index array with string "size"
So to unpack this, awk is used because the json output of tree contains extra commas after the last file or directory in a directory's contents array.
Line 65 of the input contains the report which looks like:
{"type":"report","size":1310049,"directories":24,"files":15}
It doesn't contains contents so the if should be avoiding it.
Here's some simpler test cases:
$ echo '
{"a":0, "c":[
{"a":1, "s":3},
{"a":2, "s":4}]}' | jq -c '
walk(if type == "object" and has("c") and (.c|length) > 0
then (.c|sort_by(.s)|reverse)
else . end)'
[{"a":2,"s":4},{"a":1,"s":3}]
$ echo '
{"a":0, "c":[
{"a":1,"s":3, "c":[
{"a":1,"s":5},
{"a":2,"s":6}]},
{"a":2,"s":4}]}' | jq -c '
walk(if type == "object" and has("c") and (.c|length) > 0
then (.c|sort_by(.s)|reverse)
else . end)'
jq: error (at <stdin>:1): Cannot index array with string "s"
I'm not sure I understand this error message and what I'm missing about walk.
The argument to walk in your post is incorrect.
You presumably want to update the value of .contents, i.e.:
walk(if type == "object" and has("contents")
then .contents |= (sort_by(.size)|reverse)
else . end)
(In the latest (post-1.6) version of jq, you can omit the else . clause.)
p.s.
Version 1.8 of tree fixes the trailing-comma issue.
If your tree produces invalid JSON when using the -J option, rather than using a generic text-processing tool such as awk, it might be better to use a tool such as hjson to "sanitize" the pseudo-JSON, e.g.
tree .... | hjson -j | jq ...
p.p.s.
To understand why one gets the error when the incorrect expression is used as the argument of walk, let's consider the simple test case in the OP, but with an extra debug inserted so we can see what's going on:
echo '
{"a":0, "c":[
{"a":1,"s":3, "c":[
{"a":1,"s":5},
{"a":2,"s":6}]
},
{"a":2,"s":4}]
}' | jq -c '
walk(if type == "object" and has("c")
then debug | .c | sort_by(.s) | reverse
else . end)'
(I've left out the length check as it just clutters up the code.)
This produces:
["DEBUG:",{"a":1,"s":3,"c":[{"a":1,"s":5},{"a":2,"s":6}]}]
["DEBUG:",{"a":0,"c":[[{"a":2,"s":6},{"a":1,"s":5}],{"a":2,"s":4}]}]
jq: error (at <stdin>:6): Cannot index array with string "s"
Now we can see the problem: the second DEBUG line shows that .c has become a bit of a jumble: the first item is an array. This is because we have replaced .c with an array. It is for this reason that the attempt to use sort_by(.s) is failing.
To understand this more completely, it would be necessary to check the definition of walk, which is easily done: you could google jq "def walk" or go to the source: builtin.jq

jq not replacing json value with parameter

test.sh is not replacing test.json parameter values ($input1 and $input2). result.json has same ParameterValue "$input1/solution/$input2.result"
[
{
"ParameterKey": "Project",
"ParameterValue": [ "$input1/solution/$input2.result" ]
}
]
test.sh
#!/bin/bash
input1="test1"
input2="test2"
echo $input1
echo $input2
cat test.json | jq 'map(if .ParameterKey == "Project" then . + {"ParameterValue" : "$input1/solution/$input2.result" } else . end )' > result.json
shell variables in jq scripts should be interpolated or passed as arguments via --arg name value:
jq --arg inp1 "$input1" --arg inp2 "$input2" \
'map(if .ParameterKey == "Project"
then . + {"ParameterValue" : ($inp1 + "/solution/" + $inp2 + ".result") }
else . end)' test.json
The output:
[
{
"ParameterKey": "Project",
"ParameterValue": "test1/solution/test2.result"
}
]
In your jq program, you have quoted "$input1/solution/$input2.result", and therefore it is a JSON string literal, whereas you evidently want string interpolation; you also need to distinguish between the shell variables ($input1 and $input2) on the one hand, and the corresponding jq dollar-variables (which may or may not have the same name) on the other.
Since your shell variables are strings, you could pass them in using the --arg command-line option (e.g. --arg input1 "$input1" if you chose to name the variables in the same way).
You can read up on string interpolation in the jq manual (see https://stedolan.github.io/jq/manual, but note the links at the top for different versions of jq).
There are other ways to achieve the desired results too, but using string interpolation with same-named variables, you'd write:
"\($input1)/solution/\($input2).result"
Notice that the above string is NOT itself literally a JSON string. Only after string interpolation does it become so.

Add an empty line prior to another matched line in jq?

Say I have a raw input like the following:
"```"
"include <stdio.h>"
"..."
"```"
"''some example''"
"*bob"
"**bob"
"*bob"
And I'd like to add a blank line right before the "*bob":
"```"
"include <stdio.h>"
"..."
"```"
"''some example''"
""
"*bob"
"**bob"
"*bob"
Can this be done with jq?
Yes, but to do so efficiently you'd effectively need jq 1.5 or higher:
foreach inputs as $line (0;
if $line == "*bob" then . + 1 else . end;
if . == 1 then "" else empty end,
$line)
Don't forget to use the -n command-line option!
Here is another solution which uses the -s (slurp) option
.[: .[["*bob"]][0]] + ["\n"] + .[.[["*bob"]][0]:] | .[]
that's a little unreadable but we can make it better with a few functions:
def firstbob: .[["*bob"]][0] ;
def beforebob: .[: firstbob ] ;
def afterbob: .[ firstbob :] ;
beforebob + ["\n"] + afterbob
| .[]
if the above filter is in filter.jq and the sample data is in data then
$ jq -Ms -f filter.jq data
produces
"```"
"include <stdio.h>"
"..."
"```"
"''some example''"
"\n"
"*bob"
"**bob"
"*bob"
One issue with this approach is that beforebob and afterbob won't quite work as you probably want if "*bob" is not in the input. The easiest way to address that is with an if guard:
if firstbob then beforebob + ["\n"] + afterbob else . end
| .[]
with that the input will be unaltered if "*bob" is not present.

jq: Testing if a key is in a list of predefined keys

I have a case where I need to parse quoted JSON in JSON.
I know which optional attributes will contain quoted JSON and which not.
Therefore, I want to check if the attribute keys are in a list of possible keys. I already have the following:
# attributes "a" and "b" contain quoted JSON
echo '{"a":"{\"x\":1}","y":2}' |
jq -c '
def is_json($o): ["a","b"] | (map(select(. == $o)) | length) > 0;
with_entries(if is_json(.key) then .value = (.value|fromjson) else . end)
'
This already produces the desired output: {"a":{"x":1},"y":2}. However, the checking of the attribute name looks clumsy, given that jq provides a lot built-in functions such as has, in, contains, inside, etc.
Question: Is there a better way of checking if an attribute key is in a given list?
Edit: Here is the current solution, based on peak's answer.
#!/bin/bash
to_array_string() { echo "$*" | awk -v OFS='","' 'NF > 0 {$1=$1; print "\""$0"\""}'; }
to_json_array_string() { echo "["`to_array_string "$#"`"]"; }
parse_json_jq() { jq -c "
reduce keys[] as \$key
(.; if any((`to_array_string "$#"`); . == \$key) and .[\$key] != null then .[\$key] |= fromjson else . end)
";}
There are three ways in which your program can be improved:
(efficiency) avoiding the creation of an unnecessary array (in is_json);
(efficiency) using "short-circuit" semantics to avoid
iterating unnecessarily;
(efficiency) avoid the construction/deconstruction involved with with_entries;
For the most part, I think you will agree that the alternatives offered here are simpler, more concise, or more readable.
If you have version 1.5 of jq or later, the main improvements
can be had using any/2:
def is_json($o): any( ("a","b"); . == $o );
with_entries(if is_json(.key) then .value |= fromjson else . end)
Notice also the use of '|=' where you had used '='.
If your jq does not have any/2, then you could use the following
definition, though it lacks short-circuit semantics:
def any(s): reduce s as $i (false; . == true or $i);
Finally, to avoid using with_entries, you could use reduce and eliminate is_json entirely:
reduce keys[] as $key
(.; if any(("a","b"); . == $key) then .[$key] |= fromjson else . end)