How can I avoid doing embarrassing stuff like this when trying to apply multiple regular expressions using the gsub() function in jq?
."values" | tostring | gsub("\"";"`") | gsub("\\[";"") | gsub("\\]";"") | gsub("=\\w*";"")
I want to convert the array below to a string, keep the values to the left of the equals sign and surround each value in backticks.
The jq command above works but something tells me there's a more elegant solution.
Input:
{
"values": [
"1=foo",
"2=bar",
"3=baz"
]
}
Output (expected and actual)
"`1`,`2`,`3`"
split() on a =
Take the first part
wrap in ` using string interpolation
join() with an ,:
.values | map("`\(split("=") | first)`") | join(",")
JqPlay Demo
Yet you can use gsub() such as
."values" | map("`\(gsub("=\\w+";""))`")| join(",")
Demo
.values
| map(capture( "(?<key>^[^=]*)=" ).key | "`\(.)`" )
| join(",")
Related
Given this JSON:
{
"key": "/books/OL1000072M",
"source_records": [
"ia:daywithtroubadou00pern",
"bwb:9780822519157",
"marc:marc_loc_2016/BooksAll.2016.part25.utf8:103836014:1267"
]
}
Can the following jq code be simplified?
jq -r '.key as $olid | .source_records | map([$olid, .])[] | #tsv'
The use of variable assignment feels like cheating and I'm wondering if it can be eliminated. The goal is to map the key value onto each of the source_records values and output a two column TSV.
Instead of mapping into an array, and then iterating over it (map(…)[]) just create an array and collect its items ([…]). Also, you can get rid of the variable binding (as) by moving the second part into its own context using parens.
jq -r '[.key] + (.source_records[] | [.]) | #tsv'
Alternatively, instead of using #tsv you could build your tab-separated output string yourself. Either by concatenation (… + …) or by string interpolation ("\(…)"):
jq -r '.key + "\t" + .source_records[]'
jq -r '"\(.key)\t\(.source_records[])"'
Output:
/books/OL1000072M ia:daywithtroubadou00pern
/books/OL1000072M bwb:9780822519157
/books/OL1000072M marc:marc_loc_2016/BooksAll.2016.part25.utf8:103836014:1267
It's not much shorter, but I think it's clearer than the original and clearer than the other shorter answers.
jq -r '.key as $olid | .source_records[] | [ $olid, . ] | #tsv'
Just started playing around with jq and cannot for the life of me come to terms with how I should approach this in a cleaner way. I have some data from AWS SSM Parameter Store that I receive as JSON, that I want to process.
The data is structured in the following way
[
{
"Name": "/path/to/key_value",
"Value": "foo"
},
{
"Name": "/path/to/key_value_2",
"Value": "bar"
},
...
]
I want it output in the following way: key_value=foo key_value_2=bar. My first thought was to process it as follows: map([.Name | split("/") | last, .Value] | join("=")) | join(" ") but then I get the following error: jq: error (at <stdin>:9): Cannot index array with string "Value". It's as if the reference to the Value value is lost after piping the value for the Name parameter.
Of course I could just solve it like this, but it's just plain ugly: map([.Value, .Name | split("/") | last] | reverse | join("=")) | join(" "). How do I process the value for Name without losing reference to Value?
Edit: JQ Play link
map((.Name | split("/") | last) + "=" + .Value) | join(" ")
Will output:
"key_value=foo key_value_2=bar"
Online demo
The 'trick' is to wrap the .Name | split("/") | last) into () so that .value remains available
If you prefer string interpolation (\()) over (key) + .Value, you can rewrite it as:
map("\(.Name | split("/") | last)=\(.Value)") | join(" ")
Online demo
This question already has answers here:
Get JSON string from within javascript on a html page using shell script
(2 answers)
Closed 4 years ago.
I have this json string :
{
"head": {
"url": "foobar;myid=E50DAA932C22739F92BB250C14365440"
}
}
With jq on the shell I get the content of url as an array:
jq -r '.head.url | split(";")[] '
This returns:
foobar
myid=E50DAA932C22739F92BB250C14365440
My goal is to get the id (E50DA...) after = only. I could simply use [1] to get the second element and then use a regex to get the part after the =.
But the order of elements is not safe and I'm sure there's a better way with jq already that I dont know of. Maybe create a map of the elements and use myid as a key to get the value (E50...)?
Thank you for your input!
Do you have to do it with jq only? You could further process the output with grep and cut:
jq '.head.url | split(";")[]' | grep '^myid=' | cut -d= -f2
But alas, it is easily possible by first building an object from the key value pairs and then look up the value for the key in question:
.head.url
| split(";")
| map(split("=") | { key: .[0], value: .[1] })
| from_entries
| .myid
equivalent to:
.head.url
| split(";")
| map(split("=") | { key: .[0], value: .[1] })
| from_entries["myid"]
Or without building an object, simply by selecting the first array item with matching key, then outputting its value:
.head.url | split(";")[] | split("=") | select(first == "myid")[1]
NB. x | split(y) can be expressed as x/y, e.g. .head.url/"#".
Using jq's match() with positive lookbehind to output what's after myid=:
$ jq -r '.head.url | split(";")[] | match("(?<=myid=).*;"g").string' file
E50DAA932C22739F92BB250C14365440
or drop the split() and match() after myid= until the end or ;:
$ jq -r '.head.url | match("(?<=myid=)[^;]*";"g").string' file
E50DAA932C22739F92BB250C14365440
I have the following JSON:
{
"field1":"foo",
"array":[
{
child_field1:"c1_1",
child_field2:"c1_2"
},
{
child_field1:"c2_1",
child_field2:"c2_2"
}
]...
}
and using jq, I would like to return the following output, where the value of field1 is repeated for every child element.:
foo,c1_1,c1_2
foo,c2_1,c2_2
...
I can access each field separately, am having trouble returning the desired result above.
Can this be done with jq?
jq -r '.array[] as $a | [.field1, $a.child_field1, $a.child_field2] | #csv'
Does the right thing for the sample data you provided, but I freely admit there are lots of ways to do that kind of thing in jq, and that was only the first one which sprang to mind.
I fed it through #csv because it seemed like that was what you wanted, but if you prefer the actual output, exactly as you have written, then:
jq -r '.array[] as $a | "\(.field1),\($a.child_field1),\($a.child_field2)"'
will produce it
In cases like this, there's no need for reduce or `to_entries', or to list the fields explicitly -- one can simply exploit jq's backtracking behavior:
.field1 as $f
| .array[]
| [$f, .[]]
| #csv
As pointed out by #MatthewLDaniel, there are many alternatives to using #csv here.
jq solution:
jq -r '.field1 as $f1 | .array[]
| [$f1, .[]]
| join(",")' input.json
The output:
foo,c1_1,c1_2
foo,c2_1,c2_2
I have a file with a stream of JSON objects as follows:
{"id":4496,"status":"Analyze","severity":"Critical","severityCode":1,"state":"New","code":"RNPD.DEREF","title":"Suspicious dereference of pointer before NULL check","message":"Suspicious dereference of pointer \u0027peer-\u003esctSapCb\u0027 before NULL check at line 516","file":"/home/build/branches/mmm/file1","method":"CzUiCztGpReq","owner":"unowned","taxonomyName":"C and C++","dateOriginated":1473991086512,"url":"http://xxx/yyy","issueIds":[4494]}
{"id":4497,"status":"Analyze","severity":"Critical","severityCode":1,"state":"New","code":"NPD.GEN.CALL.MIGHT","title":"Null pointer may be passed to function that may dereference it","message":"Null pointer \u0027tmpEncodedPdu\u0027 that comes from line 346 may be passed to function and can be dereferenced there by passing argument 1 to function \u0027SCpyMsgMsgF\u0027 at line 537.","file":"/home/build/branches/mmm/file1","method":"CzUiCztGpReq","owner":"unowned","taxonomyName":"C and C++","dateOriginated":1473991086512,"url":"http://xxx/yyy/zzz","issueIds":[4495]}
{"id":4498,"status":"Analyze","severity":"Critical","severityCode":1,"state":"New","code":"NPD.GEN.CALL.MIGHT","title":"Null pointer may be passed to function that may dereference it","message":"Null pointer \u0027tmpEncodedPdu\u0027 that comes from line 346 may be passed to function and can be dereferenced there by passing argument 1 to function \u0027SCpyMsgMsgF\u0027 at line 537.","file":"/home/build/branches/mmm/otherfile.c","method":"CzUiCztGpReq","owner":"unowned","taxonomyName":"C and C++","dateOriginated":1473991086512,"url":"http://xxx/yyy/zzz","issueIds":[4495]}
I would like to get with JQ (or in some other way), three lines, one each for the ids, the URLs, and the file name:
This is what I have so far:
cat /tmp/file.json | ~/bin_compciv/jq --raw-output '.id,.url,.file'
Result:
4496
http://xxx/yyy
/home/build/branches/mmm/file1
.
.
.
BUT - I would like to group them by file name, so that I will get comma-separated lists of urls and ids on the same line, like this:
4496,4497
http://xxx/yyy,http://xxx/yyy/zzz
/home/build/branches/mmm/file1
With one minor exception, you can readily achieve the stated goals using jq as follows:
jq -scr 'map({id,url,file})
| group_by(.file)
| .[]
| ((map(.id) | #csv) , (map(.url) | #csv), (.[0] | .file))'
Given your input, the output would be:
4496,4497
"http://xxx/yyy","http://xxx/yyy/zzz"
/home/build/branches/mmm/file1
4498
"http://xxx/yyy/zzz"
/home/build/branches/mmm/otherfile.c
You could then eliminate the quotation marks using a text-editing tool such as sed; using another invocation of jq; or as described below. However, this might not be such a great idea if there's ever any chance that any of the URLs contains a comma.
Here's the filter for eliminating the quotation marks with just one invocation of jq:
map({id,url,file})
| group_by(.file)
| .[]
| ((map(.id) | #csv),
([map(.url) | join(",")] | #csv | .[1:-1]),
(.[0] | .file))
Here is a solution which uses group_by and the -r, -s jq options:
group_by(.file)[]
| ([ "\(.[].id)" ] | join(",")),
([ .[].url ] | join(",")),
.[0].file