how to output all the keys and values from json using jq? - json

I am trying to out all the data from my json file that matches the value "data10=true" it does that but only grabs the names, how can i make it so it will output everything in my json file with anything that matches the "data10=true"?
this is what ive got data=$(jq -c 'to_entries[] | select (.value.data10 == "true")| [.key, .value.name]' data.json )
This is in my YAML template btw, running it as a pipeline in devops.

The detailed requirements are unclear, but hopefully you'll be able to use the following jq program as a guide:
..
| objects
| select( .data10 == "true" )
| to_entries[]
| select(.key != "data10")
| [.key, .value]
This will recursively (thanks to the initial ..) examine all the JSON objects in the input.
p.s.
If you want to make the selection based on whether .data10 is "true" or true, you could change the criterion to .data10 | . == true or . == "true".

jq 'to_entries | map(select(.value.data10=="true")) | from_entries' data.json
input data.json,
with false value:
{
"FOO": {
"data10": "false",
"name": "Donald",
"location": "Stockholm"
},
"BAR": {
"data10": "true",
"name": "Walt",
"location": "Stockholm"
},
"BAZ": {
"data10": "true",
"name": "Jack",
"location": "Whereever"
}
}
output:
{
"BAR": {
"data10": "true",
"name": "Walt",
"location": "Stockholm"
},
"BAZ": {
"data10": "true",
"name": "Jack",
"location": "Whereever"
}
}
based on: https://stackoverflow.com/a/37843822/983325

Related

using jq : how can i use the same search in other field without duplicate code?

I have the following json file for exemple:
{
"FOO": {
"name": "Donald",
"location": "Stockholm"
},
"BAR": {
"name": "Walt",
"location": "Stockholm"
},
"BAZ": {
"name": "Jack",
"location": "Whereever"
}
}
and i have this jq command :
cat json | jq .[] | {newname : select(.location=="Stockholm") | .name , contains_w : select(.location=="Stockholm") | .name | startswith("W")}
so i get the result :
{
"newname": "Donald",
"contains_w": false
}
{
"newname": "Walt",
"contains_w": true
}
my question is : is there any way to DRY my command ?
i mean how can i get the same result without duplicate the part :
select(.location=="Stockholm") | .name
how can i reuse the result of newname feild ?
i have a really big file to work with so i don't want to waste time and resources.
You are filtering multiple times during object construction. You could filter first and then do the construction on the filtered list eg.
map(select(.location=="Stockholm"))
| map({newname: .name, contains_w: (.name | startswith("W"))})
https://jqplay.org/s/aXjlgOEDnb

Print only one property of an object that is within an an array attribute as well as a property that is a sibling to the array property in jq

I have a json file that looks like so:
[
{
"code": "1234",
"files": [
{
"fileType": "pdf",
"url": "http://.../a.pdf"
},
{
"fileType": "video",
"url": "http://.../b.mp4"
}
]
},
{
"code": "4321",
"files": [
{
"fileType": "pdf",
"url": "http://.../c.pdf"
},
{
"fileType": "video",
"url": "http://.../d.mp4"
}
]
},
{
"code": "9999",
"files": [
{
"fileType": "pdf",
"url": "http://.../e.pdf"
}
]
}
]
I would like to print out only the files that are of fileType == video in the files array such that I end up with output that looks like so:
1234, "http://.../b.mp4"
4321, "http://.../d.mp4"
So far I am only able to output something that looks like this:
1234, "http://.../a.pdf", "http://.../b.mp4",
4321, "http://.../c.pdf", "http://.../d.mp4"
Using the following:
jq -r '.[] | select(.files[]?.fileType == "video") | [.code, .files[].url] | #csv'
I was wondering how I can filter the .files[] based on the fileType as I am outputting them?
The following pipeline makes the solution fairly self-explanatory, assuming one understands the basic syntax and the -r command-line option:
< input.json jq -r '
.[]
| .code as $code
| .files[]
| select(.fileType == "video")
| "\($code), \"\(.url)\""
'

Transform json to ini files using jq + bash

I'm trying to transform json (array of objects) to ini files:
[
{
"connection": {
"id": "br0",
"uuid": "ab1dd903-4786-4c7e-a4b4-3339b144d6c7",
"stable-id": "",
"type": "bridge",
"interface-name": "br0",
"autoconnect": "no",
"autoconnect-priority": "0",
"autoconnect-retries": "-1",
"auth-retries": "-1",
"timestamp": "44444",
"read-only": "no",
"permissions": "",
"zone": "WAN",
"master": "",
"slave-type": "",
"autoconnect-slaves": "1",
"secondaries": "",
"gateway-ping-timeout": "0",
"metered": "unknown",
"lldp": "default"
},
"ipv4": {
"method": "manual",
"dns": "192.168.1.1,192.168.2.1",
"dns-search": "",
"dns-options": " ",
"dns-priority": "0",
"addresses": "192.168.1.3/24",
"gateway": "",
"routes": "192.168.10.0/24 192.168.1.1",
"route-metric": "-1",
"route-table": "0",
"ignore-auto-routes": "no",
"ignore-auto-dns": "no",
"dhcp-client-id": "",
"dhcp-timeout": "0",
"dhcp-send-hostname": "yes",
"dhcp-hostname": "",
"dhcp-fqdn": "",
"never-default": "no",
"may-fail": "yes",
"dad-timeout": "-1"
}
},
{
"connection": {
...
},
}
]
OR
{
"connection": {
...
},
}
What i tried to do is:
1. Transform json to strings
data=$(jq -r 'def keyValue: to_entries[] | "[\(.key)]\\n\(.value | to_entries|map("\(.key)=\(.value)" ) | join("\\n") )\\n"; if type == "array" then keys[] as $k | "\(.[$k] | .connection.id)=\(.[$k] | keyValue)" elif type == "object" then "\(.connection.id)=\(. | keyValue)" else keys end' /tmp/json)
Provides:
br1=[connection]\nid=br1\nuuid=ab1dd903-4786-4c7e-a4b4-3339b144d6c7\nstable-id=\ntype=fff\ninterface-name=br0\nautoconnect=no\nautoconnect-priority=0\nautoconnect-retries=-1\nauth-retries=-1\ntimestamp=1525494904\nread-only=no\npermissions=\nzone=WAN\nmaster=\nslave-type=\nautoconnect-slaves=1\nsecondaries=\ngateway-ping-timeout=0\nmetered=unknown\nlldp=default\n
br1=[802-3-ethernet]\nport=\nspeed=0\nduplex=\nauto-negotiate=no\nmac-address=\ncloned-mac-address=\ngenerate-mac-address-mask=\nmac-address-blacklist=\nmtu=1500\ns390-subchannels=\ns390-nettype=\ns390-options=\nwake-on-lan=default\nwake-on-lan-password=\n....
2. Walk over strings in bash
while IFS="=" read -r key value; do [ "$oldkey" = "$key" ] && echo -en "$value" >> "/tmp/ini/$key" || echo -en "$value" > "/tmp/ini/$key" ; oldkey="$key"; done <<< "$data"
Gives:
[connection]
id=br1
uuid=ab1dd903-4786-4c7e-a4b4-3339b144d6c7
stable-id=
type=fff
interface-name=br0
autoconnect=no
autoconnect-priority=0
autoconnect-retries=-1
auth-retries=-1
timestamp=1525494904
read-only=no
permissions=
zone=WAN
master=
slave-type=
autoconnect-slaves=1
secondaries=
gateway-ping-timeout=0
metered=unknown
lldp=default
[ipv4]
method=manual
dns=192.168.1.1,192.168.2.1
dns-search=
dns-options=
dns-priority=0
addresses=192.168.1.3/24
gateway=
routes=192.168.10.0/24 192.168.1.1
route-metric=-1
route-table=0
ignore-auto-routes=no
ignore-auto-dns=no
dhcp-client-id=
dhcp-timeout=0
dhcp-send-hostname=yes
dhcp-hostname=
dhcp-fqdn=
never-default=no
may-fail=yes
dad-timeout=-1
I'm almost there! But is there possible to do it more "elegantly" and more performance way, avoiding pipes, external calls, etc.
Note: Moreover, mostly it should be done with jq + bash, because other processing tools like sed, awk is slower than i've done, but i do not reject them completely =)
P.S. - Main purpose of this transforming is the fast "bulk operation" to write ini files
To convert an array as shown to the ".ini" format could be accomplished by simply using this jq program:
def kv: to_entries[] | "\(.key)=\(.value)";
.[]
| to_entries[]
| "[\(.key)]", (.value|kv)
Putting this in a file, say program.jq, then assuming the JSON shown in the question (minus the "..." part) is in input.json, the following invocation:
jq -rf program.jq input.json
yields the corresponding ".ini" file.
If you want to ensure that the program will also handle the case when there is no enclosing array, you could modify the first line in the main program above to test whether the input is an array, so you'd have:
if type == "array" then .[] else . end
| to_entries[]
| "[\(.key)]", (.value|kv)
If the ultimate goal is to produce several .ini files, then we can reuse def kv as defined in the other answer:
def kv: to_entries[] | "\(.key)=\(.value)";
The driver program would however now be:
.[]
| [to_entries[] | "[\(.key)]", (.value|kv)]
| join("\n")
Running jq with this program then yields one JSON string for each item in the array. Using the example, the first such string would be:
"[connection]\nid=br0\nuuid=ab1dd903-4786-4c7e-a4b4-3339b144d6c7\nstable-id=\ntype=bridge\ninterface-name=br0\nautoconnect=no\nautoconnect-priority=0\nautoconnect-retries=-1\nauth-retries=-1\ntimestamp=44444\nread-only=no\npermissions=\nzone=WAN\nmaster=\nslave-type=\nautoconnect-slaves=1\nsecondaries=\ngateway-ping-timeout=0\nmetered=unknown\nlldp=default\n[ipv4]\nmethod=manual\ndns=192.168.1.1,192.168.2.1\ndns-search=\ndns-options= \ndns-priority=0\naddresses=192.168.1.3/24\ngateway=\nroutes=192.168.10.0/24 192.168.1.1\nroute-metric=-1\nroute-table=0\nignore-auto-routes=no\nignore-auto-dns=no\ndhcp-client-id=\ndhcp-timeout=0\ndhcp-send-hostname=yes\ndhcp-hostname=\ndhcp-fqdn=\nnever-default=no\nmay-fail=yes\ndad-timeout=-1"
You can then iterate over these newline-delimited strings.

How can I filter by a numeric field using jq?

I am writing a script to query the Bitbucket API and delete SNAPSHOT artifacts that have never been downloaded. This script is failing because it gets ALL snapshot artifacts, the select for the number of downloads does not appear to be working.
What is wrong with my select statement to filter objects by the number of downloads?
Of course the more direct solution here would be if I could just query the Bitbucket API with a filter. To the best of my knowledge the API does not support filtering by downloads.
My script is:
#!/usr/bin/env bash
curl -X GET --user "me:mykey" "https://api.bitbucket.org/2.0/repositories/myemployer/myproject/downloads?pagelen=100" > downloads.json
# get all values | reduce the set to just be name and downloads | select entries where downloads is zero | select entries where name contains SNAPSHOT | just get the name
#TODO i screwed up the selection somewhere its returning files that contain SNAPSHOT regardless of number of downloads
jq '.values | {name: .[].name, downloads: .[].downloads} | select(.downloads==0) | select(.name | contains("SNAPSHOT")) | .name' downloads.json > snapshots_without_any_downloads.js
#unique sort, not sure why jq gives me multiple values
sort -u snapshots_without_any_downloads.js | tr -d '"' > unique_snapshots_without_downloads.js
cat unique_snapshots_without_downloads.js | xargs -t -I % curl -Ss -X DELETE --user "me:mykey" "https://api.bitbucket.org/2.0/repositories/myemployer/myproject/downloads/%" > deleted_files.txt
A deidentified sample of the raw input from the API is:
{
"pagelen": 10,
"size": 40,
"values": [
{
"name": "myproject_1.1-SNAPSHOT_0210f77_mc_3.5.0.zip",
"links": {
"self": {
"href": "https://api.bitbucket.org/2.0/repositories/myemployer/myproject/downloads/myproject_1.1-SNAPSHOT_0210f77_mc_3.5.0.zip"
}
},
"downloads": 2,
"created_on": "2018-03-15T17:50:00.157310+00:00",
"user": {
"username": "me",
"display_name": "me",
"type": "user",
"uuid": "{3051ec5f-cc92-4bc3-b291-38189a490a89}",
"links": {
"self": {
"href": "https://api.bitbucket.org/2.0/users/me"
},
"html": {
"href": "https://bitbucket.org/me/"
},
"avatar": {
"href": "https://bitbucket.org/account/me/avatar/32/"
}
}
},
"type": "download",
"size": 430894
},
{
"name": "myproject_1.1-SNAPSHOT_thanks_for_the_reminder_charles_duffy_mc_3.5.0.zip",
"links": {
"self": {
"href": "https://api.bitbucket.org/2.0/repositories/myemployer/myproject/downloads/myproject_1.1-SNAPSHOT_0210f77_mc_3.5.0.zip"
}
},
"downloads": 0,
"created_on": "2018-03-15T17:50:00.157310+00:00",
"user": {
"username": "me",
"display_name": "me",
"type": "user",
"uuid": "{3051ec5f-cc92-4bc3-b291-38189a490a89}",
"links": {
"self": {
"href": "https://api.bitbucket.org/2.0/users/me"
},
"html": {
"href": "https://bitbucket.org/me/"
},
"avatar": {
"href": "https://bitbucket.org/account/me/avatar/32/"
}
}
},
"type": "download",
"size": 430894
},
{
"name": "myproject_1.0_mc_3.5.1.zip",
"links": {
"self": {
"href": "https://api.bitbucket.org/2.0/repositories/myemployer/myproject/downloads/myproject_1.1-SNAPSHOT_0210f77_mc_3.5.1.zip"
}
},
"downloads": 5,
"created_on": "2018-03-15T17:49:14.885544+00:00",
"user": {
"username": "me",
"display_name": "me",
"type": "user",
"uuid": "{3051ec5f-cc92-4bc3-b291-38189a490a89}",
"links": {
"self": {
"href": "https://api.bitbucket.org/2.0/users/me"
},
"html": {
"href": "https://bitbucket.org/me/"
},
"avatar": {
"href": "https://bitbucket.org/account/me/avatar/32/"
}
}
},
"type": "download",
"size": 430934
}
],
"page": 1,
"next": "https://api.bitbucket.org/2.0/repositories/myemployer/myproject/downloads?pagelen=10&page=2"
}
The output I want from this snippet is myproject_1.1-SNAPSHOT_thanks_for_the_reminder_charles_duffy_mc_3.5.0.zip - that artifact is a SNAPSHOT and has zero downloads.
I have used this intermediate step to do some debugging:
jq '.values | {name: .[].name, downloads: .[].downloads} | select(.downloads>0) | select(.name | contains("SNAPSHOT")) | unique' downloads.json > snapshots_with_downloads.js
jq '.values | {name: .[].name, downloads: .[].downloads} | select(.downloads==0) | select(.name | contains("SNAPSHOT")) | .name' downloads.json > snapshots_without_any_downloads.js
#this returns the same values for each list!
diff unique_snapshots_with_downloads.js unique_snapshots_without_downloads.js
This adjustment gives a cleaner and unique structure, it suggests that theres some sort of splitting or streaming aspect of jq that I do not fully understand:
#this returns a "unique" array like I expect, adding select to this still does not produce the desired outcome
jq '.values | [{name: .[].name, downloads: .[].downloads}] | unique' downloads.json
The data after this step looks like this. It just removed the cruft I didn't need from the raw API response:
[
{
"name": "myproject_1.0_2400a51_mc_3.4.0.zip",
"downloads": 0
},
{
"name": "myproject_1.0_2400a51_mc_3.4.1.zip",
"downloads": 2
},
{
"name": "myproject_1.1-SNAPSHOT_391f4d5_mc_3.5.0.zip",
"downloads": 0
},
{
"name": "myproject_1.1-SNAPSHOT_391f4d5_mc_3.5.1.zip",
"downloads": 2
}
]
As I understand it:
You want globally unique outputs
You want only items with downloads==0
You want only items whose name contains "SNAPSHOT"
The following will accomplish that:
jq -r '
[.values[] | {(.name): .downloads}]
| add
| to_entries[]
| select(.value == 0)
| .key | select(contains("SNAPSHOT"))'
Rather than making unique an explicit step, this version generates a map from names to download counters (adding the values together -- which means that in case of conflicts, the last one wins), and thereby both ensures that the outputs are unique.
Given your test JSON, output is:
myproject_1.1-SNAPSHOT_thanks_for_the_reminder_charles_duffy_mc_3.5.0.zip
Applied to the overall problem context, this strategy can be used to simplify the overall process:
jq -r '[.values[] | {(.links.self.href): .downloads}] | add | to_entries[] | select(.value == 0) | .key | select(contains("SNAPSHOT"))'
It simplifies the overall process by acting on the URL to the file rather than the name only. This simplifies the subsequent DELETE call. The sort and tr calls can also be removed.
Here's a solution which sums up the .download values per .name before making the selection based on the total number of downloads:
reduce (.values[] | select(.name | contains("SNAPSHOT"))) as $v
({}; .[$v.name] += $v.downloads)
| with_entries(select(.value == 0))
| keys_unsorted[]
Example:
$ jq -r -f program.jq input.json
myproject_1.1-SNAPSHOT_thanks_for_the_reminder_charles_duffy_mc_3.5.0.zip
p.s.
What is wrong with my select statement ...?
The problem that jumps out is the bit of the pipeline just before the "select" filter:
.values | {name: .[].name, downloads: .[].downloads}
The use of .[] in this manner results in the Cartesian product being formed -- that is, the above expression will emit n*n JSON sets, where n is the length of .values. You evidently intended to write:
.values[] | {name: .name, downloads: .downloads}
which can be abbreviated to:
.values[] | {name, downloads}

Bash JQ getting multiple values Issue in JSON file

I'm trying to parse a JSON file for getting multiple values. I know how to parse the specific values ( "A"/"B"/"C") in the array (.info.file.hashes[]).
For Example : When issuing the following command over the file b.json
jq -r '.info.file.hashes[] | select(.name == ("A","B","C")).value' b.json
Result :
f34d5f2d4577ed6d9ceec516c1f5a744
66031dad95dfe6ad10b35f06c4342faa
9df25fa4e379837e42aaf6d05d92012018d4b659
Where b.json:
{
"Finish": 1475668827,
"Start": 1475668826,
"info": {
"file": {
"Score": 4,
"file_subtype": "None",
"file_type": "Image",
"hashes": [
{
"name": "A",
"value": "f34d5f2d4577ed6d9ceec516c1f5a744"
},
{
"name": "B",
"value": "66031dad95dfe6ad10b35f06c4342faa"
},
{
"name": "C",
"value": "9df25fa4e379837e42aaf6d05d92012018d4b659"
},
{
"name": "D",
"value": "4a51cc531082d216a3cf292f4c39869b462bf6aa"
},
{
"name": "E",
"value": "e445f412f92b25f3343d5f7adc3c94bdc950601521d5b91e7ce77c21a18259c9"
}
],
"size": 500
}
}
}
Now, how can i get multiple values with "Finish", "Start" along with the hash values? I have tried issuing the command.
jq -r '.info.file.hashes[] | select(.name == ("A","B","C")).value','.Finish','.Start' b.json
and Im getting the result as:
f34d5f2d4577ed6d9ceec516c1f5a744
null
66031dad95dfe6ad10b35f06c4342faa
null
9df25fa4e379837e42aaf6d05d92012018d4b659
null
null
null
Expected Result :
f34d5f2d4577ed6d9ceec516c1f5a744
66031dad95dfe6ad10b35f06c4342faa
9df25fa4e379837e42aaf6d05d92012018d4b659
1475668827
1475668826
Literally just downloaded and read the manual
Try
jq '(.info.file.hashes[] |select(.name == ("A","B","C")).value), .Finish, .Start' b.json
"f34d5f2d4577ed6d9ceec516c1f5a744"
"66031dad95dfe6ad10b35f06c4342faa"
"9df25fa4e379837e42aaf6d05d92012018d4b659"
1475668827
1475668826
Note the brackets used for grouping the pipe separately from the Finish and Start values.