jq - extracting a subset of matching documents from a stream - json

I have the following JSON message. This is a single complete message. There are so many messages like this in a single file. This json message was generated from a unformated json message using jq.
{
"header": {
"user": "baskar"
},
"requests": [
{
"first_name": "mike",
"last_name": "mat"
},
{
"first_name": "mike",
"last_name": "mat"
}
],
"check": [
"Y"
]
}
{
"header": {
"user": "baskar"
},
"message": {
"header": {
"user": "baskar"
},
"response": {
"resultsList": {
"result": [
{
"first_name": "mike1",
"last_name": "mat"
}
]
},
"errorMsg": null
}
}
}
I would like to do some filtering on this. For example, when i search for first_name, mike1, I should get the header and the matching request inside request. Also the matching result inside the Response message. So, the output is expected as follows for the search string mike1.
{
"header": {
"user": "baskar"
},
"requests": [
{
"first_name": "mike1",
"last_name": "mat"
}
],
"check": [
"Y"
]
}
{
"header": {
"user": "baskar"
},
"message": {
"header": {
"user": "baskar"
},
"response": {
"resultsList": {
"result": [
{
"first_name": "mike1",
"last_name": "mat"
}
]
},
"errorMsg": null
}
}
}
Basically, i want to filter out unmatched request inside the requests array and unmatched result inside the result array.
Currently, I use the following script to get the formatted json message from the unformatted json message log file.
sed -n "/<SEARCH_STRING>/ s/.*Service - //p" $1/test.log* | jq . > ~/result.log
Thanks,
Baskar.S

jq --arg key first_name \
--arg value mike1 \
'select(.message.response.resultsList.result[]?[$key]==$value) | .message' \
<in.json
...returns only message content where the result list contains at least one first_name of mike1.

Related

Delete json block with jq command

I have json file with multiple domains which is formated as is showed below. How can I delete whole blocks with domains? For example if I will want to delete whole block in json for domain domain.tld?
I tryed this, but output is error:
jq '."http-01"."domain"[]."main"="domain.tld"' acme.json
jq: error (at acme.json:11483): Cannot iterate over null (null)
formating example file:
{
"http-01": {
"Account": {
"Email": "mail#placeholder.tld",
"Registration": {
"body": {
"status": "valid",
"contact": [
"mailto:mail#placeholder.tld"
]
},
"uri": "https://acme-v02.api.letsencrypt.org/acme/acct/110801506"
},
"PrivateKey": "main_priv_key_string",
"KeyType": "4096"
},
"Certificates": [
{
"domain": {
"main": "www.some_domain.tld"
},
"certificate": "cert_string",
"key": "key_string",
"Store": "default"
},
{
"domain": {
"main": "some_domain.tld"
},
"certificate": "cert_string",
"key": "key_string",
"Store": "default"
},
{
"domain": {
"main": "www.some_domain2.tld"
},
"certificate": "cert_string",
"key": "key_string",
"Store": "default"
},
{
"domain": {
"main": "some_domain2.tld"
},
"certificate": "cert_string",
"key": "key_string",
"Store": "default"
}
]
}
}
To delete domain block "www.some_domain.tld" :
jq '."http-01".Certificates |= map(select(.domain.main != "www.some_domain.tld"))' input.json
Your question is quite broad. What is a "block"?
Let's assume you want to delete from within the object under http-01 each field that is of type array and has at index 0 an object satisfying .domain.main == "domain.tld". Then first navigate to where you want to delete from, and update it (|=) using del and select which performs the filtered deletion.
jq '
."http-01" |= del(
.[] | select(arrays[0] | objects.domain.main == "domain.tld")
)
' acme.json
{
"http-01": {
"Account": {
"Email": "email#domain.tld",
"Registration": {
"body": {
"status": "valid",
"contact": [
"mailto:email#domain.tld"
]
},
"uri": "https://acme-v02.api.letsencrypt.org/acme/acct/110801506"
},
"PrivateKey": "long_key_string",
"KeyType": "4096"
}
}
}
Demo
If your "block" is deeper, go deeper before updating. If it is higher, the whole document for instance, there's no need to update, just start with del.

Using jq to search for a value based on a key located deep in json file

I am new to jq and I'm trying to use it to search for a value in a json file based on a key that is located deep in the json structure. Here is a sample of my json file:
{
"data": {
"inventory": {
"location": "remote",
"list": {
"content": [
{
"item": {
"name": "minivan"
},
"owner": {
"id": "12345",
"state": "CA"
}
},
{
"item": {
"name": "sedan"
},
"owner": {
"id": "67890",
"state": "AZ"
}
}
]
}
}
}
}
An example of search that I'm trying to do is:
select item.name where owner.id = "67890"
and the expected output would be:
item.name = "sedan"
I'm trying to run the following:
jq '.[] | select .owner.id = "67890" | .item.name' json
and it generates an error:
jq: error: select/0 is not defined at <top-level>, line 1:
.[] | select .owner.id = "67890" | .item.name
jq: 1 compile error
Any pointers on how to do this in jq would be much appreciated!
Thanks!
First, you have to "navigate" to where you want to make the query. This seems to be an array.
.data.inventory.list.content
[
{
"item": {
"name": "minivan"
},
"owner": {
"id": "12345",
"state": "CA"
}
},
{
"item": {
"name": "sedan"
},
"owner": {
"id": "67890",
"state": "AZ"
}
}
]
Demo
Next, let's iterate over that array's items, which gives us a stream of objects.
.[]
{
"item": {
"name": "minivan"
},
"owner": {
"id": "12345",
"state": "CA"
}
}
{
"item": {
"name": "sedan"
},
"owner": {
"id": "67890",
"state": "AZ"
}
}
Demo
From these objects we select those that match your criteria.
select(.owner.id == "67890")
{
"item": {
"name": "sedan"
},
"owner": {
"id": "67890",
"state": "AZ"
}
}
Demo
Finally, we extract the value you're interested in.
.item.name
"sedan"
Demo
Everything combined in a jq call would be:
jq '.data.inventory.list.content[] | select(.owner.id == "67890").item.name'
"sedan"
Demo
This output is still valid JSON document (containing nothing but a JSON string). If you want to process the output as raw text, use the --raw-output (or -r) option:
jq -r '.data.inventory.list.content[] | select(.owner.id == "67890").item.name'
sedan
Demo
Here's a solution that avoids having to "navigate" to the right place, and which is also quite close to your SQL-like query:
..
| objects
| select(.owner and
(.owner|type=="object" and .id == "67890"))
.item.name
or more succinctly:
..|objects|select(.owner.id? == "67890").item.name

Convert JSON Array to CSV

Trying to convert a JSON array into a CSV file using jq but not able to succeed. Following is my JSON output from curl command:
{
"requestID": "463aeb25-f4c3-40ba-a031-e62d698afc6e",
"signature": {
"id": "json",
"ph_no": "json",
"status": "json"
},
"results": [
{
"id": "9f34-66758813073c",
"ph_no": "343434325",
"status": "active"
},
{
"id": "b1a2-30a14a68c576",
"ph_no": "6767666764",
"status": "active"
},
{
"id": "9af4-5b231f05ce37",
"ph_no": "546745435",
"status": "active"
},
{
"id": "99bd-ed67fd139074",
"ph_no": "323246566",
"status": "active"
},
{
"id": "9ecc-8277c3ffa274",
"ph_no": "6753643554",
"status": "active"
}
],
"status": "success",
"metrics": {
"elapsedTime": "29.461027ms",
"executionTime": "29.364961ms",
"resultCount": 146,
"resultSize": 13856
}
}
I have tried using following referring to some solutions online but not working.
jq -r '["id","ph_no","status"],(to_entries|.[]|[.key,.value.id,.value.ph_no,.value.status)|#csv' temp.json
How should i modify the jq command to convert JSON to CSV ?
If you just want the results array of objects:
jq -r '(["id","ph_no","status"], (.results[] | [.id, .ph_no, .status])) | #csv' temp.json

Looping and searching through JSON using PowerShell

I have a JSON file that has following contents:
{
"status": "UP",
"details": {
"graphDBCheck": {
"status": "UP"
},
"ds": {
"status": "UP",
"details": {
"total": 100,
"free": 50,
"threshold": 30
}
},
"db": {
"status": "UP",
"details": {
"ADS": {
"status": "UP",
"details": {
"database": "Influx",
"hello": "Hello"
}
},
"EARDS": {
"status": "UP",
"details": {
"database": "Oracle",
"hello": "Hello"
}
},
"EFRDS": {
"status": "UP",
"details": {
"database": "Sybase",
"hello": "Hello"
}
}
}
}
}
}
I need to be able to transform this into a CSV file that has each element's name as header and it's status or value as the next row. First "status" would have column name "API_Status"
For instance:
API_Status,graphDBCheck,ds,db,ADS,EARDS,EFRDS
UP,UP,UP,UP,UP,UP,UP
Challenge here is to make this dynamic so the output will always include any other element added that has "status" in it.
I tried this and it works but I need a dynamic way do to this:
$x = Invoke-RestMethod $url -Verbose:$VerbosePreference
[pscustomobject][ordered]#{
'API_Status' = $x.status
'db' = $x.details.db.status
'ds' = $x.details.diskspace.status
'ds_Total' = $x.details.ds.details.total
'ds_Free' = $x.details.ds.details.free
'graphDBCheck' = $x.details.graphDBCheck.status
'ADS' = $x.details.db.details.auroraDataSource.status
'EARDS' = $x.details.db.details.EARDS.status
'EFRDS' = $x.details.db.details.edsFirstRowsDataSource.status
}
In an ideal world, the json would be structured like this, as an expandable array with uniform properties.
[
{
"name": "API_Status",
"status": "UP"
},
{
"name": "graphDBCheck",
"status": "UP"
},
{
"name": "ds",
"status": "UP"
},
{
"name": "db",
"status": "UP"
},
{
"name": "ADS",
"status": "UP"
},
{
"name": "EARDS",
"status": "UP"
},
{
"name": "EFRDS",
"status": "UP"
}
]
Or as a csv:
name,status
API_Status,UP
graphDBCheck,UP
ds,UP
db,UP
ADS,UP
EARDS,UP
EFRDS,UP
There's plenty of other posts about looping through powershell properties Iterate over PSObject properties in PowerShell or looping through json properties: Iterating through a JSON file PowerShell

jq add capturing group result outside

For example,
Input:
{
"id":"abc",
"name": "name-middlenane-lastname-1"
},
{
"id":"123",
"name": "fname-flast-2"
}
response:
{
"id":"abc",
"name": "name-middlename-lastname-1",
"newkey": "name-middlename-lastname"
},
{
"id":"123",
"name": "fname-flast-2",
"newkey": "fname-flast"
}
The filed name in each object is a string with characters and numbers separated by "-" hyphen. I need the complete string from beginning till the starting number. I don't want anything which is there after the number. And then the add new field with key as newkey and value should be extracted string without the number. Thus, the output should contain old fields as well as new one.
jq solution:
Sample input.json:
[
{
"id": "abc",
"name": "name-middlenane-lastname-1"
},
{
"id": "123",
"name": "fname-flast-2"
}
]
jq 'map(. + (.name | capture("(?<newkey>.+)-[0-9]+")) )' input.json
The output:
[
{
"id": "abc",
"name": "name-middlenane-lastname-1",
"newkey": "name-middlenane-lastname"
},
{
"id": "123",
"name": "fname-flast-2",
"newkey": "fname-flast"
}
]