Searching for object with jQ in command line - json

I have a JSON file of AWS security groups list. I am trying to fetch the Group Id using the Group Name. The object looks like the following:
{
"SecurityGroups": [{
"IpPermissionsEgress": [
{
"IpProtocol": "-1",
}
],
"Description": "launch-wizard-2 created 2017-10-21T09:19:40.007-04:00",
"GroupName": "MY1SG-PUBLIC-80",
"VpcId": "vpc-ceed12b7",
"OwnerId": "712503525534",
"GroupId": "sg-ee0c979c"
}]
}
With jQ my attempt is as follows:
aws ec2 describe-security-groups | jq '.GroupId' ["GroupName": "MY1SG-PUBLIC-80"]
Error:
jq: error: Could not open file [GroupName:: No such file or directory
jq: error: Could not open file MY1SG-PUBLIC-80]: No such file or directory

Issue 1: Format
https://shapeshed.com/jq-json/
The second input to jq is the file you wish to read from. If this value is - the program will read from the instream.
Issue 2: Selection
https://stedolan.github.io/jq/manual/#select(boolean_expression)
To select an element by value you should/could use a select statement
select(.GroupName == "MY1SG-PUBLIC-80")
jq 'SCOPE | select(.GroupName == "MY1SG-PUBLIC-80") | .GroupId[]' -
where SCOPE is the group you wish to look in. if SCOPE is .[], it will scan every json entry. Following this, it pipes this group into a select filter, and trims it down to only ones that have GroupName set to the given value. This result set is then piped into a key filter, where it only returns the array of matching GroupID's.

I am trying to fetch the Group Id using the Group Name.
Assuming the input has been tweaked to make it valid JSON (*), the filter:
.SecurityGroups[] | select(.GroupName=="MY1SG-PUBLIC-80") | .GroupId
produces:
"sg-ee0c979c"
It might be worthwhile considering this alternative filter:
.[][]|select(.GroupName=="MY1SG-PUBLIC-80")|.GroupId
(*) The input as originally shown has an extraneous comma.

Related

Maintain order of jq CSV output and include empty values

I am currently working on a bash script that combines the output of both the aws iam list-users and aws iam list-user-tags commands in a CSV file containing all users along with their respective information and assigned tags. To parse the JSON output of those commands I choose to use jq.
Retrieving parsing and converting (JSON to CSV) the list-user output works fine and produces the expected comma-separated list of values.
The output of list-user-tags does not quite behave that way. Its JSON output has the following schema:
{
Tags: [
{
Key: "Name",
Value: "NameOfUser"
},
{
Key: "Email",
Value: "EmailOfUser"
},
{
Key: "Company",
Value: "CompanyOfUser"
}
]
}
Unfortunately the order of the tags is not consistent across users (and possibly across queries) which currently makes it impossible for me to maintain the order defined in the CSV file. On top of that there is the possibility of one or multiple missing tags.
What I am looking for is a way to achieve the following (preferably using jq):
Select a tags "Value"-value by its "Key"-value
Check whether it is existent and if not add an empty entry
Put the value in the exact same place every time (maintain a certain order)
Repeat for every entry in the original output
Convert the resulting array of values into CSV
What I tried so far:
aws iam list-user-tags --user-name abcdef --no-cli-pager \
| jq -r '[.Tags[] | select(.Key=="Name"),select(.Key=="Email"),select(.Key=="Company") | .Value // ""] | #csv'
Any help is much appreciated!
Let's suppose your sample "schema" is in a file named schema.jq
Then
jq -n -f schema.jq | jq -r '
def get($key):
map(select(.Key == $key))
| if length == 0 then null else .[0].Value end;
.Tags | [get("Name"), get("Email"), get("Company")] | #csv
'
produces the following CSV:
"NameOfUser","EmailOfUser","CompanyOfUser"
It should be easy to adapt this illustration to your needs.

I am having trouble generating command regarding following json file I have

This is how my json file looks
"Patient":{
"Name":{
"Patient First Name":"James",
"Patient Middle Name":"Adam",
"Patient Last Name":"Manchester"
},
"Unique ID":163983,
"Demographics":{
"Sex":"M",
"Birthdate":"7/24/1940"
},
"IntakeCriteria":{
"DmDxDate":"8/25/2012",
"InitialHgbA1c":8.1,
"Comorbidity":["Diabetes Mellitus","CHF","DVT"]
},
"Labs":{
"LDLCholesterol":{
"LDLResultDate":"5/6/2013",
"LDLLevel":200
},
"SerumCreatinine":{
"CreatinineResultDate":"11/1/2016",
"CreatinineLevel":0.9
}
},
"CareLocation":{
"Facility Name":"East Side Clinic",
"Facility Contact Name":"Mary Silverman",
"Facility Contact Phone":"618-348-1891"
}
}
}
I need to prepare a query using the following fields:
Patient Unique ID number.
Patient First Name.
Patient Last Name.
Lab Test Date.
HgbA1c Level.
LDL Level.
Creatinine Level.
I tried this resulting in no result
jq - r’[.Patient.UniqueID, .Patient.Name.FirstName, .Patient.Name.LastName, .Patient.Labs.LDLCholesterol.LabResultDate, .Patient.IntakeCriteria.InitialHgbA1c, .Patient.Labs.LDLCholesterol.LDLLevel, .Patient.Labs.SerumCreatine.CreatineLevel] | #csv’ data.json > csvtest.out
When using object identifiers, you have to provide the field names exactly as used in the JSON file. Also, if they contain special characters such as white spaces (or start with a digit), surround them with double quotes for clarification:
jq -r '[
.Patient."Unique ID",
.Patient.Name."Patient First Name",
.Patient.Name."Patient Last Name",
.Patient.Labs.LDLCholesterol.LDLResultDate,
.Patient.IntakeCriteria.InitialHgbA1c,
.Patient.Labs.LDLCholesterol.LDLLevel,
.Patient.Labs.SerumCreatinine.CreatinineLevel
] | #csv' data.json > csvtest.out
163983,"James","Manchester","5/6/2013",8.1,200,0.9
Demo
To shorten your filter a bit, you may also group together some of the paths sharing common parents using subfilters with pipes | within a grouping with parentheses ():
jq -r '[.Patient |
."Unique ID",
(.Name | ."Patient First Name", ."Patient Last Name"),
.Labs.LDLCholesterol.LDLResultDate,
.IntakeCriteria.InitialHgbA1c,
(.Labs | .LDLCholesterol.LDLLevel, .SerumCreatinine.CreatinineLevel)
] | #csv' data.json > csvtest.out
163983,"James","Manchester","5/6/2013",8.1,200,0.9
Demo

Non json output from gcloud ai-platform predict. Parsing non-json outputs

I am using gcloud ai-platform predict to call an endpoint and get predictions as below using json-request and not json-response
gcloud ai-platform predict --json-request instances.json
The response is however not json and hense cannot be read further causing other complications. Below is the response.
VAL HS
0.5 {'hs_1': [[-0.134501, -0.307326, -0.151994, -0.065352, -0.14138]], 'hs_2' : [[-0.134501, -0.307326, -0.151994, -0.065352, 0.020759]]}
Can gcloud ai-platform predict return a json instead or may be parse it differently. ?
Thanks for your help.
Apparently, your output is a table with headers and two columns: a score and the (alleged) JSON content. You should extract the second column of any preferred data row (your example only has one but in general you might receive several score-JSON pairs). Maybe your API already offers functionality to extract a certain 'state', e.g. the one with the highest score. If not, a simple awk or sed script can get this job done easily.
Then, the only remaining issue before having proper JSON (which can then be queried by jq) is with the quoting style. Your output encloses field names with ' instead of " ('lstm_1' instead of "lstm_1"). Correcting thin, unfortunately, is a not-so-easy task if you can expect to receive arbitrarily complex JSON data (such as strings containing quotation marks etc.). However, if your JSON will always look as simple as in the example provided, simply substituting the wrong for the right one becomes an easy task again for tools like awk or sed.
For instance, using sed on your example output to select the second line (which is the first data row), drop everything from the beginning until but not including the first opening curly brace (which marks the beginning of the second column), make said substitutions and pipe the result into jq:
... | sed -n "2{s/^[^{]\+//;s/'/\"/g;p;q}" | jq .
{
"lstm_1": [
[
-0.13450142741203308,
-0.3073260486125946,
-0.15199440717697144,
-0.06535257399082184,
-0.1413831114768982
]
],
"lstm_2": [
[
-0.13450142741203308,
-0.3073260486125946,
-0.15199440717697144,
-0.06535257399082184,
0.02075939252972603
]
]
}
[Edited to reflect upon a comment]
If you want to utilize the score as well, let jq handle it. For instance:
... | sed -n "2{s/'/\"/g;p;q}" | jq -s '{score:first,status:last}'
{
"score": 0.548,
"status": {
"lstm_1": [
[
-0.13450142741203308,
-0.3073260486125946,
-0.15199440717697144,
-0.06535257399082184,
-0.1413831114768982
]
],
"lstm_2": [
[
-0.13450142741203308,
-0.3073260486125946,
-0.15199440717697144,
-0.06535257399082184,
0.02075939252972603
]
]
}
}
[Edited to reflect upon changes in the OP]
As changes affected only names and values but no structure, the hitherto valid approach still holds:
... | sed -n "2{s/'/\"/g;p;q}" | jq -s '{val:first,hs:last}'
{
"val": 0.5,
"hs": {
"hs_1": [
[
-0.134501,
-0.307326,
-0.151994,
-0.065352,
-0.14138
]
],
"hs_2": [
[
-0.134501,
-0.307326,
-0.151994,
-0.065352,
0.020759
]
]
}
}

How to find something in a json file using Bash

I would like to search a JSON file for some key or value, and have it print where it was found.
For example, when using jq to print out my Firefox' extensions.json, I get something like this (using "..." here to skip long parts) :
{
"schemaVersion": 31,
"addons": [
{
"id": "wetransfer#extensions.thunderbird.net",
"syncGUID": "{e6369308-1efc-40fd-aa5f-38da7b20df9b}",
"version": "2.0.0",
...
},
{
...
}
]
}
Say I would like to search for "wetransfer#extensions.thunderbird.net", and would like an output which shows me where it was found with something like this:
{ "addons": [ {"id": "wetransfer#extensions.thunderbird.net"} ] }
Is there a way to get that with jq or with some other json tool?
I also tried to simply list the various ids in that file, and hoped that I would get it with jq '.id', but that just returned null, because it apparently needs the full path.
In other words, I'm looking for a command-line json parser which I could use in a way similar to Xpath tools
The path() function comes in handy:
$ jq -c 'path(.. | select(. == "wetransfer#extensions.thunderbird.net"))' input.json
["addons",0,"id"]
The resulting path is interpreted as "In the addons field of the initial object, the first array element's id field matches". You can use it with getpath(), setpath(), delpaths(), etc. to get or manipulate the value it describes.
Using your example with modifications to make it valid JSON:
< input.json jq -c --arg s wetransfer#extensions.thunderbird.net '
paths as $p | select(getpath($p) == $s) | null | setpath($p;$s)'
produces:
{"addons":[{"id":"wetransfer#extensions.thunderbird.net"}]}
Note
If there are N paths to the given value, the above will produce N lines. If you want only the first, you could wrap everything in first(...).
Listing all the "id" values
I also tried to simply list the various ids in that file
Assuming that "id" values of false and null are of no interest, you can print all the "id" values of interest using the jq filter:
.. | .id? // empty

Search and extract value using JQ command line processor

I have a JSON file very similar to the following:
[
{
"uuid": "832390ed-58ed-4338-bf97-eb42f123d9f3",
"name": "Nacho"
},
{
"uuid": "5b55ea5e-96f4-48d3-a258-75e152d8236a",
"name": "Taco"
},
{
"uuid": "a68f5249-828c-4265-9317-fc902b0d65b9",
"name": "Burrito"
}
]
I am trying to figure out how to use the JQ command line processor to first find the UUID that I input and based on that output the name of the associated item. So for example, if I input UUID a68f5249-828c-4265-9317-fc902b0d65b9 it should search the JSON file, find the matching UUID and then return the name Burrito. I am doing this in Bash. I realize it may require some outside logic in addition to JQ. I will keep thinking about it and put an update here in a bit. I know I could do it in an overly complicated way, but I know there is probably a really simple JQ method of doing this in one or two lines. Please help me.
https://shapeshed.com/jq-json/#how-to-find-a-key-and-value
You can use select:
jq -r --arg query Burrito '.[] | select( .name == $query ) | .uuid ' tst.json