How to parse this file with jq? - json

I just started using jq and json files, and I'm trying to parse a specific file.
I'm tring to do it with jq in command line, but if there's any other way to do it properly, I'm in to give it a try.
The file itself looks like this :
{
"Status": "ok",
"Code": 200,
"Message": "",
"Result": [
{
"ID": 123456,
"Activity": 27,
"Name": Example1",
"Coordinate": {
"Galaxy": 1,
"System": 22,
"Position": 3
},
"Administrator": false,
"Inactive": false,
"Vacation": false,
"HonorableTarget": false,
"Debris": {
"Metal": 0,
"Crystal": 0,
"RecyclersNeeded": 0
},
"Moon": null,
"Player": {
"ID": 111111,
"Name": "foo",
"Rank": 4
},
"Alliance": null
},
{
"ID": 223344,
"Activity": 17,
"Name": "Example2",
"Coordinate": {
"Galaxy": 3,
"System": 44,
"Position": 5
},
"Administrator": false,
"Inactive": false,
"Vacation": false,
"StrongPlayer": false,
"HonorableTarget": false,
"Debris": {
"Metal": 0,
"Crystal": 0,
"RecyclersNeeded": 0
},
"Moon": null,
"Player": {
"ID": 765432,
"Name": "Player 2",
"Rank": 3
},
"Alliance": null
},
(...)
]
}
I would need to extract information based on the galaxy/system/position.
For example, having a script with the proper filters in it and execute something like that :
./parser --galaxy=1 --system=22 --position=3
And it would give me :
ID : 123456
Name : Example1
Activity : 27
...
I tried to do that with curl to grab my json file and jq to parse my file, but I have no idea how I can make that kind of request.

The following should be sufficient to get you on your way.
First, let's assume the JSON is in a file name galaxy.json; second, let's assume the file galaxy.jq contains the following:
.Result[]
| select(.Coordinate | (.Galaxy==$galaxy and .System==$system and .Position==$position))
Then the invocation:
jq -f so-galaxy.jq --argjson galaxy 1 --argjson system 22 --argjson position 3 galaxy.json
would yield the corresponding object:
{
"ID": 123456,
"Activity": 27,
"Name": "Example1",
"Coordinate": {
"Galaxy": 1,
"System": 22,
"Position": 3
},
"Administrator": false,
"Inactive": false,
"Vacation": false,
"HonorableTarget": false,
"Debris": {
"Metal": 0,
"Crystal": 0,
"RecyclersNeeded": 0
},
"Moon": null,
"Player": {
"ID": 111111,
"Name": "foo",
"Rank": 4
},
"Alliance": null
}
Key: Value format
If you want the output to be in key: value format, simply add -r to the command-line options, and append the following to the jq filter:
| to_entries[]
| "\(.key): \(.value)"
Output
ID: 123456
Activity: 27
Name: Example1
Coordinate: {"Galaxy":1,"System":22,"Position":3}
Administrator: false
Inactive: false
Vacation: false
HonorableTarget: false
Debris: {"Metal":0,"Crystal":0,"RecyclersNeeded":0}
Moon: null
Player: {"ID":111111,"Name":"foo","Rank":4}
Alliance: null

Related

How to filter an array of json with jq in linux?

I have the following JSON input:
{
"paging": {
"count": 0,
"total": 0,
"offset": 0,
"max": 0
},
"executions": [
{
"id": 5,
"href": "https://localhost.com.br",
"permalink": "https://localhost.com.br",
"status": "succeeded",
"project": "PROJETO",
"executionType": "scheduled",
"date-started": {
"unixtime": 1660793400012,
"date": "2022-08-18T03:30:00Z"
},
"date-ended": {
"unixtime": 1660793409694,
"date": "2022-08-18T03:30:09Z"
},
"job": {
"id": "cdkwednweoi-8745bjdf-kcjkjr8745",
"averageDuration": 0,
"name": "routine",
"group": "",
"project": "PROJECT",
"description": "",
"href": "https://localhost.com.br",
"permalink": "https://localhost.com.br"
},
"description": "runner",
"argstring": null,
"serverUUID": "jdnsdnasldnaje382nf5ubv",
"successfulNodes": [
"84jsk937nf"
]
}
]
}
First I want to select an array by a property name. And then I want to select an object of the array by the value of the propertyes.
Example of the desired informations on output:
"href"
"status"
"project"
"date-started":
"unixtime": 48298437239847,
"date": "2022-07-17"
"date-ended":
"unixtime": 48298437239847,
"date": "2022-07-17"
"job":
"name": "cleaner"
I knew how to get the firts values:
jq -r '.executions[] | [.href, .status, .project']
But the other ones I don't know how to do, I've tried with:
jq '.executions[] | with_entries( select(.value | has("date-started") ) )'
But it doesn't works.
Your first query produces a JSON array, so in this response, I'll assume it will suffice to produce an array of the eight values of interest in the order you've specified.
With your input, the following invocation produces the eight values as shown below:
jq '.executions[]
| [.href, .status, .project,
(."date-started" | (.unixtime, .date)),
(."date-ended" | (.unixtime, .date)),
.job.name]'
Output:
[
"https://localhost.com.br/rundeck/api/40/execution/2340",
"succeeded",
"PROJETO",
1660793400012,
"2022-08-18T03:30:00Z",
1660793409694,
"2022-08-18T03:30:09Z",
"proc_limpeza_saft"
]

cant use fo query json because of \" in variables

I have this result from ansible:
{
"count": 745,
"next": "/api/v2/inventories/xxxx/hosts/?page=160",
"previous": "/api/v2/inventories/xxxx/hosts/?page=14",
"results": [
{
"id": 3932944,
"type": "host",
"url": "/api/v2/hosts/39329/",
"created": "2021-02-16T20:21:58.104406Z",
"modified": "2021-03-16T23:59:28.688226Z",
"name": "machine1",
"description": "imported",
"enabled": true,
"instance_id": "",
"variables": "{\"VirtualMachine\": \"100\", \"CloneFrom\": \"mrzzzzz\", \"BuildingMachine\": \"*\", , \"__DailyCost\": \"{\\\"type\\\":\\\"TimeRate\\\",\\\"cost\\\":{\\\"type\\\":\\\"money\\\",\\\"currencyCode\\\":\\\"dz\\\",\\\"amount\\\":1.274599998125},\\\"basis\\\":{\\\"type\\\":\\\"timeSpan\\\",\\\"unit\\\":\\\"DAYS\\\",\\\"amount\\\":1}}\", \"virtualMachineState\": \"On\", \"vmDescription\": \"testj\"}",
"has_active_failures": false,
"has_inventory_sources": true,
"last_job": 862532,
"last_job_host_summary": 6309369,
},
{
"id": 3932945,
"type": "host",
"url": "/api/v2/hosts/39329/",
"created": "2021-02-16T20:21:58.104406Z",
"modified": "2021-03-16T23:59:28.688226Z",
"name": "machine2",
"description": "imported",
"enabled": true,
"instance_id": "",
"variables": "{\"VirtualMachine\": \"100\", \"CloneFrom\": \"mrzzzzz\", \"BuildingMachine\": \"*\", , \"__DailyCost\": \"{\\\"type\\\":\\\"TimeRate\\\",\\\"cost\\\":{\\\"type\\\":\\\"money\\\",\\\"currencyCode\\\":\\\"dz\\\",\\\"amount\\\":1.274599998125},\\\"basis\\\":{\\\"type\\\":\\\"timeSpan\\\",\\\"unit\\\":\\\"DAYS\\\",\\\"amount\\\":1}}\", \"virtualMachineState\": \"On\", \"vmDescription\": \"testj\"}",
"has_active_failures": false,
"has_inventory_sources": true,
"last_job": 862532,
"last_job_host_summary": 6309369,
},
]
}
I can get the value of results.name when use json_query, but I can't get value of results.varaibles.VirtualMachine for example using json,query because of '"' , so How can I format variables to be a json format , In other world I want someting like :
...
"variables": {
"VirtualMachine": "100",
"CloneFrom": "mrzzzzz",
"BuildingMachine": "*",
"DailyCost": "{"type":"TimeRate","cost":
{"type":"money","currencyCode":"dz","amount":1.274599998125},
"basis":{"type":"timeSpan","unit":"DAYS","amount":1}}",
"virtualMachineState": "On",
"vmDescription": "testj"
}
...
thank for your help at advance.
Update:
I used this Idea posted by #mdaniel (thanks a lot):
{{whatever.results | map(attribute='variables') | map('from_json') | list}}
Iget output that I need, but when I tried to filter like that:
- name: get id, name, hq_os, hq_site and hq_zone
debug:
var: dictresult_format | json_query(jmesquery)
vars:
jmesquery: "[?VirtualMachine: `100`].{From: CloneFrom, description: vmDescription}"
it output nothing :
TASK [task1t : get data from variables*********
Wednesday 17 March 2021 11:28:53 -0400 (0:00:00.049) 0:00:06.842 *******
ok: [hosttower] => {
"dictresult_format | json_query(jmesquery)": ""
}
Can I missed somting, because that "{{whatever.results | map(attribute='variables') | map('from_json') | list}}" return a list not a json?

How to find and replace json with shell variables using jq?

I have read properties with jq from a json object and have stored them to variables.
I want to now read these variables and essentially find and replace a word inside the string with a global shell variable.
I've set my json ID's from my JSON file
# Set Json ID's
TARGET_ID=$(jq '.DefaultCacheBehavior.TargetOriginId' distconfig.json)
DOMAIN_NAME=$(jq '.Origins.Items[0].DomainName' distconfig.json)
ORIGIN_ID=$(jq '.Origins.Items[0].Id' distconfig.json)
echo "$TARGET_ID"
echo "$DOMAIN_NAME"
echo "$ORIGIN_ID"
This returns
"S3-Website-stag4.example.io.s3-website.us-east-2.amazonaws.com"
"stag4.example.io.s3-website.us-east-2.amazonaws.com"
"S3-Website-stag4.example.io.s3-website.us-east-2.amazonaws.com"
I have my location id variable and would like to write it to find and replace all stag4 references in those 3 ID's.
Then I would like to write those 3 ID's to the initial json object, or create a temp version of it.
Example, if:
$DOMAIN_NAME is"stag4.example.io.s3-website.us-east-2.amazonaws.com"
I would like to essentially have it set to:
$LOCATION_NAME="stag6"
DOMAIN_LOCATION="example.io"
"$DOMAIN_NAME=S3-Website-\$LOCATION_NAME\.example.io.s3-website.us-east-2.amazonaws.com"
"$TARGET_ID=\$LOCATION_NAME\.example.io.s3-website.us-east-2.amazonaws.com"
"$ORIGIN_ID=S3-Website-\$LOCATION_NAME\.example.io.s3-website.us-east-2.amazonaws.com"
Then write those 3 to the temp or new json file so I can run my cloudformation command:
aws cloudfront create-distribution --distribution-config file://disttemp.json
I have now built out the proper variables from the initial json file like so:
$LOCATION_NAME="stag6"
DOMAIN_LOCATION="example.io"
echo "Build New IDs"
TARGET_ID_BUILT="S3-Website-$LOCATION_NAME.$DOMAIN_LOCATION.s3-website.us-east-2.amazonaws.com"
DOMAIN_NAME_BUILT="$LOCATION_NAME.$DOMAIN_LOCATION.s3-website.us-east-2.amazonaws.com"
ORIGIN_ID_BUILT="S3-Website-$LOCATION_NAME.$DOMAIN_LOCATION.s3-website.us-east-2.amazonaws.com"
echo "$TARGET_ID_BUILT"
echo "$DOMAIN_NAME_BUILT"
echo "$ORIGIN_ID_BUILT"
How do I write these variables to the json file with jq?
EDIT: Sample of distconfig.json requested – domain/creds swapped to example
{
"CallerReference": "my-test-distribution-2",
"Comment": "",
"CacheBehaviors": {
"Quantity": 0
},
"IsIPV6Enabled": true,
"Logging": {
"Bucket": "",
"Prefix": "",
"Enabled": false,
"IncludeCookies": false
},
"WebACLId": "",
"Origins": {
"Items": [
{
"OriginPath": "",
"CustomOriginConfig": {
"OriginSslProtocols": {
"Items": [
"TLSv1",
"TLSv1.1",
"TLSv1.2"
],
"Quantity": 3
},
"OriginProtocolPolicy": "http-only",
"OriginReadTimeout": 30,
"HTTPPort": 80,
"HTTPSPort": 443,
"OriginKeepaliveTimeout": 5
},
"CustomHeaders": {
"Quantity": 0
},
"Id": "S3-Website-stag4.example.io.s3-website.us-east-2.amazonaws.com",
"DomainName": "stag4.example.io.s3-website.us-east-2.amazonaws.com"
}
],
"Quantity": 1
},
}
"DefaultRootObject": "",
"PriceClass": "PriceClass_All",
"Enabled": true,
"DefaultCacheBehavior": {
"TrustedSigners": {
"Enabled": false,
"Quantity": 0
},
"LambdaFunctionAssociations": {
"Quantity": 0
},
"TargetOriginId": "S3-Website-stag4.example.io.s3-website.us-east-2.amazonaws.com",
"ViewerProtocolPolicy": "redirect-to-https",
"ForwardedValues": {
"Headers": {
"Quantity": 0
},
"Cookies": {
"Forward": "none"
},
"QueryStringCacheKeys": {
"Quantity": 0
},
"QueryString": false
},
"MaxTTL": 31536000,
"SmoothStreaming": false,
"DefaultTTL": 86400,
"AllowedMethods": {
"Items": [
"HEAD",
"GET"
],
"CachedMethods": {
"Items": [
"HEAD",
"GET"
],
"Quantity": 2
},
"Quantity": 2
},
"MinTTL": 0,
"Compress": true
},
"ViewerCertificate": {
"SSLSupportMethod": "sni-only",
"ACMCertificateArn": "xxxx",
"MinimumProtocolVersion": "TLSv1.1_2016",
"Certificate": "xxxx",
"CertificateSource": "acm"
},
"CustomErrorResponses": {
"Quantity": 0
},
"HttpVersion": "http2",
"Restrictions": {
"GeoRestriction": {
"RestrictionType": "none",
"Quantity": 0
}
},
"Aliases": {
"Quantity": 0
}
}
You should use sed to do the substitution and then inject the value back into the JSON.
echo $TARGET_ID | sed 's/stag4/stag5/g'
Outputs
S3-Website-stag5.example.io.s3-website.us-east-2.amazonaws.com
Next we'll put the value back into the original JSON, this will technically output a new JSON and does not edit the file, however, you can easily solve for this on the output by temporarily saving to a tmp file.
We will use the --arg flag to reference our bash variable and set the new value for our field
cat distconfig.json | jq --arg  TARGET_ID $TARGET_ID '.DefaultCacheBehavior.TargetOriginId = $TARGET_ID' > tmp.json && mv tmp.json distconfig.json

Ruby API call to get data from complex json

I'm making an API GET call using Ruby - the call is made to a Learning Management System and returns the following JSON:
{
"id": 12345,
"body": null,
"url": null,
"grade": "75",
"score": 75,
"submitted_at": "2020-05-02T11:30:53Z",
"assignment_id": 9876,
"user_id": 1111,
"submission_type": "online_upload",
"workflow_state": "graded",
"grade_matches_current_submission": true,
"graded_at": "2017-06-05T08:47:49Z",
"grader_id": 2222,
"attempt": 1,
"cached_due_date": "2020-05-03T15:00:00Z",
"excused": false,
"late_policy_status": null,
"points_deducted": null,
"grading_period_id": null,
"late": false,
"missing": false,
"seconds_late": 0,
"entered_grade": "75",
"entered_score": 75,
"preview_url": "https://etcetc",
"turnitin_data": {
"attachment_33333": {
"status": "scored",
"object_id": "44444444",
"similarity_score": 0,
"web_overlap": 0,
"publication_overlap": 0,
"student_overlap": 0,
"state": "none"
}
},
"attachments": [
{
"id": 33333,
"uuid": "kjsdkjhsdfkhsfd",
"folder_id": 55555,
"display_name": "Submission.pdf",
"filename": "Submission.pdf",
"content-type": "application/pdf",
"url": "https://etcetc",
"size": 2668226,
"created_at": "2020-05-02T11:30:51Z",
"updated_at": "2020-06-06T15:01:46Z",
"unlock_at": null,
"locked": false,
"hidden": false,
"lock_at": null,
"hidden_for_user": false,
"thumbnail_url": null,
"modified_at": "2020-05-02T11:30:51Z",
"mime_class": "pdf",
"media_entry_id": null,
"locked_for_user": false,
"preview_url": "api/etcetc"
}
],
"submission_comments": [
{
"id": 99999,
"comment": "here’s a comment",
"author_id": 1,
"author_name": "Mickey Mouse",
"created_at": "2020-05-15T12:54:08Z",
"edited_at": null,
"avatar_path": "/images/users/1",
"author": {
"id": 1,
"display_name": " Mickey Mouse ",
"avatar_image_url": "https://etcetc",
"html_url": "https://etcetc"
}
},
{
"id": 223344,
"comment": "another comment",
"author_id": 2,
"author_name": "Donald Duck",
"created_at": "2020-06-05T10:48:51Z",
"edited_at": null,
"avatar_path": "/images/users/2",
"author": {
"id": 2,
"display_name": "Donald Duck",
"avatar_image_url": "https://etcetc",
"html_url": "https://etcetc"
}
}
]
}
I need to be able to retrieve specific values from "submission_comments", namely the values for "comment", "author_id" and "author_name". At the moment the best I can do is retrieve "submission_comments" as one big entity. Here's how I'm getting that far:
require 'typhoeus'
require 'link_header'
require 'json'
require 'csv'
the_url = 'https://etctetc'
token = 'mytoken'
api_endpoint = '/api/etc'
output_csv = 'C:\Users\me\Desktop\Ruby Canvas course\assignment_comments.csv'
CSV.open(output_csv, 'wb') do |csv|
csv << ["user_id", "TII", "marker"]
end
request_url = "#{the_url}#{api_endpoint}"
count = 0
more_data = true
while more_data
get_comments = Typhoeus::Request.new(
request_url,
method: :get,
headers: { authorization: "Bearer #{token}" }
)
get_comments.on_complete do |response|
#get next link
links = LinkHeader.parse(response.headers['link']).links
next_link = links.find { |link| link['rel'] == 'next' }
request_url = next_link.href if next_link
if next_link && "#{response.body}" != "[]"
more_data = true
else
more_data = false
end
if response.code == 200
data = JSON.parse(response.body)
data.each do |comments|
CSV.open(output_csv, 'a') do |csv|
csv << [comments['id'], comments['turnitin_data'], comments['submission_comments']]
end
end
else
puts "Something went wrong! Response code was #{response.code}"
end
end
get_comments.run
end
puts "Script done running"
I'm new to this (the ruby code is based on an exercise so I may not fully understand it)- any help/advice would be really appreciated!
EDIT: I should also note that this isn't the total JSON response I'm dealing with - this is just one of ten items that are returned
"submission_comments": [
{
"id": 99999,
}
]
the [] means it is array. {} means it is an object.
So you probably need to do something like this:
json["submission_comments"].first["id"]
or better iterate through it:
ids = json["submission_comments"].map{|comment| comment["id"]}
I'm able to get the variables you need if you can read the JSON file in as text, then use Ruby's JSON.parse(...) method on it. I think the main problem is that JSON uses null but Ruby hashes use nil. You could do a string replace or try something like this (I did not modify your JSON, only put it into a single quoted string):
json_text = '{
"id": 12345,
"body": null,
"url": null,
"grade": "75",
"score": 75,
"submitted_at": "2020-05-02T11:30:53Z",
"assignment_id": 9876,
"user_id": 1111,
"submission_type": "online_upload",
"workflow_state": "graded",
"grade_matches_current_submission": true,
"graded_at": "2017-06-05T08:47:49Z",
"grader_id": 2222,
"attempt": 1,
"cached_due_date": "2020-05-03T15:00:00Z",
"excused": false,
"late_policy_status": null,
"points_deducted": null,
"grading_period_id": null,
"late": false,
"missing": false,
"seconds_late": 0,
"entered_grade": "75",
"entered_score": 75,
"preview_url": "https://etcetc",
"turnitin_data": {
"attachment_33333": {
"status": "scored",
"object_id": "44444444",
"similarity_score": 0,
"web_overlap": 0,
"publication_overlap": 0,
"student_overlap": 0,
"state": "none"
}
},
"attachments": [
{
"id": 33333,
"uuid": "kjsdkjhsdfkhsfd",
"folder_id": 55555,
"display_name": "Submission.pdf",
"filename": "Submission.pdf",
"content-type": "application/pdf",
"url": "https://etcetc",
"size": 2668226,
"created_at": "2020-05-02T11:30:51Z",
"updated_at": "2020-06-06T15:01:46Z",
"unlock_at": null,
"locked": false,
"hidden": false,
"lock_at": null,
"hidden_for_user": false,
"thumbnail_url": null,
"modified_at": "2020-05-02T11:30:51Z",
"mime_class": "pdf",
"media_entry_id": null,
"locked_for_user": false,
"preview_url": "api/etcetc"
}
],
"submission_comments": [
{
"id": 99999,
"comment": "here’s a comment",
"author_id": 1,
"author_name": "Mickey Mouse",
"created_at": "2020-05-15T12:54:08Z",
"edited_at": null,
"avatar_path": "/images/users/1",
"author": {
"id": 1,
"display_name": " Mickey Mouse ",
"avatar_image_url": "https://etcetc",
"html_url": "https://etcetc"
}
},
{
"id": 223344,
"comment": "another comment",
"author_id": 2,
"author_name": "Donald Duck",
"created_at": "2020-06-05T10:48:51Z",
"edited_at": null,
"avatar_path": "/images/users/2",
"author": {
"id": 2,
"display_name": "Donald Duck",
"avatar_image_url": "https://etcetc",
"html_url": "https://etcetc"
}
}
]
}'
Part I added:
ruby_hash = JSON.parse(json_text)
submission_comments = ruby_hash["submission_comments"]
submission_comments.each do |submission_comment|
comment = submission_comment["comment"]
author_id = submission_comment["author_id"]
author_name = submission_comment["author_name"]
puts "Comment: #{comment}, Author ID: #{author_id}, Author Name: #{author_name}\n\n"
end
Terminal Result:
=> Comment: here’s a comment, Author ID: 1, Author Name: Mickey Mouse
=> Comment: another comment, Author ID: 2, Author Name: Donald Duck
Edit: I added a jenky af one-liner version just for fun (presuming the json_text variable above is already initialized)
JSON.parse(json_text)["submission_comments"]
.map{|txt| puts(["comment","author_id","author_name"]
.map{|k| k.instance_eval{"#{upcase}: #{txt[to_s]}"}}.join(', '))}
COMMENT: here’s a comment, AUTHOR_ID: 1, AUTHOR_NAME: Mickey Mouse
COMMENT: another comment, AUTHOR_ID: 2, AUTHOR_NAME: Donald Duck

parse json output for primary and secondary hosts from replSetGetStatus

I've used pymongo to connect to mongo replica set and print the status of replica set using json dump. I want to parse this output and display "name" and "stateStr" into a list or array for the user to be able to pick a particular host.Here is my json dump output:
{
{
"replSetGetStatus": {
"date": "2016-10-07T14:21:25",
"members": [
{
"_id": 0,
"health": 1.0,
"name": "xxxxxxxxxxx:27017",
"optime": null,
"optimeDate": "2016-10-07T13:50:11",
"self": true,
"state": 1,
"stateStr": "PRIMARY",
"uptime": 32521
},
{
"_id": 1,
"health": 1.0,
"lastHeartbeat": "2016-10-07T14:21:24",
"lastHeartbeatRecv": "2016-10-07T14:21:24",
"name": "xxxxxxxxxxxx:27017",
"optime": null,
"optimeDate": "2016-10-07T13:50:11",
"pingMs": 0,
"state": 2,
"stateStr": "SECONDARY",
"syncingTo": "xxxxxxxxxxxx:27017",
"uptime": 27297
},
{
"_id": 2,
"health": 1.0,
"lastHeartbeat": "2016-10-07T14:21:24",
"lastHeartbeatRecv": "2016-10-07T14:21:24",
"name": "xxxxxxxxxxxxx:27020",
"pingMs": 0,
"state": 7,
"stateStr": "ARBITER",
"uptime": 32517
}
],
"myState": 1,
"ok": 1.0,
"set": "replica1"
}
}
Please try below Javascript code. It worked for me.
use admin;
var result = rs.status();
var length = result.members.length;
for (var i=0;i<length;i++){
print ("Server Name-" +result.members[i].name);
print ("Server State-" +result.members[i].stateStr);
}