cant use fo query json because of \" in variables - json

I have this result from ansible:
{
"count": 745,
"next": "/api/v2/inventories/xxxx/hosts/?page=160",
"previous": "/api/v2/inventories/xxxx/hosts/?page=14",
"results": [
{
"id": 3932944,
"type": "host",
"url": "/api/v2/hosts/39329/",
"created": "2021-02-16T20:21:58.104406Z",
"modified": "2021-03-16T23:59:28.688226Z",
"name": "machine1",
"description": "imported",
"enabled": true,
"instance_id": "",
"variables": "{\"VirtualMachine\": \"100\", \"CloneFrom\": \"mrzzzzz\", \"BuildingMachine\": \"*\", , \"__DailyCost\": \"{\\\"type\\\":\\\"TimeRate\\\",\\\"cost\\\":{\\\"type\\\":\\\"money\\\",\\\"currencyCode\\\":\\\"dz\\\",\\\"amount\\\":1.274599998125},\\\"basis\\\":{\\\"type\\\":\\\"timeSpan\\\",\\\"unit\\\":\\\"DAYS\\\",\\\"amount\\\":1}}\", \"virtualMachineState\": \"On\", \"vmDescription\": \"testj\"}",
"has_active_failures": false,
"has_inventory_sources": true,
"last_job": 862532,
"last_job_host_summary": 6309369,
},
{
"id": 3932945,
"type": "host",
"url": "/api/v2/hosts/39329/",
"created": "2021-02-16T20:21:58.104406Z",
"modified": "2021-03-16T23:59:28.688226Z",
"name": "machine2",
"description": "imported",
"enabled": true,
"instance_id": "",
"variables": "{\"VirtualMachine\": \"100\", \"CloneFrom\": \"mrzzzzz\", \"BuildingMachine\": \"*\", , \"__DailyCost\": \"{\\\"type\\\":\\\"TimeRate\\\",\\\"cost\\\":{\\\"type\\\":\\\"money\\\",\\\"currencyCode\\\":\\\"dz\\\",\\\"amount\\\":1.274599998125},\\\"basis\\\":{\\\"type\\\":\\\"timeSpan\\\",\\\"unit\\\":\\\"DAYS\\\",\\\"amount\\\":1}}\", \"virtualMachineState\": \"On\", \"vmDescription\": \"testj\"}",
"has_active_failures": false,
"has_inventory_sources": true,
"last_job": 862532,
"last_job_host_summary": 6309369,
},
]
}
I can get the value of results.name when use json_query, but I can't get value of results.varaibles.VirtualMachine for example using json,query because of '"' , so How can I format variables to be a json format , In other world I want someting like :
...
"variables": {
"VirtualMachine": "100",
"CloneFrom": "mrzzzzz",
"BuildingMachine": "*",
"DailyCost": "{"type":"TimeRate","cost":
{"type":"money","currencyCode":"dz","amount":1.274599998125},
"basis":{"type":"timeSpan","unit":"DAYS","amount":1}}",
"virtualMachineState": "On",
"vmDescription": "testj"
}
...
thank for your help at advance.
Update:
I used this Idea posted by #mdaniel (thanks a lot):
{{whatever.results | map(attribute='variables') | map('from_json') | list}}
Iget output that I need, but when I tried to filter like that:
- name: get id, name, hq_os, hq_site and hq_zone
debug:
var: dictresult_format | json_query(jmesquery)
vars:
jmesquery: "[?VirtualMachine: `100`].{From: CloneFrom, description: vmDescription}"
it output nothing :
TASK [task1t : get data from variables*********
Wednesday 17 March 2021 11:28:53 -0400 (0:00:00.049) 0:00:06.842 *******
ok: [hosttower] => {
"dictresult_format | json_query(jmesquery)": ""
}
Can I missed somting, because that "{{whatever.results | map(attribute='variables') | map('from_json') | list}}" return a list not a json?

Related

How to filter an array of json with jq in linux?

I have the following JSON input:
{
"paging": {
"count": 0,
"total": 0,
"offset": 0,
"max": 0
},
"executions": [
{
"id": 5,
"href": "https://localhost.com.br",
"permalink": "https://localhost.com.br",
"status": "succeeded",
"project": "PROJETO",
"executionType": "scheduled",
"date-started": {
"unixtime": 1660793400012,
"date": "2022-08-18T03:30:00Z"
},
"date-ended": {
"unixtime": 1660793409694,
"date": "2022-08-18T03:30:09Z"
},
"job": {
"id": "cdkwednweoi-8745bjdf-kcjkjr8745",
"averageDuration": 0,
"name": "routine",
"group": "",
"project": "PROJECT",
"description": "",
"href": "https://localhost.com.br",
"permalink": "https://localhost.com.br"
},
"description": "runner",
"argstring": null,
"serverUUID": "jdnsdnasldnaje382nf5ubv",
"successfulNodes": [
"84jsk937nf"
]
}
]
}
First I want to select an array by a property name. And then I want to select an object of the array by the value of the propertyes.
Example of the desired informations on output:
"href"
"status"
"project"
"date-started":
"unixtime": 48298437239847,
"date": "2022-07-17"
"date-ended":
"unixtime": 48298437239847,
"date": "2022-07-17"
"job":
"name": "cleaner"
I knew how to get the firts values:
jq -r '.executions[] | [.href, .status, .project']
But the other ones I don't know how to do, I've tried with:
jq '.executions[] | with_entries( select(.value | has("date-started") ) )'
But it doesn't works.
Your first query produces a JSON array, so in this response, I'll assume it will suffice to produce an array of the eight values of interest in the order you've specified.
With your input, the following invocation produces the eight values as shown below:
jq '.executions[]
| [.href, .status, .project,
(."date-started" | (.unixtime, .date)),
(."date-ended" | (.unixtime, .date)),
.job.name]'
Output:
[
"https://localhost.com.br/rundeck/api/40/execution/2340",
"succeeded",
"PROJETO",
1660793400012,
"2022-08-18T03:30:00Z",
1660793409694,
"2022-08-18T03:30:09Z",
"proc_limpeza_saft"
]

How i can get needest line " DBInstanceIdentifier" from Json output with jq or other tools

this is Json: I try to use
jq -r '.[] | .DBSnapshotIdentifier'
But it doesn't work for me
{
"DBSnapshots": [
{
"DBSnapshotIdentifier": "auto-lims-final-snapshot",
"DBInstanceIdentifier": "auto-lims",
"SnapshotCreateTime": "2018-08-15T09:59:23.332000+00:00",
"Engine": "postgres",
"AllocatedStorage": 50,
"Status": "available",
"Port": 5432,
"AvailabilityZone": "us-east-2a",
"VpcId": "vpc-e799fc8f",
"InstanceCreateTime": "2018-04-09T08:28:03.565000+00:00",
"MasterUsername": "postgres",
"EngineVersion": "9.6.6",
"LicenseModel": "postgresql-license",
"SnapshotType": "manual",
"OptionGroupName": "default:postgres-9-6",
"PercentProgress": 100,
"StorageType": "gp2",
"Encrypted": false,
"DBSnapshotArn": "arn:aws:rds:us-east-2:833682533595:snapshot:auto-lims-final-snapshot",
"IAMDatabaseAuthenticationEnabled": false,
"ProcessorFeatures": [],
"DbiResourceId": "db-ZL5T2TA2PJVG6CVOJRO7HUOAXQ",
"TagList": [
{
"Key": "Environment",
"Value": "auto"
},
{
"Key": "Application",
"Value": "LIMS"
}
],
"SnapshotTarget": "region"
},
{
"DBSnapshotIdentifier": "automation-lims-before-postgres-12-5",
"DBInstanceIdentifier": "automation-lims",
"Engine": "postgres",
"AllocatedStorage": 500,
"Status": "available",
"Port": 5432,
"AvailabilityZone": "us-east-2b",
"VpcId": "vpc-09fa88d2884ee2083",
"InstanceCreateTime": "2019-12-26T11:19:41.947000+00:00",
"MasterUsername": "lims",
"EngineVersion": "9.6.20",
"LicenseModel": "postgresql-license",
"SnapshotType": "manual",
"OptionGroupName": "default:postgres-9-6",
"PercentProgress": 100
}
]
}
How I can get only nested lines.
You first need to select the DBSnapshots object, try:
.DBSnapshots[] | .DBSnapshotIdentifier
This will output:
"auto-lims-final-snapshot"
"automation-lims-before-postgres-12-5"
As you can try in this online demo
To include the .TagList.Key we can add the following:
.DBSnapshots[] | .DBSnapshotIdentifier, .TagList[]?.Key
This will now output
"auto-lims-final-snapshot"
"Environment"
"Application"
"automation-lims-before-postgres-12-5"
As you can try in this online demo
If you'd like the DBSnapshotIdentifier to be on the same line as the TagList we can use string interpolation like so:
.DBSnapshots[] | "\(.DBSnapshotIdentifier) - \(.TagList[]?.Key)"
Thiis will output:
"auto-lims-final-snapshot - Environment"
"auto-lims-final-snapshot - Application"
As this demo demostrates
The .[]? from the above command mean:
Like .[], but no errors will be output if . is not an array or object.
More information in JQ's documentation

Using jq to fetch and show key value with quotes

I have a file that looks as below:
{
"Job": {
"Name": "sample_job",
"Description": "",
"Role": "arn:aws:iam::00000000000:role/sample_role",
"CreatedOn": "2021-10-21T23:35:23.660000-03:00",
"LastModifiedOn": "2021-10-21T23:45:41.771000-03:00",
"ExecutionProperty": {
"MaxConcurrentRuns": 1
},
"Command": {
"Name": "glueetl",
"ScriptLocation": "s3://aws-sample-s3/scripts/sample.py",
"PythonVersion": "3"
},
"DefaultArguments": {
"--TempDir": "s3://aws-sample-s3/temporary/",
"--class": "GlueApp",
"--enable-continuous-cloudwatch-log": "true",
"--enable-glue-datacatalog": "true",
"--enable-metrics": "true",
"--enable-spark-ui": "true",
"--job-bookmark-option": "job-bookmark-enable",
"--job-insights-byo-rules": "",
"--job-language": "python",
"--spark-event-logs-path": "s3://aws-sample-s3/logs"
},
"MaxRetries": 0,
"AllocatedCapacity": 100,
"Timeout": 2880,
"MaxCapacity": 100.0,
"WorkerType": "G.1X",
"NumberOfWorkers": 100,
"GlueVersion": "2.0"
}
}
I want to get key/value from "Name", "--enable-continuous-cloudwatch-log": "" and "--enable-metrics": "". So, I need to show the info like this:
"Name" "sample_job"
"--enable-continuous-cloudwatch-log" ""
"--enable-metrics" ""
UPDATE
Follow the tips from #Inian and #0stone0 I came close to it:
jq -r '(.Job ) + (.Job.DefaultArguments | { "--enable-continuous-cloudwatch-log", "--enable-metrics"}) | to_entries[] | "\"\(.key)\" \"\(.value)\""'
This extract the values I need but show all another key/values.
Since you're JSON isn't valid, I've converted it into:
{
"Job": {
"Name": "sample_job",
"Role": "sample_role_job"
},
"DefaultArguments": {
"--enable-continuous-cloudwatch-log": "test_1",
"--enable-metrics": ""
},
"Timeout": 2880,
"NumberOfWorkers": 10
}
Using the following filter:
"Name \(.Job.Name)\n--enable-continuous-cloudwatch-log \(.DefaultArguments."--enable-continuous-cloudwatch-log")\n--enable-metrics \(.DefaultArguments."--enable-metrics")"
We use string interpolation to show the desired output:
Name sample_job
--enable-continuous-cloudwatch-log test_1
--enable-metrics
jq --raw-output '"Name \(.Job.Name)\n--enable-continuous-cloudwatch-log \(.DefaultArguments."--enable-continuous-cloudwatch-log")\n--enable-metrics \(.DefaultArguments."--enable-metrics")"'
Online Demo

parse json response using ansible

I need to parse the json response from an API request using ansible and then take action on items returned in the response. Here is part of my ansible playbook -
```- name: Invoke Import API
uri:
url: "{{ atl_bitbucket_dataset_url }}/rest/api/1.0/migration/imports"
user: admin
password: "{{ atl_bitbucket_admin_password }}"
method: POST
follow_redirects: yes
force_basic_auth: yes
creates: "{{ atl_product_home_shared }}/data/migration/import/lock.file"
body: "{ \"archivePath\": \"{{ atl_bitbucket_dataset_url | basename }}\" }"
body_format: json
return_content: yes
register: response
until: response.status == 200
retries: 6
delay: 15
failed_when: response.response.json.state != 'INITIALISING'
- name: get status of import
debug: var=response```
and here is the json response i got on a previous run
```TASK [bitbucket_dataset_restore : get status of import] ************************
ok: [localhost] => {
"response": {
"attempts": 1,
"cache_control": "no-cache, no-transform",
"changed": false,
"connection": "close",
"content": "{\"id\":1,\"initiator\":{\"name\":\"admin\",\"emailAddress\":\"admin#yourcompany.com\",\"id\":1,\"displayName\":\"AdminIstrator\",\"active\":true,\"slug\":\"admin\",\"type\":\"NORMAL\",\"links\":{\"self\":[{\"href\":\"http://bbdc-test-loadbala-t6vnlr2363vl-1404860112.us-west-2.elb.amazonaws.com/users/admin\"}]}},\"nodeId\":\"e72aa995-5016-4b2c-80e8-edba5eda3ab4\",\"progress\":{\"percentage\":0},\"startDate\":1574803309090,\"state\":\"INITIALISING\",\"type\":\"com.atlassian.bitbucket.migration.import\",\"updateDate\":1574803309090}",
"content_type": "application/json;charset=UTF-8",
"cookies": {},
"cookies_string": "",
"date": "Tue, 26 Nov 2019 21:21:49 GMT",
"elapsed": 1,
"failed": false,
"failed_when_result": False,
"json": {
"id": 1,
"initiator": {
"active": True,
"displayName": "AdminIstrator",
"emailAddress": "admin#yourcompany.com",
"id": 1,
"links": {
"self": [
{
"href": "http://bbdc-test-loadbala-t6vnlr2363vl-1404860112.us-west-2.elb.amazonaws.com/users/admin"
}
]
},
"name": "admin",
"slug": "admin",
"type": "NORMAL"
},
"nodeId": "e72aa995-5016-4b2c-80e8-edba5eda3ab4",
"progress": {
"percentage": 0
},
"startDate": 1574803309090,
"state": "INITIALISING",
"type": "com.atlassian.bitbucket.migration.import",
"updateDate": 1574803309090
},
"msg": "OK (unknown bytes)",
"redirected": False,
"status": 200,
"transfer_encoding": "chunked",
"url": "http://localhost:7990/rest/api/1.0/migration/imports",
"vary": "accept-encoding,x-auserid,cookie,x-ausername,accept-encoding",
"warnings": [
"The value True (type bool) in a string field was converted to u'True' (type string). If this does not look like what you expect, quote the entire value to ensure it does not change."
],
"x_arequestid": "#XXAH7Kx1281x1x0",
"x_asen": "SEN-500",
"x_auserid": "1",
"x_ausername": "admin",
"x_content_type_options": "nosniff"
}
}```
I want to retrieve the value "state": "INITIALISING" and the "id": 1. I have tried various methods response.response.json.state or response['response']['json']['state']. And it has not worked. Can anyone help me on how i can retrieve those values so i can use in other tasks further down?
I was able to fix this by adding is defined. So my code looks like this -
failed_when: output is defined and output.json is defined and output.json.state != 'INITIALISING'

How to parse this file with jq?

I just started using jq and json files, and I'm trying to parse a specific file.
I'm tring to do it with jq in command line, but if there's any other way to do it properly, I'm in to give it a try.
The file itself looks like this :
{
"Status": "ok",
"Code": 200,
"Message": "",
"Result": [
{
"ID": 123456,
"Activity": 27,
"Name": Example1",
"Coordinate": {
"Galaxy": 1,
"System": 22,
"Position": 3
},
"Administrator": false,
"Inactive": false,
"Vacation": false,
"HonorableTarget": false,
"Debris": {
"Metal": 0,
"Crystal": 0,
"RecyclersNeeded": 0
},
"Moon": null,
"Player": {
"ID": 111111,
"Name": "foo",
"Rank": 4
},
"Alliance": null
},
{
"ID": 223344,
"Activity": 17,
"Name": "Example2",
"Coordinate": {
"Galaxy": 3,
"System": 44,
"Position": 5
},
"Administrator": false,
"Inactive": false,
"Vacation": false,
"StrongPlayer": false,
"HonorableTarget": false,
"Debris": {
"Metal": 0,
"Crystal": 0,
"RecyclersNeeded": 0
},
"Moon": null,
"Player": {
"ID": 765432,
"Name": "Player 2",
"Rank": 3
},
"Alliance": null
},
(...)
]
}
I would need to extract information based on the galaxy/system/position.
For example, having a script with the proper filters in it and execute something like that :
./parser --galaxy=1 --system=22 --position=3
And it would give me :
ID : 123456
Name : Example1
Activity : 27
...
I tried to do that with curl to grab my json file and jq to parse my file, but I have no idea how I can make that kind of request.
The following should be sufficient to get you on your way.
First, let's assume the JSON is in a file name galaxy.json; second, let's assume the file galaxy.jq contains the following:
.Result[]
| select(.Coordinate | (.Galaxy==$galaxy and .System==$system and .Position==$position))
Then the invocation:
jq -f so-galaxy.jq --argjson galaxy 1 --argjson system 22 --argjson position 3 galaxy.json
would yield the corresponding object:
{
"ID": 123456,
"Activity": 27,
"Name": "Example1",
"Coordinate": {
"Galaxy": 1,
"System": 22,
"Position": 3
},
"Administrator": false,
"Inactive": false,
"Vacation": false,
"HonorableTarget": false,
"Debris": {
"Metal": 0,
"Crystal": 0,
"RecyclersNeeded": 0
},
"Moon": null,
"Player": {
"ID": 111111,
"Name": "foo",
"Rank": 4
},
"Alliance": null
}
Key: Value format
If you want the output to be in key: value format, simply add -r to the command-line options, and append the following to the jq filter:
| to_entries[]
| "\(.key): \(.value)"
Output
ID: 123456
Activity: 27
Name: Example1
Coordinate: {"Galaxy":1,"System":22,"Position":3}
Administrator: false
Inactive: false
Vacation: false
HonorableTarget: false
Debris: {"Metal":0,"Crystal":0,"RecyclersNeeded":0}
Moon: null
Player: {"ID":111111,"Name":"foo","Rank":4}
Alliance: null