Related
need help to parse the JSON data received from Oracle Integration Cloud. The expected output is mentioned below alongwith the command i am trying to use.
JQ command
jq '[{id: .id},{integrations: [.integrations[]|{code: .code, version: .version, dependencies: .dependencies|{connections: .connections[]|{id: .id, status: .status}}, .dependencies|{lookups: .lookups}}]}]' output.json
Error :
jq: error: syntax error, unexpected FIELD (Unix shell quoting issues?) at , line 1:
[{id: .id},{integrations: [.integrations[]|{code: .code, version: .version, dependencies: .dependencies|{connections: .connections[]|{id: .id, status: .status}}, .dependencies|{lookups: .lookups}}]}]
Note : If i run below command to fetch only connections data it works fine
jq '[{id: .id},{integrations: [.integrations[]|{code: .code, version: .version, dependencies: .dependencies|{connections: .connections[]|{id: .id, status: .status}}}]}]' output.json
Expected Output:
[
{
"id": "SAMPLE_PACKAGE"
},
{
"integrations": [
{
"code": "HELLO_INTEGRATION",
"version": "01.00.0000",
"dependencies": {
"connections": {
"id": "HELLO_WORLD1",
"status": "CONFIGURED"
}
}
},
{
"code": "HELLO_INTEGRATIO_LOOKUP",
"version": "01.00.0000",
"dependencies": {
"connections": {
"id": "HELLO_WORLD1",
"status": "CONFIGURED"
},
"lookups": {
"name": "COMMON_LOOKUP_VARIABLES",
"status": "CONFIGURED"
}
}
},
{
"code": "HI_INTEGRATION",
"version": "01.00.0000",
"dependencies": {
"connections": {
"id": "HELLO_WORLD1",
"status": "CONFIGURED"
}
}
}
]
}
]
output.json file contains
{
"bartaType": "DEVELOPED",
"countOfIntegrations": 3,
"id": "SAMPLE_PACKAGE",
"integrations": [
{
"code": "HELLO_INTEGRATION",
"dependencies": {
"connections": [
{
"id": "HELLO_WORLD1",
"lockedFlag": false,
"name": "Hello World1",
"role": "SOURCE",
"status": "CONFIGURED",
"type": "rest",
"usage": 6
}
]
},
"description": "",
"eventSubscriptionFlag": false,
"filmstrip": [
{
"code": "HELLO_WORLD1",
"iconUrl": "/images/rest/rest_icon_46.png",
"name": "Hello World1",
"role": "SOURCE",
"status": "CONFIGURED"
}
],
"id": "HELLO_INTEGRATION|01.00.0000",
"lockedFlag": false,
"name": "HELLO_INTEGRATION",
"pattern": "Orchestration",
"patternDescription": "Map Data",
"payloadTracingEnabledFlag": true,
"publishFlag": false,
"scheduleApplicable": false,
"scheduleDefined": false,
"status": "ACTIVATED",
"style": "FREEFORM",
"styleDescription": "Orchestration",
"tempCopyExists": false,
"tracingEnabledFlag": true,
"version": "01.00.0000",
"warningMsg": "ACTIVATE_PUBLISH_NO_CONN"
},
{
"code": "HELLO_INTEGRATIO_LOOKUP",
"dependencies": {
"connections": [
{
"id": "HELLO_WORLD1",
"lockedFlag": false,
"name": "Hello World1",
"role": "SOURCE",
"status": "CONFIGURED",
"type": "rest",
"usage": 6
}
],
"lookups": [
{
"lockedFlag": false,
"name": "COMMON_LOOKUP_VARIABLES",
"status": "CONFIGURED",
"usage": 1
}
]
},
"description": "",
"eventSubscriptionFlag": false,
"filmstrip": [
{
"code": "HELLO_WORLD1",
"iconUrl": "/images/rest/rest_icon_46.png",
"name": "Hello World1",
"role": "SOURCE",
"status": "CONFIGURED"
}
],
"id": "HELLO_INTEGRATIO_LOOKUP|01.00.0000",
"lockedFlag": false,
"name": "HELLO_INTEGRATION_LOOKUP",
"pattern": "Orchestration",
"patternDescription": "Map Data",
"payloadTracingEnabledFlag": true,
"publishFlag": false,
"scheduleApplicable": false,
"scheduleDefined": false,
"status": "ACTIVATED",
"style": "FREEFORM",
"styleDescription": "Orchestration",
"tempCopyExists": false,
"tracingEnabledFlag": true,
"version": "01.00.0000",
"warningMsg": "ACTIVATE_PUBLISH_NO_CONN"
},
{
"code": "HI_INTEGRATION",
"dependencies": {
"connections": [
{
"id": "HELLO_WORLD1",
"lockedFlag": false,
"name": "Hello World1",
"role": "SOURCE",
"status": "CONFIGURED",
"type": "rest",
"usage": 6
}
]
},
"description": "",
"eventSubscriptionFlag": false,
"filmstrip": [
{
"code": "HELLO_WORLD1",
"iconUrl": "/images/rest/rest_icon_46.png",
"name": "Hello World1",
"role": "SOURCE",
"status": "CONFIGURED"
}
],
"id": "HI_INTEGRATION|01.00.0000",
"lockedFlag": false,
"name": "HI_INTEGRATION",
"pattern": "Orchestration",
"patternDescription": "Map Data",
"payloadTracingEnabledFlag": true,
"publishFlag": false,
"scheduleApplicable": false,
"scheduleDefined": false,
"status": "ACTIVATED",
"style": "FREEFORM",
"styleDescription": "Orchestration",
"tempCopyExists": false,
"tracingEnabledFlag": true,
"version": "01.00.0000",
"warningMsg": "ACTIVATE_PUBLISH_NO_CONN"
}
],
"isCloneAllowed": false,
"isViewAllowed": false,
"name": "SAMPLE_PACKAGE",
"type": "DEVELOPED"
}
The problem is that the lookups key is not always present so, you cannot use the [] on it. So, instead you can use the map function and provide a default before piping to the map function like below
[
{ id: .id },
{
integrations: [
.integrations[]|{
id: .id,
code: .code,
dependencies: {
connections: (.dependencies.connections//[]|map({id,status}))[0],
lookups: (.dependencies.lookups//[]|map({name,status}))[0]
}
}
]
}
]
The (.dependencies.lookups//[]|map({name,status}))[0] has the effect of passing an empty array to the map function which results in a null value when accessing the first element.
See in action https://jqplay.org/s/zQBkHtnzOd1
The provided JQ statement works fine for single elements in the array , but incase the array contains multiple elements it only fetches the first element. Also i updated the dependencies object to capture all the arrays ( connections,lookups,certificates,libraries,integrations)
Below is the modified one. Please suggest for any better options.
[
{ id: .id },
{
integrations: [
.integrations[]|{
id: .id,
code: .code,
dependencies: {
connections: (.dependencies.connections//[]|map({id,status})),
lookups: (.dependencies.lookups//[]|map({name,status})),
certificates: (.dependencies.certificates//[]|map({id,status})),
libraries: (.dependencies.libraries//[]|map({code,status,version})),
integrations: (.dependencies.integrations//[]|map({code,version}))
}
}
]
}
]|del(..|select(.==[]))
Note: To remove the empty arrays del function is added which is giving the below output :
[
{
"id": "SAMPLE_PACKAGE"
},
{
"integrations": [
{
"id": "HELLO_INTEGRATION|01.00.0000",
"code": "HELLO_INTEGRATION",
"dependencies": {
"connections": [
{
"id": "HELLO_WORLD1",
"status": "CONFIGURED"
},
{
"id": "HELLO_WORLD2",
"status": "CONFIGURED"
}
]
}
},
{
"id": "HELLO_INTEGRATIO_LOOKUP|01.00.0000",
"code": "HELLO_INTEGRATIO_LOOKUP",
"dependencies": {
"connections": [
{
"id": "HELLO_WORLD1",
"status": "CONFIGURED"
}
],
"lookups": [
{
"name": "COMMON_LOOKUP_VARIABLES",
"status": "CONFIGURED"
}
]
}
},
{
"id": "HI_INTEGRATION|01.00.0000",
"code": "HI_INTEGRATION",
"dependencies": {
"connections": [
{
"id": "HELLO_WORLD1",
"status": "CONFIGURED"
}
]
}
}
]
}
]
i have a JSON file
js.json
{
"id": "json",
"description": "test",
"packages": [
{
"Group": "group1",
"Name": "name1",
"Version": "1.0.0"
},
{
"Group": "group2",
"Name": "name2",
"Version": "1.0.0"
},
{
"Group": "group3",
"Name": "name3",
"Version": "1.0.0"
},
{
"Group": "group5",
"Name": "name5",
"Version": "1.0.0"
}
]
}
It has 4 elements in the .packages array. I want to add a fifth element "group4" to the array to get
{
"id": "json",
"description": "test",
"packages": [
{
"Group": "group1",
"Name": "name1",
"Version": "1.0.0"
},
{
"Group": "group2",
"Name": "name2",
"Version": "1.0.0"
},
{
"Group": "group3",
"Name": "name3",
"Version": "1.0.0"
},
{
"Group": "group4",
"Name": "name4",
"Version": "1.0.0"
},
{
"Group": "group5",
"Name": "name5",
"Version": "1.0.0"
}
]
}
if i'm use
jq '.packages[3] |= . + {"Group":"group4", "Name":"name4", "Version":"1.0.0"}' jq.json
{
"id": "json",
"description": "test",
"packages": [
{
"Group": "group1",
"Name": "name1",
"Version": "1.0.0"
},
{
"Group": "group2",
"Name": "name2",
"Version": "1.0.0"
},
{
"Group": "group3",
"Name": "name3",
"Version": "1.0.0"
},
{
"Group": "group4",
"Name": "name4",
"Version": "1.0.0"
}
]
}
And i'm lost group5 element.
Is it possible to add an item without losing the previous one?
I understand that I can save the output of lost indexes and insert them with the new index, but this seems wrong
Update |= the .packages field by redifining it as a new array consisting of the first three elements .[:3], the new one [{…}] and the the rest .[3:]. Technically, we are constructing an array by piecing three arrays together +.
jq '.packages |= .[:3] + [{Group: "group4", Name: "name4", Version: "1.0.0"}] + .[3:]'
{
"id": "json",
"description": "test",
"packages": [
{
"Group": "group1",
"Name": "name1",
"Version": "1.0.0"
},
{
"Group": "group2",
"Name": "name2",
"Version": "1.0.0"
},
{
"Group": "group3",
"Name": "name3",
"Version": "1.0.0"
},
{
"Group": "group4",
"Name": "name4",
"Version": "1.0.0"
},
{
"Group": "group5",
"Name": "name5",
"Version": "1.0.0"
}
]
}
Demo
You want .packages |= . + [ { ... } ], which appends an element (well, an entire array, really), to the packages array, and not .packages[3] |= { ... }, which adds/merges some keys into the object at .packages[3].
You desire to insert a record into an array such that it remains sorted. The best and easiest solution is simply to append the record than sort the array. This has the advantage that knowing the insert position isn't needed.
.packages |= (
. + [ { "Group": "group4", "Name": "name4", "Version": "1.0.0" } ] |
sort_by( .Group )
)
I have a big file that has lines as below.
{
"total": 320,
"assets": [
{
"audit": {
"created": {
"date": "2019-09-30T12:38:01.421Z"
},
"updated": {}
},
"organizationId": "12345678",
"id": 211123898,
"name": "groupId:760c47ad-c9f2958be:assetId:8o-api",
"exchangeAssetName": "8O API",
"groupId": "760c47ad-c9f2958be",
"assetId": "8o-api",
"apis": [
{
"audit": {
"created": {
"date": "2019-09-30T12:38:03.139Z"
},
"updated": {
"date": "2020-03-09T21:37:55.745Z"
}
},
"organizationId": "12345678",
"id": 15822364,
"groupId": "760c47ad-c9f2958be",
"assetId": "8o-api",
"assetVersion": "1.0.0",
"productVersion": "v1",
"description": null,
"tags": [],
"order": 1,
"providerId": null,
"deprecated": false,
"lastActiveDate": "2021-01-15T22:43:33.881Z",
"isPublic": false,
"stage": "release",
"lastActiveDelta": 7,
"pinned": false,
"activeContractsCount": 6,
"autodiscoveryInstanceName": "v1:15822364"
}
],
"totalApis": 1,
"autodiscoveryApiName": "groupId:760c47ad-c9f2958be:assetId:8o-api"
},
{
"audit": {
"created": {
"date": "2018-06-22T19:41:35.760Z"
},
"updated": {
"date": "2018-09-13T06:20:51.151Z"
}
},
"organizationId": "760c47ad-c9f2958be",
"id": 210914379,
"name": "hips-ts",
"exchangeAssetName": "hips-ts",
"groupId": "760c47ad-c9f2958be",
"assetId": "hips-ts",
"apis": [
{
"audit": {
"created": {
"date": "2018-06-22T19:41:35.759Z"
},
"updated": {
"date": "2020-03-09T21:37:55.745Z"
}
},
"organizationId": "760c47ad-c9f2958be",
"id": 15470738,
"groupId": "760c47ad-c9f2958be",
"assetId": "hips-ts",
"assetVersion": "1.0.0",
"productVersion": "v1",
"description": null,
"tags": [],
"order": 1,
"providerId": null,
"deprecated": false,
"lastActiveDate": "2021-01-15T22:43:30.004Z",
"endpointUri": null,
"isPublic": false,
"stage": "release",
"lastActiveDelta": 11,
"pinned": false,
"activeContractsCount": 1,
"autodiscoveryInstanceName": "1-test"
}
],
"totalApis": 1,
"autodiscoveryApiName": "hips-ts"
}
]
}
I am trying to use jq to just get the assetId and the activeContractsCount from this in a comma separated way.
So my output for this text should be
8o-api, 6
hips-ts, 1
I tried the following wiht jq jq -r '[.assets[].assetId, .assets[].apis[].activeContractsCount]|#csv' and I tried mapping too but nothing seems to stick.
Can you help me here? Any help is appreciated.
The following will produce exactly two columns, one for the "assetID" and one for the count:
.assets[].apis[]
| [.assetId, .activeContractsCount]
| #csv
I figured it out as soon as I posted the question. But please let me know if there's a better way to do this.
jq -r '.assets[]|[.apis[].assetId, .apis[].activeContractsCount]|#csv'
I want to read the status of clusters and servers inside it.
Below is the sample json file
"data": [{
"id": 7865,
"timeCreated": 1602589399294,
"timeUpdated": 1602748892149,
"name": "gw-ext-1",
"type": "CLUSTER",
"status": "RUNNING",
"multicastEnabled": false,
"primaryNodeId": 546,
"servers": [{
"id": 768,
"timeCreated": 1602589028419,
"timeUpdated": 1602747941321,
"name": "gw-jpg208765-1",
"type": "SERVER",
"serverType": "GATEWAY",
"status": "RUNNING",
"addresses": [{
"networkInterface": "eng123"
},
{
"networkInterface": "eng124"
}],
"clusterId": 098,
"clusterName": "gw-ext-1",
"currentClusteringPort": 897,
"runtimeInformation": {
"Information": {
"runtime": {
"name": "abctech",
"version": "1.6.8"
},
"specification": {
"vendor": "rrr",
"name": "rrrt",
"version": "1.8.89"
}
},
"osInformation": {
"name": "LX",
"version": "35",
"architecture": "klh"
},
"mExpirationDate": 098765589283662
}
},
{
"id": 876,
"timeCreated": 1602589007370,
"timeUpdated": 1602748894901,
"name": "gw-jpg208765-2",
"type": "SERVER",
"serverType": "GATEWAY",
"mVersion": "3.9.1",
"gaVersion": "3.9.1",
"agentVersion": "1.9.5",
"ExpirationDate": 32521996800000,
"ExpirationDate": 1665661007000,
"status": "DISCONNECTED",
"addresses": [{
"networkInterface": "engg"
},
{
"networkInterface": "engg"
}],
"clusterId": 768,
"clusterName": "gw-ext-1",
"serverPort": 987,
"currentClusteringPort": 987,
"runtimeInformation": {
"abcInfo": {
"runtime": {
"name": "abc",
"version": "1.2.3"
},
"specification": {
"vendor": "RRR",
"name": "RTR",
"version": "1.8.0"
}
},
"osInformation": {
"name": "LX",
"version": "4.78",
"architecture": "eng"
},
"ExpirationDate": 8765478999765
}
}],
"visibilityMap": {
"mapNodes": [{
"serverId": 765,
"visibleNodeIds": [765,
876],
"unknownNodeIps": []
},
{
"serverId": 876,
"visibleNodeIds": [765,
876],
"unknownNodeIps": []
}]
}
},
{
"id": 7865,
"timeCreated": 1602589399294,
"timeUpdated": 1602748892149,
"name": "gw-ext-2",
"type": "CLUSTER",
"status": "RUNNING",
"multicastEnabled": false,
"primaryNodeId": 546,
"servers": [{
"id": 768,
"timeCreated": 1602589028419,
"timeUpdated": 1602747941321,
"name": "gw-jpg208766-1",
"type": "SERVER",
"serverType": "GATEWAY",
"status": "RUNNING",
"addresses": [{
"networkInterface": "eng123"
},
{
"networkInterface": "eng124"
}],
"clusterId": 098,
"clusterName": "gw-ext-2",
"currentClusteringPort": 897,
"runtimeInformation": {
"Information": {
"runtime": {
"name": "abctech",
"version": "1.6.8"
},
"specification": {
"vendor": "rrr",
"name": "rrrt",
"version": "1.8.89"
}
},
"osInformation": {
"name": "LX",
"version": "35",
"architecture": "klh"
},
"mExpirationDate": 098765589283662
}
},
{
"id": 876,
"timeCreated": 1602589007370,
"timeUpdated": 1602748894901,
"name": "gw-jpg208766-2",
"type": "SERVER",
"serverType": "GATEWAY",
"mVersion": "3.9.1",
"gaVersion": "3.9.1",
"agentVersion": "1.9.5",
"ExpirationDate": 32521996800000,
"ExpirationDate": 1665661007000,
"status": "DISCONNECTED",
"addresses": [{
"networkInterface": "engg"
},
{
"networkInterface": "engg"
}],
"clusterId": 768,
"clusterName": "gw-ext-2",
"serverPort": 987,
"currentClusteringPort": 987,
"runtimeInformation": {
"abcInfo": {
"runtime": {
"name": "abc",
"version": "1.2.3"
},
"specification": {
"vendor": "RRR",
"name": "RTR",
"version": "1.8.0"
}
},
"osInformation": {
"name": "LX",
"version": "4.78",
"architecture": "eng"
},
"ExpirationDate": 8765478999765
}
}],
"visibilityMap": {
"mapNodes": [{
"serverId": 765,
"visibleNodeIds": [765,
876],
"unknownNodeIps": []
},
{
"serverId": 876,
"visibleNodeIds": [765,
876],
"unknownNodeIps": []
}]
}
}]
So in each cluster we have two servers and this json continues to have around 15 clusters.
I want to filter out the status of each cluster and server in below format
name cluster/server status
gw-ext-1 CLUSTER RUNNING
gw-jpg208765-1 SERVER RUNNING
gw-jpg208765-2 SERVER DISCONNECTED
similarly for other clusters also.
I tried few things but its not giving me the servers .. it gives only cluster's details
target_id=echo \$targetIdResponse | ${env.WORKSPACE}/jq -r '.data[] | [.name, .type, .status]'
OR
target_id=echo \$targetIdResponse | ${env.WORKSPACE}/jq -r '.data[] | [.name, .type, .status, .servers.name, .servers.type, .servers.status]'
where $targetIdResponse contains my json data
I want to know how i can filter the above json to get the required data.
You need to have the header array the required fields in a separate array and put them together in a tabular format using #tsv
jq -r '[ "name", "cluster/server", "status" ],
( .data[] | [.name, .type, .status] ),
( .data[].servers[] | [ .name, .type, .status ] ) | #tsv'
The requirement was modified since originally posted to have the server information exactly below the cluster information
jq -r '[ "name", "cluster/server", "status" ],
( .data[] | [.name, .type, .status], ( .servers[] | [.name, .type, .status] ) ) | #tsv'
I have some JSON. It looks like this:
{
"Volumes": [
{
"Attachments": [
{
"VolumeId": "vol-11111111",
"State": "attached",
"DeleteOnTermination": false,
"Device": "/dev/sdz"
}
],
"Tags": [
{
"Value": "volume1",
"Key": "Name"
},
{
"Value": "00:00",
"Key": "Start"
},
{
"Value": "00:20",
"Key": "Finish"
},
{
"Value": "2",
"Key": "Period"
}
],
"VolumeId": "vol-11111111"
},
{
"Attachments": [
{
"VolumeId": "vol-22222222",
"State": "attached",
"DeleteOnTermination": false,
"Device": "/dev/sdz"
}
],
"Tags": [
{
"Value": "volume2",
"Key": "Name"
},
{
"Value": "00:00",
"Key": "Start"
},
{
"Value": "00:20",
"Key": "Finish"
},
{
"Value": "2",
"Key": "Period"
}
],
"VolumeId": "vol-22222222"
},
{
"Attachments": [
{
"VolumeId": "vol-333333333",
"State": "attached",
"DeleteOnTermination": false,
"Device": "/dev/sdz"
}
],
"Tags": [
{
"Value": "volume3",
"Key": "Name"
},
{
"Value": "00:00",
"Key": "Start"
},
{
"Value": "00:20",
"Key": "Finish"
},
{
"Value": "2",
"Key": "Period"
}
],
"VolumeId": "vol-33333333"
}
]
}
Using jq, I am able to extract the following information:
VolumeId,Finish,Start,Period
using the jq command
cat json | jq -r '[.Volumes[]|({VolumeId}+(.Tags|from_entries))|{VolumeId,Finish,Start,Period}]'
[
{
"VolumeId": "vol-11111111",
"Finish": "00:20",
"Start": "00:00",
"Period": "2"
},
{
"VolumeId": "vol-22222222",
"Finish": "00:20",
"Start": "00:00",
"Period": "2"
},
{
"VolumeId": "vol-33333333",
"Finish": "00:20",
"Start": "00:00",
"Period": "2"
}
]
All this works fine. However I have the need to additional extract .Attachments.Device. I am looking for output for each array similar to:
[
{
"VolumeId": "vol-11111111",
"Finish": "00:20",
"Start": "00:00",
"Period": "2",
"DeviceId": "/dev/sdz"
},
{
"VolumeId": "vol-22222222",
"Finish": "00:20",
"Start": "00:00",
"Period": "2",
"DeviceId": "/dev/sdz"
},
{
"VolumeId": "vol-33333333",
"Finish": "00:20",
"Start": "00:00",
"Period": "2",
"DeviceId": "/dev/sdz"
}
]
However I can't figure out how to do this without getting an error. The most logical approach for me would be to do something like:
cat json | jq -r '[.Volumes[]|({VolumeId}+(.Attachments|from_entries)+(.Tags|from_entries))|{VolumeId,Finish,Start,Period,DeviceId}]'
However I get the error:
jq: error (at <stdin>:91): Cannot use null (null) as object key
Any help figuring out what I am not doing correct and how to fix it would be greatly appreciated.
thanks
Ultimately, the problem is that you're using from_entries on the Attachments array when it wouldn't work. from_entries takes an array of key/value pair objects to create an object with those values. However, you don't have key/value pairs, but objects. If you're just trying to combine them, you should use add.
Also, there is no property named DeviceId, it's Device. If you want to select the Device property and get it as DeviceId, you need to provide the correct name.
.Volumes | map(
({ VolumeId } + (.Attachments | add) + (.Tags | from_entries))
| { VolumeId, Finish, Start, Period, DeviceId: .Device }
)