Related
need help to parse the JSON data received from Oracle Integration Cloud. The expected output is mentioned below alongwith the command i am trying to use.
JQ command
jq '[{id: .id},{integrations: [.integrations[]|{code: .code, version: .version, dependencies: .dependencies|{connections: .connections[]|{id: .id, status: .status}}, .dependencies|{lookups: .lookups}}]}]' output.json
Error :
jq: error: syntax error, unexpected FIELD (Unix shell quoting issues?) at , line 1:
[{id: .id},{integrations: [.integrations[]|{code: .code, version: .version, dependencies: .dependencies|{connections: .connections[]|{id: .id, status: .status}}, .dependencies|{lookups: .lookups}}]}]
Note : If i run below command to fetch only connections data it works fine
jq '[{id: .id},{integrations: [.integrations[]|{code: .code, version: .version, dependencies: .dependencies|{connections: .connections[]|{id: .id, status: .status}}}]}]' output.json
Expected Output:
[
{
"id": "SAMPLE_PACKAGE"
},
{
"integrations": [
{
"code": "HELLO_INTEGRATION",
"version": "01.00.0000",
"dependencies": {
"connections": {
"id": "HELLO_WORLD1",
"status": "CONFIGURED"
}
}
},
{
"code": "HELLO_INTEGRATIO_LOOKUP",
"version": "01.00.0000",
"dependencies": {
"connections": {
"id": "HELLO_WORLD1",
"status": "CONFIGURED"
},
"lookups": {
"name": "COMMON_LOOKUP_VARIABLES",
"status": "CONFIGURED"
}
}
},
{
"code": "HI_INTEGRATION",
"version": "01.00.0000",
"dependencies": {
"connections": {
"id": "HELLO_WORLD1",
"status": "CONFIGURED"
}
}
}
]
}
]
output.json file contains
{
"bartaType": "DEVELOPED",
"countOfIntegrations": 3,
"id": "SAMPLE_PACKAGE",
"integrations": [
{
"code": "HELLO_INTEGRATION",
"dependencies": {
"connections": [
{
"id": "HELLO_WORLD1",
"lockedFlag": false,
"name": "Hello World1",
"role": "SOURCE",
"status": "CONFIGURED",
"type": "rest",
"usage": 6
}
]
},
"description": "",
"eventSubscriptionFlag": false,
"filmstrip": [
{
"code": "HELLO_WORLD1",
"iconUrl": "/images/rest/rest_icon_46.png",
"name": "Hello World1",
"role": "SOURCE",
"status": "CONFIGURED"
}
],
"id": "HELLO_INTEGRATION|01.00.0000",
"lockedFlag": false,
"name": "HELLO_INTEGRATION",
"pattern": "Orchestration",
"patternDescription": "Map Data",
"payloadTracingEnabledFlag": true,
"publishFlag": false,
"scheduleApplicable": false,
"scheduleDefined": false,
"status": "ACTIVATED",
"style": "FREEFORM",
"styleDescription": "Orchestration",
"tempCopyExists": false,
"tracingEnabledFlag": true,
"version": "01.00.0000",
"warningMsg": "ACTIVATE_PUBLISH_NO_CONN"
},
{
"code": "HELLO_INTEGRATIO_LOOKUP",
"dependencies": {
"connections": [
{
"id": "HELLO_WORLD1",
"lockedFlag": false,
"name": "Hello World1",
"role": "SOURCE",
"status": "CONFIGURED",
"type": "rest",
"usage": 6
}
],
"lookups": [
{
"lockedFlag": false,
"name": "COMMON_LOOKUP_VARIABLES",
"status": "CONFIGURED",
"usage": 1
}
]
},
"description": "",
"eventSubscriptionFlag": false,
"filmstrip": [
{
"code": "HELLO_WORLD1",
"iconUrl": "/images/rest/rest_icon_46.png",
"name": "Hello World1",
"role": "SOURCE",
"status": "CONFIGURED"
}
],
"id": "HELLO_INTEGRATIO_LOOKUP|01.00.0000",
"lockedFlag": false,
"name": "HELLO_INTEGRATION_LOOKUP",
"pattern": "Orchestration",
"patternDescription": "Map Data",
"payloadTracingEnabledFlag": true,
"publishFlag": false,
"scheduleApplicable": false,
"scheduleDefined": false,
"status": "ACTIVATED",
"style": "FREEFORM",
"styleDescription": "Orchestration",
"tempCopyExists": false,
"tracingEnabledFlag": true,
"version": "01.00.0000",
"warningMsg": "ACTIVATE_PUBLISH_NO_CONN"
},
{
"code": "HI_INTEGRATION",
"dependencies": {
"connections": [
{
"id": "HELLO_WORLD1",
"lockedFlag": false,
"name": "Hello World1",
"role": "SOURCE",
"status": "CONFIGURED",
"type": "rest",
"usage": 6
}
]
},
"description": "",
"eventSubscriptionFlag": false,
"filmstrip": [
{
"code": "HELLO_WORLD1",
"iconUrl": "/images/rest/rest_icon_46.png",
"name": "Hello World1",
"role": "SOURCE",
"status": "CONFIGURED"
}
],
"id": "HI_INTEGRATION|01.00.0000",
"lockedFlag": false,
"name": "HI_INTEGRATION",
"pattern": "Orchestration",
"patternDescription": "Map Data",
"payloadTracingEnabledFlag": true,
"publishFlag": false,
"scheduleApplicable": false,
"scheduleDefined": false,
"status": "ACTIVATED",
"style": "FREEFORM",
"styleDescription": "Orchestration",
"tempCopyExists": false,
"tracingEnabledFlag": true,
"version": "01.00.0000",
"warningMsg": "ACTIVATE_PUBLISH_NO_CONN"
}
],
"isCloneAllowed": false,
"isViewAllowed": false,
"name": "SAMPLE_PACKAGE",
"type": "DEVELOPED"
}
The problem is that the lookups key is not always present so, you cannot use the [] on it. So, instead you can use the map function and provide a default before piping to the map function like below
[
{ id: .id },
{
integrations: [
.integrations[]|{
id: .id,
code: .code,
dependencies: {
connections: (.dependencies.connections//[]|map({id,status}))[0],
lookups: (.dependencies.lookups//[]|map({name,status}))[0]
}
}
]
}
]
The (.dependencies.lookups//[]|map({name,status}))[0] has the effect of passing an empty array to the map function which results in a null value when accessing the first element.
See in action https://jqplay.org/s/zQBkHtnzOd1
The provided JQ statement works fine for single elements in the array , but incase the array contains multiple elements it only fetches the first element. Also i updated the dependencies object to capture all the arrays ( connections,lookups,certificates,libraries,integrations)
Below is the modified one. Please suggest for any better options.
[
{ id: .id },
{
integrations: [
.integrations[]|{
id: .id,
code: .code,
dependencies: {
connections: (.dependencies.connections//[]|map({id,status})),
lookups: (.dependencies.lookups//[]|map({name,status})),
certificates: (.dependencies.certificates//[]|map({id,status})),
libraries: (.dependencies.libraries//[]|map({code,status,version})),
integrations: (.dependencies.integrations//[]|map({code,version}))
}
}
]
}
]|del(..|select(.==[]))
Note: To remove the empty arrays del function is added which is giving the below output :
[
{
"id": "SAMPLE_PACKAGE"
},
{
"integrations": [
{
"id": "HELLO_INTEGRATION|01.00.0000",
"code": "HELLO_INTEGRATION",
"dependencies": {
"connections": [
{
"id": "HELLO_WORLD1",
"status": "CONFIGURED"
},
{
"id": "HELLO_WORLD2",
"status": "CONFIGURED"
}
]
}
},
{
"id": "HELLO_INTEGRATIO_LOOKUP|01.00.0000",
"code": "HELLO_INTEGRATIO_LOOKUP",
"dependencies": {
"connections": [
{
"id": "HELLO_WORLD1",
"status": "CONFIGURED"
}
],
"lookups": [
{
"name": "COMMON_LOOKUP_VARIABLES",
"status": "CONFIGURED"
}
]
}
},
{
"id": "HI_INTEGRATION|01.00.0000",
"code": "HI_INTEGRATION",
"dependencies": {
"connections": [
{
"id": "HELLO_WORLD1",
"status": "CONFIGURED"
}
]
}
}
]
}
]
I'm working on finding a solution where I need to parse a json file (input.json) and create json file as output (output.json) after removing few lines.
Only few version will have both value (ubuntu, centos), most of the version will have either ubuntu or centos.
I tried the following command;
jq '.[]' input.json
it gives the complete list But don't know how to get values under ubuntu and centos. any guidance will be beneficial.
input.json
[
{
"Version": "1.0",
"ubuntu": {
"ver": "1.0.0",
"filename": "1_0_0.log",
"sourceUrl": "https://example.com/log/1_0_0.log"
}
},
{
"Version": "1.1",
"ubuntu": {
"ver": "1.0.1",
"filename": "1_0_1.log",
"sourceUrl": "https://example.com/log/1_0_1.log"
}
},
{
"Version": "1.4",
"ubuntu": {
"ver": "1.0.4",
"filename": "1_0_4.log",
"sourceUrl": "https://example.com/log/1_0_4.log"
},
"centos": {
"ver": "1.0.4",
"filename": "1_0_4.pdf",
"sourceUrl": "https://example.com/log/1_0_4.pdf"
}
}
]
output.json
[
{
"Version": "1.0",
"ubuntu": {
"ver": "1.0.0",
"filename": "1_0_0.log"
}
},
{
"Version": "1.1",
"ubuntu": {
"ver": "1.0.1",
"filename": "1_0_1.log"
}
},
{
"Version": "1.4",
"ubuntu": {
"ver": "1.0.4",
"filename": "1_0_4.log"
},
"centos": {
"ver": "1.0.14",
"filename": "1_0_4.pdf"
}
}
]
You can use the following JQ command to remove each sourceUrl;
jq 'del(.. | .sourceUrl?)' input.json
del() docs
Output:
[
{
"Version": "1.0",
"ubuntu": {
"version": "1.0.0",
"filename": "1_0_0.log"
}
},
{
"Version": "1.1",
"ubuntu": {
"version": "1.0.1",
"filename": "1_0_1.log"
}
},
{
"Version": "1.4",
"ubuntu": {
"version": "1.0.4",
"filename": "1_0_4.log"
},
"centos": {
"version": "1.0.4",
"filename": "1_0_4.pdf"
}
}
]
Try it online!
I have been battling with this for sometime, ive looked a few articles in regards to this topic. I can't seem to get exactly what I'm aiming for. I have this original json data that I would like to transform a bit.
Group the objects based on the label.env (for now only using one env), then inserting those objects from the original json into an object named projects.
orginial_json
{
"createTime": "2020-06-25T14:16:45.720Z",
"labels": {
"cb_domain": "cloud-services",
"cb_product": "infra",
"env": "prod",
"owner": "cloud-server"
},
"lifecycleState": "ACTIVE",
"name": "projectname",
"parent": {
"id": "123456577",
"type": "folder"
},
"projectId": "projectname-324234",
"projectNumber": "962363417856"
}
{
"createTime": "2020-06-24T19:45:42.851Z",
"labels": {
"cb_domain": "cloud-services",
"cb_product": "ad",
"env": "prod",
"owner": "cloud-server"
},
"lifecycleState": "ACTIVE",
"name": "projectname2",
"parent": {
"id": "4352564765437",
"type": "folder"
},
"projectId": "projectname2-4567",
"projectNumber": "3243456324"
}
Im using jq -s 'group_by(.labels.env) | .[] | { (.[0].labels.env) : { projects: .[] | . } }'
to nest the original json data into objects grouped by env\projects, which works but seems to duplicate it instead of putting each object within projects.
{
"prod": {
"projects": {
"createTime": "2020-06-25T14:16:45.720Z",
"labels": {
"cb_domain": "cloud-services",
"cb_product": "infra",
"env": "prod",
"owner": "cloud-server"
},
"lifecycleState": "ACTIVE",
"name": "projectname",
"parent": {
"id": "770593713153",
"type": "folder"
},
"projectId": "projectname-324234",
"projectNumber": "962363417856"
}
}
}
{
"prod": {
"projects": {
"createTime": "2020-06-24T19:45:42.851Z",
"labels": {
"cb_domain": "cloud-services",
"cb_product": "ad",
"env": "prod",
"owner": "cloud-server"
},
"lifecycleState": "ACTIVE",
"name": "projectname2",
"parent": {
"id": "4352564765437",
"type": "folder"
},
"projectId": "projectname2-4567",
"projectNumber": "3243456324"
}
}
}
What I'm aiming for is
{
"prod": {
"projects": {
"createTime": "2020-06-25T14:16:45.720Z",
"labels": {
"cb_domain": "cloud-services",
"cb_product": "infra",
"env": "prod",
"owner": "cloud-server"
},
"lifecycleState": "ACTIVE",
"name": "projectname",
"parent": {
"id": "770593713153",
"type": "folder"
},
"projectId": "projectname-324234",
"projectNumber": "962363417856"
},
{
"createTime": "2020-06-24T19:45:42.851Z",
"labels": {
"cb_domain": "cloud-services",
"cb_product": "ad",
"env": "prod",
"owner": "cloud-server"
},
"lifecycleState": "ACTIVE",
"name": "projectname2",
"parent": {
"id": "4352564765437",
"type": "folder"
},
"projectId": "projectname2-4567",
"projectNumber": "3243456324"
}
}
}
Allowing for some minor discrepancies in the Q, the following variant of your jq program seems to do what you're trying to achieve:
group_by(.labels.env)
| .[]
| .[0].labels.env as $key
| { ($key) : { projects: . } }
i am wriiting an Azure ARM Template for deploying VNETs. Therefore i have a deployment file "VNET.json and a parameter file "VNET.parameter.json.
Now i want to hand over ONLY in the paramter File a variable but always got the error:
'The template variable 'IP' is not found. Please see https://aka.ms/arm-template/#variables for usage details.
or is it not "allowd" to use a variable in a parameter?
Here is my json file.
{
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"variables": {
"IP": "172.37.0.0/24"
},
"parameters": {
"VNetSettings":{
"value":{
"name":"ManagementNW",
"addressPrefixes": [
{
"name": "VPN",
"addressPrefix": "192.168.0.0/16"
},
{
"name": "Control",
"addressPrefix": "172.37.0.0/16"
},
{
"name": "Service",
"addressPrefix": "172.36.0.0/16"
},
{
"name": "DMZ",
"addressPrefix": "172.35.4.0/24"
}
],
"subnets":[
{
"name": "VPN-subnet",
"addressPrefix": "192.168.1.0/24"
},
{
"name": "Control-subnet",
"addressPrefix": "172.37.0.0/24"
},
{
"name": "Service-subnet",
"addressPrefix": "172.36.0.0/24"
},
{
"name": "DMZ-subnet",
"addressPrefix": "172.35.4.0/24"
},
{
"name": "Gateway-subnet",
"addressPrefix": "192.168.255.0/24"
},
{
"name":"AzureFirewall-subnet",
"addressPrefix":"192.168.254.0/24"
}
]
}
},
"Control-NSG-settings": {
"value": {
"securityRules": [
{
"direction": "Inbound",
"name": "VPNGW_to_Tableau_all",
"sourceAddressPrefix": "192.168.255.0/24",
"sourcePortRange": "*",
"destinationAddressPrefix": "172.37.0.41",
"destinationPortRange": "*",
"protocol": "*",
"access": "Allow",
"priority": 101,
"description": "allow RDP connections"
},
{
"direction": "Inbound",
"name": "VPNGW_to_Control_all",
"sourceAddressPrefix": "192.168.255.0/24",
"sourcePortRange": "*",
"destinationAddressPrefix": "[variables('IP')]",
"destinationPortRange": "*",
"protocol": "*",
"access": "deny",
"priority": 102,
"description": "deny"
},
KR
Marvin
In the template, you reference the value for the parameter by using the variables function.
In the ARM template json file, you use variables to simplify your template. Rather than repeating complicated expressions throughout your template, you define a variable that contains the complicated expression. However, if you only use once you do not need to use variables.
Also, as usrone said, your template looks suspicious. The normal template is as this one you could refer to.
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "",
"apiProfile": "",
"parameters": { },
"variables": { },
"functions": [ ],
"resources": [ ],
"outputs": { }
}
I have the following JSON:
[
{
"function": "ping",
"name": "start",
"servers": [
{
"load": 581.6875,
"last_heard": 2.379324197769165,
"version": "1.0",
"hidden": false,
"pid": "19735"
},
{
"load": 444.0625,
"last_heard": 1.3227169513702393,
"version": "1.0",
"hidden": false,
"pid": "12092"
}
]
},
{
"function": "pong",
"name": "middle",
"servers": [
{
"load": 581.6875,
"last_heard": 2.379324197769165,
"version": "2.0",
"hidden": false,
"pid": "19735"
},
{
"load": 444.0625,
"last_heard": 1.3227169513702393,
"version": "3.0",
"hidden": false,
"pid": "12092"
},
{
"load": 444.0625,
"last_heard": 1.3227169513702393,
"version": "3.0",
"hidden": false,
"pid": "12093"
}
]
},
{
"function": "pang",
"name": "end",
"servers": [
{
"load": 581.6875,
"last_heard": 2.379324197769165,
"version": "2.0",
"hidden": false,
"pid": "19735"
},
{
"load": 444.0625,
"last_heard": 1.3227169513702393,
"version": "2.0",
"hidden": false,
"pid": "12092"
}
]
}
]
(it's just a sample, it's hundreds of entries)
What I need is to get
[{"name": "start", "version": ["1.0"]},
{"name": "middle", "version": ["2.0", "3.0"]},
{"name": "end", "version": ["2.0"]}]
So I need to remove useless data and then get a list of names with their unique values for "version".
I can get until the point where I have
{
"name": "ping",
"version": [
"1.0",
"1.0"
]
}
{
"name": "pong",
"version": [
"2.0",
"3.0",
"3.0"
]
}
{
"name": "pang",
"version": [
"2.0",
"2.0"
]
}
using
jq '.[] | {name: .function, version: [.servers[].version]}'
But I need to get rid of the duplicated values.
Is this possible using jq?
You were almost there. Just pipe the version array to the unique function:
jq '[.[]|{name, "version": [.servers[].version]|unique}]' input
Here is a solution that uses reduce to gather version keys in a temporary object and dedup at the end using keys
[
. []
| reduce .servers[].version as $v (
{name, version:{}}
; .version[$v] = 1
)
| .version |= keys
]