I'm working on finding a solution where I need to parse a json file (input.json) and create json file as output (output.json) after removing few lines.
Only few version will have both value (ubuntu, centos), most of the version will have either ubuntu or centos.
I tried the following command;
jq '.[]' input.json
it gives the complete list But don't know how to get values under ubuntu and centos. any guidance will be beneficial.
input.json
[
{
"Version": "1.0",
"ubuntu": {
"ver": "1.0.0",
"filename": "1_0_0.log",
"sourceUrl": "https://example.com/log/1_0_0.log"
}
},
{
"Version": "1.1",
"ubuntu": {
"ver": "1.0.1",
"filename": "1_0_1.log",
"sourceUrl": "https://example.com/log/1_0_1.log"
}
},
{
"Version": "1.4",
"ubuntu": {
"ver": "1.0.4",
"filename": "1_0_4.log",
"sourceUrl": "https://example.com/log/1_0_4.log"
},
"centos": {
"ver": "1.0.4",
"filename": "1_0_4.pdf",
"sourceUrl": "https://example.com/log/1_0_4.pdf"
}
}
]
output.json
[
{
"Version": "1.0",
"ubuntu": {
"ver": "1.0.0",
"filename": "1_0_0.log"
}
},
{
"Version": "1.1",
"ubuntu": {
"ver": "1.0.1",
"filename": "1_0_1.log"
}
},
{
"Version": "1.4",
"ubuntu": {
"ver": "1.0.4",
"filename": "1_0_4.log"
},
"centos": {
"ver": "1.0.14",
"filename": "1_0_4.pdf"
}
}
]
You can use the following JQ command to remove each sourceUrl;
jq 'del(.. | .sourceUrl?)' input.json
del() docs
Output:
[
{
"Version": "1.0",
"ubuntu": {
"version": "1.0.0",
"filename": "1_0_0.log"
}
},
{
"Version": "1.1",
"ubuntu": {
"version": "1.0.1",
"filename": "1_0_1.log"
}
},
{
"Version": "1.4",
"ubuntu": {
"version": "1.0.4",
"filename": "1_0_4.log"
},
"centos": {
"version": "1.0.4",
"filename": "1_0_4.pdf"
}
}
]
Try it online!
Related
need help to parse the JSON data received from Oracle Integration Cloud. The expected output is mentioned below alongwith the command i am trying to use.
JQ command
jq '[{id: .id},{integrations: [.integrations[]|{code: .code, version: .version, dependencies: .dependencies|{connections: .connections[]|{id: .id, status: .status}}, .dependencies|{lookups: .lookups}}]}]' output.json
Error :
jq: error: syntax error, unexpected FIELD (Unix shell quoting issues?) at , line 1:
[{id: .id},{integrations: [.integrations[]|{code: .code, version: .version, dependencies: .dependencies|{connections: .connections[]|{id: .id, status: .status}}, .dependencies|{lookups: .lookups}}]}]
Note : If i run below command to fetch only connections data it works fine
jq '[{id: .id},{integrations: [.integrations[]|{code: .code, version: .version, dependencies: .dependencies|{connections: .connections[]|{id: .id, status: .status}}}]}]' output.json
Expected Output:
[
{
"id": "SAMPLE_PACKAGE"
},
{
"integrations": [
{
"code": "HELLO_INTEGRATION",
"version": "01.00.0000",
"dependencies": {
"connections": {
"id": "HELLO_WORLD1",
"status": "CONFIGURED"
}
}
},
{
"code": "HELLO_INTEGRATIO_LOOKUP",
"version": "01.00.0000",
"dependencies": {
"connections": {
"id": "HELLO_WORLD1",
"status": "CONFIGURED"
},
"lookups": {
"name": "COMMON_LOOKUP_VARIABLES",
"status": "CONFIGURED"
}
}
},
{
"code": "HI_INTEGRATION",
"version": "01.00.0000",
"dependencies": {
"connections": {
"id": "HELLO_WORLD1",
"status": "CONFIGURED"
}
}
}
]
}
]
output.json file contains
{
"bartaType": "DEVELOPED",
"countOfIntegrations": 3,
"id": "SAMPLE_PACKAGE",
"integrations": [
{
"code": "HELLO_INTEGRATION",
"dependencies": {
"connections": [
{
"id": "HELLO_WORLD1",
"lockedFlag": false,
"name": "Hello World1",
"role": "SOURCE",
"status": "CONFIGURED",
"type": "rest",
"usage": 6
}
]
},
"description": "",
"eventSubscriptionFlag": false,
"filmstrip": [
{
"code": "HELLO_WORLD1",
"iconUrl": "/images/rest/rest_icon_46.png",
"name": "Hello World1",
"role": "SOURCE",
"status": "CONFIGURED"
}
],
"id": "HELLO_INTEGRATION|01.00.0000",
"lockedFlag": false,
"name": "HELLO_INTEGRATION",
"pattern": "Orchestration",
"patternDescription": "Map Data",
"payloadTracingEnabledFlag": true,
"publishFlag": false,
"scheduleApplicable": false,
"scheduleDefined": false,
"status": "ACTIVATED",
"style": "FREEFORM",
"styleDescription": "Orchestration",
"tempCopyExists": false,
"tracingEnabledFlag": true,
"version": "01.00.0000",
"warningMsg": "ACTIVATE_PUBLISH_NO_CONN"
},
{
"code": "HELLO_INTEGRATIO_LOOKUP",
"dependencies": {
"connections": [
{
"id": "HELLO_WORLD1",
"lockedFlag": false,
"name": "Hello World1",
"role": "SOURCE",
"status": "CONFIGURED",
"type": "rest",
"usage": 6
}
],
"lookups": [
{
"lockedFlag": false,
"name": "COMMON_LOOKUP_VARIABLES",
"status": "CONFIGURED",
"usage": 1
}
]
},
"description": "",
"eventSubscriptionFlag": false,
"filmstrip": [
{
"code": "HELLO_WORLD1",
"iconUrl": "/images/rest/rest_icon_46.png",
"name": "Hello World1",
"role": "SOURCE",
"status": "CONFIGURED"
}
],
"id": "HELLO_INTEGRATIO_LOOKUP|01.00.0000",
"lockedFlag": false,
"name": "HELLO_INTEGRATION_LOOKUP",
"pattern": "Orchestration",
"patternDescription": "Map Data",
"payloadTracingEnabledFlag": true,
"publishFlag": false,
"scheduleApplicable": false,
"scheduleDefined": false,
"status": "ACTIVATED",
"style": "FREEFORM",
"styleDescription": "Orchestration",
"tempCopyExists": false,
"tracingEnabledFlag": true,
"version": "01.00.0000",
"warningMsg": "ACTIVATE_PUBLISH_NO_CONN"
},
{
"code": "HI_INTEGRATION",
"dependencies": {
"connections": [
{
"id": "HELLO_WORLD1",
"lockedFlag": false,
"name": "Hello World1",
"role": "SOURCE",
"status": "CONFIGURED",
"type": "rest",
"usage": 6
}
]
},
"description": "",
"eventSubscriptionFlag": false,
"filmstrip": [
{
"code": "HELLO_WORLD1",
"iconUrl": "/images/rest/rest_icon_46.png",
"name": "Hello World1",
"role": "SOURCE",
"status": "CONFIGURED"
}
],
"id": "HI_INTEGRATION|01.00.0000",
"lockedFlag": false,
"name": "HI_INTEGRATION",
"pattern": "Orchestration",
"patternDescription": "Map Data",
"payloadTracingEnabledFlag": true,
"publishFlag": false,
"scheduleApplicable": false,
"scheduleDefined": false,
"status": "ACTIVATED",
"style": "FREEFORM",
"styleDescription": "Orchestration",
"tempCopyExists": false,
"tracingEnabledFlag": true,
"version": "01.00.0000",
"warningMsg": "ACTIVATE_PUBLISH_NO_CONN"
}
],
"isCloneAllowed": false,
"isViewAllowed": false,
"name": "SAMPLE_PACKAGE",
"type": "DEVELOPED"
}
The problem is that the lookups key is not always present so, you cannot use the [] on it. So, instead you can use the map function and provide a default before piping to the map function like below
[
{ id: .id },
{
integrations: [
.integrations[]|{
id: .id,
code: .code,
dependencies: {
connections: (.dependencies.connections//[]|map({id,status}))[0],
lookups: (.dependencies.lookups//[]|map({name,status}))[0]
}
}
]
}
]
The (.dependencies.lookups//[]|map({name,status}))[0] has the effect of passing an empty array to the map function which results in a null value when accessing the first element.
See in action https://jqplay.org/s/zQBkHtnzOd1
The provided JQ statement works fine for single elements in the array , but incase the array contains multiple elements it only fetches the first element. Also i updated the dependencies object to capture all the arrays ( connections,lookups,certificates,libraries,integrations)
Below is the modified one. Please suggest for any better options.
[
{ id: .id },
{
integrations: [
.integrations[]|{
id: .id,
code: .code,
dependencies: {
connections: (.dependencies.connections//[]|map({id,status})),
lookups: (.dependencies.lookups//[]|map({name,status})),
certificates: (.dependencies.certificates//[]|map({id,status})),
libraries: (.dependencies.libraries//[]|map({code,status,version})),
integrations: (.dependencies.integrations//[]|map({code,version}))
}
}
]
}
]|del(..|select(.==[]))
Note: To remove the empty arrays del function is added which is giving the below output :
[
{
"id": "SAMPLE_PACKAGE"
},
{
"integrations": [
{
"id": "HELLO_INTEGRATION|01.00.0000",
"code": "HELLO_INTEGRATION",
"dependencies": {
"connections": [
{
"id": "HELLO_WORLD1",
"status": "CONFIGURED"
},
{
"id": "HELLO_WORLD2",
"status": "CONFIGURED"
}
]
}
},
{
"id": "HELLO_INTEGRATIO_LOOKUP|01.00.0000",
"code": "HELLO_INTEGRATIO_LOOKUP",
"dependencies": {
"connections": [
{
"id": "HELLO_WORLD1",
"status": "CONFIGURED"
}
],
"lookups": [
{
"name": "COMMON_LOOKUP_VARIABLES",
"status": "CONFIGURED"
}
]
}
},
{
"id": "HI_INTEGRATION|01.00.0000",
"code": "HI_INTEGRATION",
"dependencies": {
"connections": [
{
"id": "HELLO_WORLD1",
"status": "CONFIGURED"
}
]
}
}
]
}
]
i have a JSON file like following:
{
"family": "amazon",
"release": "2 (Karoo)",
"runningKernel": {
"release":"4.14.171-136.231.amzn2.x86_64",
"version": ""
},
"packages": {
"gcc": {
"name": "gcc",
"version": "7.3.1",
"release": "6.amzn2.0.4",
"arch": "x86_64"
},
"kernel": {
"name": "kernel",
"version": "4.14.173",
"release": "137.229.amzn2",
"arch": "x86_64"
},
"dmidecode": {
"name": "dmidecode",
"version": "3.0",
"release": "5.amzn2.0.2",
"arch": "x86_64"
},
"kernel": {
"name": "kernel",
"version": "4.14.171",
"release": "136.231.amzn2",
"arch": "x86_64"
}
}
}
As you can see, there are two kernel objects but I want to remove one other than the running kernel version, how can do that using CLI.
I was unable to do it in Shell script, so i handled it in python.
I have multiple JSON files with the following structures
FILE1.json
{
"Plugins": [
{
"Name": "Async Http Client",
"Installation": [
{
"Version": "1.7.24.2",
"Enabled": true,
"URL": "http://wiki.jenkins-ci.org/display/JENKINS/Async+Http+Client+Plugin",
"Server": "Server1"
}
]
},
{
"Name": "Common API for Blue Ocean",
"Installation": [
{
"Version": "1.10.2",
"Enabled": true,
"URL": "https://wiki.jenkins-ci.org/display/JENKINS/Blue+Ocean+Plugin",
"Server": "Server1"
}
]
},
.
.
.
]
}
FILE2.json
{
"Plugins": [
{
"Name": "Async Http Client",
"Installation": [
{
"Version": "1.7.24.1",
"Enabled": true,
"URL": "http://wiki.jenkins-ci.org/display/JENKINS/Async+Http+Client+Plugin",
"Server": "Server2"
}
]
},
{
"Name": "Common API for Blue Ocean",
"Installation": [
{
"Version": "1.10.2",
"Enabled": true,
"URL": "https://wiki.jenkins-ci.org/display/JENKINS/Blue+Ocean+Plugin",
"Server": "Server2"
}
]
},
.
.
.
]
}
FILE3.json
{
"Plugins": [
{
"Name": "Common API for Blue Ocean",
"Installation": [
{
"Version": "1.10.1",
"Enabled": true,
"URL": "https://wiki.jenkins-ci.org/display/JENKINS/Blue+Ocean+Plugin",
"Server": "Server3"
}
]
},
{
"Name": "Support Core Plugin",
"Installation": [
{
"Version": "2.54",
"Enabled": true,
"URL": "https://wiki.jenkins.io/display/JENKINS/Support+Core+Plugin",
"Server": "Server3"
}
]
},
.
.
.
]
}
I want to merge them into a single file with the following structure using groovy
MergedFile.json
{
"Plugins": [
{
"Name": "Async Http Client",
"Installation": [
{
"Version": "1.7.24.2",
"Enabled": true,
"URL": "http://wiki.jenkins-ci.org/display/JENKINS/Async+Http+Client+Plugin",
"Server": "Server1"
},
{
"Version": "1.7.24.1",
"Enabled": true,
"URL": "http://wiki.jenkins-ci.org/display/JENKINS/Async+Http+Client+Plugin",
"Server": "Server2"
}
]
},
{
"Name": "Common API for Blue Ocean",
"Installation": [
{
"Version": "1.10.2",
"Enabled": true,
"URL": "https://wiki.jenkins-ci.org/display/JENKINS/Blue+Ocean+Plugin",
"Server": "Server1"
},
{
"Version": "1.10.2",
"Enabled": true,
"URL": "https://wiki.jenkins-ci.org/display/JENKINS/Blue+Ocean+Plugin",
"Server": "Server2"
},
{
"Version": "1.10.1",
"Enabled": true,
"URL": "https://wiki.jenkins-ci.org/display/JENKINS/Blue+Ocean+Plugin",
"Server": "Server3"
}
]
},
{
"Name": "Support Core Plugin",
"Installation": [
{
"Version": "2.54",
"Enabled": true,
"URL": "https://wiki.jenkins.io/display/JENKINS/Support+Core+Plugin",
"Server": "Server3"
}
]
},
.
.
.
]
}
I am looking for a jenkins pipeline script using jenkins groovy. The file structure of the 3 files are the same. Only the file content maybe different. If there is any common "Name", "Installation" is merged under the same "Name" like mentioned above.
Any help is appreciated!
Edit:
I've tried something like this..
files = findFiles(glob: "*.json")
def input
def mergedjson = readJSON file:"${files[0]}"
for (f=1;f<files.size();f++)
{
input=readJSON file:"${files[f]}"
i = input.Plugins.size()
j = mergedjson.Plugins.size()
for (x=0;x<i;x++)
{
def json_status=0
for (y=0;y<j;y++)
{
if (input.Plugins[x].Name == mergedjson.Plugins[y].Name)
{
mergedjson.Plugins[y].Installation += input.Plugins[x].Installation
json_status=1
break;
}
}
if (json_status == 0)
mergedjson.Plugins[j++] = input.Plugins[x]
}
}
inventory=JsonOutput.prettyPrint(JsonOutput.toJson(mergedjson))
writeFile([file: "Inventory.json", text: inventory])
I am getting the following error
java.lang.StackOverflowError
at java.lang.reflect.InvocationTargetException.(InvocationTargetException.java:72)
at sun.reflect.GeneratedMethodAccessor544.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaBeanProperty.getProperty(MetaBeanProperty.java:59)
at groovy.lang.PropertyValue.getValue(PropertyValue.java:42)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.getProperties(DefaultGroovyMethods.java:389)
at groovy.json.JsonOutput.getObjectProperties(JsonOutput.java:327)
at groovy.json.JsonOutput.writeObject(JsonOutput.java:320)
at groovy.json.JsonOutput.writeMap(JsonOutput.java:458)
at groovy.json.JsonOutput.writeObject(JsonOutput.java:321)
at groovy.json.JsonOutput.writeMap(JsonOutput.java:458)
at groovy.json.JsonOutput.writeObject(JsonOutput.java:321)
at groovy.json.JsonOutput.writeMap(JsonOutput.java:458)
at groovy.json.JsonOutput.writeObject(JsonOutput.java:321)
.
.
.
Here is my parameter file:
"VNetSettings": {
"value": {
"name": "VNet1",
"addressPrefixes": "10.0.0.0/16",
"subnets": [
{
"name": "sub1",
"addressPrefix": "10.0.1.0/24"
},
{
"name": "sub2",
"addressPrefix": "10.0.2.0/24"
}
]
}
}
Here is my deployment file(deploy.json)
{
"contentversion": "1.0.0.0",
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"parameters": {
"VNetSettings": {
"type": "object"
},
"noofsubnets": {
"type": "int"
}
},
"resources": [
{
"apiVersion": "2015-06-15",
"type": "Microsoft.Network/virtualNetworks",
"name": "[parameters('VNetSettings').name]",
"location": "[resourceGroup().location]",
"properties": {
"addressSpace": {
"addressPrefixes": [
"[parameters('VNetSettings').addressPrefixes]"
]
},
"copy": {
"name": "subnets",
"count": "[parameters('noofsubnets')]",
"input": {
"name": "[parameters('VNetSettings').subnets[copyIndex('subnets',1)].name]",
"properties": {
"addressPrefix": "[parameters('VNetSettings').subnets[copyIndex('subnets',1)].addressPrefix]"
}
}
}
}
}
]
}
What the deployment arm template should be doing is launch the subnets with their respective address prefixes (eg: sub1 -> 10.0.1.0/24,sub2 -> 10.0.2.0/24) but when I execute the template from the powershell using the following command:
New-AzureRmResourceGroupDeployment -Name testing -ResourceGroupName rgname -TemplateFile C:\Test\deploy.json -TemplateParameterFile C:\Test\parameterfile.json
I get displayed with the following error:
The template function 'copyIndex' is not expected at this location.
The function can only be used in a resource with copy specified. Error
says there is something wrong with the copyindex() but I'm unable to
find out what exactly is wrong with it.
Copy has to look like this:
"copy": [
{
"name": "xxx",
"count": "xxx",
"input": { ... }
}
]
You are missing []
copy shouldn't be inside the properties field. It has to be at the same level as properties.
I have the following JSON:
[
{
"function": "ping",
"name": "start",
"servers": [
{
"load": 581.6875,
"last_heard": 2.379324197769165,
"version": "1.0",
"hidden": false,
"pid": "19735"
},
{
"load": 444.0625,
"last_heard": 1.3227169513702393,
"version": "1.0",
"hidden": false,
"pid": "12092"
}
]
},
{
"function": "pong",
"name": "middle",
"servers": [
{
"load": 581.6875,
"last_heard": 2.379324197769165,
"version": "2.0",
"hidden": false,
"pid": "19735"
},
{
"load": 444.0625,
"last_heard": 1.3227169513702393,
"version": "3.0",
"hidden": false,
"pid": "12092"
},
{
"load": 444.0625,
"last_heard": 1.3227169513702393,
"version": "3.0",
"hidden": false,
"pid": "12093"
}
]
},
{
"function": "pang",
"name": "end",
"servers": [
{
"load": 581.6875,
"last_heard": 2.379324197769165,
"version": "2.0",
"hidden": false,
"pid": "19735"
},
{
"load": 444.0625,
"last_heard": 1.3227169513702393,
"version": "2.0",
"hidden": false,
"pid": "12092"
}
]
}
]
(it's just a sample, it's hundreds of entries)
What I need is to get
[{"name": "start", "version": ["1.0"]},
{"name": "middle", "version": ["2.0", "3.0"]},
{"name": "end", "version": ["2.0"]}]
So I need to remove useless data and then get a list of names with their unique values for "version".
I can get until the point where I have
{
"name": "ping",
"version": [
"1.0",
"1.0"
]
}
{
"name": "pong",
"version": [
"2.0",
"3.0",
"3.0"
]
}
{
"name": "pang",
"version": [
"2.0",
"2.0"
]
}
using
jq '.[] | {name: .function, version: [.servers[].version]}'
But I need to get rid of the duplicated values.
Is this possible using jq?
You were almost there. Just pipe the version array to the unique function:
jq '[.[]|{name, "version": [.servers[].version]|unique}]' input
Here is a solution that uses reduce to gather version keys in a temporary object and dedup at the end using keys
[
. []
| reduce .servers[].version as $v (
{name, version:{}}
; .version[$v] = 1
)
| .version |= keys
]