Key value to be added in nested array - json

This is my json which i am trying to parse using jq
{
"orgs": {
"org1": [{
"space": "landscape2",
"tag": "landscape2",
"manager": {
"destination": "destination1"
}},
{
"space": "landscape3",
"tag": "landscape3",
"manager": {
"destination1": "approuter",
"destination2": "approuter2"
}},
{
"space": "landscape4",
"app": "",
"l_port_increment": 25,
"host": "",
"port": ""
}],
"org2": [
{
"space": "landscape1",
"app": "ain-hana-chisel-cs",
"l_port_increment": 13,
"host": "10.03.76.234",
"port": "30044"
},
{
"space": "landscape3",
"tag": "landscape3",
"manager1": {
"destination5": "service-v2",
"destination6": "service-v2"
},
"manager": {
"destination": "destination1"
},
"l_port_increment": 25,
"host": "",
"port": ""
} ] } }
I am looking for an output as below where tag should be one of the filter.
{
"orgs: "org1",
"space": "landscape3",
"tag": "landscape3",
"manager": {
"destination1": "approuter",
"destination2": "approuter2"
}
}
{
"orgs": "org2",
"space": "landscape3",
"tag": "landscape3",
"manager1": {
"destination5": "service-v2",
"destination6": "service-v2"
},
"manager": {
"destination": "destination1"
},
"l_port_increment": 25,
"host": "",
"port": ""
}
Using this code, I can achieve the result partially, however unable to get the orgs details.
.orgs[] | .[ ] | select(."tag"=="landscape3") | select (."manager")
My actual output is as below. How to do it by using jq? I have tried several ways but unable tot get the desired output
{
"space": "landscape3",
"tag": "landscape3",
"manager": {
"destination1": "approuter",
"destination2": "approuter2"
}
}
{
"space": "landscape3",
"tag": "landscape3",
"manager1": {
"destination5": "service-v2",
"destination6": "service-v2"
},
"manager": {
"destination": "destination1"
},
"l_port_increment": 25,
"host": "",
"port": ""
}

Your description of the task is not much clear, but I guess you're looking for something like this:
.orgs
| keys_unsorted[] as $k
| {orgs:$k} + first(.[$k][] | select(has("manager") and .tag == "landscape3"))
For each key in orgs, selects the objects which have manager as key and tag: landscape key-value pair from corresponding array, adds orgs: <key> to the first and outputs the resulting object. If multiple objects are selected, the rest except the first one is omitted.

Related

How to search for a string inside an object and replace it with another value on a condition

I have a Json file with below content, I am trying to replace serviceName string from ca-visual-node to ca-visual-node-canary if host name matches test4.analytics.io using jq utility. How can I get this done so that we will have json file updated with replace string
{
"apiVersion": "extensions/v1beta1",
"kind": "Ingress",
"metadata": {
"annotations": {
"alb.ingress.kubernetes.io/actions.ssl-redirect": "{\"Type\": \"redirect\", \"RedirectConfig\": { \"Protocol\": \"HTTPS\", \"Port\": \"443\", \"StatusCode\": \"HTTP_301\"}}"
},
"finalizers": [
"ingress.k8s.aws/resources"
],
"name": "ca-visual",
"namespace": "cloud-anaytics",
},
"spec": {
"rules": [
{
"host": "test.analytics.io",
"http": {
"paths": [
{
"backend": {
"serviceName": "ca-visual-play",
"servicePort": 9443
},
"path": "/apii/*",
"pathType": "ImplementationSpecific"
},
{
"backend": {
"serviceName": "ssl-redirect",
"servicePort": "use-annotation"
},
"path": "/*",
"pathType": "ImplementationSpecific"
},
{
"backend": {
"serviceName": "ca-visual-node",
"servicePort": 443
},
"path": "/*",
"pathType": "ImplementationSpecific"
}
]
}
},
{
"host": "test4.analytics.io",
"http": {
"paths": [
{
"backend": {
"serviceName": "ca-visual-play",
"servicePort": 9443
},
"path": "/apii/*",
"pathType": "ImplementationSpecific"
},
{
"backend": {
"serviceName": "ssl-redirect",
"servicePort": "use-annotation"
},
"path": "/*",
"pathType": "ImplementationSpecific"
},
{
"backend": {
"serviceName": "ca-visual-node",
"servicePort": 443
},
"path": "/*",
"pathType": "ImplementationSpecific"
}
]
}
}
]
},
"status": {
"loadBalancer": {
"ingress": [
{
"hostname": "16b3-cloudanaytics.us-xxxx-1.elb.amazonaws.com"
}
]
}
}
}
I am trying to replace serviceName string from ca-visual-node to ca-visual-node-canary if host name matches test4.analytics.io using jq utility. How can I get this done so that we will have json file updated with replace string.
Just walk the path from the root and use the update operator |= with the filter wrapped around (..)
(
.spec.rules[] |
select(.host == "test4.analytics.io")? |
.http.paths[].backend |
select(.serviceName == "ca-visual-node")? |
.serviceName
) |= "ca-visual-node-canary"
jqplay - demo

Converting array to objects within an object

I have been battling with this for sometime, ive looked a few articles in regards to this topic. I can't seem to get exactly what I'm aiming for. I have this original json data that I would like to transform a bit.
Group the objects based on the label.env (for now only using one env), then inserting those objects from the original json into an object named projects.
orginial_json
{
"createTime": "2020-06-25T14:16:45.720Z",
"labels": {
"cb_domain": "cloud-services",
"cb_product": "infra",
"env": "prod",
"owner": "cloud-server"
},
"lifecycleState": "ACTIVE",
"name": "projectname",
"parent": {
"id": "123456577",
"type": "folder"
},
"projectId": "projectname-324234",
"projectNumber": "962363417856"
}
{
"createTime": "2020-06-24T19:45:42.851Z",
"labels": {
"cb_domain": "cloud-services",
"cb_product": "ad",
"env": "prod",
"owner": "cloud-server"
},
"lifecycleState": "ACTIVE",
"name": "projectname2",
"parent": {
"id": "4352564765437",
"type": "folder"
},
"projectId": "projectname2-4567",
"projectNumber": "3243456324"
}
Im using jq -s 'group_by(.labels.env) | .[] | { (.[0].labels.env) : { projects: .[] | . } }'
to nest the original json data into objects grouped by env\projects, which works but seems to duplicate it instead of putting each object within projects.
{
"prod": {
"projects": {
"createTime": "2020-06-25T14:16:45.720Z",
"labels": {
"cb_domain": "cloud-services",
"cb_product": "infra",
"env": "prod",
"owner": "cloud-server"
},
"lifecycleState": "ACTIVE",
"name": "projectname",
"parent": {
"id": "770593713153",
"type": "folder"
},
"projectId": "projectname-324234",
"projectNumber": "962363417856"
}
}
}
{
"prod": {
"projects": {
"createTime": "2020-06-24T19:45:42.851Z",
"labels": {
"cb_domain": "cloud-services",
"cb_product": "ad",
"env": "prod",
"owner": "cloud-server"
},
"lifecycleState": "ACTIVE",
"name": "projectname2",
"parent": {
"id": "4352564765437",
"type": "folder"
},
"projectId": "projectname2-4567",
"projectNumber": "3243456324"
}
}
}
What I'm aiming for is
{
"prod": {
"projects": {
"createTime": "2020-06-25T14:16:45.720Z",
"labels": {
"cb_domain": "cloud-services",
"cb_product": "infra",
"env": "prod",
"owner": "cloud-server"
},
"lifecycleState": "ACTIVE",
"name": "projectname",
"parent": {
"id": "770593713153",
"type": "folder"
},
"projectId": "projectname-324234",
"projectNumber": "962363417856"
},
{
"createTime": "2020-06-24T19:45:42.851Z",
"labels": {
"cb_domain": "cloud-services",
"cb_product": "ad",
"env": "prod",
"owner": "cloud-server"
},
"lifecycleState": "ACTIVE",
"name": "projectname2",
"parent": {
"id": "4352564765437",
"type": "folder"
},
"projectId": "projectname2-4567",
"projectNumber": "3243456324"
}
}
}
Allowing for some minor discrepancies in the Q, the following variant of your jq program seems to do what you're trying to achieve:
group_by(.labels.env)
| .[]
| .[0].labels.env as $key
| { ($key) : { projects: . } }

jq select objects that have specific value in nested json bash

this is my json.
[
{
"time": "2017-06-10 00:00:48-0400,317",
"UserInfo": {
"AppId": "ONE_SEARCH",
"UsageGroupId": "92600",
},
"Items": [
{
"PublicationCode": "",
"OpenUrlRefId": "",
"ReferringUrl": "N",
"OpenAccess": "0",
"ItmId": "1328515516"
}
]
},
{
"time": "2017-06-10 00:00:48-0400,548",
"UserInfo": {
"AppId": "DIALOG",
"UsageGroupId": "1195735",
},
"Items": [
{
"Origin": "Alert",
"PublicationCode": "",
"NumberOfCopies": 1,
"ItmId": "1907446549"
},
{
"Origin": "Alert",
"PublicationCode": "",
"NumberOfCopies": 1,
"ItmId": "1907446950",
}
]
}
]
I want use jq to extract the object that have "Origin": "Alert" in its element "Items". And the result should looks like this:
{
"time": "2017-06-10 00:00:48-0400,548",
"UserInfo": {
"AppId": "DIALOG",
"UsageGroupId": "1195735",
},
"Items": [
{
"Origin": "Alert",
"PublicationCode": "",
"NumberOfCopies": 1,
"ItmId": "1907446549"
},
{
"Origin": "Alert",
"PublicationCode": "",
"NumberOfCopies": 1,
"ItmId": "1907446950",
}
]
}
Or this:
{
"Items": [
{
"Origin": "Alert",
"PublicationCode": "",
"NumberOfCopies": 1,
"ItmId": "1907446549",
"ReasonCode": ""
},
{
"Origin": "Alert",
"PublicationCode": "",
"NumberOfCopies": 1,
"ItmId": "1907446950",
}
]
}
How to do it by using jq? I have tried several ways but most of them will just return an array with all children objects that include "Origin":"Alert". I need these children objects still keep there structure, because I need to know which of them happened together and which of them happened separately.
BTW, the only value of "Origin" is "Alert". So if you have any method to select an object with a given key name, it should also work.
Thank you! :)
The filter:
.[] | select( any(.Items[]; .Origin == "Alert"))
produces the first-mentioned admissible result. If your jq does not have any/2 then I'd suggest upgrading. If that's not an option, then you could use the following simple but rather inefficient filter instead:
.[] | select( .Items | map(.Origin) | index("Alert"))
Or:
.[] | select(reduce .Items[] as $item (false; . or ($item | .Origin == "Alert")))

Query parent data (multi-level) based on a child value, on a json file, using jq

I have a ksh script that retrives (using curl) a json file similar to the one bellow:
{
"Type1": {
"dev": {
"server": [
{ "group": "APP1", "name": "DAPP1002", "ip": "10.1.1.1" },
{ "group": "APP2", "name": "DAPP2001", "ip": "10.1.1.2" }
]
},
"qa": {
"server": [
{ "group": "APP1", "name": "QAPP1002", "ip": "10.1.2.1" },
{ "group": "APP2", "name": "QAPP2001", "ip": "10.1.2.2" }
]
},
"prod": {
"proxy": "type1.prod.proxy.mydomain.com",
"server": [
{ "group": "APP1", "name": "PAPP1001", "ip": "10.1.3.1" },
{ "group": "APP1", "name": "PAPP1002", "ip": "10.1.3.2" },
{ "group": "APP2", "name": "PAPP2001", "ip": "10.1.3.3" }
]
}
},
"Type2": {
"dev": {
"server": [
{ "group": "APP8", "name": "DAPP8002", "ip": "10.2.1.1" },
{ "group": "APP9", "name": "DAPP9001", "ip": "10.2.1.2" }
]
},
"qa": {
"server": [
{ "group": "APP8", "name": "QAPP8002", "ip": "10.2.2.1" },
{ "group": "APP9", "name": "QAPP9001", "ip": "10.2.2.2" }
]
},
"prod": {
"proxy": "type2.prod.proxy.mydomain.com",
"server": [
{ "group": "APP8", "name": "PAPP8001", "ip": "10.2.3.1" },
{ "group": "APP9", "name": "PAPP9001", "ip": "10.2.3.2" },
{ "group": "APP9", "name": "PAPP9002", "ip": "10.2.3.3" }
]
}
}
}
... based on a server name (field "name") I would have to collect the following info, to pass to a function:
"Type", "name", "ip", "proxy"
(Note that the "proxy" info is optional)
I am new to json, and I am trying to get this filtered with jq but so far, I am out of lucky.
What I acomplished so far is the following jq query, when searching for "PAPP9001" :
jq '.[] | .[] | select(.server[].name=="PAPP9001") | .proxy as $proxy | .server[] | {proxy: $proxy, name: .name, ip: .ip} | select(.name=="PAPP9001")' curlreturn.json
which returns me:
{
"proxy": "type2.prod.proxy.mydomain.com",
"name": "PAPP9001",
"ip": "10.2.3.2"
}
but:
I could not get the "Type" info, at the top level
Considering the number of pipes and the 2 selects, I doubt that this is the most efficient way.
One way to retrieve the key names programmatically is using to_entries. For example, given your input, this jq filter:
to_entries[]
| .key as $type
| .value[]
| .proxy as $proxy
| .server[]
| select(.name == "PAPP9001")
| { Type: $type, name, ip, proxy: $proxy }
yields:
{
"Type": "Type2",
"name": "PAPP9001",
"ip": "10.2.3.2",
"proxy": "type2.prod.proxy.mydomain.com"
}
Variations
If, for example, you wanted these four fields as a CSV row, then you could replace the last line of the filter above with:
| [$type, .name, .ip, $proxy] | #csv
See the jq manual for how to use string interpolation.

Extract values in json object with awk/sed, but cannot get it to work

I have a file with the return of a curl statement in it, in the form of json. Each object has a set of values, but the parameters for these values are all called the same names. See code below.
These objects are part of a larger object called workflow. The Cleaning up object is the last process that runs in our workflow. For every video that passes through the workflow, a json file in this format is created. (There are more than only these three objects, this is just for illustrative purposes)
I want to take the value of completed of the object with "description": "Cleaning up" and store it as a variable $end_time. Then I want to take the value of completed of the object with "description": "Ingest" and store it as a variable $start_time. These two values are then subtracted to give me an integer time in milliseconds so I can calculate the time it took for the video to go through this part of the process. With the maths part I am fine, and know how to do it. It is the extraction of the values that I am struggling with.
I hope this makes sense? ANY help would be appreciated. Thank you in advance!
EDIT: Had to delete original code in post, due to character limitations
Here is a proper example of the file that I have to work with:
{
"workflows": {
"count": "20",
"searchTime": "1",
"startPage": "0",
"totalCount": "1",
"workflow": {
"configurations": {
"configuration": [
{
"$": "1409750880000",
"key": "schedule.start"
},
{
"$": "1409755980000",
"key": "schedule.stop"
},
{
"$": "Capture_agent",
"key": "schedule.location"
},
{
"$": "false",
"key": "trimHold"
},
{
"$": "true",
"key": "archiveOp"
},
{
"$": "false",
"key": "captionHold"
},
{
"$": "false",
"key": "videoPreview"
}
]
},
"creator": {
"organization": "mh_default_org",
"roles": [
"76b1bdde-a080-40a4-b929-bde89af6a0a8_Instructor",
"ROLE_ADMIN",
"ROLE_ANONYMOUS",
"ROLE_USER"
],
"userName": user_name
},
"description": "This workflow definition defines the steps involved in scheduling a recording, capturing it, and\n ingesting it, after which processing operations may be added.\n ",
"errors": "",
"id": "15518",
"mediapackage": {
"attachments": "",
"creators": {
"creator": "Name"
},
"id": "2d25ed19-2978-458d-a4a0-c9c56d791c68",
"license": "Creative Commons 3.0: Attribution-NonCommercial-NoDerivs",
"media": "",
"metadata": "",
"publications": {
"publication": {
"channel": "engage-player",
"id": "b7b68f91-2c33-4673-ba7c-2e9b891788f9",
"mimetype": "text/html",
"tags": "",
"url": "http://some.url.com:80/engage/ui/watch.html?id=2d25ed19-2978-458d-a4a0-c9c56d791c68"
}
},
"series": "76b1bdde-a080-40a4-b929-bde89af6a0a8",
"seriestitle": "Recording_Title_user_name",
"start": "2014-09-03T13:28:00Z",
"title": "Recording_Title"
},
"operations": {
"operation": [
{
"abortable": "false",
"completed": 1409750882092,
"configurations": {
"configuration": [
{
"$": "1409750880000",
"key": "schedule.start"
},
{
"$": "1409755980000",
"key": "schedule.stop"
},
{
"$": "Capture_agent",
"key": "schedule.location"
}
]
},
"continuable": "false",
"description": "Scheduled",
"execution-history": "",
"execution-host": "http://some.url.com:8080",
"fail-on-error": "true",
"failed-attempts": "0",
"hold-action-title": "View schedule",
"holdurl": "/workflow/hold/org.opencastproject.workflow.handler.scheduleworkflowoperationhandler",
"id": "schedule",
"job": "15519",
"max-attempts": "1",
"retry-strategy": "none",
"started": 1409750881745,
"state": "SUCCEEDED",
"time-in-queue": 0
},
{
"abortable": "false",
"configurations": "",
"continuable": "false",
"description": "Capture",
"execution-history": "",
"execution-host": "http://some.url.com:8080",
"fail-on-error": "true",
"failed-attempts": "0",
"hold-action-title": "Monitor capture",
"holdurl": "/workflow/hold/org.opencastproject.workflow.handler.captureworkflowoperationhandler",
"id": "capture",
"job": "42894",
"max-attempts": "1",
"retry-strategy": "none",
"started": 1409750884085,
"state": "SKIPPED",
"time-in-queue": 0
},
{
"completed": 1409756171224,
"configurations": "",
"description": "Ingest",
"execution-history": "",
"fail-on-error": "true",
"failed-attempts": "0",
"id": "ingest",
"max-attempts": "1",
"retry-strategy": "none",
"state": "SUCCEEDED"
},
{
"completed": 1409854379552,
"configurations": {
"configuration": {
"key": "preserve-flavors"
}
},
"description": "Cleaning up",
"execution-history": "",
"execution-host": "http://some.url.com:8080",
"fail-on-error": "false",
"failed-attempts": "0",
"id": "cleanup",
"job": "45113",
"max-attempts": "1",
"retry-strategy": "none",
"started": 1409854378128,
"state": "SUCCEEDED",
"time-in-queue": 0
}
]
},
"organization": {
"adminRole": "ROLE_ADMIN",
"anonymousRole": "ROLE_ANONYMOUS",
"id": "mh_default_org",
"name": "Opencast Project",
"properties": {
"property": [
{
"$": "true",
"key": "adminui.i18n_tab_episode.enable"
},
{
"$": "false",
"key": "adminui.i18n_tab_users.enable"
},
{
"$": "/engage/ui/img/mh_logos/OpencastLogo.png",
"key": "logo_small"
},
{
"$": "http://opencast.org/matterhorn/",
"key": "engageui.link_mobile_redirect.url"
},
{
"$": "false",
"key": "engageui.annotations.enable"
},
{
"$": "true",
"key": "engageui.links_media_module.enable"
},
{
"$": "2024",
"key": "adminui.chunksize"
},
{
"$": "false",
"key": "adminui.series_prepopulate.enable"
},
{
"$": "true",
"key": "engageui.link_download.enable"
},
{
"$": "false",
"key": "engageui.link_mobile_redirect.enable"
},
{
"$": "For more information have a look at the official site.",
"key": "engageui.link_mobile_redirect.description"
},
{
"$": "/engage/ui/img/mh_logos/MatterhornLogo_large.png",
"key": "logo_large"
}
]
},
"servers": {
"server": {
"name": "localhost",
"port": "8080"
}
}
},
"parent": {
"nil": "true"
},
"state": "SUCCEEDED",
"template": "full",
"title": "Scheduled Workflow"
}
}
}
Here is a jq example that should point you to getting what you want:
#!/bin/bash
# Assuming the json is in a file workflow.json
end_time=$( jq '.workflows.workflow.operations.operation[] | select(.description == "Cleaning up") | .completed' < workflow.json )
start_time=$( jq '.workflows.workflow.operations.operation[] | select(.description == "Ingest") | .completed' < workflow.json )
This is assuming the input you have is in an JSON array called workflow at the top level. Here's this on the command line:
$ jq '.workflows.workflow.operations.operation[] | select(.description == "Ingest") | .completed' < workflow.json
1406051539118
$ jq '.workflows.workflow.operations.operation[] | select(.description == "Cleaning up") | .completed' < workflow.json
1406051695440