The intent for the JSON data below is to update the value of the field dst with the value of src within all elements of type t, regardless of depth within the tree, while at the same time preserving the whole structure of the data.
Is this possible with jq? My several attempts have boiled down to the following command that is not working to achieve the intended purpose:
$ jq -r 'map_values(select(.. | .type? == "t" |= (.dst = .src)))'
{
"a": "b",
"c": [
{
"type": "t",
"src": "xx",
"dst": "zz"
},
{
"type": "t",
"src": "xx",
"dst": "zz"
}
],
"d": [
{
"e": [
{
"type": "t",
"src": "xx",
"dst": "zz"
}
]
},
{
"type": "t2",
"src": "xx",
"dst": "zz"
}
]
}
Is this possible with jq?
jq is Turing-complete :-)
Here's a simple solution:
walk( if type == "object" and .type == "t" then .dst = .src else . end)
If your jq does not have walk/1, then it might be a good time to upgrade (to jq 1.6); otherwise, you can snarf its def from the web, e.g. by googling: jq "def walk"
Alternatively ...
reduce paths as $x (.;
if (getpath($x)|.type? // false) == "t"
then setpath( $x + ["dst"]; getpath( $x + ["src"] ))
else . end)
Related
I got the following data:
{
"things": [
{
"name": "lkj",
"something": [
"hike"
],
"more_data": "important",
"other_stuff": "very important"
},
{
"name": "iou",
"different_more_data": "very important too",
"more_different_data": [
"even more"
]
}
]
}
Each of things has an id called "name", with jq I can edit it like:
jq '(.things[]) |= {name,something:["changed"]}'
{
"things": [
{
"name": "lkj",
"something": [
"changed"
]
},
...
Unfortunately I lose everything not declared in the right hand of the assignation operation.
Is there a way to make assignations without losing data? So that the result is like this:
{
"things": [
{
"name": "lkj",
"something": [
"changed"
],
"more_data": "important",
"other_stuff": "very important"
},
{
"name": "iou",
"something": [
"changed"
],
"different_more_data": "very important too",
"more_different_data": [
"even more"
]
}
]
}
You can simply modify your query so that it looks like:
.things[] |= (.something = ["changed"])
You can also use |= (or one of its siblings, such as +=) instead of = in the RHS expression, e.g.
.things[] |= (.something += ["changed"])
If you want to update some, but not all, items, you can still use the above forms. A straightforward approach is to use if ... then ... else ... end, for example:
.things[] |= (if .name == "lkj" then .something = ["changed"] else . end)
Using select on the LHS of |=
jq (or at least jq since version 1.4) does support the use of select on the LHS of |=, e.g.
(.things[] | select(.name=="lkj")) |= (.something += ["changed"])
With jq's map function:
jq '.things |= map(.something = ["changed"])' jsonfile
map(x) - apply specified filter x for each item of the input array
.something = ["changed"] - set key something to an object with array ["changed"] as a value
The output:
{
"things": [
{
"name": "lkj",
"something": [
"changed"
],
"more_data": "important",
"other_stuff": "very important"
},
{
"name": "iou",
"different_more_data": "very important too",
"more_different_data": [
"even more"
],
"something": [
"changed"
]
}
]
}
I have two questions:
How can I use jq to search for "name" fields that start with an underscore (like _RDS_PASSWORD) and remove the leading underscore (so it becomes RDS_PASSWORD)
How can I use jq for "name" fields that start with an underscore (like _RDS_PASSWORD) and pass the value of the value cGFzc3dvcmQK to be decoded via base64? (ex: "cGFzc3dvcmQK" | base64 --decode)
Input:
[
{
"name": "RDS_DB_NAME",
"value": "rds_db_name"
},
{
"name": "RDS_HOSTNAME",
"value": "rds_hostname"
},
{
"name": "RDS_PORT",
"value": "1234"
},
{
"name": "RDS_USERNAME",
"value": "rds_username"
},
{
"name": "_RDS_PASSWORD",
"value": "cGFzc3dvcmQK"
}
]
Desired output:
[
{
"name": "RDS_DB_NAME",
"value": "rds_db_name"
},
{
"name": "RDS_HOSTNAME",
"value": "rds_hostname"
},
{
"name": "RDS_PORT",
"value": "1234"
},
{
"name": "RDS_USERNAME",
"value": "rds_username"
},
{
"name": "RDS_PASSWORD",
"value": "password"
}
]
Q1
walk( if type=="object" and has("name") and .name[0:1] == "_"
then .name |= .[1:]
else .
end)
If your jq does not have walk/1 then you can either upgrade to a more recent version of jq than 1.5, or include its def, which can be found at https://github.com/stedolan/jq/blob/master/src/builtin.jq
Q2
.. | objects | select(has("name") and .name[0:1] == "_") | .value
If you are certain that the encoded string was a UTF-8 string, you could use jq's #base64d; otherwise, invoke jq with the -r option and pipe the results to a decoder as you indicated you planned to do.
I have a JSON document that looks like the following. Note this is a simplified example of the real JSON, which is included at bottom of question:
{
"some_array": [
{
"k1": "A",
"k2": "XXX"
},
{
"k1": "B",
"k2": "YYY"
}
]
}
I would like to change the value of all the k2 keys in the some_array array where the value of the k1 key is "B".
Is this possible using jq ?
For reference this is the actual JSON document, which is an environment variable file for use in postman / newman tool. I am attempting this conversion using JQ because the tool does not yet support command line overrides of specific environment variables
Actual JSON
{
"name": "Local-Stack-Env-Config",
"values": [
{
"enabled": true,
"key": "KC_master_host",
"type": "text",
"value": "http://localhost:8087"
},
{
"enabled": true,
"key": "KC_user_guid",
"type": "text",
"value": "11111111-1111-1111-1111-11111111111"
}
],
"timestamp": 1502768145037,
"_postman_variable_scope": "environment",
"_postman_exported_at": "2017-08-15T03:36:41.474Z",
"_postman_exported_using": "Postman/5.1.3"
}
Here is a slightly simpler version of zayquan's filter:
.some_array |= map(if .k1=="B" then .k2="changed" else . end)
Here's another solution.
jq '(.some_array[] | select(.k1 == "B") | .k2) |= "new_value"'
Output
{
"some_array": [
{
"k1": "A",
"k2": "XXX"
},
{
"k1": "B",
"k2": "new_value"
}
]
}
Here is a viable solution:
cat some.json | jq '.some_array = (.some_array | map(if .k1 == "B" then . + {"k2":"changed"} else . end))'
produces the output:
"some_array": [
{
"k1": "A",
"k2": "XXX"
},
{
"k1": "B",
"k2": "changed"
}
]
}
I have a json file that I need to convert to a csv file, but I am a little wary of trusting a json-to-csv converter site as the outputted data seems to be incorrect... so I was hoping to get some help here!
I have the following json file structure:
{
"GroupName": "GrpName13",
"Number": 3,
"Notes": "Test Group ",
"Units": [
{
"UnitNumber": "TestUnit13",
"DataSource": "Factory",
"ContractNumber": "TestContract13",
"CarNumber": "2",
"ControllerTypeMessageId" : 4,
"NumberOfLandings": 4,
"CreatedBy": "user1",
"CommissionModeMessageId": 2,
"Details": [
{
"DetailName": "TestFloor13",
"DetailNumber": "5"
}
],
"UnitDevices": [
{
"DeviceTypeMessageId": 1,
"CreatedBy": "user1"
}
]
}
]
}
The issue I think Im seeing is that the converters seem to not be able to comprehend the many nested data values. And the reason I think the converters are wrong is because when I try to convert back to json using them, I dont receive the same structure.
Does anyone know how to manually format this json into csv format, or know of a reliable converter than can handle nested values?
Try
www.json-buddy.com/convert-json-csv-xml.htm
if not working for you then you can try this tool
http://download.cnet.com/JSON-to-CSV/3000-2383_4-76680683.html
should be helpful!
I have tried your json on this for url:
http://www.convertcsv.com/json-to-csv.htm
As a result:
UnitNumber,DataSource,ContractNumber,CarNumber,ControllerTypeMessageId,NumberOfLandings,CreatedBy,CommissionModeMessageId,Details/0/DetailName,Details/0/DetailNumber,UnitDevices/0/DeviceTypeMessageId,UnitDevices/0/CreatedBy
TestUnit13,Factory,TestContract13,2,4,4,user1,2,TestFloor13,5,1,user1
Because it could save the path of the key,like the 'DeviceTypeMessageId' in list 'UnitDevices': it will named the columns name with 'UnitDevices/0/DeviceTypeMessageId', this could avoid the same name mistake, so you can get the columns name by its converter rules.
Hope helpful.
Here is a solution using jq
If the file filter.jq contains
def denormalize:
def headers($p):
keys_unsorted[] as $k
| if .[$k]|type == "array" then (.[$k]|first|headers("\($p)\($k)_"))
else "\($p)\($k)"
end
;
def setup:
[
keys_unsorted[] as $k
| if .[$k]|type == "array" then [ .[$k][]| setup ]
else .[$k]
end
]
;
def iter:
if length == 0 then []
elif .[0]|type != "array" then
[.[0]] + (.[1:] | iter)
else
(.[0][] | iter) as $x
| (.[1:] | iter) as $y
| [$x[]] + $y
end
;
[ headers("") ], (setup | iter)
;
denormalize | #csv
and data.json contains (note extra samples added)
{
"GroupName": "GrpName13",
"Notes": "Test Group ",
"Number": 3,
"Units": [
{
"CarNumber": "2",
"CommissionModeMessageId": 2,
"ContractNumber": "TestContract13",
"ControllerTypeMessageId": 4,
"CreatedBy": "user1",
"DataSource": "Factory",
"Details": [
{
"DetailName": "TestFloor13",
"DetailNumber": "5"
}
],
"NumberOfLandings": 4,
"UnitDevices": [
{
"CreatedBy": "user1",
"DeviceTypeMessageId": 1
},
{
"CreatedBy": "user10",
"DeviceTypeMessageId": 10
}
],
"UnitNumber": "TestUnit13"
},
{
"CarNumber": "99",
"CommissionModeMessageId": 99,
"ContractNumber": "Contract99",
"ControllerTypeMessageId": 99,
"CreatedBy": "user99",
"DataSource": "Another Factory",
"Details": [
{
"DetailName": "TestFloor99",
"DetailNumber": "99"
}
],
"NumberOfLandings": 99,
"UnitDevices": [
{
"CreatedBy": "user99",
"DeviceTypeMessageId": 99
}
],
"UnitNumber": "Unit99"
}
]
}
then the command
jq -M -r -f filter.jq data.json
will produce
"GroupName","Notes","Number","Units_CarNumber","Units_CommissionModeMessageId","Units_ContractNumber","Units_ControllerTypeMessageId","Units_CreatedBy","Units_DataSource","Units_Details_DetailName","Units_Details_DetailNumber","Units_NumberOfLandings","Units_UnitDevices_CreatedBy","Units_UnitDevices_DeviceTypeMessageId","Units_UnitNumber"
"GrpName13","Test Group ",3,"2",2,"TestContract13",4,"user1","Factory","TestFloor13","5",4,"user1",1,"TestUnit13"
"GrpName13","Test Group ",3,"2",2,"TestContract13",4,"user1","Factory","TestFloor13","5",4,"user10",10,"TestUnit13"
"GrpName13","Test Group ",3,"99",99,"Contract99",99,"user99","Another Factory","TestFloor99","99",99,"user99",99,"Unit99"
jq strikes again. Trying to get the value of DATABASES_DEFAULT based on the name in a json file that has a whole lot of names and I'm completely lost.
My file looks like the following (output of an aws ecs describe-task-definition) only much more complex; I've stripped this to the most basic example I can where the structure is still intact.
{
"taskDefinition": {
"status": "bar",
"family": "bar2",
"volumes": [],
"taskDefinitionArn": "bar3",
"containerDefinitions": [
{
"dnsSearchDomains": [],
"environment": [
{
"name": "bar4",
"value": "bar5"
},
{
"name": "bar6",
"value": "bar7"
},
{
"name": "DATABASES_DEFAULT",
"value": "foo"
}
],
"name": "baz",
"links": []
},
{
"dnsSearchDomains": [],
"environment": [
{
"name": "bar4",
"value": "bar5"
},
{
"name": "bar6",
"value": "bar7"
},
{
"name": "DATABASES_DEFAULT",
"value": "foo2"
}
],
"name": "boo",
"links": []
}
],
"revision": 1
}
}
I need the value of DATABASES_DEFAULT where the name is baz. Note that there are a lot of keypairs with name, I'm specifically talking about the one outside of environment.
I've been tinkering with this but only got this far before realizing that I don't understand how to access nested values.
jq '.[] | select(.name==DATABASES_DEFAULT) | .value'
which is returning
jq: error: DATABASES_DEFAULT/0 is not defined at <top-level>, line 1:
.[] | select(.name==DATABASES_DEFAULT) | .value
jq: 1 compile error
Obviously this a) doesn't work, and b) even if it did, it's independant of the name value. My thought was to return all the db defaults and then identify the one with baz, but I don't know if that's the right approach.
I like to think of it as digging down into the structure, so first you open the outer layers:
.taskDefinition.containerDefinitions[]
Now select the one you want:
select(.name =="baz")
Open the inner structure:
.environment[]
Select the desired object:
select(.name == "DATABASES_DEFAULT")
Choose the key you want:
.value
Taken together:
parse.jq
.taskDefinition.containerDefinitions[] |
select(.name =="baz") |
.environment[] |
select(.name == "DATABASES_DEFAULT") |
.value
Run it like this:
<infile jq -f parse.jq
Output:
"foo"
The following seems to work:
.taskDefinition.containerDefinitions[] |
select(
select(
.environment[] | .name == "DATABASES_DEFAULT"
).name == "baz"
)
The output is the object with the name key mapped to "baz".
$ jq '.taskDefinition.containerDefinitions[] | select(select(.environment[]|.name == "DATABASES_DEFAULT").name=="baz")' tmp.json
{
"dnsSearchDomains": [],
"environment": [
{
"name": "bar4",
"value": "bar5"
},
{
"name": "bar6",
"value": "bar7"
},
{
"name": "DATABASES_DEFAULT",
"value": "foo"
}
],
"name": "baz",
"links": []
}