For example,
Input:
{
"id":"abc",
"name": "name-middlenane-lastname-1"
},
{
"id":"123",
"name": "fname-flast-2"
}
response:
{
"id":"abc",
"name": "name-middlename-lastname-1",
"newkey": "name-middlename-lastname"
},
{
"id":"123",
"name": "fname-flast-2",
"newkey": "fname-flast"
}
The filed name in each object is a string with characters and numbers separated by "-" hyphen. I need the complete string from beginning till the starting number. I don't want anything which is there after the number. And then the add new field with key as newkey and value should be extracted string without the number. Thus, the output should contain old fields as well as new one.
jq solution:
Sample input.json:
[
{
"id": "abc",
"name": "name-middlenane-lastname-1"
},
{
"id": "123",
"name": "fname-flast-2"
}
]
jq 'map(. + (.name | capture("(?<newkey>.+)-[0-9]+")) )' input.json
The output:
[
{
"id": "abc",
"name": "name-middlenane-lastname-1",
"newkey": "name-middlenane-lastname"
},
{
"id": "123",
"name": "fname-flast-2",
"newkey": "fname-flast"
}
]
Related
I have following JSON
{
"ID": "123",
"Name": "xyz"
}
{
"ID": "1234",
"Competitor": "abc"
}
Need to convert the json objects into a array of json objects:
[
{
"ID": "123",
"Name": "xyz"
}
{
"ID": "1234",
"Competitor": "abc"
}
]
I have tried using:
. | [{ID, Name}]
But this converts into multiple arrays
[
{
"ID": "123",
"Name": "xyz"
}
]
[
{
"ID": "1234",
"Competitor": "abc"
}
]
jq applies the given filter to each JSON document/entity. A file can contain multiple such documents. If you want to handle a file with multiple documents as a single array document with each original document being an array element, then use -s/--slurp option which will read all JSON documents in your input file as one huge array:
jq -s '.' input.json
Output:
[
{
"ID": "123",
"Name": "xyz"
},
{
"ID": "1234",
"Competitor": "abc"
}
]
Note that jq requires valid JSON, so your input must be valid. The sample in the question is not. A valid JSON input would be:
{
"ID": "123",
"Name": "xyz"
}
{
"ID": "1234",
"Competitor": "abc"
}
PS Note that . | [{ID, Name}] is equivalent to [{ID, Name}] since . is the identity filter.
I am new to jq and I'm trying to use it to search for a value in a json file based on a key that is located deep in the json structure. Here is a sample of my json file:
{
"data": {
"inventory": {
"location": "remote",
"list": {
"content": [
{
"item": {
"name": "minivan"
},
"owner": {
"id": "12345",
"state": "CA"
}
},
{
"item": {
"name": "sedan"
},
"owner": {
"id": "67890",
"state": "AZ"
}
}
]
}
}
}
}
An example of search that I'm trying to do is:
select item.name where owner.id = "67890"
and the expected output would be:
item.name = "sedan"
I'm trying to run the following:
jq '.[] | select .owner.id = "67890" | .item.name' json
and it generates an error:
jq: error: select/0 is not defined at <top-level>, line 1:
.[] | select .owner.id = "67890" | .item.name
jq: 1 compile error
Any pointers on how to do this in jq would be much appreciated!
Thanks!
First, you have to "navigate" to where you want to make the query. This seems to be an array.
.data.inventory.list.content
[
{
"item": {
"name": "minivan"
},
"owner": {
"id": "12345",
"state": "CA"
}
},
{
"item": {
"name": "sedan"
},
"owner": {
"id": "67890",
"state": "AZ"
}
}
]
Demo
Next, let's iterate over that array's items, which gives us a stream of objects.
.[]
{
"item": {
"name": "minivan"
},
"owner": {
"id": "12345",
"state": "CA"
}
}
{
"item": {
"name": "sedan"
},
"owner": {
"id": "67890",
"state": "AZ"
}
}
Demo
From these objects we select those that match your criteria.
select(.owner.id == "67890")
{
"item": {
"name": "sedan"
},
"owner": {
"id": "67890",
"state": "AZ"
}
}
Demo
Finally, we extract the value you're interested in.
.item.name
"sedan"
Demo
Everything combined in a jq call would be:
jq '.data.inventory.list.content[] | select(.owner.id == "67890").item.name'
"sedan"
Demo
This output is still valid JSON document (containing nothing but a JSON string). If you want to process the output as raw text, use the --raw-output (or -r) option:
jq -r '.data.inventory.list.content[] | select(.owner.id == "67890").item.name'
sedan
Demo
Here's a solution that avoids having to "navigate" to the right place, and which is also quite close to your SQL-like query:
..
| objects
| select(.owner and
(.owner|type=="object" and .id == "67890"))
.item.name
or more succinctly:
..|objects|select(.owner.id? == "67890").item.name
I'd like to select/identity-output all objects in arrays under "emp" keys into a flat array of those objects.
[
{
"eng": {
"dev": {
"dir": {
"name": "Mickey"
},
"emp": [
{
"name": "Goofy",
"job": "laugh",
"start": "today"
},
{
"name": "Minnie",
"job": "laugh"
}
]
}
}
},
{
"mgmt": {
"dir": {
"name": "Donald"
},
"emp": [
{
"name": "Woody",
"job": "smile"
},
{
"name": "Buzz",
"job": "smile"
}
]
}
}
]
I'm looking for a flat array of arbitrary objects found in arbitrary locations within the document (in this example, under "emp" parent/keys).
In this example, it would look like
[
{
"name": "Goofy",
"job": "laugh",
"start": "today"
},
{
"name": "Minnie",
"job": "laugh"
},
{
"name": "Woody",
"job": "smile"
},
{
"name": "Buzz",
"job": "smile"
}
]
I've looked through a lot of documentation and am able to do this if I know in advance precisely where these 'emp' keys are in the document, but not if they're distributed through the document at a priori unknown locations/paths.
Use recurse to walk the structure. From all the substrucures, select objects with the emp key. Output the corresponding values and merge the resulting arrays.
jq '[recurse | select (type == "object" and .emp) | .emp ] | add' file.json
I have been playing around with jq to format a json file but I am having some issues trying to solve a particular transformation. Given a test.json file in this format:
[
{
"name": "A", // This would be the first key
"number": 1,
"type": "apple",
"city": "NYC" // This would be the second key
},
{
"name": "A",
"number": "5",
"type": "apple",
"city": "LA"
},
{
"name": "A",
"number": 2,
"type": "apple",
"city": "NYC"
},
{
"name": "B",
"number": 3,
"type": "apple",
"city": "NYC"
}
]
I was wondering, how can I format it this way using jq?
[
{
"key": "A",
"values": [
{
"key": "NYC",
"values": [
{
"number": 1,
"type": "a"
},
{
"number": 2,
"type": "b"
}
]
},
{
"key": "LA",
"values": [
{
"number": 5,
"type": "b"
}
]
}
]
},
{
"key": "B",
"values": [
{
"key": "NYC",
"values": [
{
"number": 3,
"type": "apple"
}
]
}
]
}
]
I have followed this thread Using jq, convert array of name/value pairs to object with named keys and tried to group the json using this expression
jq '. | group_by(.name) | group_by(.city) ' ./test.json
but I have not been able to add the keys in the output.
You'll want to group the items at the different levels and building out your result objects as you want.
group_by(.name) | map({
key: .[0].name,
values: (group_by(.city) | map({
key: .[0].city,
values: map({number,type})
}))
})
Just keep in mind that group_by/1 yields groups in a sorted order. You'll probably want an implementation that preserves that order.
def group_by_unsorted(key_selector):
reduce .[] as $i ({};
.["\($i|key_selector)"] += [$i]
)|[.[]];
I'm in a dire need of help for a script to basically convert JSON text to CSV text in an attempt to copy users from one AWS Cognito userpool to another.
The export JSON looks like this:
{
"Users": [
{
"Username": "user.name",
"Attributes": [
{
"Name": "sub",
"Value": "some-value"
},
{
"Name": "email_verified",
"Value": "true"
},
{
"Name": "custom:jobtitle",
"Value": Director"
},
{
"Name": "custom:user_id",
"Value": "38"
},
{
"Name": "email",
"Value": "foo.bar#email.com"
}
],
"UserCreateDate": some-value,
"UserLastModifiedDate": some-value,
"Enabled": some-value,
"UserStatus": "some-value"
}
[more lines down here]...
] }
Then the CSV file would contain these lines:
,,,,,,,,,foo.bar#email.com,TRUE,,,,,,FALSE,,,Director,,38,FALSE,foo.bar
[more lines down here]...
So, the variables would be like this for JSON:
{
"Users": [
{
"Username": "%USERNAME%",
"Attributes": [
{
"Name": "sub",
"Value": "some-value"
},
{
"Name": "email_verified",
"Value": "true"
},
{
"Name": "custom:jobtitle",
"Value": %JOB_TITLE%"
},
{
"Name": "custom:user_id",
"Value": "%USER_ID%"
},
{
"Name": "email",
"Value": %EMAIL%"
}
],
"UserCreateDate": some-value,
"UserLastModifiedDate": some-value,
"Enabled": some-value,
"UserStatus": "some-value"
}
...
]
}
And like this for CSV:
,,,,,,,,,%EMAIL%,TRUE,,,,,,FALSE,,,%JOB_TITLE%,,%USER_ID%,FALSE,%USERNAME%
where %EMAIL%, %JOB_TITLE%, %USER_ID%, and %USERNAME% are variables, everything else should be just string.
Appreciate your help in advanced guys.
Consider first this filter:
.Users[].Attributes
| map(select(.Name | . == "custom:jobtitle" or . == "custom:user_id" or . == "email") )
| from_entries
| [ .email, .["custom:jobtitle"], .["custom:user_id"] ]
| #csv
The trick used here is the use of from_entries to convert the array of Name/Value pairs to an object with the Names as keys.
Assuming valid JSON input along the lines shown in the Q, invoking jq with the -r option would yield:
"foo.bar#email.com","Director","38"
Unfortunately the precise requirements are not so clear to me, but you should be able to adapt the above in accordance with your needs.