Nest array into new object with JQ - json

I know this one should be easy but it has me stumped. I am looking to take the following json example:
[{
"Name": "Test1",
"Version": "5.0.1",
"source": "source"
},
{
"Name": "Test2",
"Version": "2.0.11",
"source": "source"
},
{
"Name": "Test3",
"Version": "2.1.2",
"source": "source"
}]
and convert it to:
{
"packages": [
{
"Name": "Test1",
"Version": "5.0.1",
"source": "source"
},
{
"Name": "Test2",
"Version": "2.0.11",
"source": "source"
},
{
"Name": "Test3",
"Version": "2.1.2",
"source": "source"
}
]
}
I've tried numerous different ways, the closest I got is using something similar to: jq '.packages += [input]'
Basically it's just moving the original JSON to be nested. Any help would be appreciated.


just do this
jq '{ packages : . }' input.json


Even things you think of as literals are really filters in jq. The filter you need in this case is simply {packages: .}:
$ echo '[{}, {}]' | jq '{packages: .}'
{
"packages": [
{},
{}
]
}

Related

Bash match 2 json files based on common id

I wish to match 2 json files based on common id
I've tried using awk, jq and the npm json package in quite a lot of different ways but nothing have gotten close to working.
The 2 json files are not sorted and do not contain all the same entries. they contain the common networkId, I only want the output to contain the entries from file2.
Hope somebody can help!
Here's an example.
file1.json:
[
{
"customerId": "id1",
"networkId": "L_653021545945744804"
},
{
"customerId": "id2",
"networkId": "L_653021545955724805"
},
{
"customerId": "id3",
"networkId": "L_655051945958724557"
},
{
"customerId": "id4",
"networkId": "L_655567989968735408"
}
]
file2.json:
[
{
"name": "a",
"networkId": "L_653021545945744804"
},
{
"name": "b",
"networkId": "L_655051945958724557"
}
]
Wanted output:
[
{
"customerId": "id1",
"name": "a",
"networkId": "L_653021545945744804"
},
{
"customerId": "id3",
"name": "b",
"networkId": "L_655051945958724557"
}
]

This is a task for INDEX, JOIN and add:
jq '[JOIN(INDEX(.networkId); input[]; .networkId; add)]' file1.json file2.json
[
{
"name": "a",
"networkId": "L_653021545945744804",
"customerId": "id1"
},
{
"name": "b",
"networkId": "L_655051945958724557",
"customerId": "id3"
}
]
Demo

select and delete attributes with JQ

Info
I have a terraform state file (json) with some deprecated attributes.
I would like to remove theses deprecated attributes.
I try to use jq and select() && del() but did not succeed to get back my full json without the deprecated attribue timeouts.
Problem
How to get my full json without the attribute timeouts for only one type of resources google_dns_record_set.
Data
{
"version": 4,
"terraform_version": "1.0.6",
"serial": 635,
"lineage": "6a9c2392-fdae-2b54-adcc-7366f262ffa4",
"outputs": {"test":"test1"},
"resources": [
{
"module": "module.resources",
"mode": "data",
"type": "google_client_config"
},
{
"module": "module.xxx.module.module1[\"cluster\"]",
"mode": "managed",
"type": "google_dns_record_set",
"name": "public_ip_ic_dns",
"provider": "module.xxx.provider[\"registry.terraform.io/hashicorp/google\"]",
"instances": [
{
"schema_version": 0,
"attributes": {
"id": "projects/xxx-xxx/managedZones/xxx--public/rrsets/*.net1.cluster.xxx--public.net.com./A",
"managed_zone": "xxx--public",
"name": "*.net1.cluster.xxx--public.net.com.",
"project": "xxx-xxx",
"rrdatas": [
"11.22.33.44"
],
"timeouts": null,
"ttl": 300,
"type": "A"
},
"sensitive_attributes": [],
"private": "xxx",
"dependencies": [
"xxx"
]
}
]
}
]
}
Command
jq -r '.resources[] | select(.type=="google_dns_record_set").instances[].attributes | del(.timeouts)' data.json

Pull the del command up front to include the whole selection as its own filter
del(.resources[] | select(.type=="google_dns_record_set").instances[].attributes.timeouts)
Demo

How to parse input with jq

I have the list of docker images in json, for example:
{
"name": "chart1",
"version": "1.1.0",
"appVersion": "1.1.0",
"dependencies": [
{
"name": "name1",
"version": "10000.1.wew2133"
},
{
"name": "name2",
"version": "10001.1.wew2133"
}
]
}
I need to convert this to list:
name1:10000.1.wew2133
name2:10001.1.wew2133
How can I do this?

Use join(":"):
jq --raw-output '.dependencies[] | join(":")'
name1:10000.1.wew2133
name2:10001.1.wew2133
JQPlay demo

Convert JSON to CSV - string manipulation (jq, bash, awk, sed, etc.)

I'm in a dire need of help for a script to basically convert JSON text to CSV text in an attempt to copy users from one AWS Cognito userpool to another.
The export JSON looks like this:
{
"Users": [
{
"Username": "user.name",
"Attributes": [
{
"Name": "sub",
"Value": "some-value"
},
{
"Name": "email_verified",
"Value": "true"
},
{
"Name": "custom:jobtitle",
"Value": Director"
},
{
"Name": "custom:user_id",
"Value": "38"
},
{
"Name": "email",
"Value": "foo.bar#email.com"
}
],
"UserCreateDate": some-value,
"UserLastModifiedDate": some-value,
"Enabled": some-value,
"UserStatus": "some-value"
}
[more lines down here]...
] }
Then the CSV file would contain these lines:
,,,,,,,,,foo.bar#email.com,TRUE,,,,,,FALSE,,,Director,,38,FALSE,foo.bar
[more lines down here]...
So, the variables would be like this for JSON:
{
"Users": [
{
"Username": "%USERNAME%",
"Attributes": [
{
"Name": "sub",
"Value": "some-value"
},
{
"Name": "email_verified",
"Value": "true"
},
{
"Name": "custom:jobtitle",
"Value": %JOB_TITLE%"
},
{
"Name": "custom:user_id",
"Value": "%USER_ID%"
},
{
"Name": "email",
"Value": %EMAIL%"
}
],
"UserCreateDate": some-value,
"UserLastModifiedDate": some-value,
"Enabled": some-value,
"UserStatus": "some-value"
}
...
]
}
And like this for CSV:
,,,,,,,,,%EMAIL%,TRUE,,,,,,FALSE,,,%JOB_TITLE%,,%USER_ID%,FALSE,%USERNAME%
where %EMAIL%, %JOB_TITLE%, %USER_ID%, and %USERNAME% are variables, everything else should be just string.
Appreciate your help in advanced guys.

Consider first this filter:
.Users[].Attributes
| map(select(.Name | . == "custom:jobtitle" or . == "custom:user_id" or . == "email") )
| from_entries
| [ .email, .["custom:jobtitle"], .["custom:user_id"] ]
| #csv
The trick used here is the use of from_entries to convert the array of Name/Value pairs to an object with the Names as keys.
Assuming valid JSON input along the lines shown in the Q, invoking jq with the -r option would yield:
"foo.bar#email.com","Director","38"
Unfortunately the precise requirements are not so clear to me, but you should be able to adapt the above in accordance with your needs.

How can I index .JSON in elasticsearch

I am starting with elasticsearch now and i don't know anything about it.
I have folowing .JSON:
[
{
"label": "Admin Law",
"tags": [
"#admin"
],
"owner": "generalTopicTagText"
},
{
"label": "Judicial review",
"tags": [
"#JR"
],
"owner": "generalTopicTagText"
},
{
"label": "Admiralty/Shipping",
"tags": [
"#shipping"
],
"owner": "generalTopicTagText"
}
]
My mapping is this:
{
"topic_tax": {
"properties": {
"label": {
"type": "string",
"index": "not_analyzed"
},
"tags": {
"type": "string",
"index_name": "tag"
},
"owner": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
I need to put the first .Json into Elasticsearch, but it does not work.
All I know is that i am defining only 1 of this:
{
"label": "Judicial review",
"tags": [
"#JR"
],
"owner": "generalTopicTagText"
}
So when i try to put all of them with my elasticsearch.init, it will not work.
But I really don't know how to declare the mapping.Json to put the all .Json, it is like i need something like a for there.

You have to insert them json after json. But what you should do is use the bulk api of elasticsearch to insert multiple documents in one request. Check this api doc to see how it works

You can do something like this
curl -XPUT 'localhost:9000/es/post/1?version=2' -d '{
"text" : "your test message!"
}'
here is the documentation for index json with elasticsearch