I have some JSON data and i want to push some of them to DB , but sometimes the json values not exists for specific devices:
from all of the following data I want just to pull , "ICCID","MDN","MSISDN","MCC","MNC","FeatureTypes","RatePlanCode","RatePlanDescription","DeviceState","BillingCycleStartDate","BillingCycleEndDate","CurrentBillCycleDataUnRatedUsage"
and if any one not exist print not-exist .
{
"categories": [{
"categoryName": "DeviceIdentifier",
"extendedAttributes": [{
"key": "ICCID",
"value": "89148000"
},
{
"key": "IMSI",
"value": "31148094"
},
{
"key": "MDN",
"value": "5514048068"
},
{
"key": "MEID",
"value": "A0000000005006"
},
{
"key": "MIN",
"value": "5514041185"
}
]
},
{
"categoryName": "DeviceAttributes",
"extendedAttributes": [{
"key": "MCC",
"value": "311"
},
{
"key": "MNC",
"value": "480"
},
{
"key": "FeatureCodes",
"value": "75802,84777,54307"
},
{
"key": "FeatureNames",
"value": "75802,84777,54307"
},
{
"key": "FeatureTypes",
"value": "4G Public Dynamic"
},
{
"key": "RatePlanCode",
"value": "4G5G "
},
{
"key": "RatePlanDescription",
"value": "4G5G"
},
{
"key": "Services",
"value": "null"
}
]
},
{
"categoryName": "Provisioning",
"extendedAttributes": [{
"key": "LastActivationDate",
"value": "2022-03-01T19:38:52Z"
},
{
"key": "CreatedAt",
"value": "2021-12-01T21:22:55Z"
},
{
"key": "DeviceState",
"value": "active"
},
{
"key": "LastDeactivationDate",
"value": "2021-12-01T21:22:55Z"
}
]
},
{
"categoryName": "Connectivity",
"extendedAttributes": [{
"key": "Connected",
"value": "true"
},
{
"key": "LastConnectionDate",
"value": "2022-09-08T03:38:55Z"
},
{
"key": "LastDisconnectDate",
"value": "2022-09-08T03:25:15Z"
}
]
},
{
"categoryName": "Billing",
"extendedAttributes": [{
"key": "BillingCycleStartDate",
"value": "2022-09-02T00:00:00Z"
},
{
"key": "BillingCycleEndDate",
"value": "2022-10-01T00:00:00Z"
},
{
"key": "DefaultRatePlan",
"value": "0"
}
]
},
{
"categoryName": "Usage",
"extendedAttributes": [{
"key": "CurrentRatedUsageRecordDate",
"value": "2022-09-04T00:00:00Z"
}, {
"key": "CurrentUnRatedUsageRecordDate",
"value": "2022-09-08T01:25:15Z"
},
{
"key": "CurrentBillCycleDataUnRatedUsage",
"value": "1698414605"
}
]
}
]
}
i'm not pushing all fields to db so i'm selecting a specific keys from that,
(what i'am selecting its fixed not changed) so the select will not change and always will be :
Expected output :
"89148000"
"5514048068"
"not-exist"
"4G Public Dynamic"
"4G5G"
"4G5G"
"active"
"2022-09-02T00:00:00Z"
"2022-10-01T00:00:00Z"
"2022-09-08T01:25:15Z"
I would like to check if the value of key is missing for this case "MSISDN" ,if not will print for me not-exist or null
any help ?
.categories[].Attributes[] |
if (.key | IN(["AAA","BBB","DDD","EEE"][]))
then .value
else "NOT-EXIST"
end
Gives the following output
"111"
"222"
"NOT-EXIST"
"444"
"555"
First we loop over the Attributes
Then we use an if to;
Check if key exist in ["AAA","BBB","DDD","EEE"]
TRUE: use .value
FALSE: use NOT-EXIST as value
Demo
Another approach, using with_entries() to update the .value before looping over all the objects to show just the value, gives the same output as above:
.categories[].Attributes[]
| select(.key | IN(["AAA","BBB","DDD","EEE"][]) | not).value = "NOT-EXIST"
| .value
Demo
I hope I understood your requirements correctly, but here is a solution that looks simple enough to understand and should be somewhat efficient. If you always expect the same 5 keys in your input, you can try:
.categories[].Attributes | from_entries as $attr
| ["AAA", "BBB", "CCC", "DDD", "EEE"]
| map($attr[.] // "NOT-EXIST")
Input:
{"categories": [
{
"categoryName": "Device",
"Attributes": [
{
"key": "AAA",
"value": "111"
},
{
"key": "BBB",
"value": "222"
},
{
"key": "DDD",
"value": "444"
},
{
"key": "EEE",
"value": "555"
}
]
}]}
Output:
[
"111",
"222",
"NOT-EXIST",
"444",
"555"
]
If you require only the values, add [] or | .[] at the end of the script or rewrite to:
.categories[].Attributes | from_entries as $attr
| "AAA", "BBB", "CCC", "DDD", "EEE"
| $attr[.] // "NOT-EXIST"
With the input from updated question, you intend to first merge all extendedAttributes array into one big array, convert to an object and then use this complete object to look up your values:
.categories | map(.extendedAttributes[]) | from_entries as $attr
| "ICCID", "MDN", "MSISDN", "MCC", "MNC", "FeatureTypes", "RatePlanCode", "RatePlanDescription", "DeviceState", "BillingCycleStartDate", "BillingCycleEndDate", "CurrentBillCycleDataUnRatedUsage"
| $attr[.] // "NOT-EXIST"
.categories | map(.extendedAttributes[]) can be rewritten as [.categories[].extendedAttributes[]] or .categories | map(.extendedAttributes) | add, which might be easier to grok.
Output:
"89148000"
"5514048068"
"NOT-EXIST"
"311"
"480"
"4G Public Dynamic"
"4G5G "
"4G5G"
"active"
"2022-09-02T00:00:00Z"
"2022-10-01T00:00:00Z"
"1698414605"
A version without an intermediate variable is also possible:
[
.categories | map(.extendedAttributes[]) | from_entries[
"ICCID",
"MDN",
"MSISDN",
"MCC",
"MNC",
"FeatureTypes",
"RatePlanCode",
"RatePlanDescription",
"DeviceState",
"BillingCycleStartDate",
"BillingCycleEndDate",
"CurrentBillCycleDataUnRatedUsage"
]
]
| map(. // "NOT-EXIST")
or
[
.categories | map(.extendedAttributes[]) | from_entries
| .["ICCID", "MDN", "MSISDN", "MCC", "MNC", "FeatureTypes", "RatePlanCode", "RatePlanDescription", "DeviceState", "BillingCycleStartDate", "BillingCycleEndDate", "CurrentBillCycleDataUnRatedUsage"]
]
| map(. // "NOT-EXIST")
How can I manipulate this chunk of json:
{
"id": "whatever",
"attributes": [
{
"key": "this",
"value": "A"
},
{
"key": "that",
"value": "B"
},
{
"key": "other",
"value": "C"
}
]
}
So that it matches on "that" and removes the key and value both in that grouping, leaving json like this:
{
"id": "whatever",
"attributes": [
{
"key": "this",
"value": "A"
},
{
"key": "other",
"value": "C"
}
]
}
I am attempting to use jq on linux.
Try this
.attributes |= map(select(.key != "that"))
Demo
Figured it out.
jq 'del(.attributes[] | select(.key == "that"))' test.json | sponge test.json
Using the command aws resourcegroupstaggingapi get-resources --profile (profile_name) returns an array of JSON object with the Resource ARNs value and their tags (another array of JSON of objects with tag's key and value.
Here's a anonymized example:
{
"ResourceTagMappingList": [
{
"ResourceARN": "arn:aws:acm:eu-west-1:123456789000:certificate/XXXXX-YYYY-8888-9999-CCCCCCCCCCCCC",
"Tags": [
{
"Key": "Environment",
"Value": "BAR"
}
]
},
{
"ResourceARN": "arn:aws:acm:eu-west-1:123456789000:certificate/XXXXX-YYYY-8888-9999-CCCCCCCCCCCCC",
"Tags": [
{
"Key": "Environment",
"Value": "FOO"
}
]
},
{
"ResourceARN": "arn:aws:ec2:eu-west-1:123456789000:elastic-ip/eipalloc-112345440809463",
"Tags": [
{
"Key": "Component",
"Value": "somethingCool"
},
{
"Key": "DeployID",
"Value": "di-01"
},
{
"Key": "Name",
"Value": "eip-nat-somethingCool-di-01"
}
]
},
{
"ResourceARN": "arn:aws:ec2:eu-west-1:123456789000:elastic-ip/eipalloc-19853410278439394i3",
"Tags": [
{
"Key": "Component",
"Value": "somethingCool"
},
{
"Key": "DeployID",
"Value": "bla-internal-goku"
},
{
"Key": "Name",
"Value": "eip-nat-somethingCool-bla-internal-goku"
}
]
},
{
"ResourceARN": "arn:aws:elasticloadbalancing:eu-west-1:123456789000:targetgroup/tf-20190624192221842800000004/oisajhiuweniƧqej82u23948u3",
"Tags": [
{
"Key": "Component",
"Value": "somethingCool"
},
{
"Key": "DeployID",
"Value": "env01-bla00"
},
{
"Key": "Environment",
"Value": "env01"
},
{
"Key": "Estate",
"Value": "something"
},
{
"Key": "Name",
"Value": "target-group-lala-somethingCool-env01-bla00-vsquad"
}
]
}
]
}
So I would like to know if I can use the cli tool jq to filter the objects based on a specific tag Value? And if it is possible to list out all Value of a specific Tag Key?
Filter objects based on a specific tag value:
aws resourcegroupstaggingapi get-resources --region REGIONCODE | jq '.ResourceTagMappingList[] | select(.Tags[0].Value == "VALUETYPE")'
List all values of a specific tag key:
aws resourcegroupstaggingapi get-tag-values --key KEYNAME
You can query for all tag values for the specified key in the specified region for the AWS account, by calling the same API, but get-tag-values.
https://docs.aws.amazon.com/resourcegroupstagging/latest/APIReference/API_GetTagValues.html
I need some help using jq to sort an array of elements where each element contains a nested
tags array of elements. My input JSON looks like this:
{
"result": [
{
"name": "ct-1",
"tags": [
{
"key": "service_name",
"value": "BaseCT"
},
{
"key": "sequence",
"value": "bb"
}
]
},
{
"name": "ct-2",
"tags": [
{
"key": "service_name",
"value": "BaseCT"
},
{
"key": "sequence",
"value": "aa"
}
]
}
]
}
I would like to sort using the value of the sequence tag in the nested tags array so that the output looks like this:
{
"result": [
{
"name": "ct-2",
"tags": [
{
"key": "service_name",
"value": "BaseCT"
},
{
"key": "sequence",
"value": "aa"
}
]
},
{
"name": "ct-1",
"tags": [
{
"key": "service_name",
"value": "BaseCT"
},
{
"key": "sequence",
"value": "bb"
}
]
}
]
}
I have tried the following jq command:
$ jq '.result |= ([.[] | .tags[] | select(.key == "sequence") | .value] | sort_by(.))' input.json
but I get the following result:
{
"result": [
"aa",
"bb"
]
}
Please let me know if you know how to deal with this scenario.
from_entries converts an array of key-value pairs to an object, you can use it with sort_by like this:
.result |= sort_by(.tags | from_entries | .sequence)
I have been playing around with jq to format a json file but I am having some issues trying to solve a particular transformation. Given a test.json file in this format:
[
{
"name": "A", // This would be the first key
"number": 1,
"type": "apple",
"city": "NYC" // This would be the second key
},
{
"name": "A",
"number": "5",
"type": "apple",
"city": "LA"
},
{
"name": "A",
"number": 2,
"type": "apple",
"city": "NYC"
},
{
"name": "B",
"number": 3,
"type": "apple",
"city": "NYC"
}
]
I was wondering, how can I format it this way using jq?
[
{
"key": "A",
"values": [
{
"key": "NYC",
"values": [
{
"number": 1,
"type": "a"
},
{
"number": 2,
"type": "b"
}
]
},
{
"key": "LA",
"values": [
{
"number": 5,
"type": "b"
}
]
}
]
},
{
"key": "B",
"values": [
{
"key": "NYC",
"values": [
{
"number": 3,
"type": "apple"
}
]
}
]
}
]
I have followed this thread Using jq, convert array of name/value pairs to object with named keys and tried to group the json using this expression
jq '. | group_by(.name) | group_by(.city) ' ./test.json
but I have not been able to add the keys in the output.
You'll want to group the items at the different levels and building out your result objects as you want.
group_by(.name) | map({
key: .[0].name,
values: (group_by(.city) | map({
key: .[0].city,
values: map({number,type})
}))
})
Just keep in mind that group_by/1 yields groups in a sorted order. You'll probably want an implementation that preserves that order.
def group_by_unsorted(key_selector):
reduce .[] as $i ({};
.["\($i|key_selector)"] += [$i]
)|[.[]];