I have below json content in a json file and "pos" value in below json is a variable and need to replace it with a constant like NNNN.
{
"qual_table": "TKGGU1.TCUSTORD",
"op_type": "INSERT",
"op_ts": "YYYY-MM-DDTHH:MI:SS.FFFFFFZ",
"pos": "G-AAAAAJcRAAAAAAAAAAAAAAAAAAAHAAoAAA==10687850.2.31.996",
"xid": "0.2.31.996",
"after": {
"CUST_CODE": "BILL",
"ORDER_DATE": "YYYY-MM-DD:HH:MI:SS",
"PRODUCT_CODE": "CAR",
"ORDER_ID": 765,
"PRODUCT_PRICE": 1500000,
"PRODUCT_AMOUNT": 3,
"TRANSACTION_ID": 100
}
}
I tried perl -i -pe 's/[A-Z]-[A-Z]*\.==*/ NNNN/g' filename.json but this is not working.
Could you suggest me a regular expression for this. Pls note this is a variable and have different length everytime.
jq is the de-facto standard tool for working with JSON from scripts and the command line. It makes updating a field of an object with a new value trivial:
$ jq '.pos = "NNNN"' input.json
{
"qual_table": "TKGGU1.TCUSTORD",
"op_type": "INSERT",
"op_ts": "YYYY-MM-DDTHH:MI:SS.FFFFFFZ",
"pos": "NNNN",
"xid": "0.2.31.996",
"after": {
"CUST_CODE": "BILL",
"ORDER_DATE": "YYYY-MM-DD:HH:MI:SS",
"PRODUCT_CODE": "CAR",
"ORDER_ID": 765,
"PRODUCT_PRICE": 1500000,
"PRODUCT_AMOUNT": 3,
"TRANSACTION_ID": 100
}
}
(Note that when standard output is a pipe or file or something other than a terminal, the JSON is printed in compact form on a single line.)
The equivalent perl one(ish)-liner would be something like
perl -MJSON::MaybeXS -MFile::Slurper=read_text -E '
my $json = JSON::MaybeXS->new;
my $obj = $json->decode(read_text($ARGV[0]));
$obj->{pos} = "NNNN";
say $json->encode($obj);' input.json
(Adjust as needed for your preferred JSON and file-reading modules)
perl -pe 's/"pos"\s*:\s*"\K[^"]*/NNNN/g' filename.json
should work for any length of pos as long as there is no quote character or newline in the string.
As Shawn pointed out it'd be better to use a json-manipulating tool instead of a string-manipulating one.
Otherwise you'll have to rely on assumptions.
I am describing a AWS security group and passing the output by jq in order to get all the CIDRs of the inbound rules.
I have reached so far:
▶ aws ec2 describe-security-groups --group-ids sg-123456789 | jq '.SecurityGroups[0].IpPermissions[0].IpRanges'
[
{
"CidrIp": "11.22.33.44/32",
"Description": "Something"
},
{
"CidrIp": "22.33.44.12/32",
"Description": "Something else"
},
{
"CidrIp": "22.11.33.55/32",
"Description": "Something different"
},
]
I know I can grep but is there a way to get just the CidrIp from each json element of this array?
Sure, change your pipeline to
jq -r '.SecurityGroups[0].IpPermissions[0].IpRanges[].CidrIp'
Demo
Note that I also added the -r flag which makes the output raw text instead of JSON.
jq '.SecurityGroups[0].IpPermissions[0].IpRanges | values[].CidrIp'
this seemed to work as well.
I have a json data as below.
{
"Data":
[
"User": [
{"Name": "Solomon", "Age":20},
{"Name": "Absolom", "Age":30},
]
"Country": [
{"Name" : "US", "Resident" : "Permanent"},
{"Name" : "UK", "Resident" : "Temporary"}
]]}
There are two tags with same keys,
in Users there is Name key and in Country also i have Name key. I need to preprocess the json file to differentiate the keys. My expected result is below. Tried through awk and sed commands, but i could not find proper solution. Any suggestion would be helpful.
Expected result:
{
"Data":
[
"User": [
{"User_Name": "Solomon", "User_Age":20},
{"User_Name": "Absolom", "User_Age":30},
]
"Country": [
{"Country_Name" : "US", "Country_Resident" : "Permanent"},
{"Country_Name" : "UK", "Country_Resident" : "Temporary"}
]]}
Tag name should be appended to the attribute name.
This is what i have tried,
jq '[.[] | .["User_Name"] = .Name]' file_name.json
But it changes for both the tages User as well as Country
with the permission of the OP, here's a jtc based solution while waiting for the jq's (assuming the input JSON is fixed):
bash $ <file.json jtc -w'<Data>l[:]<L>k<.*>L:<>k' -u'"{L}_{}";' -tc
{
"Data": {
"Country": [
{ "Country_Name": "US", "Country_Resident": "Permanent" },
{ "Country_Name": "UK", "Country_Resident": "Temporary" }
],
"User": [
{ "User_Age": 20, "User_Name": "Solomon" },
{ "User_Age": 30, "User_Name": "Absolom" }
]
}
}
bash $
Explanation of the jtc parameters:
-w'<Data>l[:]<L>k<.*>L:<>k' :
walk path (-w) selects Data label (<Data>l)
and then each of the nested elements ([:]),
and memorizes its key/label into the namespace L (<L>k),
then finds further each labeled element using REGEX label search (<.*>L:)
and finally reinterpret found element's key/label as the value (<>k)
-u'"{L}_{}";':
for each found label (in step 1) update operation (-u) is applied using template
"{L}_{}";', where {L} is interpolated with preserved in the namespace L value and {} is getting interpolated with the currently found label (at the each iteration of the walk path)
the trailing ; (or any other symbol) is required to distinguish the argument of -u from a literal JSON.
-tc is used to display JSON in a semi-compact form.
PS. I'm the creator of jtc unix JSON processing tool. The disclaimer is required by SO.
As originally posted, neither the illustrative input nor the corresponding output is valid JSON, but the following has been tested using JSON based on the shown input:
.Data |= ( (.User |= map(with_entries(.key |= ("User_" + .))))
| (.Country |= map(with_entries(.key |= ("Country_" + .)))) )
Of course, the above may need tweaking depending on the actual requirements, and can be generalized in various ways, e.g. as shown below.
A generalization
.Data |= with_entries( (.key + "_") as $newkey
| .value |= map(with_entries(.key |= ($newkey + .))))
Here is an approach using jq Streaming
fromstream(tostream | .[0] |= if length < 4 then . else .[3]="\(.[1])_\(.[3])" end)
It works by using tostream to convert your input to a stream of arrays
[["Data","Country",0,"Name"],"US"]
[["Data","Country",0,"Resident"],"Permanent"]
[["Data","Country",0,"Resident"]]
[["Data","Country",1,"Name"],"UK"]
[["Data","Country",1,"Resident"],"Temporary"]
[["Data","Country",1,"Resident"]]
[["Data","Country",1]]
[["Data","User",0,"Age"],20]
[["Data","User",0,"Name"],"Solomon"]
[["Data","User",0,"Name"]]
[["Data","User",1,"Age"],30]
[["Data","User",1,"Name"],"Absolom"]
[["Data","User",1,"Name"]]
[["Data","User",1]]
[["Data","User"]]
[["Data"]]
then applying a simple update assignment |= expression to transform the stream into
[["Data","Country",0,"Country_Name"],"US"]
[["Data","Country",0,"Country_Resident"],"Permanent"]
[["Data","Country",0,"Country_Resident"]]
[["Data","Country",1,"Country_Name"],"UK"]
[["Data","Country",1,"Country_Resident"],"Temporary"]
[["Data","Country",1,"Country_Resident"]]
[["Data","Country",1]]
[["Data","User",0,"User_Age"],20]
[["Data","User",0,"User_Name"],"Solomon"]
[["Data","User",0,"User_Name"]]
[["Data","User",1,"User_Age"],30]
[["Data","User",1,"User_Name"],"Absolom"]
[["Data","User",1,"User_Name"]]
[["Data","User",1]]
[["Data","User"]]
[["Data"]]
then reversing the transformation with fromstream.
Try it online!
I have a JSON file in which I want to append an array element, using bash and latest JQ installed. I am able to append it but the resulting string has unicode characters as can be seen below. The first element in validators array is the original and the second is the appended code. (not the whole json file)
"validators": [
{
"address": "85BAF568E7F89277E47D3FC8E111775A4F6992FA",
"pub_key": {
"type": "tendermint/PubKeyEd25519",
"value": "BCzCLcW7rZ9VJgAtEUoDN17qcZw8ZvpYbPsL6eOy3No="
},
"power": "10",
"name": ""
},
{
"address": "\u001b[32m\"F75E15A3949824B685A3C5BFCDEED7E3DA4277AE\"\u001b[0m\r",
"pub_key": "\u001b[37m{\u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[37m:\u001b[0m\u001b[32m\"tendermint/PubKeyEd25519\"\u001b[0m\u001b[37m,\u001b[0m\u001b[34;1m\"value\"\u001b[0m\u001b[37m:\u001b[0m\u001b[32m\"INeR51z41k6jPAEJ5rV+1TY+4sxnbIykc4bfJFmSCQ8=\"\u001b[0m\u001b[37m\u001b[37m}\u001b[0m\r",
"power": "10",
"name": "node2"
}
]
Printing the address element separately prints the element without any utf/unicode encoding chars.
{
"type": "tendermint/PubKeyEd25519",
"value": "BCzCLcW7rZ9VJgAtEUoDN17qcZw8ZvpYbPsL6eOy3No="
}
I merge the code using the following code:
cat genesis.json.src | jq --arg pub_key $PK --arg name node$i --arg addr $ADDR '.validators+= [{address: $addr, pub_key: $pub_key, power:"10",name:$name}]' > genesis.json.dest
I am running macOS. Any help or suggestion would be appreciated.
As #choroba mentioned in the comment, this is colour sequence characters. I removed them by adding a -M flag for JQ that disables colours.
Appologies if I've overlooked something very obvious; I've just found jq and am trying to use it to update one JSON value without affecting the surrounding data.
I'd like to pipe a curl result into jq, update a value, and pipe the updated JSON to a curl -X PUT. Something like
curl http://example.com/shipping.json | jq '.' field: value | curl -X PUT http://example.com/shipping.json
So far I've hacked it together using sed, but after looking at a few examples of the |= operator in jq I'm sure that I don't need these.
Here's a JSON sample--how would I use jq to set "local": false, while preserving the rest of the JSON?
{
"shipping": {
"local": true,
"us": true,
"us_rate": {
"amount": "0.00",
"currency": "USD",
"symbol": "$"
}
}
}
You set values of an object using the = operator. |= on the other hand is used to update a value. It's a subtle but important difference. The context of the filters changes.
Since you are setting a property to a constant value, use the = operator.
.shipping.local = false
Just note that when setting a value to a property, it doesn't necessarily have to exist. You can add new values easily this way.
.shipping.local = false | .shipping.canada = false | .shipping.mexico = true
Update a value (sets .foo.bar to "new value"):
jq '.foo.bar = "new value"' file.json
Update a value using a variable (sets .foo.bar to "hello"):
variable="hello"; jq --arg variable "$variable" '.foo.bar = $variable' file.json
a similar function to the operator |= is map.
map will be suitable to avoid the requirement of a previous filter for the array...
imagine that your data is an array (very common for this example)
[
{
"shipping": {
"local": true,
"us": true,
"us_rate": {
"amount": "1.00",
"currency": "USD",
"symbol": "$"
}
}
},
{
"shipping": {
"local": true,
"us": true,
"us_rate": {
"amount": "1.00",
"currency": "USD",
"symbol": "$"
}
}
}
]
hence it is necessary to consider the array in the code as:
http://example.com/shipping.json | jq '.[] | .shipping.local = "new place"' | curl -X PUT http://example.com/shipping.json
or to use the map function that is crafted to work in every array element as
http://example.com/shipping.json | jq 'map(.shipping.local = "new place")' | curl -X PUT http://example.com/shipping.json
Observation
For the sake of those that are learning, you also did some mistakes in the jq usage, just consider that it does "read" the 1st parameter as the program, hence all the desired commands shall be included in the very first string after calling the program.