Replace a keyword with the content of the file - json

I have a templatized json file called template.json as below:
{
"subject": "Some subject line",
"content": $CONTENT,
}
I have another file called sample.json with the json content as below:
{
"status": "ACTIVE",
"id": 217,
"type": "TEXT",
"name": "string",
"subject": "string",
"url": "contenttemplates/217",
"content": {
"text": "hello ${user_name}",
"variables": [{
"key": "${user_name}",
"value": null
}]
},
"content_footer": null,
"audit": {
"creator": "1000",
"timestamp": 1548613800000,
"product": "2",
"channel": "10",
"party": null,
"event": {
"type": null,
"type_id": "0",
"txn_id": "0"
},
"client_key": "pk6781gsfr5"
}
}
I want to replace $CONTENT from template.json with the content under the tag "content" from the content.json file . I have tried with below sed commands:
sed -i 's/$CONTENT/'$(jq -c '.content' sample.json)'/' template.json
I am getting below error:
sed: -e expression #1, char 15: unterminated `s' command
Can someone please help me to get the right sed command (or any other alternative)?

The jq Cookbook has a section on using jq with templates: https://github.com/stedolan/jq/wiki/Cookbook#using-jq-as-a-template-engine
In the present case, the first technique ("Using jq variables as template variables") matches the already-defined template file (except for the dangling comma), so you could for example write:
jq -n --arg CONTENT "$(jq -c .content sample.json)" '
{"subject": "Some subject line", "content": $CONTENT}'
or use the format:
jq -n --arg CONTENT "$(jq -c .content sample.json)" -f template.jq
(I'd only use the .json suffix for files that hold JSON or JSON streams.)

The output from jq contains spaces, you need to quote them to prevent the shell from tokenizing them.
sed -i 's/$CONTENT/'"$(jq -c '.content' sample.json)/" template.json
See further When to wrap quotes around a shell variable?

With GNU sed:
sed '/$CONTENT/{s/.*/jq -c ".content" sample.json/e}'
Replace the entire line with your command and e (GNU only) to execute the command and replace sed's pattern space with the output of the command.

Related

How to parse json values stored in shell variable

#!/bin/bash
DESCRIBE_VPC=$(aws ec2 describe-vpcs --region us-west-2)
The Json value retrieved from aws ec2 describe-vpcs --region us-west-2 stores in DESCRIBE_VPC which comes to the output format below.
> echo $DESCRIBE_VPC
{
"Vpcs": [
{
"VpcId": "vpc-12345678910",
"InstanceTenancy": "default",
"Tags": [
{
"Value": "arn:aws:cloudformation:us-west-2:12345678910:stack/vpc/0123456-vpcid",
"Key": "stack-id"
},
{
"Value": "vpc-type",
"Key": "Name"
},
],
"CidrBlockAssociationSet": [
{
"AssociationId": "vpc-cidr-123456",
"CidrBlock": "123.456.89.10",
"CidrBlockState": {
"State": "associated"
}
}
],
"State": "available",
"DhcpOptionsId": "dpt-01234567",
"OwnerId": "12345678910",
"CidrBlock": "123.456.789.10",
"IsDefault": false
}
]
}
[root#ip bin]# jq '.Vpcs' $DESCRIBE_VPC
jq: error: Could not open file {: No such file or directory
jq: error: Could not open file "Vpcs":: No such file or directory
parse error: Invalid numeric literal at line 1, column 57
Any suggestions here how to parse the json values stored in variable?
Actually you have to use the (json) content of a shell variable in place of a command's (jq) input. This input should be a file or a stream, but you have it in a shell variable. There are many ways to do this, this is a simple one:
echo "$DESCRIBE_VPC" | jq '.Vpcs'
or
printf "%s" "$DESCRIBE_VPC" | jq '.Vpcs'
or this (for bash shell):
jq '.Vpcs' <<< "$DESCRIBE_VPC"
Also jq can accept variables and json variables. For example you could do this:
jq -n --argjson x "$DESCRIBE_VPC" '$x.Vpcs'
But the first one is usually better.

How to parse asterisk in json file as string via jq

Here is a json file named test.json for testing
{
"name": "Google",
"location": {
"street": "1600 Amphitheatre Parkway",
"city": "Mountain View",
"state": "California",
"country": "US"
},
"employees": [
{
"name": "Michael",
"division": "Engineering"
},
{
"name": "Laura",
"division": "HR"
},
{
"name": "Elise",
"division": "Marketing * test"
}
]
}
if I use the jq code to parser it like below:
cat test.json | jq -r '.employees[2].division'
it will work well and give a correct result:
Marketing * test
but I use $(), the bad thing will happen!
echo $(cat test.json | jq -r '.employees[2].division')
the result will list all file names under current folder! like:
my1.json my2.json test.json test ...
I guess it $() run asterisk * as a shell script, but a string.
so how to make asterisk (*) in json file just as a string when I am using jq ?. I am using Google cloud platform and Ubuntu 17.10
Always use double-quotes around command-substitution to avoid * to be treated literally. The * is a special character in shell that is a wildcard entry that expands to all the files available in the current working directory. You need to quote it to deprive of its special meaning (Refer GNU bash man page under Parameters section).
Also jq can process the file directly, you can avoid useless cat usage.
result="$(jq -r '.employees[2].division' < test.json)"
echo "$result"
should produce the result as expected.

Cat command with sed

I have a file:
{
"test_data": [
{
"id": "1",
"pm": "30",
"mp": "40"
}
],
"test": [
"id",
"pm",
"mp"
]
}
I want to extract test_data. Output:
"test_data": [
{
"id": "1",
"pm": "30",
"mp": "40"
}
],
I try this command: cat myFile | sed -n '/^"test_data": \[$/,/^\],$/p'
But it's don't work. An idea ?
Thanks you !
jq seems the right tool for the job :
$ jq '.|{test_data:.test_data}' filename
{
"test_data": [
{
"id": "1",
"pm": "30",
"mp": "40"
}
]
}
Solution 1st: With sed
sed -n '/"test_data"/,/],/p' Input_file
OR: as per OP, OP needs to append a string/data after a line matches:
sed -n '/"test_data"/,/],/p;/],/s/$/"test"/p' Input_file
OR2: If one wants to add an another file's content to a match then following may help in same:
sed -n '/"test_data"/,/],/p;/],/r another_file' Input_file
Solution 2nd: Following simple awk may help you in same.
awk '/test_data/, /],/' Input_file
Output will be as follows.
"test_data": [
{
"id": "1",
"pm": "30",
"mp": "40"
}
],
Logic for above solutions:
For sed: -n option in sed will turn OFF the printing of any line till it is explicitly mentioned to print it, then by doing /"test_data"/,/],/ I am letting sed know that I need to get the data from test_data to till ,/] and mentioning p after that will make sure those lines which are coming in this range are getting printed here/
For awk: Simply mentioning the range from /"test_data"/,/],/ and not mentioning any action so when any line comes into this range condition becomes true and since no action mentioned so by default print of that line happens then.
You can try that with gnu
csplit -s -z infile %test_data%-1 /],/1;rm xx01;echo "Add Text here" >> xx00;cat xx00
The right way is jq tool:
jq 'del(.test)' inputfile
The output:
{
"test_data": [
{
"id": "1",
"pm": "30",
"mp": "40"
}
]
}

Replace a word after a particular line using shell

I have the following lines in json file
{"name":"images",
"enable":1,
"binary_path":"/usr/bin/docker",
"parameter":"images"
},
{"name":"pull",
"enable":1,
"binary_path":"/usr/bin/docker",
"parameter":"pull httpd"
},
I want to replace
"enable":1,
with
"enable":0,
of the json which has
"name":"pull"
I tried below codes
sed -i '/"name":"pull",/a "enable":0' test.json
The right way is to use particular json parsers like jq:
Sample test.json:
[
{
"name": "images",
"enable": 1,
"binary_path": "/usr/bin/docker",
"parameter": "images"
},
{
"name": "pull",
"enable": 1,
"binary_path": "/usr/bin/docker",
"parameter": "pull httpd"
}
]
jq '[.[] | if .name == "pull" then .enable=0 else . end]' test.json
The output:
[
{
"name": "images",
"enable": 1,
"binary_path": "/usr/bin/docker",
"parameter": "images"
},
{
"name": "pull",
"enable": 0,
"binary_path": "/usr/bin/docker",
"parameter": "pull httpd"
}
]
We can simply use the line operator in sed to achieve this. Try below,
sed '/"name":"pull"/{n;s/.*/"enable":0,/}' test.json
Add -i option to edit in the source file,
sed -i '/"name":"pull"/{n;s/.*/"enable":0,/}' test.json
awk Solution
awk '/name":"pull/{a=1;print $0;next}a==1{$0="\"enable\""":""0" ;print $0;a=0;next}1 test.json
This worked for me -
Dry run -
$ sed '/"name":"pull"/!b;n;c\ \"enable":0,' test.json
Change in file with -i -
$ sed -i '/"name":"pull"/!b;n;c\ \"enable":0,' test.json
Output -
# sed '/"name":"pull"/!b;n;c\ \"enable":0,' test.json
{"name":"images",
"enable":1,
"binary_path":"/usr/bin/docker",
"parameter":"images"
},
{"name":"pull",
"enable":0,
"binary_path":"/usr/bin/docker",
"parameter":"pull httpd"
},

Find the value of key from JSON

I'd like to extract the "id" key from this single line of JSON.
I believe this can be accomplished with grep, but I am not sure on the correct way.
If there is a better way that does not have dependencies, I would be interested.
Here is my example output:
{
"data": {
"name": "test",
"id": "4dCYd4W9i6gHQHvd",
"domains": ["www.test.domain.com", "test.domain.com"],
"serverid": "bbBdbbHF8PajW221",
"ssl": null,
"runtime": "php5.6",
"sysuserid": "4gm4K3lUerbSPfxz",
"datecreated": 1474597357
},
"actionid": "WXVAAHQDCSILMYTV"
}
If you have a grep that can do Perl compatible regular expressions (PCRE):
$ grep -Po '"id": *\K"[^"]*"' infile.json
"4dCYd4W9i6gHQHvd"
-P enables PCRE
-o retains nothing but the match
"id": * matches "id" and an arbitrary amount of spaces
\K throws away everything to its left ("variable size positive look-behind")
"[^"]*" matches two quotes and all the non-quotes between them
If your grep can't do that, you an use
$ grep -o '"id": *"[^"]*"' infile.json | grep -o '"[^"]*"$'
"4dCYd4W9i6gHQHvd"
This uses grep twice. The result of the first command is "id": "4dCYd4W9i6gHQHvd"; the second command removes everything but a pair of quotes and the non-quotes between them, anchored at the end of the string ($).
But, as pointed out, you shouldn't use grep for this, but a tool that can parse JSON – for example jq:
$ jq '.data.id' infile.json
"4dCYd4W9i6gHQHvd"
This is just a simple filter for the id key in the data object. To get rid of the double quotes, you can use the -r ("raw output") option:
$ jq -r '.data.id' infile.json
4dCYd4W9i6gHQHvd
jq can also neatly pretty print your JSON:
$ jq . infile.json
{
"data": {
"name": "test",
"id": "4dCYd4W9i6gHQHvd",
"domains": [
"www.test.domain.com",
"test.domain.com"
],
"serverid": "bbBdbbHF8PajW221",
"ssl": null,
"runtime": "php5.6",
"sysuserid": "4gm4K3lUerbSPfxz",
"datecreated": 1474597357
},
"actionid": "WXVAAHQDCSILMYTV"
}
Just pipe your data to jq and select by keys
"data": {
"name": "test",
"id": "4dCYd4W9i6gHQHvd",
"domains": [
"www.test.domain.com",
"test.domain.com"
],
"serverid": "bbBdbbHF8PajW221",
"ssl": null,
"runtime": "php5.6",
"sysuserid": "4gm4K3lUerbSPfxz",
"datecreated": 1474597357
},
"actionid": "WXVAAHQDCSILMYTV"
} | jq '.data.id'
# 4dCYd4W9i6gHQHvd
Tutorial Here
I found myself that the best way is to use python, as it handles JSON natively and is preinstalled on most systems these days, unlike jq:
$ python -c 'import sys, json; print(json.load(sys.stdin)["data"]["id"])' < infile.json
4dCYd4W9i6gHQHvd
No python ,jq, awk, sed just GNU grep:
#!/bin/bash
json='{"data": {"name": "test", "id": "4dCYd4W9i6gHQHvd", "domains": ["www.test.domain.com", "test.domain.com"], "serverid": "bbBdbbHF8PajW221", "ssl": null, "runtime": "php5.6", "sysuserid": "4gm4K3lUerbSPfxz", "datecreated": 1474597357}, "actionid": "WXVAAHQDCSILMYTV"}'
echo $json | grep -o '"id": "[^"]*' | grep -o '[^"]*$'
Tested & working here: https://ideone.com/EG7fv7
source: https://brianchildress.co/parse-json-using-grep
$ grep -oP '"id": *"\K[^"]*' infile.json
4dCYd4W9i6gHQHvd
Hopefully it will work for all. As this will work for me to print without quotes.