Cat command with sed - json

I have a file:
{
"test_data": [
{
"id": "1",
"pm": "30",
"mp": "40"
}
],
"test": [
"id",
"pm",
"mp"
]
}
I want to extract test_data. Output:
"test_data": [
{
"id": "1",
"pm": "30",
"mp": "40"
}
],
I try this command: cat myFile | sed -n '/^"test_data": \[$/,/^\],$/p'
But it's don't work. An idea ?
Thanks you !

jq seems the right tool for the job :
$ jq '.|{test_data:.test_data}' filename
{
"test_data": [
{
"id": "1",
"pm": "30",
"mp": "40"
}
]
}

Solution 1st: With sed
sed -n '/"test_data"/,/],/p' Input_file
OR: as per OP, OP needs to append a string/data after a line matches:
sed -n '/"test_data"/,/],/p;/],/s/$/"test"/p' Input_file
OR2: If one wants to add an another file's content to a match then following may help in same:
sed -n '/"test_data"/,/],/p;/],/r another_file' Input_file
Solution 2nd: Following simple awk may help you in same.
awk '/test_data/, /],/' Input_file
Output will be as follows.
"test_data": [
{
"id": "1",
"pm": "30",
"mp": "40"
}
],
Logic for above solutions:
For sed: -n option in sed will turn OFF the printing of any line till it is explicitly mentioned to print it, then by doing /"test_data"/,/],/ I am letting sed know that I need to get the data from test_data to till ,/] and mentioning p after that will make sure those lines which are coming in this range are getting printed here/
For awk: Simply mentioning the range from /"test_data"/,/],/ and not mentioning any action so when any line comes into this range condition becomes true and since no action mentioned so by default print of that line happens then.

You can try that with gnu
csplit -s -z infile %test_data%-1 /],/1;rm xx01;echo "Add Text here" >> xx00;cat xx00

The right way is jq tool:
jq 'del(.test)' inputfile
The output:
{
"test_data": [
{
"id": "1",
"pm": "30",
"mp": "40"
}
]
}

Related

Replace a keyword with the content of the file

I have a templatized json file called template.json as below:
{
"subject": "Some subject line",
"content": $CONTENT,
}
I have another file called sample.json with the json content as below:
{
"status": "ACTIVE",
"id": 217,
"type": "TEXT",
"name": "string",
"subject": "string",
"url": "contenttemplates/217",
"content": {
"text": "hello ${user_name}",
"variables": [{
"key": "${user_name}",
"value": null
}]
},
"content_footer": null,
"audit": {
"creator": "1000",
"timestamp": 1548613800000,
"product": "2",
"channel": "10",
"party": null,
"event": {
"type": null,
"type_id": "0",
"txn_id": "0"
},
"client_key": "pk6781gsfr5"
}
}
I want to replace $CONTENT from template.json with the content under the tag "content" from the content.json file . I have tried with below sed commands:
sed -i 's/$CONTENT/'$(jq -c '.content' sample.json)'/' template.json
I am getting below error:
sed: -e expression #1, char 15: unterminated `s' command
Can someone please help me to get the right sed command (or any other alternative)?
The jq Cookbook has a section on using jq with templates: https://github.com/stedolan/jq/wiki/Cookbook#using-jq-as-a-template-engine
In the present case, the first technique ("Using jq variables as template variables") matches the already-defined template file (except for the dangling comma), so you could for example write:
jq -n --arg CONTENT "$(jq -c .content sample.json)" '
{"subject": "Some subject line", "content": $CONTENT}'
or use the format:
jq -n --arg CONTENT "$(jq -c .content sample.json)" -f template.jq
(I'd only use the .json suffix for files that hold JSON or JSON streams.)
The output from jq contains spaces, you need to quote them to prevent the shell from tokenizing them.
sed -i 's/$CONTENT/'"$(jq -c '.content' sample.json)/" template.json
See further When to wrap quotes around a shell variable?
With GNU sed:
sed '/$CONTENT/{s/.*/jq -c ".content" sample.json/e}'
Replace the entire line with your command and e (GNU only) to execute the command and replace sed's pattern space with the output of the command.

Sed don't work with a special variable

I have a command that adds to a specific location of the file.
a="hellotesting"
sed "/^[[:blank:]]*\"testing_$2\": \[$/,/^[[:blank:]]*],*$/ {s/^[[:blank:]]*}$/&`echo $a`/}" $1
It's work ! Output (is a JSON file and i don't want to use jq)
{
"mytable_test": [
{
"test1": "abc",
"test2": "def",
"test2": "ghi"
}hellotesting
],
"mytable": [
"test1",
"test2",
"test3"
]
}
But I can't put the contents of a file.
My file:
yeah
is
test
my
script
And in my program:
IFS=
a=$(cat file)
sed "/^[[:blank:]]*\"data_$2\": \[$/,/^[[:blank:]]*],*$/ {s/^[[:blank:]]*}$/&`echo $a`/}" $1
I have an error:
sed: -e expression #1, char 73: unterminated `s' command
I want this output:
{
"mytable_test": [
{
"test1": "abc",
"test2": "def",
"test2": "ghi"
}yeah
is
test
my
script
],
"mytable": [
"test1",
"test2",
"test3"
]
}
But, if my file is just:
yeah
And I cat file in my variable, that work ..
I don't understand ..
Not entirely sure why you would like to corrupt a valid json, but this what you can do
$ var="yeah\nis\ntest\nmy\nscript"
$ echo "{
"mytable_test": [
{
"test1": "abc",
"test2": "def",
"test2": "ghi"
}hellotesting
],
"mytable": [
"test1",
"test2",
"test3"
]
}" | sed -E "s/hellotesting$/${var}/"
Output
{
mytable_test: [
{
test1: abc,
test2: def,
test2: ghi
}yeah
is
test
my
script
],
mytable: [
test1,
test2,
test3
]
}

Replace a word after a particular line using shell

I have the following lines in json file
{"name":"images",
"enable":1,
"binary_path":"/usr/bin/docker",
"parameter":"images"
},
{"name":"pull",
"enable":1,
"binary_path":"/usr/bin/docker",
"parameter":"pull httpd"
},
I want to replace
"enable":1,
with
"enable":0,
of the json which has
"name":"pull"
I tried below codes
sed -i '/"name":"pull",/a "enable":0' test.json
The right way is to use particular json parsers like jq:
Sample test.json:
[
{
"name": "images",
"enable": 1,
"binary_path": "/usr/bin/docker",
"parameter": "images"
},
{
"name": "pull",
"enable": 1,
"binary_path": "/usr/bin/docker",
"parameter": "pull httpd"
}
]
jq '[.[] | if .name == "pull" then .enable=0 else . end]' test.json
The output:
[
{
"name": "images",
"enable": 1,
"binary_path": "/usr/bin/docker",
"parameter": "images"
},
{
"name": "pull",
"enable": 0,
"binary_path": "/usr/bin/docker",
"parameter": "pull httpd"
}
]
We can simply use the line operator in sed to achieve this. Try below,
sed '/"name":"pull"/{n;s/.*/"enable":0,/}' test.json
Add -i option to edit in the source file,
sed -i '/"name":"pull"/{n;s/.*/"enable":0,/}' test.json
awk Solution
awk '/name":"pull/{a=1;print $0;next}a==1{$0="\"enable\""":""0" ;print $0;a=0;next}1 test.json
This worked for me -
Dry run -
$ sed '/"name":"pull"/!b;n;c\ \"enable":0,' test.json
Change in file with -i -
$ sed -i '/"name":"pull"/!b;n;c\ \"enable":0,' test.json
Output -
# sed '/"name":"pull"/!b;n;c\ \"enable":0,' test.json
{"name":"images",
"enable":1,
"binary_path":"/usr/bin/docker",
"parameter":"images"
},
{"name":"pull",
"enable":0,
"binary_path":"/usr/bin/docker",
"parameter":"pull httpd"
},

Converting JSON pretty print to one line

I have a command that I run and it gives an output like below:
{
"endpointApplications": {
"App_Name": {
"connectionState": "Disconnected",
"connectionTime": "No connection was established",
"linkAttributes": {
"ackSettings": {
"dataAckEnabled": "true",
"dataAckTimeout": "5000",
"dataNakRetryLimit": "0",
"retransmitDelay": "500"
},
"keepAliveSettings": {
"keepAliveAckTimeout": "5000",
"keepAliveInterval": "30000"
},
"logTraffic": "false",
"port": "9999",
"role": "server"
},
"protocol": "snmp"
}
},
"queueStats": {}
}
I would need the output to be in one line like below:
{"endpointApplications": {"app_name": {"connectionState": "Disconnected","connectionTime": "No connection was established","linkAttributes": {"ackSettings":{"dataAckEnabled": "true","dataAckTimeout": "5000","dataNakRetryLimit": "0","retransmitDelay": "500"},"keepAliveSettings":{"keepAliveAckTimeout": "5000","keepAliveInterval": "30000"},"logTraffic": "false","port": "9999","role": "server"},"protocol": "snmp"}},"queueStats":{}}
I tried using awk and sed combining different parameters but I can't get to work without losing the JSON format.
You should use jq for stuff like that:
jq -c . input.txt
An alternative quick a dirty solution would be to use sed & tr:
sed -e 's/^ *//' < input.txt | tr -d '\n'
although I would recommend using jq which is designed for manipulating JSON. jq is like sed for JSON. Manipulating JSON textually with sed/awk/etc is not guaranteed to produce semantically equivalent JSON.
jq or any other json aware tool is best suited for json file manipulation.However here is awk based solution.
awk -v RS= '{$1=$1}1' input.json
{ "endpointApplications": { "App_Name": { "connectionState": "Disconnected", "connectionTime": "No connection was established", "linkAttributes": { "ackSettings": { "dataAckEnabled": "true", "dataAckTimeout": "5000", "dataNakRetryLimit": "0", "retransmitDelay": "500" }, "keepAliveSettings": { "keepAliveAckTimeout": "5000", "keepAliveInterval": "30000" }, "logTraffic": "false", "port": "9999", "role": "server" }, "protocol": "snmp" } }, "queueStats": {} }
Note: This solution is mainly for the legacy systems not having tools like jq and have no chance to get them installed due to some reasons.

Update inner attribute of JSON with jq

Could somebody help me to deal with jq command line utility to update JSON object's inner value?
I want to alter object interpreterSettings.2B263G4Z1.properties by adding several key-values, like "spark.executor.instances": "16".
So far I only managed to fully replace this object, not add new properties with command:
cat test.json | jq ".interpreterSettings.\"2B188AQ5T\".properties |= { \"spark.executor.instances\": \"16\" }"
This is input JSON:
{
"interpreterSettings": {
"2B263G4Z1": {
"id": "2B263G4Z1",
"name": "sh",
"group": "sh",
"properties": {}
},
"2B188AQ5T": {
"id": "2B188AQ5T",
"name": "spark",
"group": "spark",
"properties": {
"spark.cores.max": "",
"spark.yarn.jar": "",
"master": "yarn-client",
"zeppelin.spark.maxResult": "1000",
"zeppelin.dep.localrepo": "local-repo",
"spark.app.name": "Zeppelin",
"spark.executor.memory": "2560M",
"zeppelin.spark.useHiveContext": "true",
"spark.home": "/usr/lib/spark",
"zeppelin.spark.concurrentSQL": "false",
"args": "",
"zeppelin.pyspark.python": "python"
}
}
},
"interpreterBindings": {
"2AXUMXYK4": [
"2B188AQ5T",
"2AY8SDMRU"
]
}
}
I also tried the following but this only prints contents of interpreterSettings.2B263G4Z1.properties, not full object.
cat test.json | jq ".interpreterSettings.\"2B188AQ5T\".properties + { \"spark.executor.instances\": \"16\" }"
The following works using jq 1.4 or jq 1.5 with a Mac/Linux shell:
jq '.interpreterSettings."2B188AQ5T".properties."spark.executor.instances" = "16" ' test.json
If you have trouble adapting the above for Windows, I'd suggest putting the jq program in a file, say my.jq, and invoking it like so:
jq -f my.jq test.json
Notice that there is no need to use "cat" in this case.
p.s. You were on the right track - try replacing |= with +=