I try to preprocess a JSON file and want to remove all subobjects in it.
Note that my JSON string is inline (not have \r).
In input I have:
{
"total": 2,
"result": [
{
"id":1,
"createdOn" : 123456,
"obj1": {
...:...,
...:...
},
"obj2": {
...:...,
...:...
},
"otherattribute": "..."
},
{
"id":2,
"createdOn" : 123456,
"obj1": {
...:...,
...:...
},
"obj2": {
...:...,
...:...
},
"otherattribute": "..."
}
]
}
And want to have:
{
"total": 2,
"result": [
{
"id":1,
"createdOn" : 123456,
"otherattribute": "..."
},
{
"id":2,
"createdOn" : 123456,
"otherattribute": "..."
}
]
}
I've tried to use sed command with :
sed 's/"obj1":{[^}]*//g'
It will remove the "ojb1" subobject, but let remain the endind "}," of this subobject.
I didn't find the way to also remove "},".
How?
Second question: I know the list of suboject; but is there a way to remove all theses subjects directly without knowing there name? Something like:
sed 's/".*":{[^}]*//g'
So that I will have only one sed command and not having to chain commands like :
sed 's/obj1//g' | sed 's/obj2//g' | sed 's/obj3//g' ...
I'm giving a first answer:
sed 's/"obj1":{[^}},]*//g'
It will match "obj1": until character } and then match again },, so that these last 2 chars will be also deleted (replaced with nothing).
Now I try to do something more generic to match any object name, with something like this:
sed 's/"[[:alnum:]]":{[^}},]*//g'
But it's not working...
I've already tried to escape or escape escape or escape escape escape double quote without success...
Related
How to change a value in .spec file using sed? Want to change the $nmatch value of build to someother value say "build1.1".
{
"files": [
{
"aql": {
"items.find": {
"repo": {"$eq":"app-java-repo"},
"path": "archives/test/app",
"type": "folder",
"$or": [
{
"$and": [
{
"name": {"$nmatch" : "*build*"} //change value using sed
}
]
}
]
}
}
}
]
}
Tried below command but not working
sed -i -e '/name:/{s/\($nmatch\).*/\1: "'"build1.1"'" /}'
Your document format is json, so you should use jq for editing:
jq '.files[0].aql."items.find"."$or"[0]."$and"[0].name."$nmatch"="build1.1"' spec
With sed, you can't reliably access the correct name node. Your code doesn't work because you didn't include the quotes around name in your search pattern. Using sed would be ok if you have a unique identifier for build. Example:
sed -i 's/BUILD_NUMBER/build1.1/g' spec
A more "indirect" way is to simply set everything with $nmatch:
jq '(.. | select(has("$nmatch"))? ) = "build1.1"' input.json
I'm struggling for some time and I would need some help with the following operation.
I have a JSON file and I would like to replace a string with something a bit more complex.
This is a snippet of my json file:
{ "AWS679f53fac002430cb0da5b7982bd22872D164C4C": {
"Type": "AWS::Lambda::Function",
"Properties": {
"Code": {
"S3Bucket": "hnb659fds-assets-xxccddff",
"S3Key": "68b4ffa1c39cb3733535725f85311791c09eab53b7ab8efa5152e68f8abdb005.zip"
},
"Role": {
"Fn::GetAtt": [
"AWS679f53fac002430cb0da5b7982bd2287ServiceRoleC1EA0FF2",
"Arn"
]
},
"Handler": "index.handler",
"Runtime": "nodejs12.x",
"Timeout": 120
},
"DependsOn": [
"AWS679f53fac002430cb0da5b7982bd2287ServiceRoleC1EA0FF2"
],
"Metadata": {
"aws:cdk:path": "CODE/AWS679f53fac002430cb0da5b7982bd2287/Resource",
"aws:asset:path": "asset.68b4ffa1c39cb3733535725f85311791c09eab53b7ab8efa5152e68f8abdb005",
"aws:asset:is-bundled": false,
"aws:asset:property": "Code"
}
}
}
What I need is to replace this part
"S3Bucket": "hnb659fds-assets-xxccddff",
and have the following result
"S3Bucket": {"Fn::Sub": "AAA-${AWS::Region}" },
I don't know the AWS679f53fac002430cb0da5b7982bd22872D164C4C. It is generated randomly and the string to replace is present several times in my json file.
The initial values to be replaced is stored in a variable along with the new value to be used in the replaced version as following:
cdk_bucket_name=hnb659fds-assets-xxccddff
OUTPUT_BUCKET=AAA
I need these variables because this is part of a bigger script
So I tried some sed but does not work
new_bucket_name="{"Fn::Sub\": \"$OUTPUT_BUCKET-${AWS::Region}\" }"
sed -i "s#$cdk_bucket_name#$new_bucket_name#g" my.template.json
One issue that I have is that ${AWS::Region} gets interpreted so is empty.
And second, I cannot manage the quotes in order to have my desired result.
Using sed
$ output_bucket=AAA
$ new_bucket_name="{\"Fn::Sub\": \"$output_bucket-\${AWS::Region}\" }"
$ cdk_bucket_name=hnb659fds-assets-xxccddff
$ sed s"/\"$cdk_bucket_name\"/$new_bucket_name/" input_file
{ "AWS679f53fac002430cb0da5b7982bd22872D164C4C": {
"Type": "AWS::Lambda::Function",
"Properties": {
"Code": {
"S3Bucket": {"Fn::Sub": "AAA-${AWS::Region}" },
"S3Key": "68b4ffa1c39cb3733535725f85311791c09eab53b7ab8efa5152e68f8abdb005.zip"
},
"Role": {
"Fn::GetAtt": [
"AWS679f53fac002430cb0da5b7982bd2287ServiceRoleC1EA0FF2",
"Arn"
]
},
"Handler": "index.handler",
"Runtime": "nodejs12.x",
"Timeout": 120
},
"DependsOn": [
"AWS679f53fac002430cb0da5b7982bd2287ServiceRoleC1EA0FF2"
],
"Metadata": {
"aws:cdk:path": "CODE/AWS679f53fac002430cb0da5b7982bd2287/Resource",
"aws:asset:path": "asset.68b4ffa1c39cb3733535725f85311791c09eab53b7ab8efa5152e68f8abdb005",
"aws:asset:is-bundled": false,
"aws:asset:property": "Code"
}
}
}
Using a proper JSON parser shell tool like jq:
jq '
(
.[].Properties.Code.S3Bucket |
select(. == "hnb659fds-assets-xxccddff")
) = $newS3Bucket
' input_file.json \
--argjson newS3Bucket '{"Fn::Sub":"AAA-${AWS::Region}"}'
I have some input, which you may recognize as a JSON stream, which I need to process from a simple shell script
I've opted to use sed, because half a dozen other manipulations I need to make are already working great with sed.
Below is the scenario.
input1:
{
"Expr": {
"Criteria": [
{
"Values": [
{
"displayValue": "MyObject",
"value": "UUID"
}
]
}
],
"Type": "SELECTED"
}
}
Intended result 1 (print the clause[s] containing displayValue + value pairs}:
{
"displayValue": "MyObject",
"value": "UUID"
}
Attempt:
$ sed -n -e '/"Expr":/,/"Type": "SELECTED"/ {/"Values":/,/^ }$/ {/^ {/,/^ }/p;};}' /tmp/input1.json
Actual result (good):
{
"displayValue": "MyObject",
"value": "UUID"
}
Problem is with this input2:
{
"Expr": {
"Criteria": [
{
"Values": [
{
"displayValue": "MyObject",
"value": "UUID"
}
]
}
],
"Type": "NOT_SELECTED"
}
}
Result is unexpected (since outer nest is NOT_SELECTED, do not want match unless SELECTED):
$ sed -n -e '/"Expr":/,/"Type": "SELECTED"/ {/"Values":/,/^ }$/ {/^ {/,/^ }/p;};}' /tmp/input2.json
Result 2:
{
"displayValue": "MyObject",
"value": "UUID"
}
Is it me, or is this a sed bug on my platform?
Otherwise, can you suggest a good approach to this problem?
The constraint is I do not want to print "Criteria" lines that are not within the "Type" is "SELECTED" range.
This might work for you (GNU sed):
sed -En '/"Expr"/{:a;n;H;/"Values"/h;/"Type"/!ba;/"SELECTED"/!d;g;s/.*\n((\s*)\{.*\2\}).*/\1/p}' file
Use a loop rather than a range.
Form a loop between "Expr" and "Type" and only proceed if the quoted word "SELECTED" is present.
While executing the loop save Values in the hold space and then manipulate the saved content to just show the parts needed.
N.B. This is a filtering operation so use the -n option.
Remember that sed does not look ahead in the file. So, multiline techniques must be used for this job. My approach is to "slurp" lines between toplevel { and }, which I assume they consist of a single character ({ or }, and no leading and trailing spaces), and examine the resultant pattern space:
sed '
/^{$/,/^}$/{
//!{H;d;}
g
s/.*"Values":.*\(\n[[:space:]]*{[^}]*"displayValue":[^}]*"value":[^}]*}\).*"Type":[[:space:]]"SELECTED".*/\1/p
s/.*//
h
d
}' file
Like this:
I have a json file with below format:
{
"Item": {
"summary": {
"B": "ABCDE"
},
"name": {
"S": "sider"
},
"age": {
"N": "1"
},
"data": {
"B": "abcde"
}
}
}
How can I get the object "Item" only using existed commands like sed/awk without installing any external tools in shell?
expected output:
{
"summary": {
"B": "ABCDE"
},
"name": {
"S": "sider"
},
"age": {
"N": "1"
},
"data": {
"B": "abcde"
}
}
As chepner suggests, you need to question your constraints. In some cases, you are dealt a bad hand and have to deal with it. So here is a sed approach:
Here's a revised solution with awk. Sed is thrown in also to un-indent:
awk '
/^ }/ { p = 0 ; print }
p == 1 { print }
/"Item": {/ { print " {" ; p = 1 }
' | sed 's/^ //'
Originally posted this (but noticed that the output didn't match your expectations):
sed -n '/^ "Item": {/,/^ }/p'
The above sed approach assumes that Item is indented as you have it in the sample input above.
Here's an approach using my favorite, jq:
jq '.Item'
You may do well to check if python with the json package is installed on your machine. Here's a python3 script that would suit your needs:
#!/usr/bin/env python3
import json
import sys
j = json.load( sys.stdin )
print(json.dumps(j["Item"]))
Idiomatically, it'd be incorrect to manipulate a nested data format like JSON with line-aware tools like sed/awk. However if you're limited in your choice, then the best approach is following:
convert multi-line file to a single line
using awk/sed extract your Item
Here's a sed based solution:
bash $ <file.json tr '\n' ' ' | sed -E 's/^ *{ *"Item": +//; s/ *}$//'
This is what you need:
sed -n '/^ "Item": {/,/^ }/{s/"Item": //;s/^ //;p}'
which essentially build upon Mark's solution by making two substitutions to remove "Item": and de-indent 4 spaces before printing.
I have one problem and try to solve it by sed command.
I have file.txt with text in the first line. The text for example is: # username:blablabla,password:blablabla, and I have another file named file.json with this content:
{
"Subject": {
"Data": "GitCommit login information",
"Charset": "UTF-8"
},
"Body": {
"Text": {
"Data": "This is the message body in text format.",
"Charset": "UTF-8"
},
"Html": {
"Data": "bad data"
}
}
}
Question is: How can i replace Json's value is "bad data" with the line from file.txt?
Result must be :
{
"Subject": {
"Data": "GitCommit login information",
"Charset": "UTF-8"
},
"Body": {
"Text": {
"Data": "This is the message body in text format.",
"Charset": "UTF-8"
},
"Html": {
"Data": "username:blablabla,password:blablabla"
}
}
}
Is it possible to do with bash command?
Let me preface this answer by saying you should really use a JSON parser to properly and safely change your JSON.
That being said, you can do it with sed like this:
# double backslashes (\); escape special sed-replacement chars (&/\); escape quotes (")
escape_json_for_sed() {
sed 's/\\/\\&/g; s/[&/\]/\\&/g; s/"/\\\\&/g'
}
sed 's/"Data": "bad data"/"Data": "'"$(head -1 file.txt | escape_json_for_sed)"'"/g' -i file.json
Note the helper shell function escape_json_for_sed() used on the replacement value read from file. With it, we:
escape the characters with special meaning in sed replacement string (/, \ and &); and
escape the quote characters (") which would otherwise terminate the JSON String value, as well as escape the JSON escape char (\).