Bash script - unable to replace string with double quotes and curly braces - json

I'm struggling for some time and I would need some help with the following operation.
I have a JSON file and I would like to replace a string with something a bit more complex.
This is a snippet of my json file:
{ "AWS679f53fac002430cb0da5b7982bd22872D164C4C": {
"Type": "AWS::Lambda::Function",
"Properties": {
"Code": {
"S3Bucket": "hnb659fds-assets-xxccddff",
"S3Key": "68b4ffa1c39cb3733535725f85311791c09eab53b7ab8efa5152e68f8abdb005.zip"
},
"Role": {
"Fn::GetAtt": [
"AWS679f53fac002430cb0da5b7982bd2287ServiceRoleC1EA0FF2",
"Arn"
]
},
"Handler": "index.handler",
"Runtime": "nodejs12.x",
"Timeout": 120
},
"DependsOn": [
"AWS679f53fac002430cb0da5b7982bd2287ServiceRoleC1EA0FF2"
],
"Metadata": {
"aws:cdk:path": "CODE/AWS679f53fac002430cb0da5b7982bd2287/Resource",
"aws:asset:path": "asset.68b4ffa1c39cb3733535725f85311791c09eab53b7ab8efa5152e68f8abdb005",
"aws:asset:is-bundled": false,
"aws:asset:property": "Code"
}
}
}
What I need is to replace this part
"S3Bucket": "hnb659fds-assets-xxccddff",
and have the following result
"S3Bucket": {"Fn::Sub": "AAA-${AWS::Region}" },
I don't know the AWS679f53fac002430cb0da5b7982bd22872D164C4C. It is generated randomly and the string to replace is present several times in my json file.
The initial values to be replaced is stored in a variable along with the new value to be used in the replaced version as following:
cdk_bucket_name=hnb659fds-assets-xxccddff
OUTPUT_BUCKET=AAA
I need these variables because this is part of a bigger script
So I tried some sed but does not work
new_bucket_name="{"Fn::Sub\": \"$OUTPUT_BUCKET-${AWS::Region}\" }"
sed -i "s#$cdk_bucket_name#$new_bucket_name#g" my.template.json
One issue that I have is that ${AWS::Region} gets interpreted so is empty.
And second, I cannot manage the quotes in order to have my desired result.

Using sed
$ output_bucket=AAA
$ new_bucket_name="{\"Fn::Sub\": \"$output_bucket-\${AWS::Region}\" }"
$ cdk_bucket_name=hnb659fds-assets-xxccddff
$ sed s"/\"$cdk_bucket_name\"/$new_bucket_name/" input_file
{ "AWS679f53fac002430cb0da5b7982bd22872D164C4C": {
"Type": "AWS::Lambda::Function",
"Properties": {
"Code": {
"S3Bucket": {"Fn::Sub": "AAA-${AWS::Region}" },
"S3Key": "68b4ffa1c39cb3733535725f85311791c09eab53b7ab8efa5152e68f8abdb005.zip"
},
"Role": {
"Fn::GetAtt": [
"AWS679f53fac002430cb0da5b7982bd2287ServiceRoleC1EA0FF2",
"Arn"
]
},
"Handler": "index.handler",
"Runtime": "nodejs12.x",
"Timeout": 120
},
"DependsOn": [
"AWS679f53fac002430cb0da5b7982bd2287ServiceRoleC1EA0FF2"
],
"Metadata": {
"aws:cdk:path": "CODE/AWS679f53fac002430cb0da5b7982bd2287/Resource",
"aws:asset:path": "asset.68b4ffa1c39cb3733535725f85311791c09eab53b7ab8efa5152e68f8abdb005",
"aws:asset:is-bundled": false,
"aws:asset:property": "Code"
}
}
}

Using a proper JSON parser shell tool like jq:
jq '
(
.[].Properties.Code.S3Bucket |
select(. == "hnb659fds-assets-xxccddff")
) = $newS3Bucket
' input_file.json \
--argjson newS3Bucket '{"Fn::Sub":"AAA-${AWS::Region}"}'

Related

How to query my JSON file for specific key

I am working on a project where I want to query a JSon using Jquery but I am getting an error:
jq: error: syntax error, unexpected IDENT, expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
..id
jq: 1 compile error
Error: Process completed with exit code 3.
I am using the command:
jq '..id' new_file.json
To query my JSON file which is named new_file.json
Here is my JSON file:
[
{
"type": "users_export",
"status": "pending",
"connection_id": "con_kmZIjREZWYzt39JI",
"format": "json",
"limit": 5,
"fields": [
{
"name": "user_id"
},
{
"name": "user_id",
"export_as": "provider"
},
{
"name": "username"
},
{
"name": "username",
"export_as": "provider"
},
{
"name": "nickname"
},
{
"name": "email"
},
{
"name": "email"
},
{
"name": "identities[0].connection",
"export_as": "provider"
},
{
"name": "email_verified",
"export_as": "provider"
}
],
"connection": "dev-default-evoyanbs",
"created_at": "2022-11-16T17:45:58.429Z",
"id": "job_aztDgKXWT8g8iZ5T"
}
]
I want the 'job_aztDgKXWT8g8iZ5T' as my output but I am getting the above mentioned error, can someone please help me out, thanks.
As you have an array, the command should be:
jq '.[].id' new_file.json
And the result will be: "job_aztDgKXWT8g8iZ5T".
If you want to get rid of double quotes in your output, you can add raw-output flag in your command, like this:
jq -r '.[].id' new_file.json
or
jq --raw-output '.[].id' new_file.json

how to denormalise this json structure

I have a json formatted overview of backups, generated using pgbackrest. For simplicity I removed a lot of clutter so the main structures remain. The list can contain multiple backup structures, I reduced here to just 1 for simplicity.
[
{
"backup": [
{
"archive": {
"start": "000000090000000200000075",
"stop": "000000090000000200000075"
},
"info": {
"size": 1200934840
},
"label": "20220103-122051F",
"type": "full"
},
{
"archive": {
"start": "00000009000000020000007D",
"stop": "00000009000000020000007D"
},
"info": {
"size": 1168586300
},
"label": "20220103-153304F_20220104-081304I",
"type": "incr"
}
],
"name": "dbname1"
}
]
Using jq I tried to generate a simpeler format out of this, until now without any luck.
What I would like to see is the backup.archive, backup.info, backup.label, backup.type, name combined in one simple structure, without getting into a cartesian product. I would be very happy to get the following output:
[
{
"backup": [
{
"archive": {
"start": "000000090000000200000075",
"stop": "000000090000000200000075"
},
"name": "dbname1",
"info": {
"size": 1200934840
},
"label": "20220103-122051F",
"type": "full"
},
{
"archive": {
"start": "00000009000000020000007D",
"stop": "00000009000000020000007D"
},
"name": "dbname1",
"info": {
"size": 1168586300
},
"label": "20220103-153304F_20220104-081304I",
"type": "incr"
}
]
}
]
where name is redundantly added to the list. How can I use jq to convert the shown input to the requested output? In the end I just want to generate a simple csv from the data. Even with the simplified structure using
'.[].backup[].name + ":" + .[].backup[].type'
I get a cartesian product:
"dbname1:full"
"dbname1:full"
"dbname1:incr"
"dbname1:incr"
how to solve that?
So, for each object in the top-level array you want to pull in .name into each of its .backup array's elements, right? Then try
jq 'map(.backup[] += {name} | del(.name))'
Demo
Then, generating a CSV output using jq is easy: There is a builtin called #csv which transforms an array into a string of its values with quotes (if they are stringy) and separated by commas. So, all you need to do is to iteratively compose your desired values into arrays. At this point, removing .name is not necessary anymore as we are piecing together the array for CSV output anyway. And we're giving the -r flag to jq in order to make the output raw text rather than JSON.
jq -r '.[]
| .backup[] + {name}
| [(.archive | .start, .stop), .name, .info.size, .label, .type]
| #csv
'
Demo
First navigate to backup and only then “print” the stuff you’re interested.
.[].backup[] | .name + ":" + .type

Create merged JSON array from multiple files using jq

I have multiple JSON files one.json, two.json, three.json with the below format and I want to create a consolidated array from them using jq. So, from all the files I want to extract Name and Value field inside the Parameters and use them to create an array where the id value will be constructed from the Name value and value field will be constructed using Value field value.
input:
one.json:
{
"Parameters": [
{
"Name": "id1",
"Value": "one",
"Version": 2,
"LastModifiedDate": 1581663187.36
}
]
}
two.json
{
"Parameters": [
{
"Name": "id2",
"Value": "xyz",
"Version": 2,
"LastModifiedDate": 1581663187.36
}
]
}
three.json
{
"Parameters": [
{
"Name": "id3",
"Value": "xyz",
"Version": 2,
"LastModifiedDate": 1581663187.36
}
]
}
output:
[
{
"id": "id1",
"value": "one"
},
{
"id": "id2",
"value": "xyz"
},
{
"id": "id3",
"value": "xyz"
}
]
How to achieve this using jq
You can use a reduce expression instead of slurping the whole file into memory (-s); by iterative manipulation of the input file contents and then appending the required fields one at a time.
jq -n 'reduce inputs.Parameters[] as $d (.; . + [ { id: $d.Name, value: $d.Value } ])' one.json two.json three.json
The -n flag is to ensure that we construct the output JSON data from scratch over the input file contents made available over the inputs function. Since reduce works in an iterative manner, for each of the object in the input, we create a final array, creating the KV pair as desired.

Get formatted data in shell script which is read from file containing JSON data

Writting one shell script to automatically get list of name, current and latest available version from raw json data.
I am trying to format JSON data stored in file using shell script. I tried using JQ command line JSON parser.
I want to get formatted JSON data in script. Their is advanced option provided in JQ for same scenario. I am not able to use it properly.
Example: File containing Following JSON
{
"endpoint": {
"name": "test-plugin",
"version": "0.0.1"
},
"dependencies": {
"plugin1": {
"main": {
"name": "plugin1name",
"description": "Dummy text"
},
"pkgMeta": {
"name": "plugin1name",
"version": "0.0.1"
},
"dependencies": {},
"versions": [
"0.0.5",
"0.0.4",
"0.0.3",
"0.0.2",
"0.0.1"
],
"update": {
"latest": "0.0.5"
}
},
"plugin2": {
"main": {
"name": "plugin2name",
"description": "Dummy text"
},
"pkgMeta": {
"name": "plugin2name",
"version": "0.1.1"
},
"dependencies": {},
"versions": [
"0.1.5",
"0.1.4",
"0.1.3",
"0.1.2",
"0.1.1"
],
"update": {
"latest": "0.1.5"
}
}
}
}
Trying to get result in format
[{name: "plugin1name",
c_version: "0.0.1",
n_version: "0.0.5"
},
{name: "plugin2name",
c_version: "0.1.1",
n_version: "0.1.5"}]
Can someone suggest anything ?
Your json file is not valid at: .dependencies.pkgMeta.version.
After fixing your json file, try this command:
jq '
.dependencies |
to_entries |
map(.value |
{
name: .main.name,
c_version: .pkgMeta.version,
n_version: .update.latest
}
)' input.json
The result is:
[
{
"name": "plugin1name",
"c_version": "0.0.1",
"n_version": "0.0.5"
},
{
"name": "plugin2name",
"c_version": "0.1.1",
"n_version": "0.1.5"
}
]

How to add properties to topojson file?

Given a data.tsv file such :
id code name
1 AL Alabama
2 AK Alaska
4 AZ Arizona
5 AR Arkansas
6 CA California
... ... ...
Given a topojson.json file such : (the structure is correct, the numeral values are random)
{
"type":"Topology",
"transform":
{
"scale": [0.0015484881821515486,0.0010301030103010299],
"translate":[-5.491666666666662,41.008333333333354]
},
"objects":
{
"states":
{
"type":"GeometryCollection",
"geometries":
[
{"type":"Polygon","arcs":[[0]],"properties":{"code_2":"AL"}},
{"type":"Polygon","arcs":[[1]],"properties":{"code_2":"AK"}}
]
}
},
"arcs":
[
[[2466,9916],[-25,-5],[3,-13]],
[[2357,9852],[1,-2],[1,-2]]
]
}
How to use the common fields(1) to inject the values of an other field(2) into the json file ?
1]: data.txt#code and topojson.txt.objects.states.geometries.properties.code_2
2]: data.txt#name
The end result should contains :
{"type":"Polygon","arcs":[[0]],"properties":{"code_2":"AL", "name":"Alabama" }},
{"type":"Polygon","arcs":[[1]],"properties":{"code_2":"AK", "name":"Alaska" }},
EDIT: Accepted answer:
topojson -o final.json -e data.tsv --id-property=code_2,code -p code_2,state=name -- topojson.json
Try using this:
topojson -o final.json -e data.tsv \
--id-property=code_2,code -p code_2,state=name \
-- topojson.json
Which should output:
{
"type": "Topology",
"transform": {
"scale": [
0.000016880209206372492,
0.000007005401010148724
],
"translate": [ -1.8418800213354616, 51.15278777877789 ]
},
"objects": {
"states": {
"type": "GeometryCollection",
"geometries": [
{
"type": "Polygon",
"arcs": [
[ 0 ]
],
"id": "AK",
"properties": {
"code_2": "AK",
"state": "Alaska"
}
}
]
}
},
"arcs": [
[[2466,9916],[-25,-5],[3,-13]],
[[2357,9852],[1,-2],[1,-2]]
]
}
From the Command Line Reference wiki:
--id-property name of feature property to promote to geometry id
By using the code_2 property with this option, you promote it as the feature ID.
Prepend a + in front of the input property name to coerce its value to a number.
Plus:
If the properties referenced by --id-property are null or undefined,
they are omitted from the output geometry object. Thus, the generated
objects may not have a defined ID if the input features did not have a
property with the specified name.
So, when you are using +code and +code_2, they are probably undefined, as you can't convert the AK string value to a number.
Here, the input property "FIPS" is coerced to a number and used as the
feature identifier; likewise, the column named "FIPS" is used as the
identifier in the CSV file. (If your CSV file uses a different column
name for the feature identifier, you can specify multiple id
properties, such as --id-property=+FIPS,+id.)
That's why you have to add the code to the --id-property=code_2,code option. This is how the mapping is made (the code_2 from topojson.json and the code column from data.tsv).
Then, the output property "unemployment" is generated from the
external data file, unemployment.tsv, which defines the input property
"rate"
In our case, -p code_2,state=name specifies that we will preserve the code_2 property and we will rename the name property to state. The Properties and External Properties sections in the aforementioned documentation wiki are pretty informative on the matter.
The topojson package has been deprecated. The following steps are based on the command-line cartography workflow. These interfaces are more flexible, but a little bit more complicated to use.
Install dependencies:
npm install d3-dsv ndjson-cli
Add the node_modules/.bin directory to the path so that you can easily run the commands:
PATH=$(npm bin):$PATH
Convert the tsv file into a newline-delimited json file:
tsv2json data.tsv -n > data.ndjson
{"id":"1","code":"AL","name":"Alabama"}
{"id":"2","code":"AK","name":"Alaska"}
Parse the id column as a number:
ndjson-map '{id: +d.id, code: d.code, name: d.name}' < data.ndjson > data_parsed.ndjson
{"id":1,"code":"AL","name":"Alabama"}
{"id":2,"code":"AK","name":"Alaska"}
Extract the geometries of the topojson file:
ndjson-cat topojson.json | ndjson-split 'd.objects.states.geometries' > topojson_geometries.ndjson
{"type":"Polygon","arcs":[[0]],"properties":{"code_2":"AK"}}
{"type":"Polygon","arcs":[[1]],"properties":{"code_2":"AL"}}
Join both newline-delimited json files:
ndjson-join 'd.properties.code_2' 'd.code' topojson_geometries.ndjson data_parsed.ndjson > geometries_data_join.ndjson
[{"type":"Polygon","arcs":[[0]],"properties":{"code_2":"AK"}},{"id":2,"code":"AK","name":"Alaska"}]
[{"type":"Polygon","arcs":[[1]],"properties":{"code_2":"AL"}},{"id":1,"code":"AL","name":"Alabama"}]
Add the name column to the topojson properties and only keep the topojson geometries:
ndjson-map 'd[0].properties.name = d[1].name, d[0]' < geometries_data_join.ndjson > geometries_data_merge.ndjson
{"type":"Polygon","arcs":[[0]],"properties":{"code_2":"AK","name":"Alaska"}}
{"type":"Polygon","arcs":[[1]],"properties":{"code_2":"AL","name":"Alabama"}}
Convert the previous result into an array and concat it with the original topojson file:
ndjson-join <(ndjson-cat topojson.json) <(ndjson-reduce < geometries_data_merge.ndjson) > topojson_concat.ndjson
[{
"type": "Topology",
"transform": {
"scale": [0.0015484881821515486, 0.0010301030103010299],
"translate": [-5.491666666666662, 41.008333333333354]
},
"objects": {
"states": {
"type": "GeometryCollection",
"geometries": [{
"type": "Polygon",
"arcs": [[0]],
"properties": {
"code_2": "AK"
}
}, {
"type": "Polygon",
"arcs": [[1]],
"properties": {
"code_2": "AL"
}
}
]
}
},
"arcs": [[[2466, 9916], [-25, -5], [3, -13]], [[2357, 9852], [1, -2], [1, -2]]]
}, [{
"type": "Polygon",
"arcs": [[0]],
"properties": {
"code_2": "AK",
"name": "Alaska"
}
}, {
"type": "Polygon",
"arcs": [[1]],
"properties": {
"code_2": "AL",
"name": "Alabama"
}
}
]
]
Overwrite the geometries of original topojson file and save it as a normal json file:
ndjson-map 'd[0].objects.states.geometries = d[1], d[0]' < topojson_concat.ndjson > topojson_data.json
{
"type": "Topology",
"transform": {
"scale": [0.0015484881821515486, 0.0010301030103010299],
"translate": [-5.491666666666662, 41.008333333333354]
},
"objects": {
"states": {
"type": "GeometryCollection",
"geometries": [{
"type": "Polygon",
"arcs": [[0]],
"properties": {
"code_2": "AK",
"name": "Alaska"
}
}, {
"type": "Polygon",
"arcs": [[1]],
"properties": {
"code_2": "AL",
"name": "Alabama"
}
}
]
}
},
"arcs": [[[2466, 9916], [-25, -5], [3, -13]], [[2357, 9852], [1, -2], [1, -2]]]
}
All commands in one line:
ndjson-join <(ndjson-cat topojson.json) <(ndjson-join 'd.properties.code_2' 'd.code' <(ndjson-cat topojson.json | ndjson-split 'd.objects.states.geometries') <(tsv2json data.tsv -n | ndjson-map '{id: +d.id, code: d.code, name: d.name}') | ndjson-map 'd[0].properties.name = d[1].name, d[0]' | ndjson-reduce) | ndjson-map 'd[0].objects.states.geometries = d[1], d[0]' > topojson_data.json
Notes:
I swapped "AK" and "AL" in the topojson file to check if the join really works.
The last command (before the one-liner) only works on the original output and not on the given pretty-printed version, which has newlines in it.
I tested the workflow on the subsystem for Linux since ndjson-map does not seem to work properly on Windows currently.