I am trying to fill JSON template with increasing counter to generate huge sample data set:
#!/bin/bash
for ((counter=1 ; counter<2 ;counter++ ))
do
NUMBER=${counter}
JSON=$(cat template.json | jq --arg NUMBER "$NUMBER" '.')
echo $JSON
#aws dynamodb batch-write-item --request-items "${JSON}"
done
My template.json looks like:
{
"My_Table":[{
"PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_A"},"name":{ ...
"PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_A"},"name":{ ...
"PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_A"},"name":{ ...
"PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_B"},"name":{ ...
"PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_B"},"name":{ ...
"PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_B"},"name":{ ...
"PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_C"},"name":{ ...
"PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_C"},"name":{ ...
"PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_D"},"name":{ ...
"PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_E"},"name":{ ...
}]
}
Can I get any clue to insert bash variable into JSON template? I guess I am not doing correctly using jq here:
JSON=$(cat template.json | jq --arg NUMBER "$NUMBER" '.')
EDIT
My desired output:
{
"My_Table":[{
"PutRequest":{"Item":{"type":{"S":"test-1-Type_A"},"name":{ ...
"PutRequest":{"Item":{"type":{"S":"test-1-Type_A"},"name":{ ...
"PutRequest":{"Item":{"type":{"S":"test-1-Type_A"},"name":{ ...
"PutRequest":{"Item":{"type":{"S":"test-1-Type_B"},"name":{ ...
"PutRequest":{"Item":{"type":{"S":"test-1-Type_B"},"name":{ ...
"PutRequest":{"Item":{"type":{"S":"test-1-Type_B"},"name":{ ...
"PutRequest":{"Item":{"type":{"S":"test-1-Type_C"},"name":{ ...
"PutRequest":{"Item":{"type":{"S":"test-1-Type_C"},"name":{ ...
"PutRequest":{"Item":{"type":{"S":"test-1-Type_D"},"name":{ ...
"PutRequest":{"Item":{"type":{"S":"test-1-Type_E"},"name":{ ...
}]
}
If the template is intended to be used with jq, it should be changed to work correctly, rather than trying to force jq to work with substandard input.
{
"My_Table":[{
"PutRequest":{"Item":{"type":{"S":"test-\($NUMBER)-Type_A"},"name":{ ...
"PutRequest":{"Item":{"type":{"S":"test-\($NUMBER)-Type_A"},"name":{ ...
"PutRequest":{"Item":{"type":{"S":"test-\($NUMBER)-Type_A"},"name":{ ...
"PutRequest":{"Item":{"type":{"S":"test-\($NUMBER)-Type_B"},"name":{ ...
"PutRequest":{"Item":{"type":{"S":"test-\($NUMBER)-Type_B"},"name":{ ...
"PutRequest":{"Item":{"type":{"S":"test-\($NUMBER)-Type_B"},"name":{ ...
"PutRequest":{"Item":{"type":{"S":"test-\($NUMBER)-Type_C"},"name":{ ...
"PutRequest":{"Item":{"type":{"S":"test-\($NUMBER)-Type_C"},"name":{ ...
"PutRequest":{"Item":{"type":{"S":"test-\($NUMBER)-Type_D"},"name":{ ...
"PutRequest":{"Item":{"type":{"S":"test-\($NUMBER)-Type_E"},"name":{ ...
}]
}
Now something like your original attempt will work correctly. (Name your template template.jq to emphasize that it is not actually valid JSON.)
for ((counter=1 ; counter<2 ;counter++ ))
do
JSON=$(jq -n -f template.jq --arg NUMBER "$counter")
echo "$JSON"
#aws dynamodb batch-write-item --request-items "${JSON}"
done
Unfortunately there are several parts of the question that don't quite make sense, but I believe the following should get you on your way.
First, I'll assume your template is valid JSON:
{
"My_Table": [
{"PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_A"},"name":0}}},
{"PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_A"},"name":1}}},
{"PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_A"},"name":2}}},
{"PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_B"},"name":0}}},
{"PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_B"},"name":1}}},
{"PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_B"},"name":2}}},
{"PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_C"},"name":0}}},
{"PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_D"},"name":1}}},
{"PutRequest":{"Item":{"type":{"S":"test-$NUMBER-Type_E"},"name":0}}}
]
}
Second, I'll assume you want the result along the lines of what is shown, not as described, but the following is written so you can easily adapt the code to the described problem. Specifically, the functionality for making the substitution is encapsulated
in the following definition:
def resolve(s; value):
.My_Table |= map(.PutRequest.Item.type.S |=
sub("-" + s + "-"; "-" + (value|tostring) + "-" ));
This is written using sub, the first argument of which must be a regex. So, to generate the desired output for a single substitution of "$NUMBER" by "1", one would write:
resolve("\\$NUMBER"; 1)
Since I'm not sure what your bash snippet is supposed to do exactly, I'll just suggest that you could use iteration within jq to achieve whatever result you require, rather than using bash iteration.
Related
I am writing a bash script and I am looking to replace a character within a JSON field in a JSON array. In this case, I am trying to change the "." (period) character to a "-" (hyphen) in the name field. I am using jq to parse my JSON. Any tips on how I can achieve this will greatly help. Thank you!
Bash Script so far:
RAW=$(curl ${URL})
function manip() {
# Function for string manipulation.
}
echo "${RAW}" | jq '.data | .[].name = $manip' # Unable to make a function call in there.
Sample JSON:
[
{"id":"1","name":"a.a"},
{"id":"2","name":"b.b"},
{"id":"3","name":"c.c"}
]
Expected Output:
[
{"id":"1","name":"a-a"},
{"id":"2","name":"b-b"},
{"id":"3","name":"c-c"}
]
To replace a dot with a dash, use the sub function:
jq '.[].name |= sub("\\."; "-")' file.json
I have a javascript file which prints a JSON array of objects:
// myfile.js output
[
{ "id": 1, "name": "blah blah", ... },
{ "id": 2, "name": "xxx", ... },
...
]
In my bash script, I want to iterate through each object.
I've tried following, but it doesn't work.
#!/bin/bash
output=$(myfile.js)
for row in $(echo ${output} | jq -c '.[]'); do
echo $row
done
You are trying to invoke myfile.js as a command. You need this:
output=$(cat myfile.js)
instead of this:
output=$(myfile.js)
But even then, your current approach isn't going to work well if the data has whitespace in it (which it does, based on the sample you posted). I suggest the following alternative:
jq -c '.[]' < myfile.js |
while read -r row
do
echo "$row"
done
Output:
{"id":1,"name":"blah blah"}
{"id":2,"name":"xxx"}
Edit:
If your data is arising from a previous process invocation, such as mongo in your case, you can pipe it directly to jq (to remain portable), like this:
mongo myfile.js |
jq -c '.[]' |
while read -r row
do
echo "$row"
done
How can I make jq -c '.[]' < (mongo myfile.js) work?
In a bash shell, you would write an expression along the following lines:
while read -r line ; do .... done < <(mongo myfile.js | jq -c .[])
Note that there are two occurrences of "<" in the above expression.
Also, the above assumes mongo is emitting valid JSON. If it emits //-style comments, those would have somehow to be removed.
Comparison with piping into while
If you use the idiom:
... | while read -r line ; do .... done
then the bindings of any variables in .... will be lost.
Summary
Starting from a JSON array, I'd like to replace an attribute of each item by the contents of a text file.
Example:
We have this initial JSON array:
[
{ "name": "step-a", "message": "step-a message placeholder" },
{ "name": "step-b", "message": "step-b message placeholder" }
]
And two text files matching the name attribute values (with an added .txt):
.
├── step-a.txt # contains the text: "Step A error logs ..."
└── step-b.txt # contains the text: "Step B error logs ..."
The goal is to perform the replacement and end up with this JSON array:
[
{ "name": "step-a", "message": "Step A error logs ..." },
{ "name": "step-b", "message": "Step B error logs ..." }
]
Attempt
I tried something like this:
# the variable $INITIAL contains the initial JSON array
echo $INITIAL | jq -c '.[] | .message = "$(cat .displayName+".txt")"'
Is there a way to perform an operation like this with jq or is using extra bash logic necessary?
Thank you.
You can use input_filename to achieve the goal very simply and efficiently.
For example, with the following in program.jq
# program.jq
(reduce inputs as $step ( {}; .[input_filename | rtrimstr(".txt")] = $step )) as $dict
| $a
| map(. + {message: $dict[.name] })
the following invocation would yield the expected result:
jq -n -f program.jq --argfile a array.json step-*.txt
An alternative
Depending on your requirements, you might like to replace the last line of program.jq as given above by:
| map(.message = ($dict[.name] // .message))
Alternatives to --argfile
If you prefer not to use --argfile, feel free to use --argjson or even --slurpfile, with appropriate adjustments to the above.
You can pass variables/arguments to jq using:
jq -r --arg first $1
$1 being your variable (which you can do: _fileContents="$( cat step-a.txt )" )
first is the variable name you want to create (which you can use inside the jq command like a bash variable ( $_fileContents );
An example from one of my scripts:
export techUser=$(jq -r --arg first $1 '.Environment[].Servers[] | select (.Address | contains($first)) | ."Technical User"' $serverlist)
I hope this has helped you finding a way to make it work ;)
I would like to search a JSON file for some key or value, and have it print where it was found.
For example, when using jq to print out my Firefox' extensions.json, I get something like this (using "..." here to skip long parts) :
{
"schemaVersion": 31,
"addons": [
{
"id": "wetransfer#extensions.thunderbird.net",
"syncGUID": "{e6369308-1efc-40fd-aa5f-38da7b20df9b}",
"version": "2.0.0",
...
},
{
...
}
]
}
Say I would like to search for "wetransfer#extensions.thunderbird.net", and would like an output which shows me where it was found with something like this:
{ "addons": [ {"id": "wetransfer#extensions.thunderbird.net"} ] }
Is there a way to get that with jq or with some other json tool?
I also tried to simply list the various ids in that file, and hoped that I would get it with jq '.id', but that just returned null, because it apparently needs the full path.
In other words, I'm looking for a command-line json parser which I could use in a way similar to Xpath tools
The path() function comes in handy:
$ jq -c 'path(.. | select(. == "wetransfer#extensions.thunderbird.net"))' input.json
["addons",0,"id"]
The resulting path is interpreted as "In the addons field of the initial object, the first array element's id field matches". You can use it with getpath(), setpath(), delpaths(), etc. to get or manipulate the value it describes.
Using your example with modifications to make it valid JSON:
< input.json jq -c --arg s wetransfer#extensions.thunderbird.net '
paths as $p | select(getpath($p) == $s) | null | setpath($p;$s)'
produces:
{"addons":[{"id":"wetransfer#extensions.thunderbird.net"}]}
Note
If there are N paths to the given value, the above will produce N lines. If you want only the first, you could wrap everything in first(...).
Listing all the "id" values
I also tried to simply list the various ids in that file
Assuming that "id" values of false and null are of no interest, you can print all the "id" values of interest using the jq filter:
.. | .id? // empty
I'd like to conditionally do a recursive merge. That is, if a key exists in the second object, I'd like to use it to override values in the first. For example, this does what I want:
$ echo '{"a":"value"}{"bar": {"a":"override"}}' | jq -sS '.[0] * if (.[1].foo|length) > 0 then .[1].foo else {} end'
{
"a": "value"
}
$ echo '{"a":"value"}{"foo": {"a":"override"}}' | jq -sS '.[0] * if (.[1].foo|length) > 0 then .[1].foo else {} end'
{
"a": "override"
}
In the first example, the second object does not contain a "foo" key, so the override does not happen. In the 2nd example, the second object does contain "foo", so the value is changed. (In my actual use, I always have 3 objects on the input and sometimes have a 4th which may override some of the previous values.)
Although the above works, it seems absurdly ugly. Is there a cleaner way to do this? I imagine something like jq -sS '.[0] * (.[1].foo ? .[1].foo : {}) or similar.
With -n flag specified on the command line this should do the trick:
reduce inputs as $in (input; . * ($in.foo // {}))
jqplay demo