jq read .txt file and write the values to json file - json

I want to use jq to parse a .txt file with a list of country codes and write them to the value in a JSON object.
Here is what I have so far:
cat myfile.json |
jq -R -f test_id.txt 'select(.country == []).country = "test_id.txt"' > newfile.json
Where .txt file looks like this:
"NSC"
"KZC"
"KCC"
"KZL"
"NZG"
"VRU"
"ESM"
"KZF"
"SFU"
"EWF"
"KQY"
"KQV"
and my JSON looks like this:
{
"scsRequestId": null,
"includeMetadata": true,
"includeHoldings": true,
"country": [],
"region": [],
"oclcSymbol": []
}
Here is the error I am getting:
jq: error: syntax error, unexpected QQSTRING_START, expecting $end (Unix shell quoting issues?) at <top-level>, line 2:
"KZC"
jq: 1 compile error
I want the list of country codes to go into the country array.

-f's argument is the file to read the filter to run from. If you want to read data from a file, that's a use for --slurpfile, not -f.
Thus:
jq --slurpfile countries test_id.txt '.country=$countries' <myfile.json >newfile.json
When run with your provided inputs, the resulting contents in newfile.json are:
{
"scsRequestId": null,
"includeMetadata": true,
"includeHoldings": true,
"country": [
"NSC",
"KZC",
"KCC",
"KZL",
"NZG",
"VRU",
"ESM",
"KZF",
"SFU",
"EWF",
"KQY",
"KQV"
],
"region": [],
"oclcSymbol": []
}

Related

Retrieving value from JSON object in bash using JQ

I have the following script:
#!/bin/bash
CONFIG_RECORDER=`aws configservice describe-configuration-recorders`
NAME=$(jq -r ‘.ConfigurationRecorders[].name’ <<<“$CONFIG_RECORDER”)
echo $NAME
I am trying to retrieve the value for name from the following JSON object:
{
"ConfigurationRecorders": [{
"name": "default",
"roleARN": "arn:aws:iam::xxxxxxxxxxxx:role/Config-Recorder",
"recordingGroup": {
"allSupported": true,
"includeGlobalResourceTypes": true,
"resourceTypes": []
}
}]
}
When running the script, I am getting an error stating that jq could not open the file. That is because I am trying to pass in the result stored in a variable. How can I move past this? Below is the error:
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end
(Unix shell quoting issues?) at <top-level>, line 1:
‘.ConfigurationRecorders[].name’
jq: 1 compile error
NAME=$(jq -r '.ConfigurationRecorders[].name' <<<"$CONFIG_RECORDER")
<<< is known as here-string.
-r removes the quotes in the result.
[] in the jq query is necessary as ConfigurationRecorders is a JSON array.
$( … ) is just an other form of command substitution than backticks. I prefer this one as it is more readable to me.

How to add a new JSON object in an existing JSON file using jq and variable arguments

I have a json here which looks like:
{
"cluster": "bvt-rtp-123",
"state": "installed",
"timestamp": "2022-02-14T10:23:01Z"
}
I want to use parameters/environment variables to dynamically add an object to that JSON object using jq
The result should look like:
{
"cluster": "bvt-rtp-123",
"state": "installed",
"timestamp": "2022-02-14T10:23:01Z",
"aiops": {
"catalog_source": "abc.com/123",
"channel": "dev"
}
}
wherein aiops, catalog_source and channel are parameterized by environment variables in this way:
parent_key=aiops
child_key=catalog_source
child_val=abc.com/123
I already tried this method,
cat test.json | jq --arg parent "$parent_key" --arg child "$child_key" --arg child_val "$payload_val" '.[$key].[$child] = $child_val'
But it throws this error:
jq: error: syntax error, unexpected '[', expecting FORMAT or QQSTRING_START (Unix shell quoting issues?) at <top-level>, line 1:
.[$key].[$child] = $child_val
jq: 1 compile error
Define them using --arg, then set them using +=. That way you can always add more subitems.
jq --arg parent_key aiops \
--arg child_key catalog_source \
--arg child_val abc.com/123 \
'.[$parent_key] += {($child_key): $child_val}' input.json
{
"cluster": "bvt-rtp-123",
"state": "installed",
"timestamp": "2022-02-14T10:23:01Z",
"aiops": {
"catalog_source": "abc.com/123"
}
}
Demo for the jq filter (simulating argument input with variable declaration)

jq bash Adding a json field to a json file

i'm stuck on jq input problem. I have a json file that looks like this:
{
"main_object": {
"child1": ["banana", "apple", "orange"]
}
}
I need to add another child object and rewrite this file, the problem is that this child object needs to be generated dynamically. so i'm doing this:
added_string=$(printf '.main_object += {%s: %s}' "$child_name" "$fruits")
Then I wrote this line, which worked well on my mac shell:
edited_json=$(cat $json_variable_file | jq $added_string)
When i tried to run all of this from a bash script i got this error:
jq: error: Could not open file +=: No such file or directory
jq: error: Could not open file {"child2":: No such file or directory
jq: error: Could not open file ["orange","potato","watermelon"]}: No such file or directory
So I tried many things so far, most of them still give me the same error, also tried doing this:
edited_json=$(cat $json_variable_file | jq <<< $added_string)
The error i got is this:
parse error: Invalid numeric literal at line 1, column 23
Really appreciate your time, the weird thing here is that it works completely fine, generating the needed json on my zsh but it does not work on bash.
With bash and zsh:
child_name="child2"
fruits='["orange","potato","watermelon"]'
added_string=$(printf '.main_object += {%s: %s}' "$child_name" "$fruits")
cat file | jq "$added_string" # quotes are important
Output:
{
"main_object": {
"child1": [
"banana",
"apple",
"orange"
],
"child2": [
"orange",
"potato",
"watermelon"
]
}
}

How to dynamically parse a JSON object with shell jq

I got a question about shell's jq. So my JSON object is:
{"1543446000": {"name": "John", "company": "foo"}, "1543359600": {"name": "Kate", "company": "bar"}}
The numers 1543446000 and 1543359600 are UNIX timestamps. How can I parse one of the JSON objects by the timestamp with a shell variable?
My shell script so far:
#!/bin/sh
URL="https://pastebin.com/raw/w7awz7kZ"
export DATE=$(date -d "$today 0" +%s)
JSON=$(curl -H "Accept: application/json" $API_URL)
JSON=$(echo $JSON | jq --arg date $DATE '.$date')
echo $JSON
Doesn't seem to work. My intention is to select the inner JSON object described by one of the timestamps, which are basically midnight of today. So I want to select today's data set.
Any suggestions?
Greets,
Innoberger
You need to use the full syntax for key access, as the dollar sign preclude you using the shorter form. The error message should provide this suggestion.
$ jq --arg date 1543359600 '.$date' tmp.json
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.$date
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.$date
jq: 2 compile errors
Note the error message
'try .["field"] instead of .field'.
You won't need the quotes, though, as that would be how you specify a literal key $date.
$ jq --arg date 1543359600 '.[$date]' tmp.json
{
"name": "Kate",
"company": "bar"
}

Need help! - Unable to load JSON using COPY command

Need your expertise here!
I am trying to load a JSON file (generated by JSON dumps) into redshift using copy command which is in the following format,
[
{
"cookieId": "cb2278",
"environment": "STAGE",
"errorMessages": [
"70460"
]
}
,
{
"cookieId": "cb2271",
"environment": "STG",
"errorMessages": [
"70460"
]
}
]
We ran into the error - "Invalid JSONPath format: Member is not an object."
when I tried to get rid of square braces - [] and remove the "," comma separator between JSON dicts then it loads perfectly fine.
{
"cookieId": "cb2278",
"environment": "STAGE",
"errorMessages": [
"70460"
]
}
{
"cookieId": "cb2271",
"environment": "STG",
"errorMessages": [
"70460"
]
}
But in reality most JSON files from API s have this formatting.
I could do string replace or reg ex to get rid of , and [] but I am wondering if there is a better way to load into redshift seamlessly with out modifying the file.
One way to convert a JSON array into a stream of the array's elements is to pipe the former into jq '.[]'. The output is sent to stdout.
If the JSON array is in a file named input.json, then the following command will produce a stream of the array's elements on stdout:
$ jq ".[]" input.json
If you want the output in jsonlines format, then use the -c switch (i.e. jq -c ......).
For more on jq, see https://stedolan.github.io/jq