This question already has answers here:
Modify a key-value in a json using jq in-place
(8 answers)
Closed 1 year ago.
I have a daemon.json file which contains one line as below
{ "insecure-registries":["192.X.X.X:8123"] }
I am trying to use a variable to change generically to the current IP address. In bash script normally
I'd store in a variable like
myip=hostname -I | awk '{print $1}'
{ "insecure-registries":["$myip:8123"] }
How to use a kind of variable in JSON file?
If you have access to jq, I would recommend storing not JSON, but a jq filter, like
{"insecure-registries": ["\($ip):8123"]}
Assume the preceding is in file named foo.jq; then using jq as follows to produce JSON from the filter.
$ myip=$(hostname -I | awk '{print $1}') # 192.0.2.42, e.g.
$ jq -nf foo.jq --arg ip "$myip"
{
"insecure-registries": [
"192.0.2.42:8123"
]
}
JSON itself doesn't have a notion of substitution, and bash itself isn't really suitable for making substitutions like this.
Related
This question already has answers here:
How to remove double-quotes in jq output for parsing json files in bash?
(2 answers)
Closed 1 year ago.
The Problem:
I am trying to fetch credentials from AWS Secrets Manager in my terminal, however the Keys and Values I want needs to be in JSON, however they come with a lot of escape characters due to quotes.
The Scenario:
After I fire the aws secretsmanager get-secret-value --secret-id snowflake-access-uat command, I get the credentials as below:
{
"ARN": "arn:aws:secretsmanager:ap-regionnm-1:111111111111:secret:my-secret",
"Name": "snowflake-access-uat",
"VersionId": "dont-care",
"SecretString": "{\"sf-user\":\"USER_123_ADMIN\",\"sf-password\":\"FooBaarPassword\",\"sf-db\":\"MY_SPL_DB\",\"wh_name\":\"JOB_EXECUTOR\",\"sf-role\":\"JOB_EXECUTOR_ROLE\",\"sf-account\":\"icy-party\"}",
"VersionStages": [
"AWSCURRENT"
],
"CreatedDate": 1627104812.142
}
However, I am interested in Secret String only, for which I fire aws secretsmanager get-secret-value --secret-id snowflake-programmatic-access-uat | jq '.SecretString' command and receive this:
"{\"sf-user\":\"USER_123_ADMIN\",\"sf-password\":\"FooBaarPassword\",\"sf-db\":\"MY_SPL_DB\",\"wh_name\":\"JOB_EXECUTOR\",\"sf-role\":\"JOB_EXECUTOR_ROLE\",\"sf-account\":\"icy-party\"}"
But since it has multiple escape characters, I am unable to leverage it with jq tree. I tried to get from this link for reference but I'm unable to make it work. Besides, I need the Keys and Values to be variables in my bash session.
NOTE: I cannot use any third party tools, since I need to automate this on CodeBuild (Run time fresh instance will be selected)
The escape characters are there because you don't use -r with jq '.SecretString'. Change it to jq -r '.SecretString' and your output will instead be:
{"sf-user":"USER_123_ADMIN","sf-password":"FooBaarPassword","sf-db":"MY_SPL_DB","wh_name":"JOB_EXECUTOR","sf-role":"JOB_EXECUTOR_ROLE","sf-account":"icy-party"}
...which, being valid JSON, you can feed back into jq -r to retrieve individual fields.
SecretStringJson=$(... | jq -r '.SecretString')
### one jq call per field isn't the most efficient possible way but it's easy
sfUser=$(jq -r '.["sf-user"]' <<<"$SecretStringJson")
sfDb=$(jq -r '.["sf-db"]' <<<"$SecretStringJson")
# ...etc
This question already has an answer here:
"Argument list too long" while slurping JSON files [duplicate]
(1 answer)
Closed 1 year ago.
There are more than 6k JSON files, each containing exactly one JSON object. I want to prepare one list of objects from these JSONs.
When I am running below jq command I am getting an error.
Kedar.Javalkar#KD2806 MINGW64 /c/zz
$ jq -s '.' inventoryItem_*.json > inventory_items_result_$(date +"%Y%m%d_%H%M%S").json
bash: /usr/bin/jq: Argument list too long
I tried ulimit -s unlimited but the same error
I am using a windows 10 git bash
This is a job that xargs is created to fix -- splitting lists of items into individual command lines that are within the permitted limit.
Because running jq -s a single time is different from concatenating the results of multiple smaller runs, it's appropriate to use xargs to combine cat invocations using the manner described in the linked duplicate.
printf '%s\0' inventoryItem_*.json \
| xargs -0 cat \
| jq -s . \
>"inventory_items_result_$(date +"%Y%m%d_%H%M%S").json"
This question already has answers here:
Parsing JSON with Unix tools
(45 answers)
Bash script store command output into variable
(2 answers)
Closed 3 years ago.
I am using curl to create a session to log into the switch.
Below the script that i use
curl --noproxy 10.23.193.1 -X POST http://10.23.193.1:80/rest/v3/login-sessions -d '{"userName":"admin", "password":"password"}'
After this is executed I get the following output
{"uri":"/login-sessions","cookie":"sessionId=DfZNCFbfoc3LDuMgjLXRiS8ZmEo4MWENCOJM0Iu14R1uMT9kKqbe1Rx6AedmoeT"}
My requirement here is to first only print this part of the string "sessionId=DfZNCFbfoc3LDuMgjLXRiS8ZmEo4MWENCOJM0Iu14R1uMT9kKqbe1Rx6AedmoeT"
Secondly I would want to know how to store the above string in a variable so that I could call the same variable for subsequent operations
I ran the following, but I am not getting any output.
curl --noproxy 10.23.193.1 -X POST http://10.23.193.1:80/rest/v3/login-sessions -d '{"userName":"admin", "password":"password"}' | grep -`E ""cookie":"
Avoid using simple tools like grep or sed to parse JSON as they won't handle things like quotes or multi-line data correctly. It's best to use a JSON-aware program such as jq.
With jq it's simple and robust:
curl ... | jq '.cookie'
To store the cookie in a variable use the -r flag to have JQ print out the raw, unquoted string.
cookie=$(curl ... | jq -r '.cookie')
Further reading:
jq Manual
Parsing JSON with Unix tools
How to parse JSON with shell scripting in Linux?
This question already has answers here:
Passing bash variable to jq
(10 answers)
Closed 5 years ago.
json processor jq for a package.json
{
"someparent" : {
"somechild" : "oldvalue"
}
}
If I run the following command (to change oldvalue to somevalue):
jq '.someparent["somechild"] = "somevalue" "$aDirection/package.json"'
It is successfully changed. However, if I give a variable instead of someValue:
aVar="anotherValue"
jq '.someparent["somechild"] = "$aVar" "$aDirection/package.json"'
It is not working.
What I already tried:
["aVar"] #interpreted as string
["$aVar"] #interpreted as string
"$aVar" # aVar is not defined
$avar #aVar is not defined
The single quotes around the jq filter are in the wrong place. In your attempt made it is also enclosing the actual JSON file input. The right syntax when applying filter over files is
jq '<filter>' json-file
In your case the right filter is just
.someparent["somechild"] = $aVar
and to use a shell variable in to the jq, you need to use the --arg field. Putting together both of the options. The first variable after --arg is used in the context of jq and the one following it is the actual shell variable.
aVar="anotherValue"
jq --arg aVar "$aVar" '.someparent["somechild"] = $aVar' "$aDirection/package.json"
# ^^^^ Remember the single quote
# terminating here and not extended
Just to be clear: using the OP's approach can be made to work, e.g.
jq '.someparent["somechild"] = "'"$aVar"'"' "$aDirection/package.json"
but using --arg (for strings) and/or --argjson is far better. An additional option for environment variables is to use the jq function env.
This question already has answers here:
Read JSON data in a shell script [duplicate]
(4 answers)
Closed 7 years ago.
Let's say that we have this kind of JSON file:
{
...
"quotes":{
"SOMETHING":10,
...
"SOMETHING_ELSE":120.4,
...
} }
How can I obtain those values and use them in order to add them together?
Am I able to do even this?
#!/bin/bash
#code ...
echo "$SOMETHING + $SOMETHING_ELSE" | bc
#code ...
#exit
I will obtain the JSON file with wget command. All I want is the content from this file.
Can you help me, please? I am a beginner in shell programming.
I usually use jq, a really fast json parser, to do this kind of things (because parsing a json file with tools like awk or sed is really error-prone).
Given an input file like this:
# file: input.json
{
"quotes":{
"SOMETHING":10,
"SOMETHING_ELSE":120.4
}
}
You can obtain the sum of the 2 fields with a simple filter:
jq '.quotes.SOMETHING + .quotes.SOMETHING_ELSE' input.json
# output -> 130.4
NOTE: jq is available in every major linux distribution. In a debian-derivative system you can install with a sudo apt-get install jq.
This will print out the sum of the selected lines' floats.
#!/bin/bash
awk '{ if ($1 ~ /"SOMETHING":/) {print}; if ($1 ~ /"SOMETHING_ELSE":/) {print} }' $1 | cut -d: -f2 | cut -d, -f1 | awk '{s+=$1};END{print s}'
This finds the lines you want, the plucks out the numbers, and adds them.
You should look up and learn jq as shown in Read the json data in shell script.
The tools in a "normal" shell installation like awk and sed all predate JSON by decades, and are a very very bad fit. jq is worth the time to learn.
Or use Python instead.