This question already has answers here:
Parsing JSON with Unix tools
(45 answers)
Closed 5 years ago.
Inside my bash script I've created a function that returns the following JSON array:
{
"access_token": "sjdhjdYdjsdbhggjhSJ78dbdsdbMhd=",
"scopes": "repository:delete repository:admin repository:write project:write team:write account:write",
"expires_in": 3600,
"refresh_token": "skdjKJDJDBNBDs",
"token_type": "bearer"
}
Now I wan't to grep the access token between the "". So I got sjdhjdYdjsdbhggjhSJ78dbdsdbMhd=
I've created a regex like below but this returns a grep: invalid repetition count(s) error.
grep -P '\{\s*"access_token"\s*:\s*(.+?)\s*\"' foobar.txt
Since you're dependending on nonstandard, nonportable functionality (in the form of grep -P), you might as well switch to a tool that's built for the job at hand (and actually understands JSON syntax, so it'll work even if your input has, say, a newline between the key and value, so long as it's syntactically valid):
jq -r .access_token <foobar.txt
Related
This question already has answers here:
change json file by bash script
(6 answers)
Closed 6 months ago.
I have a file called usr.json, and this file contains a JSON object. The object looks as following:
{
"users": {
"accessRights":"FFFFF",
"name": "admin"
}
}
Now I want to replace the string for accessRights with a new value which is stored in a variable called newUsrCfg.
Since I'm working with a remote device I need to ssh to it, and for that I have a script.
However, I call all this in Python code:
def test_tc():
newUsrCfg = "F7FFF"
cmd = f'sed -ie \'s/\"accessRights\":.*/\"accessRights\": \"{newUsrCfg}\"/g\' "/etc/hgp/runtime/user/user1.json"'
subprocess.call(["bash", pathToSshCommand, cmd])
The commands need to be either called with single quotes or double ones. And when opening the file the output of this is:
{
"users":{
"accessRights": "f7fff"
I tried many other sed commands with different formats, but nothing worked.
If jq is available to you, you can
jq '.users.accessRights = "f7fff"' usr.json
As also commented below, manipulating json with text replacement is likely to break if your input does not exactly meet your expectations (for instance though differences in whitespace). If you have a tool like jq which understands json, that is always the preferable option.
However, in case you still want to do it with sed you could adjust your regex to be less inclusive. Since you want to replace 5 hex values, something along the following lines should work (I hope I escaped everything correctly, can't test since I'm on the way).
cmd = 'sed -ie \'s/\"accessRights\":\s*\"[0-9,a-f,A-F]\{5\}\"/\"accessRights\": \"{newUsrCfg}\"/g\' "/etc/hgp/runtime/user/user1.json"'
See regex101.
A third option would to both load the file via ssh and manipulate the json in Python, or do it directly in Python on the target system, if Python is available there as well.
This question already has answers here:
Passing bash variable to jq
(10 answers)
"Invalid numeric literal" error from jq trying to modify JSON with variable
(1 answer)
Closed 4 years ago.
Here is the example json
{
"app": "K8s",
"version": "1.8",
"date": "2018-10-10"
}
In order to get the value of app, I can do this in jq as
jq '.app'
But what I want is, I want to pass the key to jq as a bash variable, i.e
bash_var="app"
jq '."${bash_var}"'
I'm getting the output as null instead of the value. What is the correct syntax to achieve this?
First, you need to port the bash variable into jq's context usign the --arg flag and access it inside the [..]
jq --arg keyvar "$bash_var" '.[$keyvar]' json
This question already has answers here:
How to remove double-quotes in jq output for parsing json files in bash?
(2 answers)
Closed 1 year ago.
The Problem:
I am trying to fetch credentials from AWS Secrets Manager in my terminal, however the Keys and Values I want needs to be in JSON, however they come with a lot of escape characters due to quotes.
The Scenario:
After I fire the aws secretsmanager get-secret-value --secret-id snowflake-access-uat command, I get the credentials as below:
{
"ARN": "arn:aws:secretsmanager:ap-regionnm-1:111111111111:secret:my-secret",
"Name": "snowflake-access-uat",
"VersionId": "dont-care",
"SecretString": "{\"sf-user\":\"USER_123_ADMIN\",\"sf-password\":\"FooBaarPassword\",\"sf-db\":\"MY_SPL_DB\",\"wh_name\":\"JOB_EXECUTOR\",\"sf-role\":\"JOB_EXECUTOR_ROLE\",\"sf-account\":\"icy-party\"}",
"VersionStages": [
"AWSCURRENT"
],
"CreatedDate": 1627104812.142
}
However, I am interested in Secret String only, for which I fire aws secretsmanager get-secret-value --secret-id snowflake-programmatic-access-uat | jq '.SecretString' command and receive this:
"{\"sf-user\":\"USER_123_ADMIN\",\"sf-password\":\"FooBaarPassword\",\"sf-db\":\"MY_SPL_DB\",\"wh_name\":\"JOB_EXECUTOR\",\"sf-role\":\"JOB_EXECUTOR_ROLE\",\"sf-account\":\"icy-party\"}"
But since it has multiple escape characters, I am unable to leverage it with jq tree. I tried to get from this link for reference but I'm unable to make it work. Besides, I need the Keys and Values to be variables in my bash session.
NOTE: I cannot use any third party tools, since I need to automate this on CodeBuild (Run time fresh instance will be selected)
The escape characters are there because you don't use -r with jq '.SecretString'. Change it to jq -r '.SecretString' and your output will instead be:
{"sf-user":"USER_123_ADMIN","sf-password":"FooBaarPassword","sf-db":"MY_SPL_DB","wh_name":"JOB_EXECUTOR","sf-role":"JOB_EXECUTOR_ROLE","sf-account":"icy-party"}
...which, being valid JSON, you can feed back into jq -r to retrieve individual fields.
SecretStringJson=$(... | jq -r '.SecretString')
### one jq call per field isn't the most efficient possible way but it's easy
sfUser=$(jq -r '.["sf-user"]' <<<"$SecretStringJson")
sfDb=$(jq -r '.["sf-db"]' <<<"$SecretStringJson")
# ...etc
This question already has answers here:
Parsing JSON with Unix tools
(45 answers)
Bash script store command output into variable
(2 answers)
Closed 3 years ago.
I am using curl to create a session to log into the switch.
Below the script that i use
curl --noproxy 10.23.193.1 -X POST http://10.23.193.1:80/rest/v3/login-sessions -d '{"userName":"admin", "password":"password"}'
After this is executed I get the following output
{"uri":"/login-sessions","cookie":"sessionId=DfZNCFbfoc3LDuMgjLXRiS8ZmEo4MWENCOJM0Iu14R1uMT9kKqbe1Rx6AedmoeT"}
My requirement here is to first only print this part of the string "sessionId=DfZNCFbfoc3LDuMgjLXRiS8ZmEo4MWENCOJM0Iu14R1uMT9kKqbe1Rx6AedmoeT"
Secondly I would want to know how to store the above string in a variable so that I could call the same variable for subsequent operations
I ran the following, but I am not getting any output.
curl --noproxy 10.23.193.1 -X POST http://10.23.193.1:80/rest/v3/login-sessions -d '{"userName":"admin", "password":"password"}' | grep -`E ""cookie":"
Avoid using simple tools like grep or sed to parse JSON as they won't handle things like quotes or multi-line data correctly. It's best to use a JSON-aware program such as jq.
With jq it's simple and robust:
curl ... | jq '.cookie'
To store the cookie in a variable use the -r flag to have JQ print out the raw, unquoted string.
cookie=$(curl ... | jq -r '.cookie')
Further reading:
jq Manual
Parsing JSON with Unix tools
How to parse JSON with shell scripting in Linux?
This question already has answers here:
Parsing JSON with Unix tools
(45 answers)
Closed 6 years ago.
Below is my JSON string available in file, out of which I need to extract the value for status in shell script.
Expected Output: status=success
response.json
{"eventDate":null,"dateProccessed":null,"fileName":null,"status":"success"}
Please let me know, if you need any information
Look into jq. It is a very handy json parser.
If you want to extract just the status you can do something like:
jq -r '.status' response.json
# output
success
You can format your output as well to follow the results you want.
jq -r '"status=\(.status)"' response.json
# output
status=success
You can try sed:
sed -n 's|.*"status":"\([^"]*\)".*|status=\1|p' response.json