This question already has answers here:
How to parse json response in the shell script?
(4 answers)
Extract json value with sed
(5 answers)
Closed 2 years ago.
This command
ubus -S call system board
gives me this output
{"kernel":"4.14.195","hostname":"OpenWrt","system":"ARMv7 Processor rev 1 (v7l)","model":"Linksys WRT32X","board_name":"linksys,venom","release":{"distribution":"OpenWrt","version":"19.07.4","revision":"r11208-ce6496d796","target":"mvebu/cortexa9","description":"OpenWrt 19.07.4 r11208-ce6496d796"}}
I want to just extract the model and, if there's a space, replace it with an underscore so I end up with
Linksys_WRT32X
Your command output is json, you can extract the value of the "model" field in pure text and use sed to replace any spaces with underscore.
<command> | jq -r '.model' | sed 's/ /_/g'
or using one jq command to select and replace the text value (thanks #Cyrus)
<command> | jq -r '.model | sub(" "; "_")'
If you don't have jq here is an awk for this
awk -v RS=, -F: '$1 ~ /model/{gsub(/\"/,""); gsub(" ","_"); print $2}'
Have in mind, that one awk or sed for this specific output is fine, but if this was for any json, it could break, json is not text, it can be printed in various lines, it can have additional spaces in places etc.
There are many ways to do this. Here's another one:
ubus -S call system board | sed 's/.*"model":"\([^"]*\)".*$/\1/' | tr ' ' _
I would attempt to answer this using my own version. Since GNU grep has support for Perl regex, we can get the result using:
ubus -S call system board | grep -oP '"model":"\K[^"]+' | tr ' ' _
We attempt to take advantage of the pattern that the JSON key-value pair is of the form "key-name":"<value>" and try to capture the <value> part using grep.
Related
This question already has an answer here:
"Argument list too long" while slurping JSON files [duplicate]
(1 answer)
Closed 1 year ago.
There are more than 6k JSON files, each containing exactly one JSON object. I want to prepare one list of objects from these JSONs.
When I am running below jq command I am getting an error.
Kedar.Javalkar#KD2806 MINGW64 /c/zz
$ jq -s '.' inventoryItem_*.json > inventory_items_result_$(date +"%Y%m%d_%H%M%S").json
bash: /usr/bin/jq: Argument list too long
I tried ulimit -s unlimited but the same error
I am using a windows 10 git bash
This is a job that xargs is created to fix -- splitting lists of items into individual command lines that are within the permitted limit.
Because running jq -s a single time is different from concatenating the results of multiple smaller runs, it's appropriate to use xargs to combine cat invocations using the manner described in the linked duplicate.
printf '%s\0' inventoryItem_*.json \
| xargs -0 cat \
| jq -s . \
>"inventory_items_result_$(date +"%Y%m%d_%H%M%S").json"
This question already has answers here:
Modify a key-value in a json using jq in-place
(8 answers)
Closed 1 year ago.
I have a daemon.json file which contains one line as below
{ "insecure-registries":["192.X.X.X:8123"] }
I am trying to use a variable to change generically to the current IP address. In bash script normally
I'd store in a variable like
myip=hostname -I | awk '{print $1}'
{ "insecure-registries":["$myip:8123"] }
How to use a kind of variable in JSON file?
If you have access to jq, I would recommend storing not JSON, but a jq filter, like
{"insecure-registries": ["\($ip):8123"]}
Assume the preceding is in file named foo.jq; then using jq as follows to produce JSON from the filter.
$ myip=$(hostname -I | awk '{print $1}') # 192.0.2.42, e.g.
$ jq -nf foo.jq --arg ip "$myip"
{
"insecure-registries": [
"192.0.2.42:8123"
]
}
JSON itself doesn't have a notion of substitution, and bash itself isn't really suitable for making substitutions like this.
Given the following string:
arn:aws:secretsmanager:us-east-1:3264873466873:secret:foo/bar 1564681234.974 foo/bar {"username":"admin","password":"admin123","secret_key":"KASJDFJHAKHFKAHASDF"} 4e397333-3797-4f0b-ad7e-8c1cc0ed041c VERSIONSTAGES AWSCURRENT
Within a shell script, how do you extract just the JSON portion to end up like this:
{"username":"admin","password":"admin123","secret_key":"KASJDFJHAKHFKAHASDF"}
I was able to do it using two sed commands:
echo $longString | sed 's/^.*{/{/' | sed 's/}.*$/}/'
but was wondering if there is a way to do it using only one command.
To extract continuous part of the input, you can use grep with its -o option (if supported on your system). It tells grep to only output the matching part.
grep -o '{.*}'
For extracting columns, use awk:
echo $longString | awk '{print $4}'
Or cut:
echo $longString | cut -f 4 -d ' '
Beware if you have spaces in your JSON data. You might be better off using jq to process the results of aws secretsmanager list-secrets and similar.
You can use
echo $longString | sed -n 's|.*\({.*}\).*|\1|p'
to match and print the desired pattern
you can just join the sed commands to a single command
sed 's/^.*{/{/;s/}.*$/}/'
This awk should do. I will handle if there are any space in the string.
echo $string | awk -F"[{}]" '{print $2}'
"username":"admin","password":"admin123","secret_key":"KASJDFJHAKHFKAHASDF"
I have a json result and I would like to extract a string without double quotes
{"value1":5.0,"value2":2.5,"value3":"2019-10-24T15:26:00.000Z","modifier":[]}
With this regex I can extract the value3 (019-10-24T15:26:00.000Z) correctly
sed -e 's/^.*"endTime":"\([^"]*\)".*$/\1/'
How can I extract the "value2" result, a string without double quotes?
I need to do with sed so can’t install jq. That’s my problem
With GNU sed for -E to enable EREs:
$ sed -E 's/.*"value3":"?([^,"]*)"?.*/\1/' file
2019-10-24T15:26:00.000Z
$ sed -E 's/.*"value2":"?([^,"]*)"?.*/\1/' file
2.5
With any POSIX sed:
$ sed 's/.*"value3":"\{0,1\}\([^,"]*\)"\{0,1\}.*/\1/' file
2019-10-24T15:26:00.000Z
$ sed 's/.*"value2":"\{0,1\}\([^,"]*\)"\{0,1\}.*/\1/' file
2.5
The above assumes you never have commas inside quoted strings.
Just run jq a Command-line JSON processor
$ json_data='{"value1":5.0,"value2":2.5,"value3":"2019-10-24T15:26:00.000Z","modifier":[]}'
$ jq '.value2' <(echo "$json_data")
2.5
with the key .value2 to access the value you are interested in.
This link summarize why you should NOT use, regex for parsing json
(the same goes for XML/HTML and other data structures that are in
theory can be infinitely nested)
Regex for parsing single key: values out of JSON in Javascript
If you do not have jq available:
you can use the following GNU grep command:
$ echo '{"value1":5.0,"value2":2.5,"value3":"2019-10-24T15:26:00.000Z","modifier":[]}' | grep -zoP '"value2":\s*\K[^\s,]*(?=\s*,)'
2.5
using the regex detailed here:
"value2":\s*\K[^\s,]*(?=\s*,)
demo: https://regex101.com/r/82J6Cb/1/
This will even work if the json is not linearized!!!!
With python it is also pretty direct and you should have it installed by default on your machine even if it is not python3 it should work
$ cat data.json
{"value1":5.0,"value2":2.5,"value3":"2019-10-24T15:26:00.000Z","modifier":[]}
$ cat extract_value2.py
import json
with open('data.json') as f:
data = json.load(f)
print(data["value2"])
$ python extract_value2.py
2.5
You can try this :
creds=$(eval aws secretsmanager get-secret-value --region us-east-1 --secret-id dpi/dev/hivemetastore --query SecretString --output text )
passwd=$(/bin/echo "${creds}" | /bin/sed -n 's/.*"password":"\(.*\)",/\1/p' | awk -F"\"" '{print $1}')
it is definitely possible to remove the AWK part though ...
To extract all values in proper list form to a file using sed(LINUX).
sed 's/["{}\]//g' <your_file.json> | sed 's/,/\n/g' >> <your_new_file_to_save>
sed 's/regexp/replacement/g' inputFileName > outputFileName
In some versions of sed, the expression must be preceded by -e to indicate that an expression follows.
The s stands for substitute, while the g stands for global, which means that all matching occurrences in the line would be replaced.
I've put [ ] inside it as elements that you wanna remove from .json file.
The pipe character | is used to connect the output from one command to the input of another.
Then, the last thing I did is substitute , and add a \n, known as line breaker.
If you want to show a single value see below command:
sed 's/["{}\]//g' <your_file.json> | sed 's/,/\n/g' | sed 's/<ur_value>//p'
p is run; this is equivalent to /pattern match/! p as per above; i.e., "if the line does not match /pattern match/, print it". So the complete command prints all the lines from the first occurrence of the pattern to the last line, but suppresses the ones that match.
if your data in 'd' file, try gnu sed
sed -E 's/[{,]"\w+":([^,"]+)/\1\n/g ;s/(.*\n).*".*\n/\1/' d
This question already has answers here:
Read JSON data in a shell script [duplicate]
(4 answers)
Closed 7 years ago.
Let's say that we have this kind of JSON file:
{
...
"quotes":{
"SOMETHING":10,
...
"SOMETHING_ELSE":120.4,
...
} }
How can I obtain those values and use them in order to add them together?
Am I able to do even this?
#!/bin/bash
#code ...
echo "$SOMETHING + $SOMETHING_ELSE" | bc
#code ...
#exit
I will obtain the JSON file with wget command. All I want is the content from this file.
Can you help me, please? I am a beginner in shell programming.
I usually use jq, a really fast json parser, to do this kind of things (because parsing a json file with tools like awk or sed is really error-prone).
Given an input file like this:
# file: input.json
{
"quotes":{
"SOMETHING":10,
"SOMETHING_ELSE":120.4
}
}
You can obtain the sum of the 2 fields with a simple filter:
jq '.quotes.SOMETHING + .quotes.SOMETHING_ELSE' input.json
# output -> 130.4
NOTE: jq is available in every major linux distribution. In a debian-derivative system you can install with a sudo apt-get install jq.
This will print out the sum of the selected lines' floats.
#!/bin/bash
awk '{ if ($1 ~ /"SOMETHING":/) {print}; if ($1 ~ /"SOMETHING_ELSE":/) {print} }' $1 | cut -d: -f2 | cut -d, -f1 | awk '{s+=$1};END{print s}'
This finds the lines you want, the plucks out the numbers, and adds them.
You should look up and learn jq as shown in Read the json data in shell script.
The tools in a "normal" shell installation like awk and sed all predate JSON by decades, and are a very very bad fit. jq is worth the time to learn.
Or use Python instead.