Extract json value with sed - json

I have a json result and I would like to extract a string without double quotes
{"value1":5.0,"value2":2.5,"value3":"2019-10-24T15:26:00.000Z","modifier":[]}
With this regex I can extract the value3 (019-10-24T15:26:00.000Z) correctly
sed -e 's/^.*"endTime":"\([^"]*\)".*$/\1/'
How can I extract the "value2" result, a string without double quotes?
I need to do with sed so can’t install jq. That’s my problem

With GNU sed for -E to enable EREs:
$ sed -E 's/.*"value3":"?([^,"]*)"?.*/\1/' file
2019-10-24T15:26:00.000Z
$ sed -E 's/.*"value2":"?([^,"]*)"?.*/\1/' file
2.5
With any POSIX sed:
$ sed 's/.*"value3":"\{0,1\}\([^,"]*\)"\{0,1\}.*/\1/' file
2019-10-24T15:26:00.000Z
$ sed 's/.*"value2":"\{0,1\}\([^,"]*\)"\{0,1\}.*/\1/' file
2.5
The above assumes you never have commas inside quoted strings.

Just run jq a Command-line JSON processor
$ json_data='{"value1":5.0,"value2":2.5,"value3":"2019-10-24T15:26:00.000Z","modifier":[]}'
$ jq '.value2' <(echo "$json_data")
2.5
with the key .value2 to access the value you are interested in.
This link summarize why you should NOT use, regex for parsing json
(the same goes for XML/HTML and other data structures that are in
theory can be infinitely nested)
Regex for parsing single key: values out of JSON in Javascript
If you do not have jq available:
you can use the following GNU grep command:
$ echo '{"value1":5.0,"value2":2.5,"value3":"2019-10-24T15:26:00.000Z","modifier":[]}' | grep -zoP '"value2":\s*\K[^\s,]*(?=\s*,)'
2.5
using the regex detailed here:
"value2":\s*\K[^\s,]*(?=\s*,)
demo: https://regex101.com/r/82J6Cb/1/
This will even work if the json is not linearized!!!!
With python it is also pretty direct and you should have it installed by default on your machine even if it is not python3 it should work
$ cat data.json
{"value1":5.0,"value2":2.5,"value3":"2019-10-24T15:26:00.000Z","modifier":[]}
$ cat extract_value2.py
import json
with open('data.json') as f:
data = json.load(f)
print(data["value2"])
$ python extract_value2.py
2.5

You can try this :
creds=$(eval aws secretsmanager get-secret-value --region us-east-1 --secret-id dpi/dev/hivemetastore --query SecretString --output text )
passwd=$(/bin/echo "${creds}" | /bin/sed -n 's/.*"password":"\(.*\)",/\1/p' | awk -F"\"" '{print $1}')
it is definitely possible to remove the AWK part though ...

To extract all values in proper list form to a file using sed(LINUX).
sed 's/["{}\]//g' <your_file.json> | sed 's/,/\n/g' >> <your_new_file_to_save>
sed 's/regexp/replacement/g' inputFileName > outputFileName
In some versions of sed, the expression must be preceded by -e to indicate that an expression follows.
The s stands for substitute, while the g stands for global, which means that all matching occurrences in the line would be replaced.
I've put [ ] inside it as elements that you wanna remove from .json file.
The pipe character | is used to connect the output from one command to the input of another.
Then, the last thing I did is substitute , and add a \n, known as line breaker.
If you want to show a single value see below command:
sed 's/["{}\]//g' <your_file.json> | sed 's/,/\n/g' | sed 's/<ur_value>//p'
p is run; this is equivalent to /pattern match/! p as per above; i.e., "if the line does not match /pattern match/, print it". So the complete command prints all the lines from the first occurrence of the pattern to the last line, but suppresses the ones that match.

if your data in 'd' file, try gnu sed
sed -E 's/[{,]"\w+":([^,"]+)/\1\n/g ;s/(.*\n).*".*\n/\1/' d

Related

Insert comma into string at the place you want

I'm trying to extract the balance from this string (which I did already) and add a comma to the string like 6841,12691421 (already done that also) BUT! theres a problem with doing it the way I did.
{
"address": "NKNXyCmatuYuAnMFufdDnLL82qmvgB4uAYt6",
"count_transactions": 59606,
"first_transaction": "2020-08-07 17:25:51",
"last_transaction": "2021-05-02 09:09:24",
"balance": 684112691421,
"name": []
}
I did it with (excuse the noob code):
sed -n -r 's/(^.*balance":)([^"]+)".*/\2/p' | sed -e 's/[",]//g' | sed 's/./&,/4'
The problem:
The sed 's/./&,/4' is a static thing. When the balance is lower by one character the output is wrong then, example `68411269142 the balance should be 684,11269142.
I need a solution to count the comm`a insert place from the right, 8 characters in.
Two jq-only solutions:
a) without any regex overhead:
jq -r '.balance | tostring | .[:-8]+ ","+ .[-8:]'
b) with regex:
jq -r 'tostring|sub("(?<tail>[0-9]{8}$)"; ",\(.tail)" )'
Caveat
Unfortunately these jq-only solutions will only work reliably for integers with fewer than 16 digits unless you have a sufficiently recent version of jq (after 1.6).
You may use this single sed:
sed -E 's/(^.*balance":)([^",]+).*/\2/; s/[0-9]{8}$/,&/' file
6841,12691421
s/[0-9]{8}$/,&/ matches 8 trailing digits and inserts a comma before it
With jq and sed:
jq '.balance' file.json | sed -E 's/.{8}$/,&/'
Output:
6841,12691421

How to extract JSON string from a longer mixed string in a shell script

Given the following string:
arn:aws:secretsmanager:us-east-1:3264873466873:secret:foo/bar 1564681234.974 foo/bar {"username":"admin","password":"admin123","secret_key":"KASJDFJHAKHFKAHASDF"} 4e397333-3797-4f0b-ad7e-8c1cc0ed041c VERSIONSTAGES AWSCURRENT
Within a shell script, how do you extract just the JSON portion to end up like this:
{"username":"admin","password":"admin123","secret_key":"KASJDFJHAKHFKAHASDF"}
I was able to do it using two sed commands:
echo $longString | sed 's/^.*{/{/' | sed 's/}.*$/}/'
but was wondering if there is a way to do it using only one command.
To extract continuous part of the input, you can use grep with its -o option (if supported on your system). It tells grep to only output the matching part.
grep -o '{.*}'
For extracting columns, use awk:
echo $longString | awk '{print $4}'
Or cut:
echo $longString | cut -f 4 -d ' '
Beware if you have spaces in your JSON data. You might be better off using jq to process the results of aws secretsmanager list-secrets and similar.
You can use
echo $longString | sed -n 's|.*\({.*}\).*|\1|p'
to match and print the desired pattern
you can just join the sed commands to a single command
sed 's/^.*{/{/;s/}.*$/}/'
This awk should do. I will handle if there are any space in the string.
echo $string | awk -F"[{}]" '{print $2}'
"username":"admin","password":"admin123","secret_key":"KASJDFJHAKHFKAHASDF"

not able to store sed output to variable

I am new to bash script.
I am getting some json response and i get only one property from the response. I want to save it to a variable but it is not working
token=$result |sed -n -e 's/^.*access_token":"//p' | cut -d'"' -f1
echo $token
it returns blank line.
I cannot use jq or any third party tools.
Please let me know what I am missing.
Your command should be:
token=$(echo "$result" | sed -n -e 's/^.*access_token":"//p' | cut -d'"' -f1)
You need to use echo to print the contents of the variable over standard output, and you need to use a command substitution $( ) to assign the output of the pipeline to token.
Quoting your variables is always encouraged, to avoid problems with white space and glob characters like *.
As an aside, note that you can probably obtain the output using something like:
token=$(jq -r .access_token <<<"$result")
I know you've said that you can't use jq but it's a standalone binary (no need to install it) and treats your JSON in the correct way, not as arbitrary text.
Give this a try:
token="$(sed -E -n -e 's/^.*access_token": ?"//p' <<<"$result" | cut -d'"' -f1)"
Explanation:
token="$( script here )" means that $token is set to the output/result of the script run inside the subshell through a process known as command substituion
-E in sed allows Extended Regular Expressions. We want this because JSON generally contains a space after the : and before the next ". We use the ? after the space to tell sed that the space may or may not be present.
<<<"$result" is a herestring that feeds the data into sed as stdin in place of a file.

Find and replace text in JSON with sed [duplicate]

I am trying to change the values in a text file using sed in a Bash script with the line,
sed 's/draw($prev_number;n_)/draw($number;n_)/g' file.txt > tmp
This will be in a for loop. Why is it not working?
Variables inside ' don't get substituted in Bash. To get string substitution (or interpolation, if you're familiar with Perl) you would need to change it to use double quotes " instead of the single quotes:
# Enclose the entire expression in double quotes
$ sed "s/draw($prev_number;n_)/draw($number;n_)/g" file.txt > tmp
# Or, concatenate strings with only variables inside double quotes
# This would restrict expansion to the relevant portion
# and prevent accidental expansion for !, backticks, etc.
$ sed 's/draw('"$prev_number"';n_)/draw('"$number"';n_)/g' file.txt > tmp
# A variable cannot contain arbitrary characters
# See link in the further reading section for details
$ a='foo
bar'
$ echo 'baz' | sed 's/baz/'"$a"'/g'
sed: -e expression #1, char 9: unterminated `s' command
Further Reading:
Difference between single and double quotes in Bash
Is it possible to escape regex metacharacters reliably with sed
Using different delimiters for sed substitute command
Unless you need it in a different file you can use the -i flag to change the file in place
Variables within single quotes are not expanded, but within double quotes they are. Use double quotes in this case.
sed "s/draw($prev_number;n_)/draw($number;n_)/g" file.txt > tmp
You could also make it work with eval, but don’t do that!!
This may help:
sed "s/draw($prev_number;n_)/draw($number;n_)/g"
You can use variables like below. Like here, I wanted to replace hostname i.e., a system variable in the file. I am looking for string look.me and replacing that whole line with look.me=<system_name>
sed -i "s/.*look.me.*/look.me=`hostname`/"
You can also store your system value in another variable and can use that variable for substitution.
host_var=`hostname`
sed -i "s/.*look.me.*/look.me=$host_var/"
Input file:
look.me=demonic
Output of file (assuming my system name is prod-cfm-frontend-1-usa-central-1):
look.me=prod-cfm-frontend-1-usa-central-1
I needed to input github tags from my release within github actions. So that on release it will automatically package up and push code to artifactory.
Here is how I did it. :)
- name: Invoke build
run: |
# Gets the Tag number from the release
TAGNUMBER=$(echo $GITHUB_REF | cut -d / -f 3)
# Setups a string to be used by sed
FINDANDREPLACE='s/${GITHUBACTIONSTAG}/'$(echo $TAGNUMBER)/
# Updates the setup.cfg file within version number
sed -i $FINDANDREPLACE setup.cfg
# Installs prerequisites and pushes
pip install -r requirements-dev.txt
invoke build
Retrospectively I wish I did this in python with tests. However it was fun todo some bash.
Another variant, using printf:
SED_EXPR="$(printf -- 's/draw(%s;n_)/draw(%s;n_)/g' $prev_number $number)"
sed "${SED_EXPR}" file.txt
or in one line:
sed "$(printf -- 's/draw(%s;n_)/draw(%s;n_)/g' $prev_number $number)" file.txt
Using printf to build the replacement expression should be safe against all kinds of weird things, which is why I like this variant.

extra characters at end of command in shell

I am trying to run the following script:
sed -E -n '/"data"/,/}/{/[{}]/d;s/^[[:space:]]*"([^"]+)":[[:space:]]*"([^"]+)".*$/\1|\2/g;p}' /tmp/data.json | while IFS="|" read -r item val;do item="${item^^}"; item="${val}"; export "${item}"; echo ${item}; done
This basically exports data from inside a JSON as environment variables.
That is,
Here, the key data will have a list (of different lengths) of key-value pairs within itself wherein the key is not fixed. Now, I want to read every key in the list and export its value. For example, I want these commands to be executed as part of the shell script.
export HELLO1
export SAMPLEKEY
However, when I run this, it gives the error: sed: 1: "/"data"/,/}/{/[{}]/d;s/ ...": extra characters at the end of p command. What might be the reason for this?
Rather than trying to use sed to parse .json files (which can rapidly grow beyond reasonable sed parsing), instead use a tool made for parsing json (like jq -- json query). You can easily obtain the keys for values under data, and then parse with your shell tools.
(note: your questions should be tagged bash since you use the parameter expansion for character-case which is a bashism, e.g. ${item^^})
Using jq, you could do something like the following:
jq '.data' /tmp/data.json | tail -n+2 | head -n-1 |
while read -r line; do line=${line#*\"}; line=${line%%\"*}; \
printf "export %s " ${line^^}; done; echo ""
Which results in the output:
export HELLO1 export SAMPLEKEY
(there are probably even cleaner way to do this with jq -- and there was)
You can have jq output the keys for data one per line with:
jq -r '.data | to_entries[] | (.key|ascii_upcase)' /tmp/data.json
This allows you to shorten your command to generate export in from of the keys with:
while read -r key; do \
printf "export %s " $key; \
done < <(jq -r '.data | to_entries[] | (.key|ascii_upcase)' /tmp/data.json); \
echo ""
(note: to effect your actual environment, you would need to export the values as part of your shell startup)