I have json file with the following structure
{ "tool_first":"1.1.1","tool_second":"2.2.2","tool_three":"3.3.3" }
And I want to retrieve version from it with bash grep. I create something like this
cat myjson.json | grep -Po '"tool_second":\K"[A-Za-z0-9/._]*"'
which give me output
"2.2.2"
How to use variable instead of string "tool_second"? I want to have something like
cat myjson.json | grep -Po '"$x":\K"[A-Za-z0-9/._]*"'
where $x is the variable; x = "tool_second". I can't retrieve information from it with variable. How to escape variable properly in this way? I need just version number, without "".
grep is NOT the right tool for parsing JSON text. Use a more syntax aware tool like jq. Use the answer below only for trivial purposes.
You are not escaping your double-quotes in your string to search present in variable x,
x="\"tool_second\""
grep -Po "$x:\K\"[A-Za-z0-9/._]*\"" file
"2.2.2"
and it can work for other strings too!
x="\"tool_first\""
grep -Po "$x:\K\"[A-Za-z0-9/._]*\"" file
"1.1.1"
Related
I know we should use JQ for parsing json data, but I want to parse it using regex. I want to fetch the value of a json key into a variable in my shell script. As of now, I am using JQ for parsing.
So my abc.json is
{"value1":5.0,"value2":2.5,"value3":"2019-10-24T15:26:00.000Z","modifier":[],"value4":{"value41":{"value411":5}}}
Currently, my XYZ.sh has these lines to fetch the data
data1 =$(cat abc.json | jq -r '.value4.value41.value411')
I want data1 to have value of value411. How can I achieve this?
ps- The JSON is mutable. The above JSON is just a part of the JSON file that I want to fetch.
Is your json structure immutable? If you have to use it, consider the following
┌──[root#vms83.liruilongs.github.io]-[~]
└─$cat abc.json | awk -F: '{print $NF}' | grep -o '[[:digit:]]'
5
I think your problem was you had a space between data and =. There can't be a space there.
This works as you want it to (I removed the unnecessary cat)
data1=$(jq -r '.value4.value41.value411' abc.json)
echo $data1
This is working as expected:
$ cat /etc/tfe-config/sources/fluent-bit.conf.tpl | jq -R -s
$ "[OUTPUT]\n Name cloudwatch_logs\n Match *\n region eu-central-1\n log_group_name TFE-LogForwarding\n log_stream_name TFE-AllLogs"
However, assignment to a variable does not work:
$ MY_VARIABLE=$(cat /etc/tfe-config/sources/fluent-bit.conf.tpl | jq -R -s)
$ echo $MY_VARIABLE
jq - commandline JSON processor [version 1.5]
Usage: jq [options] <jq filter> [file...]
jq is a tool for processing JSON inputs, applying the
given filter to its JSON text inputs and producing the
filter's results as JSON on standard output.
The simplest filter is ., which is the identity filter,
copying jq's input to its output unmodified (except for
formatting).
For more advanced filters see the jq(1) manpage ("man jq")
and/or https://stedolan.github.io/jq
Some of the options include:
-c compact instead of pretty-printed output;
.... trimmed
I am on AWS EC2 machine with the latest Amazon Linux 2 image.
What is going on here?
The file looks like this:
[OUTPUT]
Name cloudwatch_logs
Match *
region eu-central-1
log_group_name TFE-LogForwarding
log_stream_name TFE-AllLogs
Two things:
You have to specify a filter for jq – just . to get the entire input
Once in a variable with whitespace, you must quote the string, else it shows up differently when you print it
var=$(cat /etc/tfe-config/sources/fluent-bit.conf.tpl | jq -R -s '.')
echo "$var"
Relevant Q&A:
How to use `jq` in a shell pipeline? (and also this GitHub issue)
I just assigned a variable, but echo $variable shows something else
I have this JSON String:
{"name":"http://someUrl/ws/someId","id":"someId"}
I just want to get value for "id" key and store it in some variable. I succesfully tried using jq. But due to some constraints, I need to achieve this just by using grep and string matching.
I tried this so far: grep -Po '"id":.*?[^\\]"'; But that is giving "id":"ws-4c906698-03a2-49c3-8b3e-dea829c7fdbe" as output. I just need the id value. Please help
With a PCRE regex, you may use lookarounds. Thus, you need to put "id":" into the positive lookbehind construct, and then match 1 or more chars other than ":
grep -Po '(?<="id":")[^"]+'
where
(?<="id":") - requires a "id":" to appear immediately to the left of the current position (but the matched text is not added to the match value) and
[^"]+ - matches and adds to the match 1 or more chars other than ".
To get the values with escaped quotes:
grep -Po '(?<="id":")[^"\\]*(?:\\.[^"\\]*)*'
Here, (?<="id":") will still match the position right after "id":" and then the following will get matched:
[^"\\]* - zero or more chars other than " and \
(?:\\.[^"\\]*)* - zero or more consequent sequences of:
\\. - a \ and any char (any escape sequence)
[^"\\]* - zero or more chars other than " and \
See Jshon, it is a command line Json parser for shell script usage.
echo '{"name":"http://someUrl/ws/someId","id":"someId"}' | jshon -e id
"someId"
Just noticed I read past the section stating you needed to use standard tools available, if your admin doesn't allow Jshon it is very likely that the system will have Python available which you could use.
echo '{"name":"http://someUrl/ws/someId","id":"someId"}' | python -c 'import sys, json; print json.load(sys.stdin)["id"]'
someId
Using grep for this is just asking for trouble, I would avoid it and opt for a proper Json parser as above.
I am running the following command:
sudo clustat | grep primary | awk 'NF{print $1",""server:"$2 ",""status:"$3}'
Results are:
service:servicename,server:servername,status:started
service:servicename,server:servername,status:started
service:servicename,server:servername,status:started
service:servicename,server:servername,status:started
service:servicename,server:servername,status:started
My desired result is:
{"service":"servicename","server":"servername","status":"started"}
{"service":"servicename","server":"servername","status":"started"}
{"service":"servicename","server":"servername","status":"started"}
{"service":"servicename","server":"servername","status":"started"}
{"service":"servicename","server":"servername","status":"started"}
I can't seem to put the qoutation marks withour srewing up my output.
Use jq:
sudo clustat | grep primary |
jq -R 'split(" ")|{service:.[0], server:.[1], status:.[2]}'
The input is read as raw text, not JSON. Each line is split on a space (the argument to split may need to be adjusted depending on the actual input). jq ensures that values are properly quoted when constructing the output objects.
Don't do this: Instead, use #chepner's answer, which is guaranteed to generate valid JSON as output with all possible inputs (or fail with a nonzero exit status if no JSON representation is possible).
The below is only tested to generate valid JSON with the specific inputs shown in the question, and will quite certainly generate output that is not valid JSON with numerous possible inputs (strings with literal quotes, strings ending in literal backslashes, etc).
sudo clustat |
awk '/primary/ {
print "{\"service\":\"" $1 "\",\"server\":\"" $2 "\",\"status\":\""$3"\"}"
}'
For JSON conversion of common shell commands, a good option is jc (JSON Convert)
There is no parser for clustat yet though.
clustat output does look table-like, so you may be able to use the --asciitable parser with jc.
I'd like a unix one-liner that will output the current execution environment as a JSON structure like: { "env-var" : "env-value", ... etc ... }
This kinda works:
(echo "{"; printenv | sed 's/\"/\\\"/g' | sed -n 's|\(.*\)=\(.*\)|"\1"="\2"|p' | grep -v '^$' | paste -s -d"," -; echo "}")
but has some extra lines and I think won't work if the environment values or variables have '=' or newlines in them.
Would prefer pure bash/sh, but compact python / perl / ruby / etc one-liners would also be appreciated.
Using jq 1.5 (e.g. jq 1.5rc2 -- see http://stedolan.github.io/jq):
$ jq -n env
This works for me:
python -c 'import json, os;print(json.dumps(dict(os.environ)))'
It's pretty simple; the main complication is that os.environ is a dict-like object, but it is not actually a dict, so you have to convert it to a dict before you feed it to the json serializer.
Adding parentheses around the print statement lets it work in both Python 2 and 3, so it should work for the forseeable future on most *nix systems (especially since Python comes by default on any major distro).
#Alexander Trauzzi asked: "Wondering if anyone knows how to do this, but only passing a subset of the current environment's variables?"
I just found the way to do this:
jq -n 'env | {USER, HOME, PS1}'