I have a JSON structure like this which I've read into a bash variable as a string:
{
"elem1": "val1",
"THEELEM": "THEVAL",
"elem3": "val3"
}
I want to use regex to match on "THEELEM": "THEVAL". It works if I try individual words, where the JSON is stored in result as a string:
[[ $result =~ THEVAL ]] && echo "yes"
But I want to match on the key-pair like this:
[[ $result =~ "THEELEM": "THEVAL" ]] && echo "yes"
That gives me syntax issues. I've tried escaping, single-quotes and triple quotes to no avail. Any help appreciated.
Quoting works for me.
[[ $result =~ '"THEELEM": "THEVAL"' ]] && echo "yes"
Note that quoting the pattern disables recognition of special regular expression characters, and just searches for the literal substring. This is not a problem here, since you don't have any wildcards or other non-literal pattern characters. But if you did, you'd have to put the pattern in a variable, as in #noah's answer.
You can create a variable $expr to hold the string you want to match to and then use that for the regex.
expr='"THEELEM": "THEVAL"'
[[ $result =~ $expr ]] && echo "yes"
Inspired by this stack overflow post
Related
I have this code
#!/bin/bash
today=$(date +'%Y-%m-%d')
date_in_config=$(cat config.json | jq ".date")
echo $date_in_config
echo $today
if [ $date_in_config == $today ];then
echo "Same date"
else
echo "different"
fi
config.json
{
"date": "2021-03-24",
"session": "0"
}
I am stuck at comparing the dates. The output that I get from the above code is
"2021-03-24"
2021-03-24
different
What am I doing wrong?
Clearly you have already detected what is wrong: For Bash (and any other programming language I can think of), "x" is different from x.
Put the double-quotes in today:
today=$(date +'"%Y-%m-%d"')
Or remove the quotes when retrieving the JSON information:
date_in_config=$(jq -r ".date" < config.json)
And remember to quote your variable expansions, as Shellcheck would have told you. Also get rid of the == bashism (optional, for portability).
[ "$date_in_config" = "$today" ]
I have this problem in my script but I don't have ideas for resolve this !
when I launch the script :
boak#boak-LX:~/Documents$ perl dl-sound.pl --url http://soundcloud.com/alexorion/bigger-room-radio-015
Missing or empty input at dl-sound.pl line 100.
the script :
sub fetch_music_info {
my ($self, $music_url) = #_;
$music_url ||= $self->{url};
my $page = $self->_get_content($music_url);
my $jsmusic = $1 if ($page =~ m{window.SC.bufferTracks.push\((.*)}i);
$jsmusic =~ s/;//g if defined($jsmusic);
$jsmusic =~ s/\)//g if defined($jsmusic);
my $music_info = JSON::Tiny::decode_json($jsmusic);
return $music_info;
}
It is unlikely to he the cause of your problem, but you shouldn't make declarations conditional. The behaviour is undefined, and it can lead to all sorts of nonsense.
So this
my $jsmusic = $1 if $page =~ m{window.SC.bufferTracks.push\((.*)}i;
should be
my $jsmusic;
$jsmusic = $1 if $page =~ /window\.SC\.bufferTracks\.push\(([^)]*)/i;
Note also that I have changed the regex to escape the dots, and to capture only the characters up to the next closing parenthesis. That means the following substitutions shouldn't be necessary if I understand your data properly
Update
In fact, looking again, the whole purpose of your subroutine is invalidated if the pattern doesn't match, so you should write it more like
sub fetch_music_info {
my ($self, $music_url) = #_;
$music_url ||= $self->{url};
my $page = $self->_get_content($music_url);
if ($page =~ /window\.SC\.bufferTracks\.push\(([^)]*)/i) {
return JSON::Tiny::decode_json($1);
}
else {
die "Music Info not found";
}
}
I guess that $jsmusic is not defined here:
my $music_info = JSON::Tiny::decode_json($jsmusic);
Change to:
my $music_info;
$music_info = JSON::Tiny::decode_json($jsmusic) if defined $jsmusic;
return $music_info;
I agree with Borodin, but there's a more Perlish way to write this:
my ($jsmusic) = ($page =~ m{window.SC.bufferTracks.push\((.*)}i);
That's because the =~ operator returns a list of the matchvars ($1,$2, etc.).
In this case, $jsmusic will always be declared but it will be undefined if the regexp doesn't match. You need the parentheses around ($jsmusic) to force the operator to list context.
I have the following perl search and replace :
With Escapes :
perl -pi -w -e 's/ onclick=\"img=document.getElementById(\'img_1\')\; img.style.display = (img.style.display == \'none\' ? \'block\' : \'none\');return false"//' test.html
Without Escapes :
perl -pi -w -e 's/ onclick="img=document.getElementById('img_1'); img.style.display = (img.style.display == 'none' ? 'block' : 'none');return false"//' test.html
My objective is to replace onclick="img=document.getElementById('img_1'); img.style.display = (img.style.display == 'none' ? 'block' : 'none');return false" with nothing in the file test.html. What am I messing up?
I keep on getting the error : -sh: syntax error near unexpected token)'` which I cannot help but feel is because of some stupid escaping on my part. Please help me out.
[ You didn't specify for which shell you are building the command. I'm going to assume sh or bash. ]
Problem #1: Converting text into a regex pattern
Many characters have a special meaning in regex patterns. Among those you used, (, ), . and ? are special. These need to be escaped.
If you want to match
onclick="img=document.getElementById('img_1'); img.style.display = (img.style.display == 'none' ? 'block' : 'none');return false"
You need to use the pattern
onclick="img=document\.getElementById\('img_1'\); img\.style\.display = \(img\.style\.display == 'none' \? 'block' : 'none'\);return false"
So your Perl code is
s/onclick="img=document\.getElementById\('img_1'\); img\.style\.display = \(img\.style\.display == 'none' \? 'block' : 'none'\);return false"//
Problem #2: Converting text (a Perl program) into a shell literal
This is the problem you asked about.
Sections quoted by single quotes ("'") end at the next single quote (unconditionally), so the following is wrong:
echo 'foo\'bar' # Third "'" isn't matched.
If you wanted to output "foo'bar", you could use
echo 'foo'\''bar' # Concatenation of "foo", "'" and "bar".
foo and bar are quoted by single quotes, and the single quote is escaped using \. \ works here because it's outside of single quotes.
So, the lesson is basically use '\'' instead of ' inside single quotes.
you want to pass the following program to Perl:
s/onclick="img=document\.getElementById\('img_1'\); img\.style\.display = \(img\.style\.display == 'none' \? 'block' : 'none'\);return false"//
To do so, you'll need to create a shell literal that produces the correct argument, meaning we want to surround the whole with single quotes and escape the existing single quotes. We get the following:
's/onclick="img=document\.getElementById\('\''img_1'\''\); img\.style\.display = \(img\.style\.display == '\''none'\'' \? '\''block'\'' : '\''none'\''\);return false"//'
As such, the entire command would be
perl -i -wpe's/onclick="img=document\.getElementById\('\''img_1'\''\); img\.style\.display = \(img\.style\.display == '\''none'\'' \? '\''block'\'' : '\''none'\''\);return false"//' test.html
Using the next:
perl -lnE 'say quotemeta $_'
and feed it with your plain input:
onclick="img=document.getElementById('img_1'); img.style.display = (img.style.display == 'none' ? 'block' : 'none');return false"
you will get:
onclick\=\"img\=document\.getElementById\(\'img_1\'\)\;\ img\.style\.display\ \=\ \(img\.style\.display\ \=\=\ \'none\'\ \?\ \'block\'\ \:\ \'none\'\)\;return\ false\"
So using it:
perl -i -pe "s/onclick\=\"img\=document\.getElementById\(\'img_1\'\)\;\ img\.style\.display\ \=\ \(img\.style\.display\ \=\=\ \'none\'\ \?\ \'block\'\ \:\ \'none\'\)\;return\ false\"//"
should work.
Some things are easier when you don't use a one-liner. Less things to escape, and not as fragile.
use strict;
use warnings;
my $literal_string = q{ onclick="img=document.getElementById('img_1'); img.style.display = (img.style.display == 'none' ? 'block' : 'none');return false"};
while (<>) {
s/\Q$literal_string//;
print;
}
Then execute the script:
perl -i thescript.pl test.html
This question already has answers here:
Parsing JSON with Unix tools
(45 answers)
Closed 6 years ago.
In shell I have a requirement wherein I have to read the JSON response which is in the following format:
{ "Messages": [ { "Body": "172.16.1.42|/home/480/1234/5-12-2013/1234.toSort", "ReceiptHandle": "uUk89DYFzt1VAHtMW2iz0VSiDcGHY+H6WtTgcTSgBiFbpFUg5lythf+wQdWluzCoBziie8BiS2GFQVoRjQQfOx3R5jUASxDz7SmoCI5bNPJkWqU8ola+OYBIYNuCP1fYweKl1BOFUF+o2g7xLSIEkrdvLDAhYvHzfPb4QNgOSuN1JGG1GcZehvW3Q/9jq3vjYVIFz3Ho7blCUuWYhGFrpsBn5HWoRYE5VF5Bxc/zO6dPT0n4wRAd3hUEqF3WWeTMlWyTJp1KoMyX7Z8IXH4hKURGjdBQ0PwlSDF2cBYkBUA=", "MD5OfBody": "53e90dc3fa8afa3452c671080569642e", "MessageId": "e93e9238-f9f8-4bf4-bf5b-9a0cae8a0ebc" } ] }
Here I am only concerned with the "Body" property value. I made some unsuccessful attempts like:
jsawk -a 'return this.Body'
or
awk -v k="Body" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}
But that did not suffice. Can anyone help me with this?
There is jq for parsing json on the command line:
jq '.Body'
Visit this for jq: https://stedolan.github.io/jq/
tl;dr
$ cat /tmp/so.json | underscore select '.Messages .Body'
["172.16.1.42|/home/480/1234/5-12-2013/1234.toSort"]
Javascript CLI tools
You can use Javascript CLI tools like
underscore-cli:
json:select(): CSS-like selectors for JSON.
Example
Select all name children of a addons:
underscore select ".addons > .name"
The underscore-cli provide others real world examples as well as the json:select() doc.
Similarly using Bash regexp. Shall be able to snatch any key/value pair.
key="Body"
re="\"($key)\": \"([^\"]*)\""
while read -r l; do
if [[ $l =~ $re ]]; then
name="${BASH_REMATCH[1]}"
value="${BASH_REMATCH[2]}"
echo "$name=$value"
else
echo "No match"
fi
done
Regular expression can be tuned to match multiple spaces/tabs or newline(s). Wouldn't work if value has embedded ". This is an illustration. Better to use some "industrial" parser :)
Here is a crude way to do it: Transform JSON into bash variables to eval them.
This only works for:
JSON which does not contain nested arrays, and
JSON from trustworthy sources (else it may confuse your shell script, perhaps it may even be able to harm your system, You have been warned)
Well, yes, it uses PERL to do this job, thanks to CPAN, but is small enough for inclusion directly into a script and hence is quick and easy to debug:
json2bash() {
perl -MJSON -0777 -n -E 'sub J {
my ($p,$v) = #_; my $r = ref $v;
if ($r eq "HASH") { J("${p}_$_", $v->{$_}) for keys %$v; }
elsif ($r eq "ARRAY") { $n = 0; J("$p"."[".$n++."]", $_) foreach #$v; }
else { $v =~ '"s/'/'\\\\''/g"'; $p =~ s/^([^[]*)\[([0-9]*)\](.+)$/$1$3\[$2\]/;
$p =~ tr/-/_/; $p =~ tr/A-Za-z0-9_[]//cd; say "$p='\''$v'\'';"; }
}; J("json", decode_json($_));'
}
use it like eval "$(json2bash <<<'{"a":["b","c"]}')"
Not heavily tested, though. Updates, warnings and more examples see my GIST.
Update
(Unfortunately, following is a link-only-solution, as the C code is far
too long to duplicate here.)
For all those, who do not like the above solution,
there now is a C program json2sh
which (hopefully safely) converts JSON into shell variables.
In contrast to the perl snippet, it is able to process any JSON,
as long as it is well formed.
Caveats:
json2sh was not tested much.
json2sh may create variables, which start with the shellshock pattern () {
I wrote json2sh to be able to post-process .bson with Shell:
bson2json()
{
printf '[';
{ bsondump "$1"; echo "\"END$?\""; } | sed '/^{/s/$/,/';
echo ']';
};
bsons2json()
{
printf '{';
c='';
for a;
do
printf '%s"%q":' "$c" "$a";
c=',';
bson2json "$a";
done;
echo '}';
};
bsons2json */*.bson | json2sh | ..
Explained:
bson2json dumps a .bson file such, that the records become a JSON array
If everything works OK, an END0-Marker is applied, else you will see something like END1.
The END-Marker is needed, else empty .bson files would not show up.
bsons2json dumps a bunch of .bson files as an object, where the output of bson2json is indexed by the filename.
This then is postprocessed by json2sh, such that you can use grep/source/eval/etc. what you need, to bring the values into the shell.
This way you can quickly process the contents of a MongoDB dump on shell level, without need to import it into MongoDB first.
I'm outputting some values to JSON format, and it appears if a value starts with a '.' it isn't valid JSON (The API doesn't seem to like these int's inside " "). What would be the best way to check if my value has anything in front of a '.', and if not, put a 0 there?
i.e
value = .53
newvalue = 0.53
I'm not very good at doing anything more than simple functions in BASH at the moment, still trying to learn awk/sed and other useful things such as cut.
There might be a number of possible solutions given the nature of the input. However, given those unknowns an easy workaround would be to say:
[[ $value == \.* ]] && newvalue=0${value}
Example:
$ value=.42
$ [[ $value == \.* ]] && newvalue=0${value}
$ echo $newvalue
0.42