Invalid JSON with json_encode and shell_exec - json

I'm using Symfony's console component to write some command line tools, one of which uses WP-CLI to set up a WordPress site. Specifically using wp option update I'm running into issues with both JSON and quotes.
For example, running something like:
shell_exec("wp option update blogdescription ''");
Results in literal '' in the database. In fact, any time I use that command, if it's successful, whatever I've tried to set in the database is wrapped in the single quotes.
And then something like this:
$github_updater_args = [
'github_access_token'=>'',
'bitbucket_username'=>'username',
'bitbucket_password'=>'password',
'all-in-one-seo-populate-keywords'=>'1'
];
$github_updater_args = json_encode($github_updater_args);
var_dump($github_updater_args);
shell_exec("wp option update github_updater '$github_updater_args' --format=json");
The dump results in:
string(200) "{"github_access_token":"","bitbucket_username":"devs#webspecdesign.com","bitbucket_password":"xxxxxxxxx","webspec-design-wordpress-core":"1","all-in-one-seo-populate-keywords":"1","webspec-smtp":"1"}"
Which is valid JSON, but the wp command resulted in the following:
Error: Invalid JSON: '{github_access_token:,bitbucket_username:username,bitbucket_password:password,all-in-one-seo-populate-keywords:1}'
You'll notice the quotes have been stripped out, which I assume is what it's complaining about? Or is that just how it dumps out the error?
Either way, I then decided I would hardcode the JSON, so just:
shell_exec('wp option update github_updater \'{"github_access_token": "", "bitbucket_username": "username", "bitbucket_password": "password"}\' --format=json');
Which gave me the following error:
Error: Too many positional arguments: , bitbucket_username: username, bitbucket_password: password}'
I wager the two issues are related. I thought maybe it was a WP-CLI issue, so I opened an issue there, but it was closed without reaching an answer. The more I stare at it, however, the more I think it might by a Symfony thing. Perhaps I definitely need to use the Process Component rather than shell_exec? Is that what it's designed for?
Update: Tried the Symfony Process Component:
$process = new Process("wp option update blogdescription ''");
$process->run();
Still got my two quotes. So nothing doing there.

You are looking for escapeshellarg() to escape your json data. escapeshellcmd() is for sanitizing an entire command, not a single argument. Try something like this:
$github_updater_args = [
'github_access_token'=>'',
'bitbucket_username'=>'username',
'bitbucket_password'=>'password',
'all-in-one-seo-populate-keywords'=>'1'
];
$github_updater_args = json_encode($github_updater_args);
$cmd = sprintf("wp option update github_updater %s --format=json", escapeshellarg($github_updater_args));
shell_exec($cmd);
Edit
There is something missing from your question or there is a bug in the CLI script you are executing. I wrote two quick test php scripts to illustrate that the json escaping works correctly.
test.php
<?php
$github_updater_args = [
'github_access_token'=>'',
'bitbucket_username'=>'username',
'bitbucket_password'=>'password',
'all-in-one-seo-populate-keywords'=>'1'
];
$github_updater_args = json_encode($github_updater_args);
$cmd = sprintf("php j.php %s", escapeshellarg($github_updater_args));
echo shell_exec($cmd);
echo PHP_EOL;
j.php
<?php
var_dump(json_decode($argv[1]));
output
~$ php test.php
object(stdClass)#1 (4) {
["github_access_token"]=>
string(0) ""
["bitbucket_username"]=>
string(8) "username"
["bitbucket_password"]=>
string(8) "password"
["all-in-one-seo-populate-keywords"]=>
string(1) "1"
}

Related

500 Can't connect to url with lwp in perl

I'm trying to parse some json data with the fandom wikia API. When I browse to my marvel.fandom.com/api request I get following JSON output: {"batchcomplete":"","query":{"pages":{"45910":{"pageid":45910,"ns":0,"title":"Uncanny X-Men Vol 1 171"}}}}
Nothing to fancy to begin with and running it through a JSON parser online gives following output:
{
"batchcomplete":"",
"query":{
"pages":{
"45910":{
"pageid":45910,
"ns":0,
"title":"Uncanny X-Men Vol 1 171"
}
}
}
}
which seems to be ok as far as I can see
I want to get the pageid for several other requests but I can't seem to get the same output through Perl.
The script:
#!/usr/bin/perl
use strict;
use warnings;
use LWP::Simple;
use JSON;
use Data::Dumper;
my $url = "https://marvel.fandom.com/api.php?action=query&titles=Uncanny%20X-Men%20Vol%201%20171&format=json";
my $json = getprint( $url);
die "Could not get $url!" unless defined $json;
my $decoded_json = decode_json($json);
print Dumper($decoded_json);
but this gives following error:
Could not get https://marvel.fandom.com/api.php?action=query&titles=Uncanny%20X-Men%20Vol%201%20171&format=json! at ./marvelScraper.pl line 11.
When I change the get to getprint for some extra info, I get this:
500 Can't connect to marvel.fandom.com:443
<URL:https://marvel.fandom.com/api.php?action=query&titles=Uncanny%20X-Men%20Vol%201%20171&format=json>
malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "(end of string)") at ./script.pl line 13.
I tried this on another computer and still get the same errors.
The versions of LWP::Simple and LWP::Protocol::https
/usr/bin/perl -MLWP::Simple -E'say $LWP::Simple::VERSION'
6.15
/usr/bin/perl -MLWP::Protocol::https -E'say $LWP::Protocol::https::VERSION'
6.09
Appearantly it has something to do with the Bash Ubuntu on Windows since on a Ubuntu 18.04 I get (with the same script) following response:
JSON text must be an object or array (but found number, string, true, false or null, use allow_nonref to allow this) at ./test.pl line 13.
{"batchcomplete":"","query":{"pages":{"45910":{"pageid":45910,"ns":0,"title":"Uncanny X-Men Vol 1 171"}}}}
Actually, the very same script works from my Bash Ubuntu on Windows with the get() command instead of the getprint() you gave after editing your question.
orabig#Windows:~/DEV$ ./so.pl
$VAR1 = {
'query' => {
'pages' => {
'45910' => {
'pageid' => 45910,
'ns' => 0,
'title' => 'Uncanny X-Men Vol 1 171'
}
}
},
'batchcomplete' => ''
};
So maybe you have another issue that has nothing to do with Perl or Ubuntu.
Can you try this for example ?
curl -v 'https://marvel.fandom.com/api.php?action=query&titles=Uncanny%20X-Men%20Vol%201%20171&format=json'
Maybe you just hit the site too much, and the 500 error is just a result of some anti-leech protection ?

Bash JSON string into variable

my idea is to put json string into variable JSON, i have a command whom takes a console login to user in AWS IAM, the comand is : aws iam create-login-profile --cli-input-json file://create-login-profile.json
in create-login-profile.json is the json following content :
{
"UserName": "roberto.viquezzz",
"Password": "aaaaaaaaaaa",
"PasswordResetRequired": true}
i try to write bash script whom contains json var "JSON" like following code:
JSON="{\"UserName\": \"roberto.viquezzz\",\"Password\": \"aaaaaaaaaaa\",\"PasswordResetRequired\" : true}"
aws iam create-login-profile --cli-input-json $JSON
and if i type in console ./file.sh the file execute create a console user.
if it try to execute this code i get an error : Unknown options: "aaaaaaaaaaa","PasswordResetRequired", :, true}, "roberto.viquezzz","Password":
but if i execute this code from command line like :
aws iam create-login-profile --cli-input-json "{\"UserName\": \"roberto.viquezzz\",\"Password\": \"aaaaaaaaaaa\",\"PasswordResetRequired\": true}"
all is be ok , maybe whom know whats wrong ? please suggest!
Put quotes around $JSON:
aws iam create-login-profile --cli-input-json "$JSON"
The quotes that are there during the assignment get consumed by the shell. You can verify this by issuing echo $JSON.
By adding the quotes you will make sure that the entire string is passed to the command "aws" as a single argument.

Shell script function with global variable

yesterday I got a very easy task, but unfortunatelly looks like i can't do with a nice code.
The task briefly: I have a lot of parameters, that I want to ask with whiptail "interactive" mode in the installer script.
The detail of code:
#!/bin/bash
address="192.168.0.1" # default address, what the user can modify
addressT="" # temporary variable, that I want to check and modify, thats why I don't modify in the function the original variable
port="1234"
portT=""
... #there is a lot of other variable that I need for the installer
function parameter_set {
$1=$(whiptail --title "Installer" --inputbox "$2" 12 60 "$3" 3>&1 1>&2 2>&3) # thats the line 38
}
parameter_set "addressT" "Please enter IP address!" "$address"
parameter_set "portT" "Please enter PORT!" "$port"
But i got the following error:
"./install.sh: line: 38: address=127.0.0.1: command not found"
If I modify the variable to another (not a parameter of function), works well.
function parameter_set {
foobar=$(whiptail --title "Installer" --inputbox "$2" 12 60 "$3" 3>&1 1>&2 2>&3)
echo $foobar
}
I try to use global retval variable, and assign to outside of the function to the original variable, it works, but I think it's not the nicest solution for this task.
Could anybody help me, what I do it wrong? :)
Thanks in advance (and sorry for my bad english..),
Attila
It seems that your whiptail command is not producing anyoutput because of the redirections. So the command substitution leaves the value empty. Try removing them. Also it's better to save the new value to a local variable first:
parameter_set() {
local NAME=$1
local NEWVALUE=$(whiptail --title "Installer" --inputbox "$2" 12 60 "$3")
export $NAME="$NEWVALUE"
}

Parsing JSON from shell script using JSON.sh

I'm working on parsing JSON data using JSON.sh. And I wanted to read data from json file (test.json) whose content will be something like,
{
"/home/ukrishnan/projects/test.yml": {
"LOG_DRIVER": "syslog",
"IMAGE": "mysql:5.6"
},
"/home/ukrishnan/projects/mysql/app.xml": {
"ENV_ACCOUNT_BRIDGE_ENDPOINT": "/u01/src/test/sample.txt"
}
}
And I try to parse this JSON using JSON.sh by using,
test_parser=`sh ./lib/JSON.sh < test/test.json`
echo $test_parser
It prints,
["/home/ukrishnan/projects/test.yml","LOG_DRIVER"] "syslog" ["/home/ukrishnan/projects/test.yml","IMAGE"] "mysql:5.6" ["/home/ukrishnan/projects/test.yml"] {"LOG_DRIVER":"syslog","IMAGE":"mysql:5.6"} ["/home/ukrishnan/projects/mysql/app.xml","ENV_ACCOUNT_BRIDGE_ENDPOINT"] "/u01/src/test/sample.txt" ["/home/ukrishnan/projects/mysql/app.xml"] {"ENV_ACCOUNT_BRIDGE_ENDPOINT":"/u01/src/test/sample.txt"} [] {"/home/ukrishnan/projects/test.yml":{"LOG_DRIVER":"syslog","IMAGE":"mysql:5.6"},"/home/ukrishnan/projects/mysql/app.xml":{"ENV_ACCOUNT_BRIDGE_ENDPOINT":"/u01/src/test/sample.txt"}}
Whereas, the same command (sh ./lib/JSON.sh < test/test.json), if I run through terminal, it is printing with line breaks,
["/home/ukrishnan/projects/test.yml","LOG_DRIVER"] "syslog"
["/home/ukrishnan/projects/test.yml","IMAGE"] "mysql:5.6"
["/home/ukrishnan/projects/test.yml"] {"LOG_DRIVER":"syslog","IMAGE":"mysql:5.6"}
["/home/ukrishnan/projects/mysql/app.xml","ENV_ACCOUNT_BRIDGE_ENDPOINT"] "/u01/src/test/sample.txt"
["/home/ukrishnan/projects/mysql/app.xml"] {"ENV_ACCOUNT_BRIDGE_ENDPOINT":"/u01/src/test/sample.txt"}
[] {"/home/ukrishnan/projects/test.yml":{"LOG_DRIVER":"syslog","IMAGE":"mysql:5.6"},"/home/ukrishnan/projects/mysql/app.xml":{"ENV_ACCOUNT_BRIDGE_ENDPOINT":"/u01/src/test/sample.txt"}}
I wanted to read this and assign to bash variables like,
file_name='/home/ukrishnan/projects/test.yml'
key='LOG_DRIVER'
value='syslog'
As I'm almost completely new to shell script and grep or awk, I don't have much idea of how to achieve this. Any help on this would be greatly appreciated.
I wrote a JSON serializer / deserializer for gawk, if you're interested. Save that script and modify it, replacing everything above # === FUNCTIONS === with the following:
#!/usr/bin/gawk -f
# capture JSON string from beginning to end into a scalar variable
{ json = json ORS $0 }
END {
# objectify JSON string to the multilevel array "obj"
deserialize(json, obj)
for (filename in obj) {
print "file_name=" quote(filename)
for (key in obj[filename]) {
# print key="value"
print key "=" quote(obj[filename][key])
}
}
}
Do chmod 755 json.awk and execute it. Output will resemble this:
$ ./json.awk test5.json
file_name="/home/ukrishnan/projects/mysql/app.xml"
ENV_ACCOUNT_BRIDGE_ENDPOINT="/u01/src/test/sample.txt"
file_name="/home/ukrishnan/projects/test.yml"
LOG_DRIVER="syslog"
IMAGE="mysql:5.6"
Hopefully the logic is reasonably easy to follow. If you prefer to output filename=, key=, and value= on every loop iteration, modify the nested for loops accordingly:
for (filename in obj) {
for (key in obj[filename]) {
print "file_name=" quote(filename)
print "key=" quote(key)
print "value=" quote(obj[filename][key])
}
}
That change will result in the following output:
$ ./json.awk test5.json
file_name="/home/ukrishnan/projects/mysql/app.xml"
key="ENV_ACCOUNT_BRIDGE_ENDPOINT"
value="/u01/src/test/sample.txt"
file_name="/home/ukrishnan/projects/test.yml"
key="LOG_DRIVER"
value="syslog"
file_name="/home/ukrishnan/projects/test.yml"
key="IMAGE"
value="mysql:5.6"
Anyway, with that output, you can do something silly in BASH like this to populate and act upon the variables:
#!/bin/bash
./test.awk test5.json | while read -r line; do {
eval $line
[ "${line/=*/}" = "value" ] && {
echo "bash: file_name=$file_name"
echo "bash: key=$key"
echo "bash: value=$value"
echo "------"
}
}; done
It'd probably be more graceful just to do all processing within gawk from start to finish and not mess with the polyglot handoff, though.
Getting back to json.awk, if you prefer to keep json.awk modular for easy reuse in future projects, you could remove everything above # === FUNCTIONS ===, create a separate main.awk containing the code block at the top of this answer, and #include "json.awk" as a helper library pretty much anywhere outside of END {...} (just below the shbang, for example).
JSON.sh (from http://json.org) offers a nice bash friendly means of flattening out a JSON file. Which you've already provided how it looks in your question. So, the flatten form is the format:
[node] tab value
You have to think in UNIX script in extracting the information you want, you'll note the lines you're interested in actually follow this pattern:
["filename","key"] tab ["value"]
In regex notation, we replace:
filename with (.*)
key with (.*)
tab with \t
value with (.*)
We can retrieve the first, second and third matching groups with \1, \2, \3 respectively.
When used in sed we also note that these symbols []() need to be escaped with a backslash \, resulting in the following script:
./lib/JSON.sh < test/test.json | sed 's/\["\(.*\)","\(.*\)\"]\t"\(.*\)"/\1,\2,\3/;t;d'
/home/ukrishnan/projects/test.yml,LOG_DRIVER,syslog
/home/ukrishnan/projects/test.yml,IMAGE,mysql:5.6
/home/ukrishnan/projects/mysql/app.xml,ENV_ACCOUNT_BRIDGE_ENDPOINT,/u01/src/test/sample.txt
Now we put the lines in a loop and for each line, we can extract out filename,key,value:
for line in $(./lib/JSON.sh < test/test.json | sed 's/\["\(.*\)","\(.*\)\"]\t"\(.*\)"/\1,\2,\3/;t;d')
do
IFS="," read -ra arr <<< $line
filename=${arr[0]}
key=${arr[1]}
value=${arr[2]}
cat <<EOF
filename : $filename
key : $key
value : $value
EOF
done
Which outputs:
filename : /home/ukrishnan/projects/test.yml
key : LOG_DRIVER
value : syslog
filename : /home/ukrishnan/projects/test.yml
key : IMAGE
value : mysql:5.6
filename : /home/ukrishnan/projects/mysql/app.xml
key : ENV_ACCOUNT_BRIDGE_ENDPOINT
value : /u01/src/test/sample.txt

Perl's REST Client error when trying to use Crontab

Script works well when run manually, but when I schdule it in cronjob it shows :
malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "<html>\r\n<head><tit...") at /usr/local/lib/perl5/site_perl/5.14.2/JSON.pm line 171.
script itself:
#rest config vaiables
$ENV{'PERL_LWP_SSL_VERIFY_NONE'} = 0;
print "test\n";
my $client = REST::Client->new();
$client->addHeader('Authorization', 'Basic YWRtaW46cmFyaXRhbg==');
$client->addHeader('content_type', 'application/json');
$client->addHeader('accept', 'application/json');
$client->setHost('http://10.10.10.10');
$client->setTimeout(1000);
$useragent = $client->getUseragent();
print "test\n";
#Getting racks by pod
$req = '/api/v2/racks?name_like=2t';
#print " rekvest {$req}\n";
$client->request('GET', qq($req));
$racks = from_json($client->responseContent());
$datadump = Dumper (from_json($client->responseContent()));
crontab -l
*/2 * * * * /usr/local/bin/perl /folder/api/2t.pl > /dmitry/api/damnitout 2>&1
Appreciate any suggestion
Thank you,
Dmitry
It is difficult to say what is really happening, but in my experience 99% issues of running stuff in crontab stems from differences in environment variables.
Typical way to debug this: in the beginning of your script add block like this:
foreach my $key (keys %ENV) {
print "$key = $ENV{$key}\n";
}
Run it in console, look at the output, save it in log file.
Now, repeat the same in crontab and save it into log file (you have already done that - this is good).
See if there is any difference in environment variables when trying to run it both ways and try to fix it. In Perl, probably easiest is to alter environment by changing %ENV. After all differences are sorted out, there is no reason for this to not work right.
Good luck!