Escape special characters in JSON string - json

I have Perl script which contains variable $env->{'arguments'}, this variable should contain a JSON object and I want to pass that JSON object as argument to my other external script and run it using backticks.
Value of $env->{'arguments'} before escaping:
$VAR1 = '{"text":"This is from module and backslash \\ should work too"}';
Value of $env->{'arguments'} after escaping:
$VAR1 = '"{\\"text\\":\\"This is from module and backslash \\ should work too\\"}"';
Code:
print Dumper($env->{'arguments'});
escapeCharacters(\$env->{'arguments'});
print Dumper($env->{'arguments'});
my $command = './script.pl '.$env->{'arguments'}.'';
my $output = `$command`;
Escape characters function:
sub escapeCharacters
{
#$env->{'arguments'} =~ s/\\/\\\\"/g;
$env->{'arguments'} =~ s/"/\\"/g;
$env->{'arguments'} = '"'.$env->{'arguments'}.'"';
}
I would like to ask you what is correct way and how to parse that JSON string into valid JSON string which I can use as argument for my script.

You're reinventing a wheel.
use String::ShellQuote qw( shell_quote );
my $cmd = shell_quote('./script.pl', $env->{arguments});
my $output = `$cmd`;
Alternatively, there's a number of IPC:: modules you could use instead of qx. For example,
use IPC::System::Simple qw( capturex );
my $output = capturex('./script.pl', $env->{arguments});
Because you have at least one argument, you could also use the following:
my $output = '';
open(my $pipe, '-|', './script.pl', $env->{arguments});
while (<$pipe>) {
$output .= $_;
}
close($pipe);
Note that current directory isn't necessarily the directory that contains the script that executing. If you want to executing script.pl that's in the same directory as the currently executing script, you want the following changes:
Add
use FindBin qw( $RealBin );
and replace
'./script.pl'
with
"$RealBin/script.pl"

Piping it to your second program rather than passing it as an argument seems like it would make more sense (and be a lot safer).
test1.pl
#!/usr/bin/perl
use strict;
use JSON;
use Data::Dumper;
undef $/;
my $data = decode_json(<>);
print Dumper($data);
test2.pl
#!/usr/bin/perl
use strict;
use IPC::Open2;
use JSON;
my %data = ('text' => "this has a \\backslash", 'nums' => [0,1,2]);
my $json = JSON->new->encode(\%data);
my ($chld_out, $chld_in);
print("Executing script\n");
my $pid = open2($chld_out, $chld_in, "./test1.pl");
print $chld_in "$json\n";
close($chld_in);
my $out = do {local $/; <$chld_out>};
waitpid $pid, 0;
print(qq~test1.pl output =($out)~);

Related

How to convert a string to hash in perl without using regex or split

I have a function i cannot control which returns a string which is acutally a hash. It looks something like below:
{"offset":0,"limit":500,"count":0,"virtual_machines":[]}
I need to check if the count is greater than 0. Because the output is a string and not a hash, i am trying to split the string and get the output from it.
The snippet for the same is below:
my $output = '{"offset":0,"limit":500,"count":0,"virtual_machines":[]}';
$output =~ s/ +/ /g;
my #words = split /[:,"\s\/]+/, $output;
print Dumper(#words);
The output for this is:
$VAR1 = '{';
$VAR2 = 'offset';
$VAR3 = '0';
$VAR4 = 'limit';
$VAR5 = '500';
$VAR6 = 'count';
$VAR7 = '0';
$VAR8 = 'virtual_machines';
$VAR9 = '[]}';
Now, i can get the value $VAR7 and get the count.
Is there a way to convert a string to hash and then use the keys to get the values instead of using regex and split. Can someone help me out here!
That string is in JSON format. I'd simply do
use strict;
use warnings;
use JSON::PP qw(decode_json);
my $output = '{"offset":0,"limit":500,"count":0,"virtual_machines":[]}';
my $data = decode_json $output;
print $data->{count}, "\n";
If all colons are just separators, then you can replace them with '=>'s and eval the string.
That's probably unrealistic, though. So you can use JSON ... looks like the string is in JSON format. Try the following (worked for me :-):
#!/usr/bin/perl
use JSON::Parse 'parse_json';
# the string is JSON
my $jstr = '{"offset":0,"limit":500,"count":0,"virtual_machines":[]}';
# oversimplified (not using json ... o.k. if no colons anywhere but as separators
my $sstr = $jstr;
$sstr =~ s/:/=>/g;
my $href = eval "$sstr";
printf("From oversimplified eval, limit == %d\n", $href->{limit});
# using JSON (looks like string is in JSON format).
# get JSON::Parse from CPAN (sudo cpan JSON::Parse)
my $jref = parse_json($jstr);
printf("From JSON::Parse, limit == %d\n", $jref->{limit});
1;
Output:
From oversimplified eval, limit == 500
From JSON::Parse, limit == 500

How do I use encode_json with string in Perl?

Here is my code that I try to open the file to get data and change it to UTF-8, then read each line and store it in variable my $abstract_text and send it back in JSON structure.
my $fh;
if (!open($fh, '<:encoding(UTF-8)',$path))
{
returnApplicationError("Cannot read abstract file: $path ($!)\nERRORCODE|111|\n");
}
printJsonHeader;
my #lines = <$fh>;
my $abstract_text = '';
foreach my $line (#lines)
{
$abstract_text .= $line;
}
my $json = encode_json($abstract_text);
close $fh;
print $json;
By using that code, I get this error;
hash- or arrayref expected (not a simple scalar, use allow_nonref to allow this)
error message also point out that the problem is in this line;
my $json = encode_json($abstract_text);
I want to send the data back as a string (which is in UTF-8). Please help.
I assume you're using either JSON or JSON::XS.
Both allow for non-reference data, but not via the procedural encode_json routine.
You'll need to use the object-oriented approach:
use strict; # obligatory
use warnings; # obligatory
use JSON::XS;
my $encoder = JSON::XS->new();
$encoder->allow_nonref();
print $encoder->encode('Hello, world.');
# => "Hello, world."

JSON encode/decode utf8 strings in Perl

I pass a utf8 encoded string from my command line into a Perl program:
> ./test.pl --string='ḷet ūs try ṭhiñgs'
which seems to recognize the string correctly:
use utf8;
GetOptions(
'string=s' => \$string,
) or die;
print Dumper($string);
print Dumper(utf8::is_utf8($string));
print Dumper(utf8::valid($string));
prints
$VAR1 = 'ḷet ūs try ṭhiñgs';
$VAR1 = '';
$VAR1 = 1;
When I store this string into a hash and call encode_json on it, the string seems to be again encoded whereas to_json seems to work (if I read the output correctly):
my %a = ( 'nāme' => $string ); # Note the Unicode character
print Dumper(\%a);
print Dumper(encode_json(\%a));
print Dumper(to_json(\%a));
prints
$VAR1 = {
"n\x{101}me" => 'ḷet ūs try ṭhiñgs'
};
$VAR1 = '{"nāme":"ḷet Å«s try á¹­hiñgs"}';
$VAR1 = "{\"n\x{101}me\":\"\x{e1}\x{b8}\x{b7}et \x{c5}\x{ab}s try \x{e1}\x{b9}\x{ad}hi\x{c3}\x{b1}gs\"}";
Turning this back into the original hash, however, doesn't seem to work with either methods and in both cases hash and string and broken:
print Dumper(decode_json(encode_json(\%a)));
print Dumper(from_json(to_json(\%a)));
prints
$VAR1 = {
"n\x{101}me" => "\x{e1}\x{b8}\x{b7}et \x{c5}\x{ab}s try \x{e1}\x{b9}\x{ad}hi\x{c3}\x{b1}gs"
};
$VAR1 = {
"n\x{101}me" => "\x{e1}\x{b8}\x{b7}et \x{c5}\x{ab}s try \x{e1}\x{b9}\x{ad}hi\x{c3}\x{b1}gs"
};
A hash lookup $a{'nāme'} now fails.
Question: How do I handle utf8 encoding and strings and JSON encode/decode correctly in Perl?
You need to decode your input:
use Encode;
my $string;
GetOptions('string=s' => \$string) or die;
$string = decode('UTF-8', $string);
Putting it all together, we get:
use strict;
use warnings;
use 5.012;
use utf8;
use Encode;
use Getopt::Long;
use JSON;
my $string;
GetOptions('string=s' => \$string) or die;
$string = decode('UTF-8', $string);
my %hash = ('nāme' => $string);
my $json = encode_json(\%hash);
my $href = decode_json($json);
binmode(STDOUT, ':encoding(utf8)');
say $href->{nāme};
Example:
$ perl test.pl --string='ḷet ūs try ṭhiñgs'
ḷet ūs try ṭhiñgs
Make sure your source file is actually encoded as UTF-8!

Parsing JSON Data::Dumper output array in Perl

I'm trying to edit an old perl script and I'm a complete beginner. The request from the server returns as:
$VAR1 = [
{
'keywords' => [
'bare knuckle boxing',
'support group',
'dual identity',
'nihilism',
'support',
'rage and hate',
'insomnia',
'boxing',
'underground fighting'
],
}
];
How can I parse this JSON string to grab:
$keywords = "bare knuckle boxing,support group,dual identity,nihilism,support,rage and hate,insomnia,boxing,underground fighting"
Full perl code
#!/usr/bin/perl
use LWP::Simple; # From CPAN
use JSON qw( decode_json ); # From CPAN
use Data::Dumper; # Perl core module
use strict; # Good practice
use warnings; # Good practice
use WWW::TheMovieDB::Search;
use utf8::all;
use Encode;
use JSON::Parse 'json_to_perl';
use JSON::Any;
use JSON;
my $api = new WWW::TheMovieDB::Search('APIKEY');
my $img = $api->type('json');
$img = $api->Movie_imdbLookup('tt0137523');
my $decoded_json = decode_json( encode("utf8", $img) );
print Dumper $decoded_json;
Thanks.
Based on comments and on your recent edit, I would say that what you are asking is how to navigate a perl data structure, contained in the variable $decoded_json.
my $keywords = join ",", #{ $decoded_json->[0]{'keywords'} };
say qq{ #{ $arrayref->[0]->{'keywords'} } };
As TLP pointed out, all you've shown is a combination of perl arrays/hashes. But you should look at the JSON.pm documentation, if you have a JSON string.
The result you present is similar to json, but the Perl-variant of it. (ie => instead of : etc). I don't think you need to look into the json part of it, As you already got the data. You just need to use Perl to join the data into a text string.
Just to eleborate on the solution to vol7ron :
#get a reference to the list of keywords
my $keywords_list = $decoded_json->[0]{'keywords'};
#merge this list with commas
my $keywords = join(',', #$keywords_list );
print $keywords;

Perl HTML Cleaning

How can i make the HTML::Clean Module work in this perl script.
#!/usr/bin/env perl
use strict;
use warnings;
require LWP::UserAgent;
open FH, "<", "text.txt";
my $ua = LWP::UserAgent->new;
$ua->timeout(10);
foreach my $line (<FH>) {
my $response = $ua->get($line);
my $h = new HTML::Clean(\$response);
if ($response->is_success) {
print $response->decoded_content;
}
else {
die $response->status_line;
}
}
close FH;
The HTML::Clean constructor method takes an argument which is either a filename or a reference to a string containing HTML. You seem to be passing it an HTTP::Response object. Try passing it the HTML instead.
my $response = $ua->get($line);
my $html = $response->decoded_content;
my $h = HTML::Clean->new(\$html);