Creating JSON in perl - json

Using this code, I am only getting an array of values name and extension listed one after the other, how do I get a separate {name:"tom", extension:"012345"} for each hash I am creating? i.e I would like [{name:"tom", extension:"012345"}, {name:"tim", extension:"66666"}]
#!/usr/bin/perl
use strict;
use warnings;
use lib $ENV{HOME} . '/perl/lib/perl5';
use lib '/home/accounts/lib';
use Data::Dump qw(dump);
use Lib::Phonebook;
use JSON;
my $ldap = Lib::Phonebook->new();
my (#names, #numbers, $count, $name_number_count, #result);
#names = $ldap->list_telephone_account_names();
#numbers = $ldap->list_telephone_account_numbers();
$name_number_count = #names;
$count = 0;
for $count (0 .. $name_number_count - 1) {
my %hash = ( "name" => $names[$count], "extension" => $numbers[$count] );
push( #result, %hash );
}
my $json = encode_json \#result;
print dump $json;

Instead of
push( #result, %hash );
do:
push( #result, \%hash );
The original statement copies the entire hash to a list (as you discovered that becomes key, value, key, value, ...), whereas adding the backslash pushes a reference to the hash (without copying anything else) and serializing all that to JSON creates a nested structure.

Related

Escape special characters in JSON string

I have Perl script which contains variable $env->{'arguments'}, this variable should contain a JSON object and I want to pass that JSON object as argument to my other external script and run it using backticks.
Value of $env->{'arguments'} before escaping:
$VAR1 = '{"text":"This is from module and backslash \\ should work too"}';
Value of $env->{'arguments'} after escaping:
$VAR1 = '"{\\"text\\":\\"This is from module and backslash \\ should work too\\"}"';
Code:
print Dumper($env->{'arguments'});
escapeCharacters(\$env->{'arguments'});
print Dumper($env->{'arguments'});
my $command = './script.pl '.$env->{'arguments'}.'';
my $output = `$command`;
Escape characters function:
sub escapeCharacters
{
#$env->{'arguments'} =~ s/\\/\\\\"/g;
$env->{'arguments'} =~ s/"/\\"/g;
$env->{'arguments'} = '"'.$env->{'arguments'}.'"';
}
I would like to ask you what is correct way and how to parse that JSON string into valid JSON string which I can use as argument for my script.
You're reinventing a wheel.
use String::ShellQuote qw( shell_quote );
my $cmd = shell_quote('./script.pl', $env->{arguments});
my $output = `$cmd`;
Alternatively, there's a number of IPC:: modules you could use instead of qx. For example,
use IPC::System::Simple qw( capturex );
my $output = capturex('./script.pl', $env->{arguments});
Because you have at least one argument, you could also use the following:
my $output = '';
open(my $pipe, '-|', './script.pl', $env->{arguments});
while (<$pipe>) {
$output .= $_;
}
close($pipe);
Note that current directory isn't necessarily the directory that contains the script that executing. If you want to executing script.pl that's in the same directory as the currently executing script, you want the following changes:
Add
use FindBin qw( $RealBin );
and replace
'./script.pl'
with
"$RealBin/script.pl"
Piping it to your second program rather than passing it as an argument seems like it would make more sense (and be a lot safer).
test1.pl
#!/usr/bin/perl
use strict;
use JSON;
use Data::Dumper;
undef $/;
my $data = decode_json(<>);
print Dumper($data);
test2.pl
#!/usr/bin/perl
use strict;
use IPC::Open2;
use JSON;
my %data = ('text' => "this has a \\backslash", 'nums' => [0,1,2]);
my $json = JSON->new->encode(\%data);
my ($chld_out, $chld_in);
print("Executing script\n");
my $pid = open2($chld_out, $chld_in, "./test1.pl");
print $chld_in "$json\n";
close($chld_in);
my $out = do {local $/; <$chld_out>};
waitpid $pid, 0;
print(qq~test1.pl output =($out)~);

How to store the output of a Perl script as a JSON array

My program logs in to a list of IPs, identifies the software running on it, and prints the output.
I want the output to be in the form of a JSON array.
Can a JSON encode function be used for this?
use strict;
use warnings;
use QA::unit::testbedinfo;
my #machines_under_test = ( ... ); # list of ip's listed here
sub test_1_get_install_info_of_machines_under_test {
my ( $self ) = #_;
my %output;
foreach my $ip ( #machines_under_test ) {
my $output = $self->{'queryObj'}->get_install_info( $ip );
push #{ $output{$output} }, $ip;
INFO( ' software version running on machine ' . $ip . ' : ' . $output );
}
return 1;
}
So it looks like you're building a hash, %output, which has software version numbers for keys and (references to) arrays of IP addresses for values, correct? To output that structure as JSON, just use the JSON module and print the output of the to_json function:
#!/usr/bin/env perl
use warnings;
use strict;
use 5.010;
use JSON 'to_json';
my %output = (
'1.0' => [ qw( 1.2.3.4 5.6.7.8 ) ],
'1.1' => [ qw( 192.168.0.3 192.168.37.42 192.168.0.123 ) ]
);
# Note that to_json takes a reference to the structure, not the raw hash
say to_json(\%output);
Which produces the output:
{"1.0":["1.2.3.4","5.6.7.8"],"1.1":["192.168.0.3","192.168.37.42","192.168.0.123"]}

DBI convert fetched arrayref to hash

I'm trying to write a program to fetch a big MySQL table, rename some fields and write it to JSON. Here is what I have for now:
use strict;
use JSON;
use DBI;
# here goes some statement preparations and db initialization
my $rowcache;
my $max_rows = 1000;
my $LIMIT_PER_FILE = 100000;
while ( my $res = shift( #$rowcache )
|| shift( #{ $rowcache = $sth->fetchall_arrayref( undef, $max_rows ) } ) ) {
if ( $cnt % $LIMIT_PER_FILE == 0 ) {
if ( $f ) {
print "CLOSE $fname\n";
close $f;
}
$filenum++;
$fname = "$BASEDIR/export-$filenum.json";
print "OPEN $fname\n";
open $f, ">$fname";
}
$res->{some_field} = $res->{another_field}
delete $res->{another_field}
print $f $json->encode( $res ) . "\n";
$cnt++;
}
I used the database row caching technique from
Speeding up the DBI
and everything seems good.
The only problem I have for now is that on $res->{some_field} = $res->{another_field}, the row interpreter complains and says that $res is Not a HASH reference.
Please could anybody point me to my mistakes?
If you want fetchall_arrayref to return an array of hashrefs, the first parameter should be a hashref. Otherwise, an array of arrayrefs is returned resulting in the "Not a HASH reference" error. So in order to return full rows as hashref, simply pass an empty hash:
$rowcache = $sth->fetchall_arrayref({}, $max_rows)

I want to load json/csv into hbase

I want to load JSON OR CSV into HBASE with out using any mapreduce programs as well as HIVEQL/pig support, is it possible and which one is more efficient hive-hbase or mapreduce-hbase.
I use a Perl Script to do this;
This is my (perl generated) JSON file
{"c3":"c","c4":"d","c5":"tim","c2":"b","c6":"andrew","c1":"a"},"CURRENTLY20140131":{"c2":"tim2","c1":"bill2"},"THERE20140131"::{"c3":"c","c4":"d","c9":"bill2","c10":"tim2","c2":"b","c6":"andrew","c7":"bill","c5":"tim","c1":"a","c8":"tom"},"TODAY20140131":{"c2":"bill","c1":"tom"}}
I am Sharding on a STRING, with multiple columns depending who/what has referenced the key object.
use strict;
use warnings;
use Data::Dumper;
use JSON::XS qw(encode_json decode_json);
use File::Slurp qw(read_file write_file);
my %words = ();
my $debug = 0;
sub ReadHash {
my ($filename) = #_;
my $json = read_file( $filename, { binmode => ':raw' } );
%words = %{ decode_json $json };
}
# Main Starts here
ReadHash("Save.json");
foreach my $key (keys %words)
{
printf("put 'test', '$key',");
my $cnt=0;
foreach my $key2 ( keys %{ $words{$key} } ) {
my $val = $words{$key}{$key2};
print "," if $cnt>0;
printf("'cf:$key2', '$val'");
++$cnt;
}
print "\n";
}
Generate the Hbase commands and then execute them.
Alternativly - I would look at happybase (Python) which also loads large data sets very quickly.
Hope this helps
This should produce output like .....
put 'test', 'WHERE20140131','cf:c2', 'bill2','cf:c1', 'tim2'
put 'test', 'OMAN20140131','cf:c3', 'c','cf:c4', 'd','cf:c5', 'tim','cf:c2', 'b','cf:c1', 'a','cf:c6', 'andrew'
put 'test', 'CURRENTLY20140131','cf:c2', 'tim2','cf:c1', 'bill2'
Maybe you can refer to bulk loading. Here is the link. bulk loading

Parsing JSON Data::Dumper output array in Perl

I'm trying to edit an old perl script and I'm a complete beginner. The request from the server returns as:
$VAR1 = [
{
'keywords' => [
'bare knuckle boxing',
'support group',
'dual identity',
'nihilism',
'support',
'rage and hate',
'insomnia',
'boxing',
'underground fighting'
],
}
];
How can I parse this JSON string to grab:
$keywords = "bare knuckle boxing,support group,dual identity,nihilism,support,rage and hate,insomnia,boxing,underground fighting"
Full perl code
#!/usr/bin/perl
use LWP::Simple; # From CPAN
use JSON qw( decode_json ); # From CPAN
use Data::Dumper; # Perl core module
use strict; # Good practice
use warnings; # Good practice
use WWW::TheMovieDB::Search;
use utf8::all;
use Encode;
use JSON::Parse 'json_to_perl';
use JSON::Any;
use JSON;
my $api = new WWW::TheMovieDB::Search('APIKEY');
my $img = $api->type('json');
$img = $api->Movie_imdbLookup('tt0137523');
my $decoded_json = decode_json( encode("utf8", $img) );
print Dumper $decoded_json;
Thanks.
Based on comments and on your recent edit, I would say that what you are asking is how to navigate a perl data structure, contained in the variable $decoded_json.
my $keywords = join ",", #{ $decoded_json->[0]{'keywords'} };
say qq{ #{ $arrayref->[0]->{'keywords'} } };
As TLP pointed out, all you've shown is a combination of perl arrays/hashes. But you should look at the JSON.pm documentation, if you have a JSON string.
The result you present is similar to json, but the Perl-variant of it. (ie => instead of : etc). I don't think you need to look into the json part of it, As you already got the data. You just need to use Perl to join the data into a text string.
Just to eleborate on the solution to vol7ron :
#get a reference to the list of keywords
my $keywords_list = $decoded_json->[0]{'keywords'};
#merge this list with commas
my $keywords = join(',', #$keywords_list );
print $keywords;