Parsing JSON data in Perl - json

I am parsing JSON data which is in .json file. Here I have 2 formats of JSON data files.
I could parse first JSON file - file is shown below:
file1.json
{
"sequence" : [ {
"type" : "type_value",
"attribute" : {
"att1" : "att1_val",
"att2" : "att2_val",
"att3" : "att3_val",
"att_id" : "1"
}
} ],
"current" : 0,
"next" : 1
}
Here is my script:
#/usr/lib/perl
use strict;
use warnings;
use Data::Dumper;
use JSON;
my $filename = $ARGV[0]; #Pass json file as an argument
print "FILE:$filename\n";
my $json_text = do {
open(my $json_fh, "<:encoding(UTF-8)", $filename)
or die("Can't open \$filename\": $!\n");
local $/;
<$json_fh>
};
my $json = JSON->new;
my $data = $json->decode($json_text);
my $aref = $data->{sequence};
my %Hash;
for my $element (#$aref) {
my $a = $element->{attribute};
next if(!$a);
my $aNo = $a->{att_id};
$Hash{$aNo}{'att1'} = $a->{att1};
$Hash{$aNo}{'att2'} = $a->{att2};
$Hash{$aNo}{'att3'} = $a->{att3};
}
print Dumper \%Hash;
Everything is getting stored in %Hash and when I print Dumper of the %Hash I am getting following result.
$VAR1 = {
'1' => {
'att1' => 'att1_val',
'att2' => 'att2_val',
'att3' => 'att3_val'
}
};
But when I parse second set of JSON file, I am getting empty hash by using the above script.
Output:
$VAR1 = {};
Here is the JSON file -
file2.json
{
"sequence" : [ {
"type" : "loop",
"quantity" : 8,
"currentIteration" : 0,
"sequence" : [ {
"type" : "type_value",
"attribute" : {
"att1" : "att1_val",
"att2" : "att2_val",
"att3" : "att3_val",
"att_id" : "1"
}
} ]
} ]
}
We can see two sequence in above JSON data file, which is causing the problem.
Can somebody tell me what I am missing in the script inorder to parse file2.json.

One possibility might be to check the type field to differentiate between the two file formats:
# [...]
for my $element (#$aref) {
if ( $element->{type} eq "loop" ) {
my $aref2 = $element->{sequence};
for my $element2 ( #$aref2 ) {
get_attrs( $element2, \%Hash );
}
}
else {
get_attrs( $element, \%Hash );
}
}
sub get_attrs {
my ( $element, $hash ) = #_;
my $a = $element->{attribute};
return if(!$a);
my $aNo = $a->{att_id};
$hash->{$aNo}{'att1'} = $a->{att1};
$hash->{$aNo}{'att2'} = $a->{att2};
$hash->{$aNo}{'att3'} = $a->{att3};
}

Please see the following code if it fits your requirements
#!/usr/bin/env perl
#
# vim: ai:sw=4:ts=4
#
use strict;
use warnings;
use feature 'say';
use Data::Dumper;
use JSON;
my $debug = 0; # debug flag
my $filename = shift; # Pass json file as an argument
say "FILE: $filename";
open(my $json_fh, "<:encoding(UTF-8)", $filename)
or die("Can't open \$filename\": $!\n");
my $json_data = do { local $/; <$json_fh> };
close $json_fh;
my $json = JSON->new;
my $data = $json->decode($json_data);
say Dumper($data) if $debug;
my $data_ref;
my %Hash;
$data_ref = $data->{sequence}[0]{attribute}
if $filename eq 'file1.json';
$data_ref = $data->{sequence}[0]{sequence}[0]{attribute}
if $filename eq 'file2.json';
say Dumper($data_ref) if $debug;
my #fields = qw/att1 att2 att3/;
my $aNo = $data_ref->{att_id};
my %data_hash;
#data_hash{#fields} = $data_ref->{#fields};
$Hash{$aNo} = \%data_hash;
say Dumper(\%Hash);

Related

Get request not the same decoded

Hi all i'm new at perl but i have few experience in other language.
So i made a simple code that get JSON file from internet here a telegram bot, but when i display it i got no probleme but when i decoded it with dedcode_json i dont have at all the same output :///
Here the output of the server :
Received reply: {"ok":true,"result":{"id":0000,"first_name":"[MAGA]"}}
and now the output of the decoded anwser :
$VAR1 = {
'ok' => bless( do{\(my $o = 1)}, 'JSON::PP::Boolean' ),
'result' => {
'id' => 0000,
'username' => 'MAGA_bot',
'first_name' => '[MAGA]'
}
};
how can i just get the 'result' part of the decoded json ?
here my code :
#!/usr/bin/perl
use warnings;
use LWP::UserAgent;
use Data::Dumper;
use JSON;
my $ua = LWP::UserAgent->new;
my $destination = "http://api.telegram.org/bot<TOKEN>/getMe";
my $req = HTTP::Request->new(GET => $destination);
my $succes;
my $json;
my $resp = $ua->request($req);
if ($resp->is_success) {
my $message = $resp->decoded_content;
print "Received reply: $message\n";
$succes = "yes";
$json = $message;
} else {
print "HTTP POST error code: ", $resp->code, "\n";
print "HTTP POST error message: ", $resp->message, "\n";
}
print "Encoding the JSON file \n";
if ($succes eq "yes") {
my $decoded_json = decode_json($json);
print Dumper($decoded_json);
} elsif ($succes ne "yes") {
print "Parsing JSON failed\n";
}
Since the decoded JSON is converted into a Perl hash reference in this case, you access it as such:
my $result = $decoded_json->{result};
print "$result->{first_name}\n";
Output:
[MAGA]
If you only want to display part of a complex data structure, then just print that part of the data structure.
print Dumper $decoded_json->{result};

File path into JSON data structure

I'm doing a disk space report that uses File::Find to collect cumulative sizing in a directory tree.
What I get (easily) from File::Find is the directory name.
e.g.:
/path/to/user/username/subdir/anothersubdir/etc
I'm running File::Find to collect sizes beneath:
/path/to/user/username
And build a cumulative size report of the directory and each of the subdirectories.
What I've currently got is:
while ( $dir_tree ) {
%results{$dir_tree} += $blocks * $block_size;
my #path_arr = split ( "/", $dir_tree );
pop ( #path_arr );
$dir_tree = join ( "/", #path_arr );
}
(And yes, I know that's not very nice.).
The purpose of doing this is so when I stat each file, I add it's size to the current node and each parent node in the tree.
This is sufficient to generate:
username,300M
username/documents,150M
username/documents/excel,50M
username/documents/word,40M
username/work,70M
username/fish,50M,
username/some_other_stuff,30M
But I'd like to now turn that in to JSON more like this:
{
"name" : "username",
"size" : "307200",
"children" : [
{
"name" : "documents",
"size" : "153750",
"children" : [
{
"name" : "excel",
"size" : "51200"
},
{
"name" : "word",
"size" : "81920"
}
]
}
]
}
That's because I'm intending to do a D3 visualisation of this structure - loosely based on D3 Zoomable Circle Pack
So my question is this - what is the neatest way to collate my data such that I can have cumulative (and ideally non cumulative) sizing information, but populating a hash hierarchically.
I was thinking in terms of a 'cursor' approach (and using File::Spec this time):
use File::Spec;
my $data;
my $cursor = \$data;
foreach my $element ( File::Spec -> splitdir ( $File::Find::dir ) ) {
$cursor -> {size} += $blocks * $block_size;
$cursor = $cursor -> {$element}
}
Although... that's not quite creating the data structure I'm looking for, not least because we basically have to search by hash key to do the 'rolling up' part of the process.
Is there a better way of accomplishing this?
Edit - more complete example of what I have already:
#!/usr/bin/env perl
use strict;
use warnings;
use File::Find;
use Data::Dumper;
my $block_size = 1024;
sub collate_sizes {
my ( $results_ref, $starting_path ) = #_;
$starting_path =~ s,/\w+$,/,;
if ( -f $File::Find::name ) {
print "$File::Find::name isafile\n";
my ($dev, $ino, $mode, $nlink, $uid,
$gid, $rdev, $size, $atime, $mtime,
$ctime, $blksize, $blocks
) = stat($File::Find::name);
my $dir_tree = $File::Find::dir;
$dir_tree =~ s|^$starting_path||g;
while ($dir_tree) {
print "Updating $dir_tree\n";
$$results_ref{$dir_tree} += $blocks * $block_size;
my #path_arr = split( "/", $dir_tree );
pop(#path_arr);
$dir_tree = join( "/", #path_arr );
}
}
}
my #users = qw ( user1 user2 );
foreach my $user (#users) {
my $path = "/home/$user";
print $path;
my %results;
File::Find::find(
{ wanted => sub { \&collate_sizes( \%results, $path ) },
no_chdir => 1
},
$path
);
print Dumper \%results;
#would print this to a file in the homedir - to STDOUT for convenience
foreach my $key ( sort { $results{$b} <=> $results{$a} } keys %results ) {
print "$key => $results{$key}\n";
}
}
And yes - I know this isn't portable, and does a few somewhat nasty things. Part of what I'm doing here is trying to improve on that. (But currently it's a Unix based homedir structure, so that's fine).
If you do your own dir scanning instead of using File::Find, you naturally get the right structure.
sub _scan {
my ($qfn, $fn) = #_;
my $node = { name => $fn };
lstat($qfn)
or die $!;
my $size = -s _;
my $is_dir = -d _;
if ($is_dir) {
my #child_fns = do {
opendir(my $dh, $qfn)
or die $!;
grep !/^\.\.?\z/, readdir($dh);
};
my #children;
for my $child_fn (#child_fns) {
my $child_node = _scan("$qfn/$child_fn", $child_fn);
$size += $child_node->{size};
push #children, $child_node;
}
$node->{children} = \#children;
}
$node->{size} = $size;
return $node;
}
Rest of the code:
#!/usr/bin/perl
use strict;
use warnings;
no warnings 'recursion';
use File::Basename qw( basename );
use JSON qw( encode_json );
...
sub scan { _scan($_[0], basename($_[0])) }
print(encode_json(scan($ARGV[0] // '.')));
In the end, I have done it like this:
In the File::Find wanted sub collate_sizes:
my $cursor = $data;
foreach my $element (
File::Spec->splitdir( $File::Find::dir =~ s/^$starting_path//r ) )
{
$cursor->{$element}->{name} = $element;
$cursor->{$element}->{size} += $blocks * $block_size;
$cursor = $cursor->{$element}->{children} //= {};
}
To generate a hash of nested directory names. (The name subelement is probably redundant, but whatever).
And then post process it with (using JSON):
my $json_structure = {
'name' => $user,
'size' => $data->{$user}->{size},
'children' => [],
};
process_data_to_json( $json_structure, $data->{$user}->{children} );
open( my $json_out, '>', "homedir.json" ) or die $!;
print {$json_out} to_json( $json_structure, { pretty => 1 } );
close($json_out);
sub process_data_to_json {
my ( $json_cursor, $data_cursor ) = #_;
if ( ref $data_cursor eq "HASH" ) {
print "Traversing $key\n";
my $newelt = {
'name' => $key,
'size' => $data_cursor->{$key}->{size},
};
push( #{ $json_cursor->{children} }, $newelt );
process_data_to_json( $newelt, $data_cursor->{$key}->{children} );
}
}

perl fast json parser program

My JSON file contains some 3000 lines of content like below:
{
"product": [
{
"data": [
{
"number":"111",
"price":"3170",
"stock":"1"
},
{
"number":"222",
"price":"3170",
"stock":"1"
},
{
"number":"333",
"price":"3749",
"stock":"1"
}
],
"object":"apple",
"id":"54529"
},
{
"data":[],
"object":"orange",
"id":"54524"
}
]
}
I need to parse them really quick.
Below is my code. It's not working ..
use strict;
use warnings;
use JSON qw( );
my $filename = 'mob.json';
my $json_text = do
{
open(my $json_fh, "<:encoding(UTF-8)", $filename);
local $/;
<$json_fh>
};
my $json = JSON->new;
my $data = $json->decode($json_text);
for ( #{$data->{'product'}} )
{
print $_->{data}[0]->{number};
}
I need to get the number, price, stock and object, id as well.
Your code works fine. Almost. I made a couple of tweaks.
You alluded to speed at the beginning. Not clear if you wanted a quick answer, or a quicker way to parse lots of information. If it's the former, read on. If it's the latter, make sure you have JSON::XS installed.
Style-wise I find it painful to look at.
The use of a do{} to read the file makes me want to hurt myself. But, you used 3-param open. Kudos.
You need to deference the array value from the hash
You need to handle empty values in the data or you'll keep getting warnings
This code parses your JSON and outputs it, substuting empty vals with 'undefined':
use strict;
use warnings;
use JSON qw( );
my $filename = 'mob.json';
my $json_text = do {
open(my $json_fh, "<:encoding(UTF-8)", $filename);
local $/;
<$json_fh>;
};
my $json = JSON->new()->utf8(1);
my $data = $json->decode($json_text);
for my $product ( #{$data->{'product'}} ){
my ($name, $id) = map { $product->{$_} // 'undefined' } qw(name id);
print sprintf("Product: %s (%s)\n", $name, $id);
foreach my $data ( #{$product->{'data'}} ) {
my ($number, $price, $stock) =
map { $data->{$_}//'undefined' } qw(number price stock);
print sprintf(
" number: %s, price: %s, stock: %s\n",
$number,
$price,
$stock,
);
}
print "\n";
}

json formatting string to number

My Json output generates;
[
{
"a1_id":"7847TK10",
"output2":"7847TK10",
"output4":"something",
"output5":"3stars.gif",
"output9": "269000",
...
etc. etc.
The google visualization api asks for a number format for the output9 element e.g.:
"output9": 269000 instead of "output9": "269000". How can I achieve this for this element?
My json.php generates the json output like this:
?>
{
"total": <?php echo $total ?>,
"success": true,
"rows": [
// Iterate over the rows
$nextRow= $result->nextRow();
$r = 1;
$info = array();
while ( $nextRow ) {
$nextColumn = $result->nextColumn();
// Has this column been printed already
if ( $unique )
{
$d = $result->getDataForField($unique);
if ( array_key_exists($d, $already) )
{
$nextRow= $result->nextRow();
continue;
}
$already[$d] = true;
}
echo '{';
// Iterate over the columns in each row
while ( $nextColumn )
{
// Get the variable
$variable = $result->getOutputVariable();
$name = $variable->getName(true);
$data = $result->getDataForField();
if ( !isset($info[$name]) ) {
$info[$name]['translate'] = $variable->shouldTranslate();
$info[$name]['type'] = $variable->getDataType();
$info[$name]['linkable'] = $variable->isLinkable();
}
// Translate the data if requested
if ( $info[$name]['translate'] ) {
$data = LQMTemplate::_($data);
}
$data = $variable->format($data, false);
$type = $info[$name]['type'];
if ( ($type == 'bool') or ($type == 'boolean') )
{
$data = $data ? '1' : '0';
echo "'$name':$data";
} elseif ( $encode ) {
// Can we use json_encode ?
// str_replace because some versions of PHP have a bug that will over escape forward slashes
echo "\"$name\":".str_replace('\\/', '/', json_encode($data));
} else {
$data = LQMUtility::jsonEscape($data, '"');
//echo "'$name':\"$data\"";
echo "\"$name\":\"$data\"";
}
// Conditionally print the next column
$nextColumn = $result->nextColumn();
if ( $nextColumn ) echo ",\n ";
}
// Conditionally print the next column
$nextRow = $result->nextRow();
echo $nextRow ? "},\n" : "}\n";
$r++;
}
unset($result);
echo ']}';
}
}
This depends on how you are generating your JSON.
For example, if you were using a Ruby backend, you could call:
"output9" => output9.to_i
There are various helper methods in different languages (e.g. Java and Javascript have parseInt() functions) to change a string into an integer.
Edit:
If your JSON is being generated by PHP, cast the string to an integer:
$json['output9'] = int($output9_value);
That should get rid of the quotation marks.

Perl to convert data into JSON format

I have a problem, converting my data into json, and I don't know why.
Here is some Code that works:
#constructor
sub new {
my $class = shift;
my $Titel = shift;
my $Text = shift;
my $Time = localtime;
my $self = {};
$self->{text} = $Text;
$self->{create} = $Time;
$self->{edit} = $Time;
my $json = JSON::XS->new();
open WF, '>> $file' || die "Error : $!";
print WF $json->encode($self)."\n";
close WF;
bless $self, $class;
}
I create an 'object' and save the data in a textfile (via JSON), too.
I have problems, if I try to edit some data:
sub edit {
my $self = shift;
my $Text = shift;
my $ID = shift;
my $Time = localtime;
my $json = JSON::XS->new();
$json->allow_blessed(1);
$self->{text} = $Text; #edit text
$self->{edit} = $Time; # edit date
open INPUT, '< $file' || die "Error : $!";
my #data = <INPUT>;
close(INPUT);
open WF, '> $file' || die "Error : $!";
for (my $Count=0; $Count<=$#data; $Count++){
chomp($data[$Count]);
if($Count == $ID){#if line of data found, who is going to be edited
print WF $json->encode($self)."\n";
}else{
print WF $data[$Count]."\n";
}
}
close WF;
}
What I try to do is to edit just one line in the textfile.. (if you have a better idea, please show me :D)
I see no difference between my procedure in the code shown first and that one....
it just writes "null" back in the textfile...
Any ideas?
I'm no JSON expert, but the encode method is having trouble with a blessed reference. Using an unblessed reference seems like a valid workaround:
if($Count == $ID){#if line of data found, who is going to be edited
print WF $json->encode( {%$self} )."\n";
...
I second the notion (as you have already found) that the problem is the blessed reference, however I offer you another solution (the is Perl after all: TIMTOWTDI). The Acme::Damn module allows you to unbless (i.e. damn) an object. Therefore you should be able to:
print WF $json->encode(damn($self))."\n";
Also I felt I had to share, since the method is so cleverly named.
Following the last mob's suggestion, here is simple example how to serialize blessed references.
package Node;
sub new {
my $class = shift;
bless { #_ }, $class;
}
sub TO_JSON {
my $self = shift;
return { class => 'Node', data => { %$self } };
}
package main;
use JSON;
my $node_hash = {
a => [ 'text1', 'text2' ],
b => [ 'what', 'is', 'this' ],
c => [ Node->new(some => 'data') ],
};
print to_json($node_hash, { convert_blessed => 1 });
However you need to pay attention in decoding. It is possible to use filter_json_single_key_object to implement full round-trip.