Need to add new data to JSON array in perl - json

I'm trying to create a subroutine to append new values to a json file, but can't seem to put things together correctly. Eventually I would like to be able to plot this data. ideally, each time this routine is run, the newest data passed as an argument is added to the array so it can be plotted.
Here's what I've got so far.
# Subroutine to push CPU data to customer/device json file
sub push_json_data
{
my $json = JSON->new;
#get these as arguements
my $json_cpu_user = $_[0];
my $json_cpu_system = $_[1];
my $json_cpu_idle = $_[2];
my $filename = 'data/custid_devid.json';
#open file for read/write results
#open (JSONFILE, "+>$filename");
#print JSONFILE "$json_data";
#close (JSONFILE);
{
local $/; #Enable 'slurp' mode
open my $fh, "<", "$filename";
$json = <$fh>;
close $fh;
}
my $data = decode_json($json);
my $newdata = {user=>"$json_cpu_user",system=>"$json_cpu_system",idle=>"$json_cpu_idle"};
my $previousdata = $data->{'CPU'};
my $newjsondata = {"CPU"=>[$previousdata,$newdata]};
print Dumper $newjsondata;
open my $fh, ">", "data/custid_devid.json";
print $fh encode_json($newjsondata);
close $fh;
};
That outputs something like this...
$VAR1 = {
'CPU' => [
[
[
{
'system' => '0',
'user' => '1',
'idle' => '97'
}
],
{
'system' => '0',
'user' => '0',
'idle' => '98'
}
],
{
'system' => '0',
'user' => '0',
'idle' => '98'
}
]
};
I'm getting the extra array for each value, how can I get around that?

If I start with a file like:
{"CPU":[]}
and modify your code to do:
my $newdata = {user=>"$json_cpu_user",system=>"$json_cpu_system",idle=>"$json_cpu_idle"};
my $data = decode_json($json);
push #{ $data->{'CPU'} }, $newdata;
...
print $fh encode_json($data);
it does what I think you want.

Related

Parsing JSON data in Perl

I am parsing JSON data which is in .json file. Here I have 2 formats of JSON data files.
I could parse first JSON file - file is shown below:
file1.json
{
"sequence" : [ {
"type" : "type_value",
"attribute" : {
"att1" : "att1_val",
"att2" : "att2_val",
"att3" : "att3_val",
"att_id" : "1"
}
} ],
"current" : 0,
"next" : 1
}
Here is my script:
#/usr/lib/perl
use strict;
use warnings;
use Data::Dumper;
use JSON;
my $filename = $ARGV[0]; #Pass json file as an argument
print "FILE:$filename\n";
my $json_text = do {
open(my $json_fh, "<:encoding(UTF-8)", $filename)
or die("Can't open \$filename\": $!\n");
local $/;
<$json_fh>
};
my $json = JSON->new;
my $data = $json->decode($json_text);
my $aref = $data->{sequence};
my %Hash;
for my $element (#$aref) {
my $a = $element->{attribute};
next if(!$a);
my $aNo = $a->{att_id};
$Hash{$aNo}{'att1'} = $a->{att1};
$Hash{$aNo}{'att2'} = $a->{att2};
$Hash{$aNo}{'att3'} = $a->{att3};
}
print Dumper \%Hash;
Everything is getting stored in %Hash and when I print Dumper of the %Hash I am getting following result.
$VAR1 = {
'1' => {
'att1' => 'att1_val',
'att2' => 'att2_val',
'att3' => 'att3_val'
}
};
But when I parse second set of JSON file, I am getting empty hash by using the above script.
Output:
$VAR1 = {};
Here is the JSON file -
file2.json
{
"sequence" : [ {
"type" : "loop",
"quantity" : 8,
"currentIteration" : 0,
"sequence" : [ {
"type" : "type_value",
"attribute" : {
"att1" : "att1_val",
"att2" : "att2_val",
"att3" : "att3_val",
"att_id" : "1"
}
} ]
} ]
}
We can see two sequence in above JSON data file, which is causing the problem.
Can somebody tell me what I am missing in the script inorder to parse file2.json.
One possibility might be to check the type field to differentiate between the two file formats:
# [...]
for my $element (#$aref) {
if ( $element->{type} eq "loop" ) {
my $aref2 = $element->{sequence};
for my $element2 ( #$aref2 ) {
get_attrs( $element2, \%Hash );
}
}
else {
get_attrs( $element, \%Hash );
}
}
sub get_attrs {
my ( $element, $hash ) = #_;
my $a = $element->{attribute};
return if(!$a);
my $aNo = $a->{att_id};
$hash->{$aNo}{'att1'} = $a->{att1};
$hash->{$aNo}{'att2'} = $a->{att2};
$hash->{$aNo}{'att3'} = $a->{att3};
}
Please see the following code if it fits your requirements
#!/usr/bin/env perl
#
# vim: ai:sw=4:ts=4
#
use strict;
use warnings;
use feature 'say';
use Data::Dumper;
use JSON;
my $debug = 0; # debug flag
my $filename = shift; # Pass json file as an argument
say "FILE: $filename";
open(my $json_fh, "<:encoding(UTF-8)", $filename)
or die("Can't open \$filename\": $!\n");
my $json_data = do { local $/; <$json_fh> };
close $json_fh;
my $json = JSON->new;
my $data = $json->decode($json_data);
say Dumper($data) if $debug;
my $data_ref;
my %Hash;
$data_ref = $data->{sequence}[0]{attribute}
if $filename eq 'file1.json';
$data_ref = $data->{sequence}[0]{sequence}[0]{attribute}
if $filename eq 'file2.json';
say Dumper($data_ref) if $debug;
my #fields = qw/att1 att2 att3/;
my $aNo = $data_ref->{att_id};
my %data_hash;
#data_hash{#fields} = $data_ref->{#fields};
$Hash{$aNo} = \%data_hash;
say Dumper(\%Hash);

Api Response and Json laravel format

I'm using Laravel 5.7. and GuzzleHttp 6.0 to get API response
from endpoint
I'm passing query data from Blade form to this function.
public static function prhmulti($multisearch, $start ,$end)
{ $city = $multisearch['city'];
$client = new Client([
'base_uri' => 'https://avoindata.prh.fi/tr/',
'query' => [
'totalResults' => 'true',
'maxResults' => '1000',
'registeredOffice'=> $city,
'companyForm'=>'OY',
'companyRegistrationFrom'=>$start,
'companyRegistrationTo'=>$end,
],
'defaults'=>[
'timeout' => 2.0,
'cookies' => true,
'headers' => [
'content-type' => 'application/json',
'User-Agent' =>"GuzzleHttp/Laravel-App-5.7, Copyright MikroMike"
]]]);
$res = $client->request('GET','v1');
$ResData = json_decode($res->getBody()->getContents());
dd ($ResData) gives all data from API response.
But I am not able to return JSON back to other function
return $this->multisave($ResData);
public static function multisave (data $ResData)
This will parse JSON and
{
foreach ($data->results as $company) {
$name = $company->name;
$Addr = $company->addresses;
$businessId = $company->businessId;
$companyForm = $company->companyForm;
$registrationDate = $company->registrationDate;
foreach ($company->addresses as $Addr) {
$city = $Addr->city;
$postcode = $Addr->postCode;
$street = $Addr->street;
}
}
save data to Mysql.
$NewCompany = new Company();
$NewCompany = Company::updateOrCreate($array,[
[ 'vat_id', $businessId],
[ 'name', $name],
[ 'form',$companyForm],
[ 'street', $Addr],
[ 'postcode', $postcode],
[ 'city', $city],
[ 'regdate', $registrationDate],
]);
}
IF Parse part and Save part is inside same function code works ok(save only one company),
but I need to separate them because later on it's easier to maintain.
Error which I am getting to return $ResData
" Using $this when not in object context"
Information is in JSON array.
Also foreach part save ONLY one company ?
foreach ($data->results as $company) {
$name = $company->name;
$Addr = $company->addresses;
$businessId = $company->businessId;
$companyForm = $company->companyForm;
$registrationDate = $company->registrationDate;
foreach ($company->addresses as $Addr) {
$city = $Addr->city;
$postcode = $Addr->postCode;
$street = $Addr->street;
}
So : 1) What is best way to create own function for parse JSON
and other for save data to DB?
2) As foreach loop save only one company data, What is
best way to fix it?
Thanks MikroMike.
Resolved my own question for saving companies to db
First get total number inside Array
use for-loop to make counting
use foreach-loop extract information per single company as object.
$data = json_decode($res->getBody()->getContents());
$total = $data->totalResults;
for ($i = 0; $i < $total; $i++){
$NewCompany = new Company();
foreach ($data->results as $company)
{
$name = $company->name;
$businessId = $company->businessId;
$companyForm = $company->companyForm;
$registrationDate = $company->registrationDate;
$array = [];
Arr::set($array, 'vat_id', $businessId);
Arr::set($array, 'name', $name );
Arr::set($array, 'form', $companyForm);
Arr::set($array, 'regdate', $registrationDate);
$NewCompany = Company::updateOrCreate($array,[
[ 'vat_id', $businessId],
[ 'name', $name],
[ 'form',$companyForm],
[ 'regdate', $registrationDate],
]);
}// END OF MAIN FOREACH
}// END OF For loop
}// END OF FUCNTION
} // END OF CLASS

File path into JSON data structure

I'm doing a disk space report that uses File::Find to collect cumulative sizing in a directory tree.
What I get (easily) from File::Find is the directory name.
e.g.:
/path/to/user/username/subdir/anothersubdir/etc
I'm running File::Find to collect sizes beneath:
/path/to/user/username
And build a cumulative size report of the directory and each of the subdirectories.
What I've currently got is:
while ( $dir_tree ) {
%results{$dir_tree} += $blocks * $block_size;
my #path_arr = split ( "/", $dir_tree );
pop ( #path_arr );
$dir_tree = join ( "/", #path_arr );
}
(And yes, I know that's not very nice.).
The purpose of doing this is so when I stat each file, I add it's size to the current node and each parent node in the tree.
This is sufficient to generate:
username,300M
username/documents,150M
username/documents/excel,50M
username/documents/word,40M
username/work,70M
username/fish,50M,
username/some_other_stuff,30M
But I'd like to now turn that in to JSON more like this:
{
"name" : "username",
"size" : "307200",
"children" : [
{
"name" : "documents",
"size" : "153750",
"children" : [
{
"name" : "excel",
"size" : "51200"
},
{
"name" : "word",
"size" : "81920"
}
]
}
]
}
That's because I'm intending to do a D3 visualisation of this structure - loosely based on D3 Zoomable Circle Pack
So my question is this - what is the neatest way to collate my data such that I can have cumulative (and ideally non cumulative) sizing information, but populating a hash hierarchically.
I was thinking in terms of a 'cursor' approach (and using File::Spec this time):
use File::Spec;
my $data;
my $cursor = \$data;
foreach my $element ( File::Spec -> splitdir ( $File::Find::dir ) ) {
$cursor -> {size} += $blocks * $block_size;
$cursor = $cursor -> {$element}
}
Although... that's not quite creating the data structure I'm looking for, not least because we basically have to search by hash key to do the 'rolling up' part of the process.
Is there a better way of accomplishing this?
Edit - more complete example of what I have already:
#!/usr/bin/env perl
use strict;
use warnings;
use File::Find;
use Data::Dumper;
my $block_size = 1024;
sub collate_sizes {
my ( $results_ref, $starting_path ) = #_;
$starting_path =~ s,/\w+$,/,;
if ( -f $File::Find::name ) {
print "$File::Find::name isafile\n";
my ($dev, $ino, $mode, $nlink, $uid,
$gid, $rdev, $size, $atime, $mtime,
$ctime, $blksize, $blocks
) = stat($File::Find::name);
my $dir_tree = $File::Find::dir;
$dir_tree =~ s|^$starting_path||g;
while ($dir_tree) {
print "Updating $dir_tree\n";
$$results_ref{$dir_tree} += $blocks * $block_size;
my #path_arr = split( "/", $dir_tree );
pop(#path_arr);
$dir_tree = join( "/", #path_arr );
}
}
}
my #users = qw ( user1 user2 );
foreach my $user (#users) {
my $path = "/home/$user";
print $path;
my %results;
File::Find::find(
{ wanted => sub { \&collate_sizes( \%results, $path ) },
no_chdir => 1
},
$path
);
print Dumper \%results;
#would print this to a file in the homedir - to STDOUT for convenience
foreach my $key ( sort { $results{$b} <=> $results{$a} } keys %results ) {
print "$key => $results{$key}\n";
}
}
And yes - I know this isn't portable, and does a few somewhat nasty things. Part of what I'm doing here is trying to improve on that. (But currently it's a Unix based homedir structure, so that's fine).
If you do your own dir scanning instead of using File::Find, you naturally get the right structure.
sub _scan {
my ($qfn, $fn) = #_;
my $node = { name => $fn };
lstat($qfn)
or die $!;
my $size = -s _;
my $is_dir = -d _;
if ($is_dir) {
my #child_fns = do {
opendir(my $dh, $qfn)
or die $!;
grep !/^\.\.?\z/, readdir($dh);
};
my #children;
for my $child_fn (#child_fns) {
my $child_node = _scan("$qfn/$child_fn", $child_fn);
$size += $child_node->{size};
push #children, $child_node;
}
$node->{children} = \#children;
}
$node->{size} = $size;
return $node;
}
Rest of the code:
#!/usr/bin/perl
use strict;
use warnings;
no warnings 'recursion';
use File::Basename qw( basename );
use JSON qw( encode_json );
...
sub scan { _scan($_[0], basename($_[0])) }
print(encode_json(scan($ARGV[0] // '.')));
In the end, I have done it like this:
In the File::Find wanted sub collate_sizes:
my $cursor = $data;
foreach my $element (
File::Spec->splitdir( $File::Find::dir =~ s/^$starting_path//r ) )
{
$cursor->{$element}->{name} = $element;
$cursor->{$element}->{size} += $blocks * $block_size;
$cursor = $cursor->{$element}->{children} //= {};
}
To generate a hash of nested directory names. (The name subelement is probably redundant, but whatever).
And then post process it with (using JSON):
my $json_structure = {
'name' => $user,
'size' => $data->{$user}->{size},
'children' => [],
};
process_data_to_json( $json_structure, $data->{$user}->{children} );
open( my $json_out, '>', "homedir.json" ) or die $!;
print {$json_out} to_json( $json_structure, { pretty => 1 } );
close($json_out);
sub process_data_to_json {
my ( $json_cursor, $data_cursor ) = #_;
if ( ref $data_cursor eq "HASH" ) {
print "Traversing $key\n";
my $newelt = {
'name' => $key,
'size' => $data_cursor->{$key}->{size},
};
push( #{ $json_cursor->{children} }, $newelt );
process_data_to_json( $newelt, $data_cursor->{$key}->{children} );
}
}

Perl string to Json

package Emp;
sub new
{
my $class = shift;
my $self = {
OrderID => shift,
OrderDate => shift,
CustomerID => shift,
ShipName => shift,
Freight => shift,
};
bless $self, $class;
return $self;
}
sub TO_JSON { return { %{ shift() } }; }
package main;
use JSON;
my $JSON = JSON->new->utf8;
$JSON->convert_blessed(1);
$e = new Emp( "10248", "1996-07-04", "WILMK","Vins","10");
$json = $JSON->encode($e);
print "$json\n";
I try to convert my String to JSON by using above example. The output is like below:
{"Freight":"10","OrderDate":"1996-07-04","CustomerID":"WILMK","OrderID":"10248","ShipName":"Vins"}
Where do I change if I wanted my JSON to be like below:
{
"rows":[
{"OrderID":"10248","OrderDate":"1996-07-04","CustomerID":"WILMK","ShipName":"Vins et alcools Chevalier","Freight":"32.3800"},
{"OrderID":"10276","OrderDate":"1996-08-08","CustomerID":"TORTU","ShipName":"Tortuga Restaurante","Freight":"13.8400"},
{"OrderID":"10277","OrderDate":"1996-08-09","CustomerID":"MORGK","ShipName":"Morgenstern Gesundkost","Freight":"125.7700"}
]
}
Any advice or reference links is highly appreciated.
Create the data structure you are looking for, and then call JSON->encode
In your main package, try the following:
use JSON;
my $JSON = JSON->new->utf8;
$JSON->convert_blessed(1);
my $data = { rows => [] };
push #{$data->{rows}}, new Emp( "10248", "1996-07-04", "WILMK","Vins","32");
push #{$data->{rows}}, new Emp( "10276", "1996-08-08", "TORTU","Tortuga","13");
push #{$data->{rows}}, new Emp( "10277", "1996-08-09", "MORGK","Morgenstern","125");
$json = $JSON->encode($data);
print "$json\n";
Output:
{"rows":[{"Freight":"32","OrderDate":"1996-07-04","CustomerID":"WILMK","OrderID":"10248","ShipName":"Vins"},
{"Freight":"13","OrderDate":"1996-08-08","CustomerID":"TORTU","OrderID":"10276","ShipName":"Tortuga"},
{"Freight":"125","OrderDate":"1996-08-09","CustomerID":"MORGK","OrderID":"10277","ShipName":"Morgenstern"}]}

Perl to convert data into JSON format

I have a problem, converting my data into json, and I don't know why.
Here is some Code that works:
#constructor
sub new {
my $class = shift;
my $Titel = shift;
my $Text = shift;
my $Time = localtime;
my $self = {};
$self->{text} = $Text;
$self->{create} = $Time;
$self->{edit} = $Time;
my $json = JSON::XS->new();
open WF, '>> $file' || die "Error : $!";
print WF $json->encode($self)."\n";
close WF;
bless $self, $class;
}
I create an 'object' and save the data in a textfile (via JSON), too.
I have problems, if I try to edit some data:
sub edit {
my $self = shift;
my $Text = shift;
my $ID = shift;
my $Time = localtime;
my $json = JSON::XS->new();
$json->allow_blessed(1);
$self->{text} = $Text; #edit text
$self->{edit} = $Time; # edit date
open INPUT, '< $file' || die "Error : $!";
my #data = <INPUT>;
close(INPUT);
open WF, '> $file' || die "Error : $!";
for (my $Count=0; $Count<=$#data; $Count++){
chomp($data[$Count]);
if($Count == $ID){#if line of data found, who is going to be edited
print WF $json->encode($self)."\n";
}else{
print WF $data[$Count]."\n";
}
}
close WF;
}
What I try to do is to edit just one line in the textfile.. (if you have a better idea, please show me :D)
I see no difference between my procedure in the code shown first and that one....
it just writes "null" back in the textfile...
Any ideas?
I'm no JSON expert, but the encode method is having trouble with a blessed reference. Using an unblessed reference seems like a valid workaround:
if($Count == $ID){#if line of data found, who is going to be edited
print WF $json->encode( {%$self} )."\n";
...
I second the notion (as you have already found) that the problem is the blessed reference, however I offer you another solution (the is Perl after all: TIMTOWTDI). The Acme::Damn module allows you to unbless (i.e. damn) an object. Therefore you should be able to:
print WF $json->encode(damn($self))."\n";
Also I felt I had to share, since the method is so cleverly named.
Following the last mob's suggestion, here is simple example how to serialize blessed references.
package Node;
sub new {
my $class = shift;
bless { #_ }, $class;
}
sub TO_JSON {
my $self = shift;
return { class => 'Node', data => { %$self } };
}
package main;
use JSON;
my $node_hash = {
a => [ 'text1', 'text2' ],
b => [ 'what', 'is', 'this' ],
c => [ Node->new(some => 'data') ],
};
print to_json($node_hash, { convert_blessed => 1 });
However you need to pay attention in decoding. It is possible to use filter_json_single_key_object to implement full round-trip.