return json structure using perl - json

i have the following sql statement inside a function..
my $sth = $dbh->prepare(qq[SELECT device_uuid,device_name FROM ].DB_SCHEMA().qq[.user_device WHERE user_id = ?]);
$sth->execute($user_id) || die $dbh->errstr;
the results are being fetched using the following statement
while(my $data = $sth->fetchrow_arrayref()) {
}
my question is how can i create and return a json structure containing objects for every row being fetched?something like this
{
object1:{
"device_uuid1":"id1",
"device_name1":"name1"
},
object2:{
"device_uuid2":"id2",
"device_name2":"name2"
},
object3:{
"device_uuid3":"id3",
"device_name3":"name3"
}
}
the total number of json objects will be equal to the number of rows returned by the sql statement.
i have managed to build the structure like this
$VAR1 = [{"device_name":"device1","device_id":"device_id1"},{"device_name":"device2","device_id":"device_id2"}]
how can i iterate through the array refs and get "device_name" and "device_id" values?

For your needs, this library should work well. What you need to do is have a scalar variable defined as below and push the element for each iteration in while loop
my $json = JSON->new->utf8->space_after->encode({})
while(my $data = $sth->fetchrow_arrayref()) {
#Push new element here in $json using incr_parse method
#or using $json_text = $json->encode($perl_scalar)
}
Hope this helps you.

finally what i did was to create an array ref and push the fetched rows which are being returned as hash refs
my #device = ();
while(my $data = $sth->fetchrow_hashref()) {
push(#device, $data);
}
last i convert the #device array ref to json and return the outcome
return encode_json(\#device);

The statement handle method fetchall_arrayref() can return an array reference where each element in the referenced array is a hash reference containing details of one row in the resultset. This seems to me to be exactly the data structure that you want. So you can just call that method and pass the returned data structure to a JSON encoding function.
# Passing a hash ref to fetchall_arrayref() tells it to
# return each row as a hash reference.
my $json = encode_json($sth->fetchall_arrayref({});

Your sample JSON is incorrect - JSON is actually quite nicely represented by perl data structures - [] denotes array, {} denotes key-value (very similar to hash).
I would rather strongly suggest though, that what you've asked for is probably not what you want - you've seemingly gone for globally unique keys, which ... isn't good style when they're nested.
Why? well, so you can do things like this:
print $my_data{$_}->{'name'} for keys %my_data;
Far better to go for something like:
#!/usr/bin/env perl
use strict;
use warnings;
use Data::Dumper;
use JSON;
my %my_data = (
object1 => {
uuid => "id1",
name => "name1"
},
object2 => {
uuid => "id2",
name => "name2"
},
object3 => {
uuid => "id3",
name => "name3"
},
);
print Dumper \%my_data;
print to_json ( \%my_data, { 'pretty' => 1 } )."\n";
Now, that does assume your 'object1' is a unique key - if it isn't, you can instead do something like this - an array of anonymous hashes (for bonus points, it preserves ordering)
my #my_data = (
{ object1 => {
uuid => "id1",
name => "name1"
}
},
{ object2 => {
uuid => "id2",
name => "name2"
}
},
{ object3 => {
uuid => "id3",
name => "name3"
}
},
);
Now, how to take your example and extend it? Easy peasy really - assemble what you want to add to your structure in your loop, and insert it into the structure:
while(my $data = $sth->fetchrow_arrayref()) {
my $objectname = $data -> [0]; #assuming it's this element!
my $uuid = $data -> [1];
my $name = $data -> [2];
my $new_hash = { uuid => $uuid, name => $name };
$mydata{$objectname} = $new_hash;
}

Related

DBIx to JSON - Wrong format

In a Catalyst application, I need to generate JSON from DBIx::Class:Core objects.
Such a class definition looks like this:
use utf8;
package My::Schema::Book;
use strict;
use warnings;
use Moose;
use MooseX::NonMoose;
use MooseX::MarkAsMethods autoclean => 1;
extends 'DBIx::Class::Core';
__PACKAGE__->load_components("InflateColumn::DateTime");
__PACKAGE__->table("books");
__PACKAGE__->add_columns(
"id",
{
data_type => "uuid",
default_value => \"uuid_generate_v4()",
is_nullable => 0,
size => 16,
},
"title"
);
__PACKAGE__->set_primary_key("id");
__PACKAGE__->meta->make_immutable;
sub TO_JSON {
my $self = shift;
{book => {
id => $self->id,
title => $self->title,
}}
}
1;
After queriyng the books from database I do the encoding of the blessed objects:
$c->stash(books_rs => $c->model('My::Schema::Book'));
$c->stash(books => [$c->stash->{books_rs}->search(
{},
{order_by => 'title ASC'})]
);
$c->stash(json => $json->convert_blessed->encode($c->stash->{books}));
$c->forward('View::JSON');
The JSON output of the query is this:
{"json":"[{\"book\":{\"id\":\"ae355346-8e19-46ee-88ee-773ac30938a9\",\"title\":\"TITLE1\"}},{\"book\":{\"id\":\"9a20f526-d4cd-4e7d-a726-55e78bc3c0ac\",\"title\":\"TITLE2\"}},{\"book\":{\"title\":\"TITLE3\",\"id\":\"1ddb2d27-3ec6-46c1-a1a7-0b151fe44597\"}}]"}
The value of the json key and each particular book key got double quotes what can not be parsed by jQuery. It complains about format exception.
$json->convert_blessed->encode($c->stash->{books}) returns a string. It looks like View::JSON also encodes json.
Try to pass your data as is: $c->stash(json => $c->stash->{books});. You may also need to configure expose_stash and json_encoder_args to handle the right keys from your stash and correctly convert your objects.
See
https://metacpan.org/pod/Catalyst::View::JSON#CONFIG-VARIABLES

Building ordered JSON in perl

I'm having a bit of a fiddle with JSON and D3, trying to represent some disk usage information in a 'bubble' style.
Based on this initially: http://bl.ocks.org/mbostock/4063269
The JSON is pretty simple - hierarchical data looking a bit like this:
http://bl.ocks.org/mbostock/raw/4063530/flare.json
{
"name": "flare",
"children": [
{
"name": "analytics",
"children": [
{
"name": "cluster",
"children": [
{"name": "AgglomerativeCluster", "size": 3938},
{"name": "CommunityStructure", "size": 3812},
{"name": "HierarchicalCluster", "size": 6714},
{"name": "MergeEdge", "size": 743}
]
}
]
}
]
}
Now, what I'm trying to do is take a quotas report from a NetApp - in XML form.
I have multiple 'per server' XML files looking approximately like:
<quotas>
<quota>
<quota-target>/vol/vol1/qtree1</quota-target>
<volume>vol1</volume>
<disk-used>554444</disk-used>
<disk-limit>2000000</disk-limit>
</quota>
<quota>
<quota-target>/vol/vol1/qtree2</quota-target>
<volume>vol1</volume>
<disk-used>1235655</disk-used>
<disk-limit>2000000</disk-limit>
</quota>
<quota>
<quota-target>/vol/vol2/qtree1</quota-target>
<volume>vol2</volume>
<disk-used>987664</disk-used>
<disk-limit>2000000</disk-limit>
</quota>
</quotas>
What I'm trying to do is assemble some JSON for use with D3 that's hierarchical:
site
server
volume
quota-target
disk-used
I'm doing ok with a foreach loop:
#!/usr/bin/env perl
use strict;
use warnings;
use XML::Twig;
use JSON;
my %sites = (
'site1' => [qw ( servera serverb )],
'site2' => [qw ( s2serverc s2serverd)],
);
my $data;
$data->{'name'} = "quotas";
foreach my $sitename ( keys %sites ) {
my $site = { 'name' => $sitename };
push( #{ $data->{'children'} }, $site );
foreach my $server ( #{ $sites{$sitename} } ) {
my $server = { 'name' => $server };
push( #{ $site->{'children'} }, $server );
$twig->parsefile("$server.quotas.xml");
foreach my $quota ( $twig->get_xpath('//quota') ) {
push(
#{ $server->{'children'} },
{ 'name' => $quota->first_child_text('quota-target'),
'size' => $quota->first_child_text('disk-used')
}
)
}
}
}
open( my $output, ">", "quotas.json" ) or die $!;
print {$output} to_json( $data, { 'pretty' => 1 } );
close($output);
This is broadly working, and producing me pretty pictures.
However I'm having two problems:
Ordering of the JSON changes each run, because I'm using a hash. Whilst not a show stopper - is there a way I can enforce an order in the JSON output? (Not necessarily just 'sorted' alphabetically)
Similarly - I'm looking at how to insert a 'volume' level node which isn't currently present, as I'm creating new anonymous hashes to insert into children at each layer of the foreach loop. This feels a bit clunky, but what I'm thinking is:
Extract a list of volumes with get_xpath('//volume') and uniqueify it.
Either iterate per volume finding subnodes that match (Is there an xpath expression to specify a child value?)
Or create a 'staging' hash of hashes that I then 'merge' into children in the JSON.
Does anyone have any better suggestions?
I can quite easily create a hash of the desired structure e.g.
$stuff{$site}{$server}{$volume}{$qtree} = $size;
But then would have to turn that into the appropriate JSON (which I suppose might be a better approach overall).
is there a way I can enforce an order in the JSON output?
Yeah, use an array instead of an object. JSON objects are unordered.
An object is an unordered set of name/value pairs.
But it seems you're already using arrays for your lists.
Maybe you're want to be able to perform diffs, in which case JSON.pm provides a coarse means of specifying keys in the form of sort_by. Useful if you want to perform diffs.
If you just want to "enforce" some sorting by the hash key, you can use the "canonical" feature, as also said here.

Can indexby be used on associations directly in the query?

I have a number of situations where I need to cross-reference various records by ID, and find it's easiest to do so when the array is indexed by that ID. For example, Divisions hasMany Teams, Divisions hasMany Games, and Games belongTo HomeTeam and AwayTeam. When I want to read all of the teams and games in a division, I do something like this:
$division = $this->Divisions->get($id, [
'contain' => ['Teams', 'Games']
]);
I don't do
$division = $this->Divisions->get($id, [
'contain' => ['Teams', 'Games' => ['HomeTeam', 'AwayTeam']]
]);
because it seems that would increase memory requirements, especially when I'm further containing other models (People, etc.) in the Teams. So, instead I do
$division->teams = collection($division->teams)->indexBy('id')->toArray();
after the get to reindex that array, and then when I'm iterating through $division->games, to get the home team record I use $division->teams[$game->home_team_id]. This is all well and good (except that it sets the teams property as being dirty, a minor inconvenience).
But it seems that the queryBuilder functionality of the ORM is pretty magical, and I know that I can do
$teams = $this->Divisions->Teams->find()
->where(['division_id' => $id])
->indexBy('id')
->toArray();
to get an array of teams indexed how I want, so I'm wondering if there's some way to include indexBy on the associations. I tried
$division = $this->Divisions->get($id, [
'contain' => [
'Teams' => [
'queryBuilder' => function (Query $q) {
return $q->indexBy('id');
},
],
'Games',
]
]);
but, unsurprisingly, this didn't work. Any ideas?
Just for the record, guess you know this already, indexBy() doesn't belong to the query, but to the result set, so being able to call it requires the query to be executed first. It's not possible to use this for an association query builder, as it must return a query, not a result set.
While it would be possible to use result formatters for the associations and modify the result set accordingly, the problem is that the result set will hold all team results for all divisions, and when the team entities are being distributed on the various division entities that they belong to, the arrays will be "reindexed", respectively, the arrays will be populated without respect to the indices of the result set, so long story short, that won't work.
Global result formatter
However, a result formatter for the main query should work fine, and as you probably already figured, you can simply reset the dirty state afterwards in case it causes any problems, something like
$division = $this->Divisions
->find()
->contain([
'Teams'
])
->where([
'Divisions.id' => $id
])
->formatResults(function($results) {
/* #var $results \Cake\Datasource\ResultSetInterface|\Cake\Collection\CollectionInterface */
return $results
->map(function ($row) {
if (isset($row['teams'])) {
$row['teams'] = collection($row['teams'])->indexBy('id')->toArray();
}
if ($row instanceof EntityInterface) {
$row->dirty('teams', false);
}
return $row;
});
})
->firstOrFail();
Custom association and association specific result formatters
Another option would be to use a custom association class, which overrides ExternalAssociationTrait::_buildResultMap(), so that it respects the indices of the result set, as this is where the problem starts.
By default the associated entities are fetched from the result set and appended to a new array, which is later assigned to the respective association property on the entity the results belong to. So this is where the the keys from the possible custom indexed result set are being lost.
Here's an example, the change is really small, but I'm not sure about possible side effects!
src/Model/Association/IndexAwareHasMany.php
namespace App\Model\Association;
use Cake\ORM\Association\HasMany;
class IndexAwareHasMany extends HasMany
{
protected function _buildResultMap($fetchQuery, $options)
{
$resultMap = [];
$key = (array)$options['foreignKey'];
// grab the index here
foreach ($fetchQuery->all() as $index => $result) {
$values = [];
foreach ($key as $k) {
$values[] = $result[$k];
}
// and make use of it here
$resultMap[implode(';', $values)][$index] = $result;
}
return $resultMap;
}
}
Original: https://github.com/cakephp/...ORM/Association/ExternalAssociationTrait.php#L109
Now you must of course make use of the new association, to simplify it for this example, let's just override the table class' default hasMany() method
public function hasMany($associated, array $options = [])
{
$options += ['sourceTable' => $this];
$association = new \App\Model\Association\IndexAwareHasMany($associated, $options);
return $this->_associations->add($association->name(), $association);
}
And now, finally, you could use a result formatter for the association:
$division = $this->Divisions->get($id, [
'contain' => [
'Teams' => [
'queryBuilder' => function (Query $query) {
return $query
->formatResults(function($results) {
/* #var $results \Cake\Datasource\ResultSetInterface|\Cake\Collection\CollectionInterface */
return $results->indexBy('id');
});
}
],
'Games',
]
]);

What is the best means to get the unique hash data into a JSON object without using an array when cycling through a DB Dataset?

I need to get the data into a JSON object but because I'm using the %data hash and it has the same address I'm getting the same data repeatedly in my JSON object.
This is the code that produces the JSON.
while (my ($orderID, $possessorName, $itemDescription, $...) = $sth->fetchrow_array)
{
%data = (orderID => $orderID, possessorName => $possessorName, itemDescription => $itemDescription,...);
$query_results{"job$index"} = {"data" => \%data};
$index++;
}
return $json_obj->pretty->encode(\%query_results, {ascii => 1, pretty => 1});
The problem is that the last item in my data set is masking all the previous items so I end up with one large JSON of the same exact data. I could use an array of hashes I suppose but this seems really messy and sloppy. How do I write the cleanest code to get my data? If an array of hashes is the best way to go please let me know and I'll do it. I all ready know how or can figure it out on my own.
What happens when you try:
my $index = 0;
my %query_results;
while (my ($orderID, $possessorName, $itemDescription, $...) = $sth->fetchrow_array) {
my %data = (orderID => $orderID, possessorName => $possessorName, itemDescription => $itemDescription,...);
$query_results{"job$index"}{'data'} = \%data;
$index++;
}
Previously, you used a %data hash declared in an outside scope; or worse, you didn't use strict; use warnings so %data was in fact an implicit global. Now, we declare the %data inside the loop which makes all the hashes distinct.
You could also copy the hash into a new hashref by {%data}.
That said, you don't even need that variable:
$query_results{"job$index"}{data} = {
# anonymous hashref here
orderID => $orderId,
possessorName => $possessorName,
itemDescription => ...
};

Accessing values of json structure in perl

I have a json structure that I'm decoding that looks like this:
person => {
city => "Chicago",
id => 123,
name => "Joe Smith",
pets => {
cats => [
{ age => 6, name => "cat1", type => "siamese", weight => "10 kilos" },
{ age => 10, name => "cat2", type => "siamese", weight => "13 kilos" },
],
dogs => [
{ age => 7, name => "dog1", type => "siamese", weight => "20 kilos" },
{ age => 5, name => "dog2", type => "siamese", weight => "15 kilos" },
],
},
},
}
I'm able to print the city, id, name by doing:
foreach my $listing ($decoded->{person})
{
my $city = $listing->{city};
my $name = $listing->{name};
name - $city - \n";
}
however, I'm unsure of how to print the pets->cats or pets->dogs. I'm able to do a dump of them by:
my #pets = $listing->{pets}->{cats};
dump #pets;
but I'm not sure how to access them through the hash structure.
Digging into a big structure is pretty simple, once you know the rules:
Wrap hash keys in {}
Wrap array indexes in []
If your top level variable is a reference, use -> before the first identifier.
After the first set of braces or brackets, additional arrows (->) are optional.
So:
* $data->{person}{name} returns 'Joe Smith'
* $data->{person}->{name} also returns 'Joe Smith'
* $data->{pets}{cats}[0]{age} returns 6.
For way more detail on this topic, see the Perl Data Structures Cookbook (perldoc perldsc)
When you work with big structures like this there are some important things to be aware of. The biggest of these is autovivification. Autoviv means that Perl will automatically make data structure elements pop into existence for you to make your life easier. Unfortunately it can also make things difficult.
For example, autoviv is great when I do this:
my $data;
$data->{horse}[0]{color} = 'brown';
Autoviv magically turns $data into a hashref that contains the key horse with an array ref as its value. The array ref gets populated by a hash ref. The final hash ref then gets the key value pair of color => brown.
The problem comes in when you are walking a structure and do deep tests for existence:
# Code from above continues:
if( exists $data->{cat}[5]{color} ) {
print "Cat 5 has a color\n";
}
use Data::Dumper;
print Dumper $data;
Here, autovivification burns you by creating a bunch of junk in data, here's the program output:
$VAR1 = {
'cat' => [
undef,
undef,
undef,
undef,
undef,
{}
],
'horse' => [
{
'color' => 'brown'
}
]
};
Now you can guard against this kind of thing by carefully testing each layer of your structure for existence, but it's a huge pain in the butt. Instead, I prefer to use Data::Diver.
use Data::Diver qw( Dive );
my $dog_20_color = Dive( $data, 'dog', 20, 'color' );
print "Dog 20 is $dog_20_color\n" if defined $dog_20_color;
$data is unchanged here.
Also, you may have noticed that since Dive takes a list of keys or indexes, that means its easy to programatically build up a list of keys/indexes and descend an arbitrary path in your code.
Data::Diver can be a real life saver when you have to do a lot of manipulation of big, wonky data structures.
Assuming your $listing is a person you have to dereference array and hash refs.
# as long as we are assuming $listing is a person
# this goes inside the foreach you posted in your
# question.
# this will print all cats' names
foreach my $cat ( #{ $listing->{pets}->{cats} } )
{
# here $cat is a hash reference
say $cat->{name}; # cat's name
}
and so on for other stuff.
To access them from the structure you can do:
say $listing->{pets}->{cats}->[0]->{name}; # this will print 'cat1'
my #pets = $listing->{pets}->{cats};
This isn't doing what you think it is. $listing->{pets}->{cats} contains a reference to an array. Your new #pets array ends up contains just one element - the array reference.
What you actually need is
my #pets = #{ $listing->{pets}{cats} };
This deferences the array reference and gets you the actual array. Notice that I've also dropped the optional second arrow in the expression.
Once you've got the array, each element of it is a hash reference.
foreach (#pets) {
say $_->{name};
# etc ...
}
Of course, you don't need the intermediate array at all.
foreach (#{ $listing->{pets}{cats} }) {
say $_->{name};
}