getting JSON from DBI:Pg selectall_arrayref - json

I have an array
my #cols = ("accountid", "balance");
and a dataset
my $rowsref=$dbh->selectall_arrayref($_[0]);
foreach my $row (#$rowsref) {
print join(", ", map {defined $_ ? $_ : "(null)"} #$row), "\n";
}
which prints "1, 150".
I would like to get a JSON output like [{"accountid": 1, "balance": 150},{..}].
I have the JSON module loaded, but unsure how to merge #cols with each $row.

edit: added explanation of column name transaction in 2nd example
edit: Fixed for your requirement of a single JSON encoding of the whole resultset.
edit: forgot keys in cols mapping code in 2nd example.
edit: typo'd hashref everywhere to arrayref. :-
Firstly, use selectall_hashref instead, so that it already contains the key names.
Then use one of the JSON encoding modules to encode each row.
(making the same assumptions as your code....)
Using the list-of-hashrefs from selectall_hashref() as-is:
use JSON::XS;
use 5.10.0;
my $rowsref = $dbh->selectall_hashref($_[0]);
print JSON::XS::encode_json($rowsref),"\n";
Performing translation on colnames from selectall_hashref():
If the column names from the database aren't the same as your column names, then you'll need a mapping:
use JSON::XS;
use 5.10.0;
my $trans = { account => 'accountid', amount => 'balance' };
my $rowsref = $dbh->selectall_hashref($_[0]);
my $output = [];
for my $row (#$rowsref) {
push #$output, {
map {
my $colname = exists($trans->{$_}) ? $trans->{$_} : $_;
my $value = $row->{$_};
$colname => $value;
} keys %$row
});
}
print JSON::XS::encode_json($output),"\n";
For each $row above of the resultset, keys %$row gives back the column names in the row as returned from the database.
The map operation takes each of those column names and producues 2 scalar values; (1) $colname is either the original database column name or (if it's found in the $trans hashref) a 'translation' of the column name; (2) $value is the value returned by the database for this column name in this particular $row. $colname => $value returns both the $colname and $value from the map as a 'flattened' pair of scalar values. That means that the map operation returns a list of scalar values twice as long as the original list of column names returned by keys %$row.
Finally, the push #$output, { ... } creates an anonymous hash reference from that list of scalar values in key,value,key,value,... order and adds it to the end of the $output array reference.
Blind translation from selectall_arrayref()
If (for some reason) you have a pathological aversion to querying hashrefs from your database, I guess you could do:
use 5.10.0;
my #cols = ("accountid", "balance");
my $rowsref = $dbh->selectall_arrayref($_[0]);
my $output = [];
for my $row (#$rowsref) {
my %row = map { $cols[$_] => $row->[$_] } (0 .. $#cols);
push #$output, \%row;
}
print JSON::XS::encode_json($output),"\n";
That assumes there are only two columns coming back from the query, though.
"Defined or" operator:
By the way... assuming a late enough perl, you can replace this sort of thing:
defined $_ ? $_ : "(null)"
with this:
$_ // "(null)"
Your code editor (e.g: Vim) might not syntax highlight it correctly if it's not up to date with perl. (e.g: it might treat it as an m// construct).

Note that PostgreSQL can also generate JSON. If it is an option for you then Perl JSON module is redundant.

Related

Can't read: variable is array

I am trying to pull back all the data from my sql statement but I am only able to pull back the first row data in the first put statement. So what I am trying to do is do a for loop through the $data variable. The for loop won't run because it says that the variable is an array.
orasql crsr $sql
orafetch $crsr -datarray data -indexbyname
puts $data(customer)
foreach value $data {
puts $data(customer)
}
The problem is when you do foreach value $data; since the data variable is an array, you can't read it as a simple value. To print the keys and values in the array, do this:
foreach {key value} [array get data] {
puts "$key => $value"
}
The array get command takes the name of an array variable and gets a list of keys and values from it, which can be iterated over with a two-variable foreach. (The parray command does a somewhat more sophisticated version of this; you could use parray data in place of that loop above, but then you'd have no real other route into processing the row.)
You usually ought to know the names of the columns you're getting back from an SQL query so that you don't need to loop over it like that. Also, the order of columns doesn't really matter, but neither does the order of entries in the array. (The order of rows in the result set of the query matters, but orafetch only gets one at a time.)

Perl / DBI query doesn't preserve integer values for JSON output

I can't get this Perl code to return true integer values for integers in the table. The MySQL table columns are correctly specified as integers, yet the JSON output here wraps all query values in quotes. How can I correctly preserve data-types (esp. integers and boolean values) as specified?
use strict;
use warnings;
use DBI;
use JSON;
my $sth = "SELECT id, name, age FROM table";
my $data = $dbh->selectall_arrayref($sth, {Slice => {}});
my $response = encode_json($data);
print $response;
## outputs: {"id":"1","name":"Joe Blodge","age":"42"}
What am I doing wrong here? How can I get this to output the correctly formatted JSON:
{"id":1,"name":"Joe Blodge","age":42}
DBD::mysql returns all results as strings (see https://github.com/perl5-dbi/DBD-mysql/issues/253). Normally Perl doesn't care, encoding to JSON is one of the few times when it matters. You can either use Cpanel::JSON::XS::Type to provide type declarations for your JSON structure:
use Cpanel::JSON::XS;
use Cpanel::JSON::XS::Type;
my $response = encode_json($data, {id => JSON_TYPE_INT, name => JSON_TYPE_STRING, age => JSON_TYPE_INT});
or you can go through and numify the appropriate elements before JSON encoding.
$data->{$_} += 0 for qw(id age);
It is possible to check the type (as indicated by MySQL) of each returned column, if you construct and execute your query using a statement handle then the type will be available as an array in $sth->{TYPE}, but this is pretty complex and may not be reliable.

how to specify the order of attributes of json object in perl?

I'm parsing a table in html file to make it a json file.
I set up list of attribute names. Each time designated element was found i store it in a perl hash with next attibute name in the list(see code below). Then us JSON module to encode the hash. But the order of each attribute in a object was not the order they were inserted.
sub scan_line
{
my($elem) = #_; # HTML::Element
my %result = ();
my #tds = $elem->find("td");
my $index = 0; # of attrnames
foreach my $td (#tds){
$result{$attrnames[$index]} = $td->as_text();
$index++;
}
my $text = $json->encode(\%result);
print TARGET $text;
}
Are there methods in perl that can specify the order of attribute or add attribute manually like "$jobj->add_attr($attr, $value)"?
You can use:
my $text = $json->sort_by(sub { $JSON::PP::a cmp $JSON::PP::b })->encode(\%result);
Of course you can use <=> instead of cmp or any other sorting function. Note that if your JSON version is below 2.0, you have to explicitly create $json as JSON::PP->new (not JSON->new).
Also be aware that this will decrease performance (and not only of the encode call, but of all $json usages, AFAIU).

JSON giving unwanted row index number - Active Record - Codeigniter

I don't know why when I echo json_encode a query result set I get the number of the result row before each object. I just want to count the number of total rows returns and have them displayed only once in the beginning of the JSON string and then just the rows returns afterwards. I.e. using the following code:
//...active record query
$result = $this->db->get();
$data = array();
$count = 1;
foreach($result->result() as $row)
{
$data['count'] = $count;
$entry = array();
$entry['firstname'] = $row->first_name;
$entry['lastname'] = $row->last_name;
$entry['jobtitle'] = $row->title;
$entry['dept'] = $row->dept_name;
$entry['deptid'] = $row->dept_no;
if($row->emp_no == null)
{
$entry['ismanager'] = 0;
}
else
{
$entry['ismanager'] = 1;
}
$data[] = $entry;
$count++;
}
return $data;
and then json_encode it in the controller, I get:
{"count":35,"0":{"firstname":"Georgi","lastname":"Facello","jobtitle":"Senior Engineer","dept":"Development","deptid":"d005","ismanager":0},"1":{"firstname":"Kirk","lastname":"Facello","jobtitle":"Senior Engineer","dept":"Development","deptid":"d005","ismanager":0},....rest of the query results
What I don't want is the "0" and "1" etc, before the row results. I already have the total count of the returned results so I don't need the individual row numbers.
If someone could kindly help me out I would appreciate it, thanks.
If you try to serialize an array as JSON, it would become something like this:
[elem1, elem2, elem3, ...]
But if that "array" have other fields then it will be serialized as an object:
{"field":value, "0":elem1, "1":elem2, "2":elem3, ...}
Since there's no way to serialize field using the array syntax, and json_encode can not simply discard it, then it uses the object syntax. As stated in the docs:
Note:
When encoding an array, if the keys are not a continuous numeric sequence starting from 0, all keys are encoded as strings, and specified explicitly for each key-value pair.
A possible workaround for this would be separating the count from the list of elements:
$data = array();
$list = array();
$data['list'] = list;
$count = 1;
foreach($result->result() as $row)
{
$data['count'] = $count;
$entry = array();
...
$list[] = $entry;
$count++;
}
That would serialize to something like:
{"count":35,"list":[{"firstname":"Georgi","lastname":"Facello","jobtitle":"Senior Engineer","dept":"Development","deptid":"d005","ismanager":0},{"firstname":"Kirk","lastname":"Facello","jobtitle":"Senior Engineer","dept":"Development","deptid":"d005","ismanager":0},....rest of the query results]}
It looks like you might be using JSON_FORCE_OBJECT on your json_encode which will always make your numerical index show up as a property. You should show your json_encode step in your question.
If you turn option off and go with a default encoding, you will still need to nest your numerically indexed array in its own property or the numerical indexes will show up as properties in order to make valid JSON. For perhaps do something like this when assigning your rows to the object:
$data['records'][] = $entry;

Problems parsing Reddit's JSON

I'm working on a perl script that parses reddit's JSON using the JSON module.
However I do have the problem of being very new to both perl and json.
I managed to parse the front page and subreddits successfully, but the comments have a different structure and I can't figure out how to access the data I need.
Here's the code that successfully finds the "data" hash for the front page and subreddits:
foreach my $children(#{$json_text->{"data"}->{"children"}}) #For values of children.
{
my $data = $children->{"data"}; #accessing each data hash.
my %phsh = (); #my hash to collect and print.
$phsh{author} = $data->{"author"};#Here I get the "author" value from "data"
*Etc....
This successfully gets what I need from http://www.reddit.com/.json
But when I go to the json of a comment, this one for example, it has a different format and I can't figure out how to parse it. If I try the same thing as before my parser crashes, saying it is not a HASH reference.
So my question is: How do access the "children" in the second JSON? I need to get both the data for the Post and the data for the comments. Can anybody help?
Thanks in advance!
(I know it may be obvious, but I'm running on very little sleep XD)
You need to either look at the JSON data or dump the decoded data to see what form it takes. The comment data, for example is an array at the top level.
Here is some code that prints the body field of all top-level comments. Note that a comment may have an array of replies in its replies field, and each reply may also have replies in turn.
Depending on what you want to do you may need to check whether a reference is to an array or a hash by checking the value returned by the ref operator.
use strict;
use warnings;
binmode STDOUT, ':utf8';
use JSON;
use LWP;
use Data::Dump;
my $ua = LWP::UserAgent->new;
my $resp = $ua->get('http://www.reddit.com/r/funny/comments/wx3n5/caption_win.json');
die $resp->status_line unless $resp->is_success;
my $json = $resp->decoded_content;
my $data = decode_json($json);
die "Error: $data->{error}" if ref $data eq 'HASH' and exists $data->{error};
dd $data->[1]{data}{children}[0];
print "\n\n";
my $children = $data->[1]{data}{children};
print scalar #$children, " comments:\n\n";
for my $child (#$children) {
print $child->{data}{body}, "\n";
}