How can I gracefully iterate over a key - value hash generated from JSON containing values set to 'undef' and 'null'? - json

I am pulling a third party's json response and sometimes the values of the fields are literally 'undef' or 'null'. If I try to do a print of the key and value of each object in this json, whenever there is a undef value it will throw an uninitialized value error.
Is there something I can add to the initial $json->decode to change those null/undefs to something perl can handle? Or maybe even just have it exclude the value pairs that are null/undef from being deposited into $json_text?
my $json_text = $json->decode($content);
foreach my $article(#{$json_text->{data}->{articles}}){
while (my($k, $v) = each ($article)){
print "$k => $v\n";
}
}

$_ // "" will translate undef values to empty string,
my $json_text = $json->decode($content);
foreach my $article (#{$json_text->{data}->{articles}}) {
while (my($k, $v) = map { $_ // "" } each %$article) {
print "$k => $v\n";
}
}

Since you are running a version of Perl that allows each to be applied to a hash reference, you can also use the defined-or operator //.
An expression like a // b evaluates to a if a is defined, otherwise b.
You can use it like this.
my $json_text = $json->decode($content);
for my $article (#{$json_text->{data}{articles}}) {
while (my ($k, $v) = each $article) {
printf "%s => %s\n", $k, $v // 'null';
}
}

Try printf "%s => %s\n", $k || "empty", $v || "empty";
or even
$k ||= "empty";
$v ||= "empty";
print "$k => $v\n";

Related

implode Warning: array to string conversion in PHP-Nuke Titanium function deepPurifier

This works without any warnings in every version of PHP except 8
I think they have changed something with implode and I have tried all the examples to no avail.
Perhaps I could get this done some other way. I need some PHP 8 eyes as I'm very new to PHP 8 and up.
The warning in my function is on the following line:
Warning: array to string conversion
$test = implode(', ', $data);
It is very hard to make sense of certain things when you use so many different languages with similar syntax. I fear that this is just a minor brain fart on my part.
This code appears to be working all though I have the warning, I am wondering if this is just a bug in PHP 8 and 8.1 etc
function deepPurifier($data)
{
global $html_auth, $admin;
static $config, $purifier;
# Error check
if(empty($data) || !isset($data))
return $data;
if(!is_array($data))
return stripslashes((string) $data);
// THIS IS WHERE MY WARNING IS
// warning: array to string conversion
$test = implode(', ', $data);
if(!preg_match('[<|>]', $test))
{
return $data;
}
if(!isset($config) || empty($config))
{
set_include_path(NUKE_BASE_DIR. get_include_path() );
require_once(NUKE_VENDOR_DIR.'ezyang/htmlpurifier/library/HTMLPurifier/Bootstrap.php');
require_once(NUKE_VENDOR_DIR.'ezyang/htmlpurifier/library/HTMLPurifier.autoload.php');
$config = HTMLPurifier_Config::createDefault();
$config->set('Core.Encoding', 'UTF-8');
$config->set('HTML.Doctype', 'HTML 4.01 Transitional');
if(!is_god($admin) || (is_god($admin) && !$html_auth))
{
$config->set('HTML.Trusted', true);
$config->set('HTML.SafeObject', true);
$config->set('HTML.SafeEmbed', true);
$config->set('HTML.AllowedAttributes','img#height,img#width,img#src,iframe#src,iframe#allowfullscreen');
$config->set('HTML.AllowedAttributes', 'src, height, width, alt');
$config->set('HTML.AllowedElements', ['img', 'iframe', 'div', 'script', 'object', 'p', 'span', 'pre', 'b', 'i', 'u', 'strong', 'em', 'sup', 'a', 'img', 'table', 'tr', 'td', 'tbody', 'thead', 'param']);
$config->set('Output.FlashCompat', true);
$config->set('Attr.EnableID', true);
$config->set('Filter.Custom', [new HTMLPurifier_Filter_YouTube()]);
}
$def = $config->getHTMLDefinition(true);
$def->addAttribute('iframe', 'allowfullscreen', 'Bool');
$purifier = new HTMLPurifier($config);
}
# Loop through the data
foreach ($data as $k => $v) {
# If its an array
if (is_array($data[$k])) {
# Go though this function again
$data[$k] = array_map('deepStrip', $data[$k]);
} elseif (is_numeric($v) || empty($v)) {
$data[$k] = $v;
} else {
if (isset($_GET['op']) && $_GET['op'] == 'Configure' && isset($_GET['sub']) && $_GET['sub'] == '11') {
$data[$k] = $v;
continue;
} elseif ($k == 'xsitename' || $k == 'xslogan') {
$data[$k] = $v;
continue;
} elseif (isset($_GET['name'])) {
# If forum post let it pass to the forum html security
if ($_GET['name'] == 'Forums' && (isset($_GET['file']) && ($_GET['file'] == 'posting')) && ($k == 'message' || $k == 'subject')) {
$data[$k] = $v;
continue;
}
# If PM let it pass to the forum html security
if ($_GET['name'] == 'Private_Messages' && ($k == 'message' || $k == 'subject')) {
$data[$k] = $v;
continue;
}
# If SIG let it pass to the forum html security
if ($_GET['name'] == 'Profile' && (isset($_GET['mode']) && ($_GET['mode'] == 'signature')) && $k == 'signature') {
$data[$k] = $v;
continue;
}
}
# If its a strip lets purify it
if (!is_god($admin) || (is_god($admin) && !$html_auth)) {
$data[$k] = $purifier->purify($v);
}
$data[$k] = str_replace('\n', "\n", (string) $data[$k]);
# Get the registered globals also
global ${$k};
if (isset(${$k}) && !empty(${$k})) {
${$k} = $data[$k];
}
}
}
return $data;
}
var_dump($test);
string(20) "Forums,viewtopic,284" string(20) "Forums,viewtopic,284"
The warning appears in PHP 8 when the array being imploded is nested. From the docs, the $array argument should be an array of strings, not an array of arrays.
For example, the following produces no warnings in both PHP 7.4 and PHP 8.1:
$data = ["a", "b"];
print(implode(" ", $data));
Whereas the following gives the warning Array to string conversion (note the arrays within the first array):
$data = ["a", "b" => ["c"], "d" => ["e"]];
print(implode(" ", $data));
You can verify the behaviour with docker and different PHP versions:
docker run --rm php:7.4 -r 'print(implode(" ", ["a", "b" => ["c"], "d" => ["e"]]));'
docker run --rm php:8.1 -r 'print(implode(" ", ["a", "b" => ["c"], "d" => ["e"]]));'
Both produce the output:
a Array Array
But PHP 8 will now raise warnings:
Warning: Array to string conversion in Command line code on line 1
To get rid of the warning you must change the following:
From:
$test = implode(',', $data);
To:
$test = json_encode($data);
Also Change Improper Syntax:
From:
if(!preg_match('[<|>]', $test))
{
return $data;
}
To:
if(!preg_match('/[<|>]/', $test))
{
return $data;
}

Parsing JSON data in Perl

I am parsing JSON data which is in .json file. Here I have 2 formats of JSON data files.
I could parse first JSON file - file is shown below:
file1.json
{
"sequence" : [ {
"type" : "type_value",
"attribute" : {
"att1" : "att1_val",
"att2" : "att2_val",
"att3" : "att3_val",
"att_id" : "1"
}
} ],
"current" : 0,
"next" : 1
}
Here is my script:
#/usr/lib/perl
use strict;
use warnings;
use Data::Dumper;
use JSON;
my $filename = $ARGV[0]; #Pass json file as an argument
print "FILE:$filename\n";
my $json_text = do {
open(my $json_fh, "<:encoding(UTF-8)", $filename)
or die("Can't open \$filename\": $!\n");
local $/;
<$json_fh>
};
my $json = JSON->new;
my $data = $json->decode($json_text);
my $aref = $data->{sequence};
my %Hash;
for my $element (#$aref) {
my $a = $element->{attribute};
next if(!$a);
my $aNo = $a->{att_id};
$Hash{$aNo}{'att1'} = $a->{att1};
$Hash{$aNo}{'att2'} = $a->{att2};
$Hash{$aNo}{'att3'} = $a->{att3};
}
print Dumper \%Hash;
Everything is getting stored in %Hash and when I print Dumper of the %Hash I am getting following result.
$VAR1 = {
'1' => {
'att1' => 'att1_val',
'att2' => 'att2_val',
'att3' => 'att3_val'
}
};
But when I parse second set of JSON file, I am getting empty hash by using the above script.
Output:
$VAR1 = {};
Here is the JSON file -
file2.json
{
"sequence" : [ {
"type" : "loop",
"quantity" : 8,
"currentIteration" : 0,
"sequence" : [ {
"type" : "type_value",
"attribute" : {
"att1" : "att1_val",
"att2" : "att2_val",
"att3" : "att3_val",
"att_id" : "1"
}
} ]
} ]
}
We can see two sequence in above JSON data file, which is causing the problem.
Can somebody tell me what I am missing in the script inorder to parse file2.json.
One possibility might be to check the type field to differentiate between the two file formats:
# [...]
for my $element (#$aref) {
if ( $element->{type} eq "loop" ) {
my $aref2 = $element->{sequence};
for my $element2 ( #$aref2 ) {
get_attrs( $element2, \%Hash );
}
}
else {
get_attrs( $element, \%Hash );
}
}
sub get_attrs {
my ( $element, $hash ) = #_;
my $a = $element->{attribute};
return if(!$a);
my $aNo = $a->{att_id};
$hash->{$aNo}{'att1'} = $a->{att1};
$hash->{$aNo}{'att2'} = $a->{att2};
$hash->{$aNo}{'att3'} = $a->{att3};
}
Please see the following code if it fits your requirements
#!/usr/bin/env perl
#
# vim: ai:sw=4:ts=4
#
use strict;
use warnings;
use feature 'say';
use Data::Dumper;
use JSON;
my $debug = 0; # debug flag
my $filename = shift; # Pass json file as an argument
say "FILE: $filename";
open(my $json_fh, "<:encoding(UTF-8)", $filename)
or die("Can't open \$filename\": $!\n");
my $json_data = do { local $/; <$json_fh> };
close $json_fh;
my $json = JSON->new;
my $data = $json->decode($json_data);
say Dumper($data) if $debug;
my $data_ref;
my %Hash;
$data_ref = $data->{sequence}[0]{attribute}
if $filename eq 'file1.json';
$data_ref = $data->{sequence}[0]{sequence}[0]{attribute}
if $filename eq 'file2.json';
say Dumper($data_ref) if $debug;
my #fields = qw/att1 att2 att3/;
my $aNo = $data_ref->{att_id};
my %data_hash;
#data_hash{#fields} = $data_ref->{#fields};
$Hash{$aNo} = \%data_hash;
say Dumper(\%Hash);

Get request not the same decoded

Hi all i'm new at perl but i have few experience in other language.
So i made a simple code that get JSON file from internet here a telegram bot, but when i display it i got no probleme but when i decoded it with dedcode_json i dont have at all the same output :///
Here the output of the server :
Received reply: {"ok":true,"result":{"id":0000,"first_name":"[MAGA]"}}
and now the output of the decoded anwser :
$VAR1 = {
'ok' => bless( do{\(my $o = 1)}, 'JSON::PP::Boolean' ),
'result' => {
'id' => 0000,
'username' => 'MAGA_bot',
'first_name' => '[MAGA]'
}
};
how can i just get the 'result' part of the decoded json ?
here my code :
#!/usr/bin/perl
use warnings;
use LWP::UserAgent;
use Data::Dumper;
use JSON;
my $ua = LWP::UserAgent->new;
my $destination = "http://api.telegram.org/bot<TOKEN>/getMe";
my $req = HTTP::Request->new(GET => $destination);
my $succes;
my $json;
my $resp = $ua->request($req);
if ($resp->is_success) {
my $message = $resp->decoded_content;
print "Received reply: $message\n";
$succes = "yes";
$json = $message;
} else {
print "HTTP POST error code: ", $resp->code, "\n";
print "HTTP POST error message: ", $resp->message, "\n";
}
print "Encoding the JSON file \n";
if ($succes eq "yes") {
my $decoded_json = decode_json($json);
print Dumper($decoded_json);
} elsif ($succes ne "yes") {
print "Parsing JSON failed\n";
}
Since the decoded JSON is converted into a Perl hash reference in this case, you access it as such:
my $result = $decoded_json->{result};
print "$result->{first_name}\n";
Output:
[MAGA]
If you only want to display part of a complex data structure, then just print that part of the data structure.
print Dumper $decoded_json->{result};

perl fast json parser program

My JSON file contains some 3000 lines of content like below:
{
"product": [
{
"data": [
{
"number":"111",
"price":"3170",
"stock":"1"
},
{
"number":"222",
"price":"3170",
"stock":"1"
},
{
"number":"333",
"price":"3749",
"stock":"1"
}
],
"object":"apple",
"id":"54529"
},
{
"data":[],
"object":"orange",
"id":"54524"
}
]
}
I need to parse them really quick.
Below is my code. It's not working ..
use strict;
use warnings;
use JSON qw( );
my $filename = 'mob.json';
my $json_text = do
{
open(my $json_fh, "<:encoding(UTF-8)", $filename);
local $/;
<$json_fh>
};
my $json = JSON->new;
my $data = $json->decode($json_text);
for ( #{$data->{'product'}} )
{
print $_->{data}[0]->{number};
}
I need to get the number, price, stock and object, id as well.
Your code works fine. Almost. I made a couple of tweaks.
You alluded to speed at the beginning. Not clear if you wanted a quick answer, or a quicker way to parse lots of information. If it's the former, read on. If it's the latter, make sure you have JSON::XS installed.
Style-wise I find it painful to look at.
The use of a do{} to read the file makes me want to hurt myself. But, you used 3-param open. Kudos.
You need to deference the array value from the hash
You need to handle empty values in the data or you'll keep getting warnings
This code parses your JSON and outputs it, substuting empty vals with 'undefined':
use strict;
use warnings;
use JSON qw( );
my $filename = 'mob.json';
my $json_text = do {
open(my $json_fh, "<:encoding(UTF-8)", $filename);
local $/;
<$json_fh>;
};
my $json = JSON->new()->utf8(1);
my $data = $json->decode($json_text);
for my $product ( #{$data->{'product'}} ){
my ($name, $id) = map { $product->{$_} // 'undefined' } qw(name id);
print sprintf("Product: %s (%s)\n", $name, $id);
foreach my $data ( #{$product->{'data'}} ) {
my ($number, $price, $stock) =
map { $data->{$_}//'undefined' } qw(number price stock);
print sprintf(
" number: %s, price: %s, stock: %s\n",
$number,
$price,
$stock,
);
}
print "\n";
}

Perl to convert data into JSON format

I have a problem, converting my data into json, and I don't know why.
Here is some Code that works:
#constructor
sub new {
my $class = shift;
my $Titel = shift;
my $Text = shift;
my $Time = localtime;
my $self = {};
$self->{text} = $Text;
$self->{create} = $Time;
$self->{edit} = $Time;
my $json = JSON::XS->new();
open WF, '>> $file' || die "Error : $!";
print WF $json->encode($self)."\n";
close WF;
bless $self, $class;
}
I create an 'object' and save the data in a textfile (via JSON), too.
I have problems, if I try to edit some data:
sub edit {
my $self = shift;
my $Text = shift;
my $ID = shift;
my $Time = localtime;
my $json = JSON::XS->new();
$json->allow_blessed(1);
$self->{text} = $Text; #edit text
$self->{edit} = $Time; # edit date
open INPUT, '< $file' || die "Error : $!";
my #data = <INPUT>;
close(INPUT);
open WF, '> $file' || die "Error : $!";
for (my $Count=0; $Count<=$#data; $Count++){
chomp($data[$Count]);
if($Count == $ID){#if line of data found, who is going to be edited
print WF $json->encode($self)."\n";
}else{
print WF $data[$Count]."\n";
}
}
close WF;
}
What I try to do is to edit just one line in the textfile.. (if you have a better idea, please show me :D)
I see no difference between my procedure in the code shown first and that one....
it just writes "null" back in the textfile...
Any ideas?
I'm no JSON expert, but the encode method is having trouble with a blessed reference. Using an unblessed reference seems like a valid workaround:
if($Count == $ID){#if line of data found, who is going to be edited
print WF $json->encode( {%$self} )."\n";
...
I second the notion (as you have already found) that the problem is the blessed reference, however I offer you another solution (the is Perl after all: TIMTOWTDI). The Acme::Damn module allows you to unbless (i.e. damn) an object. Therefore you should be able to:
print WF $json->encode(damn($self))."\n";
Also I felt I had to share, since the method is so cleverly named.
Following the last mob's suggestion, here is simple example how to serialize blessed references.
package Node;
sub new {
my $class = shift;
bless { #_ }, $class;
}
sub TO_JSON {
my $self = shift;
return { class => 'Node', data => { %$self } };
}
package main;
use JSON;
my $node_hash = {
a => [ 'text1', 'text2' ],
b => [ 'what', 'is', 'this' ],
c => [ Node->new(some => 'data') ],
};
print to_json($node_hash, { convert_blessed => 1 });
However you need to pay attention in decoding. It is possible to use filter_json_single_key_object to implement full round-trip.