I'm trying to parse JSON data in Perl. it is request to Cisco Prime Service. My script works, but parsing doesn't work. And I have a warning,
malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "HTTP::Response=HASH(...") at InternetBSP.pl line 39.
It is here:
my $json_text = $json->allow_nonref->utf8->relaxed->escape_slash->loose->allow_singlequote->allow_barekey->decode($res);
have no Idee how should I fix it...
use strict;
use warnings;
use JSON -support_by_pp;
use LWP 5.64;
use LWP::UserAgent;
use MIME::Base64;
use REST::Client;
use IO::Socket::SSL;
#So dass es auch ohne SSL Sertifizierung funktioniert
BEGIN { $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0 }
#Create a user agent object
my $ua = LWP::UserAgent->new(
ssl_opts => {
SSL_verify_mode => SSL_VERIFY_NONE(),
verify_hostname => 0,
}
);
#Create a request
my $req = HTTP::Request->new( GET => 'https://10.10.10.10/webacs/api/v1/data/AccessPoints.json?.full=true' );
$req->content_type('application/json');
$req->authorization_basic( "Username", "Password" );
#Pass request to the user agent and get a response back
my $res = $ua->request($req);
#Check the outcome of the Response
if ( $res->is_success ) {
print $res->content;
} else {
print $res->status_line, "n";
}
my $json = new JSON;
my $json_text = $json->allow_nonref->utf8->relaxed->escape_slash->loose->allow_singlequote->allow_barekey->decode($res);
#my try to pasre the data
foreach my $ap ( #{ $json_text->{queryResponse}->{'entity'} } ) {
print "------------------------\nAccess Point " . $ap->{'accessPointsDTO'}->{'#id'} . "\n";
print "Model:" . $ap->{'accessPointsDTO'}->{'model'} . "\n";
print "MAC Address:" . $ap->{'accessPointsDTO'}->{'macAddress'} . "\n";
print "Serial Number:" . $ap->{'accessPointsDTO'}->{'serialNumber'} . "\n";
print "Software Version:" . $ap->{'accessPointsDTO'}->{'softwareVersion'} . "\n";
print "Status:" . $ap->{'accessPointsDTO'}->{'status'} . "\n";
print "Location:" . $ap->{'accessPointsDTO'}->{'location'} . "\n";
}
I have this like outcome:
{"queryResponse":{"#last":"7","#first":"0","#count":"8","#type":"AccessPoints","#responseType":"listEntityInstances","#requestUrl":"https:\/\/10.66.1.23\/webacs\/api\/v1\/ data\/AccessPoints?.full=true","#rootUrl":"https:\/\/10.66.1.23\/webacs\/api\/v1\/data","entity":[{"#dtoType":"accessPointsDTO","#type":"AccessPoints","#url":"https:\/\/10 .66.1.23\/webacs\/api\/v1\/data\/AccessPoints\/205320"
But it shoud be smth like:
{"queryResponse":
{"#type":"AccessPoints",
"#rootUrl":"https://172.18.138.90/webacs/api/v1/data",
"#requestUrl":"https://172.18.138.90/webacs/api/v1/data/AccessPoints?.full=true",
"#responseType":"listEntityInstances",
"entity":[
{"#url":"https://172.18.138.90/webacs/api/v1/data/AccessPoints/13544533",
"#type":"AccessPoints",
"#dtoType":"accessPointsDTO",
"accessPointsDTO":
{"#id":"13544533",
"#displayName":"13544533",
"adminStatus":"ENABLE",
"bootVersion":"12.4.23.0",
"clientCount":0,
After update :)
------------------------
Access Point 205320
Model:AIR-LAP1142N-E-K9
MAC Address:6c:9c:ed:b5:45:60
Serial Number:FCZ1544W51B
Software Version:7.6.130.0
Status:CLEARED
Location:de.bw.stu.
------------------------
Access Point 205322
Model:AIR-CAP3502I-E-K9
MAC Address:0c:f5:a4:ee:70:10
Serial Number:FCZ184680VB
Software Version:7.6.130.0
Status:CLEARED
Location:de.bw.stu.
------------------------
Access Point 205324
Model:AIR-LAP1142N-E-K9
MAC Address:6c:9c:ed:86:9d:20
Serial Number:FCZ1544W50Y
Software Version:7.6.130.0
Status:CLEARED
Location:de.bw.stu.
malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "HTTP::Response=HASH(...")
This error message means that the data you are giving to decode is not JSON.
You are passing $res to decode, which is an HTTP::Response object (see above, emphasis mine). You need to use $res->content, which you use for debugging output a few lines above.
if ($res->is_success) {
print $res->content;
} else {print $res->status_line, "n";
}
I would rewrite that whole block of code to this.
die $res->status_line unless $res->is_success;
my $json = JSON->new->allow_nonref
->utf8->relaxed
->escape_slash->loose
->allow_singlequote->allow_barekey;
my $json_text = $json->decode( $res->content );
Instead of printing some debug output and then going on anyway if things went wrong you can just die if the request was not successful.
After that, create your JSON object and configure it. This is way more readable than this long line of code, and we're using a method call to new instead of indirect object notation.
Finally, we are decodeing $res->content.
Related
I´m getting an issue when trying to store a json comming from a String into a Hash. Take a look at this example:
use strict;
use warnings;
use utf8;
use JSON;
use Data::Dumper;
my %hash1 = %{get_hash_from_json()};
print "Final Dump:\n";
print Dumper \%hash1 . "\n";
print "Keys:\n";
for (keys %hash1) {printf "key is $_\n";}
sub get_hash_from_json (){
my $json = '{"hello": "world", "hello2": "world"}';
print "Dumper without variable:\n";
print Dumper (from_json($json)) . "\n";
print "Dumper with variable:\n";
my $hash_ref = from_json($json);
my %hash = %{$hash_ref};
print Dumper \%hash . "\n";
return from_json($json);
}
And the output is:
main::get_hash_from_json() called too early to check prototype at example.pl line 10.
Dumper without variable:
$VAR1 = {
'hello' => 'world',
'hello2' => 'world'
};
Dumper with variable:
$VAR1 = 'HASH(0x29c88e0)
';
Final Dump:
$VAR1 = 'HASH(0x2512438)
';
Keys:
key is hello2
key is hello
Does anyone understand why is this happening? Somehow the hash is there but Data::Dumper won´t take it?
Precedence issue.
print Dumper \%hash . "\n";
means
print(Dumper(\%hash . "\n"));
but you want
print(Dumper(\%hash) . "\n");
The thing is, the value returned by Dumper will already end with a new line, so you really don't need another one. The following would do:
print(Dumper(\%hash));
If you want to omit the parens, there's no harm here.
print Dumper \%hash;
As for the prototype-related warning, you are getting it because a call to a sub with a prototype is encountered by the compiler before a declaration for that sub is encountered. In that situation, the call did not use the prototype, so Perl is letting you know this happened.
The simple solution is to just remove the useless prototype (the () in sub get_hash_from_json ()).
You are falling victim to precedence.
print Dumper \%hash1 . "\n";
The . concatenates \%hash1 and the newline, and that's what Dumper outputs. Put parentheses around it to make it work.
print Dumper(\%hash1) . "\n";
Or use say.
Here is my code that I try to open the file to get data and change it to UTF-8, then read each line and store it in variable my $abstract_text and send it back in JSON structure.
my $fh;
if (!open($fh, '<:encoding(UTF-8)',$path))
{
returnApplicationError("Cannot read abstract file: $path ($!)\nERRORCODE|111|\n");
}
printJsonHeader;
my #lines = <$fh>;
my $abstract_text = '';
foreach my $line (#lines)
{
$abstract_text .= $line;
}
my $json = encode_json($abstract_text);
close $fh;
print $json;
By using that code, I get this error;
hash- or arrayref expected (not a simple scalar, use allow_nonref to allow this)
error message also point out that the problem is in this line;
my $json = encode_json($abstract_text);
I want to send the data back as a string (which is in UTF-8). Please help.
I assume you're using either JSON or JSON::XS.
Both allow for non-reference data, but not via the procedural encode_json routine.
You'll need to use the object-oriented approach:
use strict; # obligatory
use warnings; # obligatory
use JSON::XS;
my $encoder = JSON::XS->new();
$encoder->allow_nonref();
print $encoder->encode('Hello, world.');
# => "Hello, world."
Edit: modified code and output to make it more clear
Edit 2: Added example input for reproduction
I have a JSON file and a CSV file and I am running comparisons on the two. The problem is that $asset_ip is correctly defined in the outer foreach loop, but when in the nested loop $asset_ip becomes undefined.
Why is $asset_ip becoming undefined?
#!/usr/bin/perl
# perl -e'use CPAN; install "Text::CSV"'
use strict;
use warnings;
use JSON::XS;
use File::Slurp;
use Text::CSV;
my $csv = Text::CSV->new( { sep_char => ',' } );
my $csv_source = "servers.csv";
my $json_source = "assets.json";
my $dest = "servers_for_upload.csv";
# defined these here as I need to use them in foreach loop and if statement:
my $csv_ip;
my #fields;
open( my $csv_fh, '<', $csv_source ) or die "$! error trying to read";
open( my $dest_fh, '>', $dest ) or die "$! error trying to read";
my $json = read_file($json_source);
my $json_array = decode_json $json;
foreach my $item (#$json_array) {
my $id = $item->{id};
my $asset_ip = $item->{interfaces}->[0]->{ip_addresses}->[0]->{value};
# test the data is there:
if ( defined $asset_ip ) {
print "id: " . $id . "\nip: " . $asset_ip . "\n";
}
while (my $line = <$csv_fh>) {
chomp $line;
if ( $csv->parse($line) ) {
#fields = $csv->fields();
$csv_ip = $fields[0];
}
else {
warn "Line could not be parsed: $line\n";
}
if ( $csv_ip eq $asset_ip ) {
# preppend id to csv array and write these lines to new file
unshift( #fields, $id );
print $dest_fh join( ", ", #fields );
}
}
}
close $csv_fh;
Output:
Use of uninitialized value $asset_ip in string eq at script.pl line 43, <$csv_fh> line 1.
Use of uninitialized value $asset_ip in string eq at script.pl line 43, <$csv_fh> line 2.
Use of uninitialized value $asset_ip in string eq at script.pl line 43, <$csv_fh> line 3.
id: 1003
ip: 192.168.0.2
id: 1004
ip: 192.168.0.3
id: 1005
ip: 192.168.0.4
assets.json:
[{"id":1001,"interfaces":[]},{"id":1003,"interfaces":[{"ip_addresses":[{"value":"192.168.0.2"}]}]},{"id":1004,"interfaces":[{"ip_addresses":[{"value":"192.168.0.3"}]}]},{"id":1005,"interfaces":[{"ip_addresses":[{"value":"192.168.0.4"}]}]}]
Note, that for the first iteration, $asset_ip will be undefined. I will therefore alter the code to only run the eq comparison if $asset_ip is defined. However, for this example I am not doing the check because all iterations are undefined.
servers.csv:
192.168.0.3,Brian,Germany
192.168.0.4,Billy,UK
192.168.0.5,Ben,UK
I think your problem will be this:
foreach my $line (<$csv_fh>) {
You execute this within our outer loop. But when you do this, your $csv_fh ends up at the end of file.
Once you have done this, subsequent iterations of your outer loop will not execute this inner loop, because there's nothing left for it to read from $csv_fh.
An easy test if this is your problem is to add a seek e.g. seek ( $csv_fh, 0, 0 );.
But this isn't an efficient thing to do, because then you'll be looping through the file multiple times - you should instead read it into a data structure and use that.
Edit: Here is your problem:
[{"id":1001,"interfaces":[]},{"id":1003,"interfaces":[{"ip_addresses":[{"value":"192.168.0.2"}]}]},{"id":1004,"interfaces":[{"ip_addresses":[{"value":"192.168.0.3"}]}]},{"id":1005,"interfaces":[{"ip_addresses":[{"value":"192.168.0.4"}]}]}]
And specifically:
[{"id":1001,"interfaces":[]}
Your first element in that array doesn't have a $asset_ip defined.
This means - on your first pass - $asset_ip is undefined and generates the errors. (no line is printed because of your if defined test).
But then - the code proceeds to traverse $csv_fh - reading to the end of file - looking for matches (and fails 3 times, generating 3 error messages.
Second iteration - for id 1002 - the IP isn't in the file anyway, but $csv_fh has already been read to end-of-file (EOF) - so that foreach loop doesn't execute at all.
This can be made workable by:
adding else next; after that if defined.
adding seek to after the while loop.
But really - a rewrite would be in order so you're not re-reading a file over and over anyway.
Very crudely:
#!/usr/bin/perl
# perl -e'use CPAN; install "Text::CSV"'
use strict;
use warnings;
use JSON::XS;
use File::Slurp;
use Text::CSV;
my $csv = Text::CSV->new( { sep_char => ',' } );
my $csv_source = "servers.csv";
my $json_source = "assets.json";
my $dest = "servers_for_upload.csv";
# defined these here as I need to use them in foreach loop and if statement:
my $csv_ip;
my #fields;
open( my $csv_fh, '<', $csv_source ) or die "$! error trying to read";
open( my $dest_fh, '>', $dest ) or die "$! error trying to read";
my $json = read_file($json_source);
my $json_array = decode_json $json;
foreach my $item (#$json_array) {
my $id = $item->{id};
my $asset_ip = $item->{interfaces}->[0]->{ip_addresses}->[0]->{value};
# test the data is there:
if ( defined $asset_ip ) {
print "id: " . $id . "\nip: " . $asset_ip . "\n";
}
else {
print "asset_ip undefined for id $id\n";
next;
}
while ( my $line = <$csv_fh> ) {
chomp $line;
if ( $csv->parse($line) ) {
#fields = $csv->fields();
$csv_ip = $fields[0];
}
else {
warn "Line could not be parsed: $line\n";
}
if ( $csv_ip eq $asset_ip ) {
# preppend id to csv array and write these lines to new file
unshift( #fields, $id );
print {$dest_fh} join( ", ", #fields ),"\n";
}
}
seek( $csv_fh, 0, 0 );
}
close $csv_fh;
I would suggest this also needs:
change of while so you're not re-reading the file each time
You're using Text::CSV so using a print join ( ","... doesn't seem a consistent choice. If your data warrants Text::CSV it's worth keeping it for output too.
how to decode the json file if a value has more than one line
a.json file:
{
"sv1" : {
"output" : "Hostname: abcd
asdkfasfjsl",
"exp_result" : "xyz"
}
}
when I try to read the above json file, I am hitting with an error "invalid character encountered while parsing JSON string, at character offset 50 (before "\n ...")"
code to read the above json file:
#!/volume/perl/bin/perl -w
use strict;
use warnings;
use JSON;
local $/;
open(AA,"<a.json") or die "can't open json file : $!\n";
my $json = <AA>;
my $data = decode_json($json);
print "reading output $data->{'sv1'}->{'output'}\n";
print "reading output $data->{'sv1'}->{'exp_result'}\n";
close AA;
Besides from whether the JSON is valid or not (see comments on question), you're reading only the first line from the file.
my $json = <AA>;
This is a scalar variable and receives only one line.
Use an array to get all lines:
my #json = <AA>;
my $json = join "\n", #json;
or even better: use File::Slurp::read_file to get the whole content of the file with one simple command.
use File::Slurp qw/read_file/;
my $json = read_file( "a.json" );
When trying to run my perl script via cmd prompt my json string is returning [] I have read other posts, and fixed my database to be utf8 and the error still persists. I have tried two didfferent way to encode my perl string the first was $json = encode_json #temp_array which returns this error hash- or arrayref expected (not a simple scalar, use allow_nonref to allow this However, when I use this line $json_text = $json->encode(#temp_array) I just get []
Here is my perl:
my $json_text;
my $json = JSON->new->utf8;
my #temp_array =[];
my $temp_array;
while (#data = $query_handle->fetchrow_array())
{
my %json_hash = ();
my $hash_ref;
%json_hash = (
"User ID" => $data[0],
"Status" => $data[1],
"Last Password Reset" => $data[2],
"Reset Needed" => $data[3]
);
$hash_ref = \%json_hash;
push (#temp_array, $hash_ref);
}
print $json = encode_json #temp_array . "\n"; #encode with error
print $json_text = $json->encode(#temp_array) . "\n"; #encode with []
print $cgi->header(-type => "application/json", -charset => "utf-8");
print $json_text; #Prints []
So in my own testing, via the cmd prompt I know the while is retrieving the data from my db correctly and is building a hash, which I am assuming is correct.
Is it the fact I am pushing my hash reference to the array instead of the hash itself? Once I get this string built correctly, I will be calling it to an html via jquery
Thank you.
JSON expects references:
print $json = encode_json(\#temp_array) . "\n";
print $json_text = $json->encode(\#temp_array) . "\n";
Edit: Unless you enable allow_nonref.
Another edit: This line is wrong--
my #temp_array =[]; ## should be my #temp_array = ();
and this line overwrites the $json variable:
print $json = encode_json #temp_array . "\n"; ## the next line in your script shouldn't work
Last edit - untested:
my $json = JSON->new->utf8;
my #temp_array;
while (my #data = $query_handle->fetchrow_array()) {
my %json_hash = (
"User ID" => $data[0],
"Status" => $data[1],
"Last Password Reset" => $data[2],
"Reset Needed" => $data[3]
);
push (#temp_array, \%json_hash);
}
print $json->encode(\#temp_array) . "\n";