perl json encode converting numbers to strings [duplicate] - json

This question already has answers here:
Converting array of numbers to json in perl, getting array of strings
(3 answers)
Closed 8 years ago.
I am trying to encode a perl nested hash and send it to some web application.
Somehow the json encoder converts the numbers or floats to strings.
The web application sees the data as strings and can't plot the chart. I can add code in the web application to convert them back to numbers, but I am looking for better solution of not having the numbers as strings in the first place.
Here is the code:
use strict;
use warnings;
use CGI qw/param/;
use JSON::XS;
my $json_obj = JSON::XS->new->allow_nonref;
## Build some Perl data
my %perl_data;
$perl_data{'numbers'}{'nested'} = [qw/1 -2 4 2 5 6/] ;
$perl_data{'mix'}{'AnotherLevel'} = [qw/null "Temp" 4 2 5 6/] ;
print "Content-type: text/html\n\n";
print $json_obj->pretty->encode(\%perl_data);
Here is the output where everything is just stringified:
Content-type: text/html
{
"numbers" : {
"nested" : [
"1",
"-2",
"4",
"2",
"5",
"6"
]
},
"mix" : {
"AnotherLevel" : [
"null",
"\"Temp\"",
"4",
"2",
"5",
"6"
]
}
}
In the above code, I even tried the following, but to no avail.
use JSON;
my $json_obj = JSON;
Any help is greatly appreciated.

The JSON::XS documentation actually has a good section which describes how Perl data structures are serialized as JSON. In particular, it says for scalars:
JSON::XS will encode undefined scalars as JSON null values, scalars that have last
been used in a string context before encoding as JSON strings, and anything else
as number values.
When you define your array of numbers using qw, you are using them in a string context. (qw means "quote word" and is generally used to save some typing when defining a list of words.)
Note also that null in JSON is represented by the undef value in Perl. When you say qw/null/, you're just creating the literal string 'null'.
So you have two options.
Define your array like this:
$perl_data{'numbers'}{'nested'} = [1, -2, 4, 2, 5, 6] ;
$perl_data{'mix'}{'AnotherLevel'} = [undef, "Temp", 4, 2, 5, 6] ;
Or, force-numify all your numbers by adding zero to them before you serialize. E.g.
$perl_data{'numbers'}{'nested'} = [ map { $_ + 0 } qw/1 -2 4 2 5 6/ ];

Don't initialize them as strings, and you won't have a problem:
use strict;
use warnings;
use CGI qw/param/;
use JSON::XS;
my $json_obj = JSON::XS->new->allow_nonref;
## Build some Perl data
my %perl_data = (
'numbers' => {'nested' => [1, -2, 4, 2, 5, 6]},
'mix' => {'AnotherLevel' => [qw/null "Temp"/, 4, 2, 5, 6]},
);
print "Content-type: text/html\n\n";
print $json_obj->pretty->encode(\%perl_data);
Outputs:
Content-type: text/html
{
"numbers" : {
"nested" : [
1,
-2,
4,
2,
5,
6
]
},
"mix" : {
"AnotherLevel" : [
"null",
"\"Temp\"",
4,
2,
5,
6
]
}
}

Related

How to extract certain data using Perl from a file?

I have data that needs to be extracted from a file, the lines I need for the moment are name,location and host. This is example of the extract. How would I go about getting these lines into a separate file? I have the Original file and the new file i want to create as the input/output file, there are thousands of devices contained within the output file and they are all the same formatting as in my example.
#!/usr/bin/perl
use strict;
use warnings;
use POSIX qw(strftime);
#names of files to be input output
my $inputfile = "/home/nmis/nmis_export.csv";
my $outputfile = "/home/nmis/nmis_data.csv";
open(INPUT,'<',$inputfile) or die $!;
open(OUTPUT, '>',$outputfile) or die $!;
my #data = <INPUT>;
close INPUT;
my $line="";
foreach $line (#data)
{
======Sample Extract=======
**"group" : "NMIS8",
"host" : "1.2.3.4",
"location" : "WATERLOO",
"max_msg_size" : 1472,
"max_repetitions" : 0,
"model" : "automatic",
"netType" : "lan",
"ping" : 1,
"polling_policy" : "default",
"port" : 161,
"rancid" : 0,
"roleType" : "access",
"serviceStatus" : "Production",
"services" : null,
"threshold" : 1,
"timezone" : 0,
"version" : "snmpv2c",
"webserver" : 0
},
"lastupdate" : 1616690858,
"name" : "test",
"overrides" : {}
},
{
"activated" : {
"NMIS" : 1
},
"addresses" : [],
"aliases" : [],
"configuration" : {
"Stratum" : 3,
"active" : 1,
"businessService" : "",
"calls" : 0,
"cbqos" : "none",
"collect" : 0,
"community" : "public",
"depend" : [
"N/A"
],
"group" : "NMIS8",
"host" : "1.2.3.5",
"location" : "WATERLOO",
"max_msg_size" : 1472,
"max_repetitions" : 0,
"model" : "automatic",
"netType" : "lan",
"ping" : 1,
"polling_policy" : "default",
"port" : 161,
"rancid" : 0,
"roleType" : "access",
"serviceStatus" : "Production",
"services" : null,
"threshold" : 1,
"timezone" : 0,
"version" : "snmpv2c",
"webserver" : 0
},
"lastupdate" : 1616690858,
"name" : "test2",
"overrides" : {}
},**
I would use jq for this not Perl. You just need to query a JSON document. That's what jq is for. You can see an example here
The jq query I created is this one,
.[] | {name: .name, group: .configuration.group, location: .configuration.location}
This breaks down into
.[] # iterate over the array
| # create a filter to send it to
{ # that produces an object with the bellow key/values
.name,
group: .configuration.group,
location: .configuration.location
}
It provides an output like this,
{
"name": "test2",
"group": "NMIS8",
"location": "WATERLOO"
}
{
"name": "test2",
"group": "NMIS8",
"location": "WATERLOO"
}
You can use this to generate a csv
jq -R '.[] | [.name, .configuration.group, .configuration.location] | #csv' ./file.json
Or this to generate a csv with a header,
jq -R '["name","group","location"], (.[] | [.name, .configuration.group, .configuration.location]) | #csv' ./file.json
You can use the JSON distribution for this. Read the entire file in one fell swoop to put the entire JSON string into a scalar (as opposed to putting it into an array and iterating over it), then simply decode the string into a Perl data structure:
use warnings;
use strict;
use JSON;
my $file = 'file.json';
my $json_string;
{
local $/; # Locally reset line endings to nothing
open my $fh, '<', $file or die "Can't open file $file!: $!";
$json_string = <$fh>; # Slurp in the entire file
}
my $perl_data_structure = decode_json $json_string;
As what you have there is JSON, you should parse it with a JSON parser. JSON::PP is part of the standard Perl distribution. If you want something faster, you could install something else from CPAN.
Update: I included a link to JSON::PP in my answer. Did you follow that link? If you did, you would have seen the documentation for the module. That has more information about how to use the module than I could include in an answer on SO.
But it's possible that you need a little more high-level information. The documentation says this:
JSON::PP is a pure perl JSON decoder/encoder
But perhaps you don't know what that means. So here's a primer.
JSON is a text format for storing complex data structures. The format was initially used in Javascript (the acronym stands for "JavaScript Object Notation") but it is now a standard that is used across pretty much all programming languages.
You rarely want to actually deal with JSON in a program. A JSON document is just text and manipulating that would require some complex regular expressions. When dealing with JSON, the usual approach is to "decode" the JSON into a data structure inside your program. You can then manipulate the data structure however you want before (optionally) "encoding" the data structure back into JSON so you can write it to an output file (in your case, you don't need to do that as you want your output as CSV).
So there are pretty much only two things that a Perl JSON library needs to do:
Take some JSON text and decode it into a Perl data structure
Take a Perl data structure and encode it into JSON text
If you look at the JSON::PP documentation you'll see that it contains two functions, encode_json() and decode_json() which do what I describe above. There's also an OO interface, but let's not overcomplicate things too quickly.
So your program now needs to have the following steps:
Read the JSON from the input file
Decode the JSON into a Perl data structure
Walk the Perl data structure to extract the items that you need
Write the required items into your output file (for which Text::CSV will be useful
Having said all that, it really does seem to me that the jq solution suggested by user157251 is a much better idea.

Perl json keys with spaces [duplicate]

This question already has answers here:
Which Perl module would you recommend for JSON manipulation?
(6 answers)
Closed 2 years ago.
how can i parse perl json object which has spaces in its keys
{
"abc" : [
"lmn" : {
"Ab Cd" : "Xy Zw",
"Ef Gh" : "Pq Rs",
}
]
}
By definition, one parses JSON using a JSON parser. There exists multiple JSON parsers on CPAN, including Cpanel::JSON::XS. It handles keys with spaces in them without issue, as should every other JSON parser.
Note that what you have isn't JSON. I'm assuming the errors are typos since you asked about JSON.
Spaces in a key will present no problems at all to any JSON parser.
There are, however, two problems in your JSON that will cause problems for any parser. Others have noted the extra comma after "Pq Rs", but you also have an array that contains a key/value pair (with the key "lnm") which needs to be inside an object.
Originally, I just removed the comma and ran this code:
#!/usr/bin/perl
use strict;
use warnings;
use feature 'say';
use Data::Dumper;
use JSON;
my $json = '{
"abc" : [
"lmn" : {
"Ab Cd" : "Xy Zw",
"Ef Gh" : "Pq Rs"
}
]
}';
my $data = decode_json($json);
say Dumper $data;
This gives an error:
, or ] expected while parsing array, at character offset 28 (before ": {\n "Ab C...")
I fixed it, by inserting { ... } around the lnm object.
#!/usr/bin/perl
use strict;
use warnings;
use feature 'say';
use Data::Dumper;
use JSON;
my $json = '{
"abc" : [ {
"lmn" : {
"Ab Cd" : "Xy Zw",
"Ef Gh" : "Pq Rs"
}
} ]
}';
my $data = decode_json($json);
say Dumper $data;
And then I got this output:
$VAR1 = {
'abc' => [
{
'lmn' => {
'Ab Cd' => 'Xy Zw',
'Ef Gh' => 'Pq Rs'
}
}
]
};
Which is, I think, what you are expecting.

Return empty string or 0 (zero) in case of missing key with jq

I am having problems with my json to csv conversion.
I have json file of this structure:
{
"key": [
{
"key1": 1,
"key2": 1,
"key3": {
"1": 1,
"2": 2,
"3": 3,
"4": 4
}
},
{
"key1": 2,
"key2": 2,
"key3": {
"2": 2
}
}
...
],
...
}
I was using this jq call to convert my json to csv:
bin\jq-win64 ".key[] | [.key1, .key2, .key3.\"1\", .key3.\"2\", .key3.\"3\", .key3.\"4\" ] | tostring] | join(\";\")" source.json > output.tmp
I can't use standard #csv, because it's not good for my locale settings. But back to the problem. In some cases key3 might not be full 4 element object (4 is the max with keys 1/2/3/4, just like in example). I have a problem with those missing subkeys, because jq returns "null" and that does not work well with CSV evaluation in excel or calc. Is there a way to force empty string or numeric 0 as output in such case?
In the end, I can try using some other command line text processor, but I'd be glad if I could do that with single tool.
EDIT:
I had the wrong json structure example and now the actual problem has changed a bit. Message is updated.
You could tweak your pipeline by adding
map(. // 0)
right after forming the array. If you want to preserve false, then you would have to add
map(if . == null then 0 else . end)
instead.
If you wanted a solution that was agnostic about the key names, you could use something along the lines of:
def resize($n): [range(0;$n) as $i | .[$i] // 0];
This would truncate or expand the input array. If you don’t ever want to truncate, then tweak accordingly.

jq split string by a pattern

I have a json object with one of field having values for example "countries-sapi-1.0", "inventory-list-api-1.0-snapshot"
Note that the first one has sapi and the other one has api.
Using jq, how can i get countries-sapi or inventory-list-api I mean whatever is there before the version. the version can be as simple as 1.0 or 1.0.1-snapshot etc..
I got here searching for how to split by regex instead of substring in jq, but I found out that you have to give two arguments to the split function (where the second argument contains flags for the regex, but it can be just an empty string).
$ jq -n '"axxbxxxc"|split("x+";"")'
[
"a",
"b",
"c"
]
From the manual:
split
Splits an input string on the separator argument.
jq 'split(", ")'
"a, b,c,d, e, "
=> ["a","b,c,d","e",""]
[...]
split(regex; flags)
For backwards compatibility, split splits on a string, not a regex.
It looks like you need to study up on regular expressions (regex); see for example https://regexone.com/ or https://github.com/zeeshanu/learn-regex or dozens of others.
Using jq, in your particular case, you could start with:
sub(" *- *[0-9]+\\.[0-9]+.*$"; "")
Note that two backslashes are required here because the "from" expression must be (or evaluate to) a valid JSON string.
For input "countries-sapi-1.0", use: .[] | match( "\\d"; "ig") which will give you the following output:
{
"offset": 15,
"length": 1,
"string": "1",
"captures": []
}
{
"offset": 17,
"length": 1,
"string": "0",
"captures": []
}
This uses the first object's offset value and tries to slice it from the starting position to the received offset.
Slice from beginning: $ jq -c '.[:15]'
In our case we got 15 as the offset for the first object so we used :15 for the slice.

keeping track of type with JSON.pm

say i have the following inputted JSON object
{
"field1": 21,
"field2": "21",
"field3": "hello"
}
is there any way with decode_json or from_json to know what the original type was (number verses string) of the values? I know Perl generally doesnt care about type, but I have a need to know what the original type was. I also know that perl does keep track of the type when creating a JSON object (so it does distinguish between "21" and 21 when creating a JSON object, so Im hoping there is a way to keep that information when decoding/'from'ming it.
I don't want to base it on the field name, because im trying to write something that will be used somewhat generically, and fieldnames could change.
When using JSON::XS, the type of the value in the scalar matches the type of the value in the document.
$ perl -e'
use strict;
use warnings;
use B qw( svref_2object SVf_IOK SVf_NOK SVf_POK );
use JSON::XS qw( decode_json );
my $data = decode_json(q{[ "4", 4, 4.0, 20000000000000000000 ]});
for my $i (0..$#$data) {
my $sv = svref_2object(\( $data->[$i] ));
my $flags = $sv->FLAGS;
printf("Scalar %s has %s\n",
$i,
join(",",
$flags & SVf_POK ? "PV" : (),
$flags & SVf_IOK ? "IV" : (),
$flags & SVf_NOK ? "NV" : (),
),
);
}
'
Scalar 0 has PV
Scalar 1 has IV
Scalar 2 has NV
Scalar 3 has PV
As you can see, the fourth scalar is an exception when using JSON::XS. JSON::XS stores very large numbers as strings to avoid loosing precision.
You get similar results with JSON::PP:
Scalar 0 has PV
Scalar 1 has IV
Scalar 2 has NV
Scalar 3 has NV