Perl Out Of Memory when parsing json with regexp inside - json

I am parsing a very simple json file and perl is running out of memory. I am using the JSON lib from cpan. I am reading the json from a file named "data.json". I am also using the File::Slurper module.
use JSON;
use File::Slurper;
my $file = read_text("data.json");
my #data = decode_json($file);
print #data;
In my data.json file I have an array of objects with regexp inside each obj.
[
{
"val": "test",
"reg": "m/^(match)/"
}
...3 more entries
]
WHen i run this it says
Out of memory!
how can i bypass this?

It is not clear what causes the error in your code. See the following sample code which does same job in a little different way.
use strict;
use warnings;
use feature 'say';
use JSON;
use Data::Dumper;
my $file = do { local $/; <DATA> };
my #data = decode_json($file);
say Dumper(\#data);
__DATA__
[
{
"val": "test1",
"reg": "m/^(match)/"
},
{
"val": "test2",
"reg": "m/^(match)/"
},
{
"val": "test3",
"reg": "m/^(match)/"
},
{
"val": "test4",
"reg": "m/^(match)/"
}
]
Output
$VAR1 = [
[
{
'val' => 'test1',
'reg' => 'm/^(match)/'
},
{
'val' => 'test2',
'reg' => 'm/^(match)/'
},
{
'val' => 'test3',
'reg' => 'm/^(match)/'
},
{
'reg' => 'm/^(match)/',
'val' => 'test4'
}
]
];

Related

How to loop through a JSON object in Perl?

I'm trying to create an api using perl, the api is for a react native, when the user submits a form on the app I'll get the following object,
I'm new to perl and I'm lost trying to loop thro the object :/
{
checkboxes: [
{
id: "1",
fullname: "Name 1",
color: "red",
res: false
},
{
color: "green",
fullname: "Name 2",
id: "2",
res: false
},
{
color: "blue",
id: "3",
fullname: "Name 3",
res: false
}
]
}
my $data_decoded = decode_json($data);
I'm trying this loop, but it only prints the full object.
foreach $a (#data) {
print "value of a: $a\n";
}
You turned your JSON into a reference Perl data structure with decode_json (from somewhere, and Mojo::JSON is such a place):
use Mojo::JSON qw(decode_json);
my $data = ...;
my $data_decoded = decode_json($data);
Now you have to figure out how to access whatever you have in $data_decoded. You can look at its structure by dumping it
use Mojo::Util qw(dumper);
print dumper( $data_decoded );
You'll see that the Perl structure is the same as the JSON structure. You have a hash (JSON's Object) that has a checkboxes key that points to an array. The array elements are hashes.
Using Perl v5.24's postfix dereferencing notation, you get all of the array elements:
# uglier circumfix notation #{$data_decoded->{checkboxes}}
my #elements = $data_decoded->{checkboxes}->#*;
You might put that in a loop:
foreach my $hash ( $data_decoded->{checkboxes}->#* ) {
...
}
Now you get each hash element in $hash and you can do whatever you like with it. That part you haven't told us about yet. :)
The Perl Data Structures Cookbook (perldsc) has a lot of examples of the generation and iteration of various combinations of hash and array references.
You say that you want to do something when the value of the res key is true. In that case, you can use next to skip the items where res is false:
foreach my $hash ( $data_decoded->{checkboxes}->#* ) {
next unless $hash->{res};
say "I'm doing something when res is true";
}
Following demo code demonstrates looping through these particular data
use strict;
use warnings;
use feature 'say';
use JSON;
use Data::Dumper;
my $input = do { local $/; <DATA> };
my $data = from_json($input);
say Dumper($data);
for my $obj ( #{$data->{checkboxes}} ) {
say join(",\t", $obj->#{qw/id fullname color/});
}
__DATA__
{
"checkboxes": [
{
"id": "1",
"fullname": "Name 1",
"color": "red",
"res": false
},
{
"color": "green",
"fullname": "Name 2",
"id": "2",
"res": false
},
{
"color": "blue",
"id": "3",
"fullname": "Name 3",
"res": false
}
]
}
Output
$VAR1 = {
'checkboxes' => [
{
'res' => bless( do{\(my $o = 0)}, 'JSON::PP::Boolean' ),
'color' => 'red',
'fullname' => 'Name 1',
'id' => '1'
},
{
'fullname' => 'Name 2',
'color' => 'green',
'res' => $VAR1->{'checkboxes'}[0]{'res'},
'id' => '2'
},
{
'id' => '3',
'color' => 'blue',
'res' => $VAR1->{'checkboxes'}[0]{'res'},
'fullname' => 'Name 3'
}
]
};
1, Name 1, red
2, Name 2, green
3, Name 3, blue

API POST request in Julia

I am trying to convert some Python code to Julia. Here is the Python code:
url = "http://api.scb.se/OV0104/v1/doris/sv/ssd/START/BE/BE0101/BE0101G/BefUtvKon1749"
json = {
"query": [
{
"code": "Kon",
"selection": {
"filter": "item",
"values": [
"1",
"2"
]
}
},
{
"code": "ContentsCode",
"selection": {
"filter": "item",
"values": [
"000000LV"
]
}
}
],
"response": {
"format": "px"
}
}
r = requests.post(url=url, json=json)
Below is the Julia code, that is not working, with this error message:
syntax: { } vector syntax is discontinued around path:8
top-level scope at population_data.jl:8
using DataFrames, DataFramesMeta, HTTP, JSON3
url = "http://api.scb.se/OV0104/v1/doris/sv/ssd/START/BE/BE0101/BE0101G/BefUtvKon1749"
json = {
"query": [
{
"code": "Kon",
"selection": {
"filter": "item",
"values": [
"1",
"2",
"1+2"
]
}
},
{
"code": "ContentsCode",
"selection": {
"filter": "item",
"values": [
"000000LV"
]
}
}
],
"response": {
"format": "px"
}
}
r = HTTP.post(url, json)
My attempts to solve this are the following:
Convert the json variable to a string using """ around it.
Converting the JSON string to Julia data types, using JSON3.read()
Passing the converted JSON string to the POST request. This gives the following error:
IOError(Base.IOError("read: connection reset by peer (ECONNRESET)", -54) during request(http://api.scb.se/OV0104/v1/doris/sv/ssd/START/BE/BE0101/BE0101G/BefUtvKon1749)
None of it works, and I am not even sure that it is about the JSON format. It could be that I am passing the wrong parameters to the POST request. What should I do?
One way of solving this consists in building the parameters as native julia data structures, and use JSON to convert and use them as the body of your PUT request:
Dictionaries in julia are built using a syntax like Dict(key => value). Arrays are built using a standard syntax: [a, b, c]. The julia native data structure equivalent to your parameters would look like this:
params = Dict(
"query" => [
Dict("code" => "Kon",
"selection" => Dict(
"filter" => "item",
"values" => [
"1",
"2",
"1+2"
]),
),
Dict("code"=> "ContentsCode",
"selection" => Dict(
"filter" => "item",
"values" => [
"000000LV"
]),
),
],
"response" => Dict(
"format" => "px"
))
Then, you can use JSON.json() to build the JSON representation of it as a string and pass it to the HTTP request:
using HTTP
using JSON
url = "http://api.scb.se/OV0104/v1/doris/sv/ssd/START/BE/BE0101/BE0101G/BefUtvKon1749"
# send the request
r = HTTP.request("POST", url,
["Content-Type" => "application/json"],
JSON.json(params))
# retrieve the response body as a string
b = String(r.body)

How to add keys to values and convert to json in PowerShell?

I have a string like this:
$string = "PackageName1,1,C:\Path
PackageName2,12,C:\Path2
PackageName3,3,C:\Path3"
(is a file with multilines, when I get the content I have the string above)
I want to convert this string to Json:
[
{
Pacakge: "PackageName1",
Branch: = "1",
LocalPath = "C:\Path"
}
{
Pacakge: "PackageName2",
Branch: = "2",
LocalPath = "C:\Path2"
}
]
I can get the values with this code:
$spiltted = $string.Split(' ')
ForEach ($s in $splitted)
{
$values = $s.Split(',')
}
How can I add the Json keys to each value and convert it to Json object?
Thank you.
As your String looks like a csv without headers:
$string | ConvertFrom-Csv -Header Package,Branch,LocalPath|ConvertTo-Json
Sample output
[
{
"Package": "PackageName1",
"Branch": "1",
"LocalPath": "C:\\Path"
},
{
"Package": "PackageName2",
"Branch": "12",
"LocalPath": "C:\\Path2"
},
{
"Package": "PackageName3",
"Branch": "3",
"LocalPath": "C:\\Path3"
}
]

Hash::Ordered versus Tie::IxHash with JSON::XS encode

I'm trying Hash::Ordered instead of Tie::IxHash, because it seems to be faster.
While Tie::IxHash is working fine, I struggle with some problems with Hash::Ordered. The point is to have the hashes ordered (which are usually random in Perl).
use Hash::Ordered;
use JSON::XS;
use Data::Dumper;
use strict;
use warnings;
my $json = JSON::XS->new;
my $oh = Hash::Ordered->new;
$oh->push('result' => { 'counter' => "123" }, 'number' => { 'num' => '55' });
my #r = $oh->as_list;
$json->pretty(1);
my $jsondata = $json->encode(\#r);
print Dumper $jsondata;
The result is odd:
[
"result",
{
"counter" : "123"
},
"number",
{
"num" : "55"
}
]
Here is the working example with Tie::IxHash, I try to get the same results with Hash::Ordered.
use Data::Dumper;
use Tie::IxHash;
use JSON::XS;
use strict;
use warnings;
my $json = JSON::XS->new;
my %h;
tie(%h, 'Tie::IxHash', result => { counter => "123" }, number => { num => '55' });
$json->pretty(1);
my $pretty_json = $json->encode(\%h);
print Dumper $pretty_json;
Output
{
"result" : {
"counter" : "123"
},
"number" : {
"num" : "55"
}
}
The object-oriented interface of Hash::Ordered is much faster that the tied interface, but some utilities (like $json->encode) require a real hash reference
The way to get the best of both worlds is to tie a hash for use with those utilities, and use tied to extract the underlying Hash::Ordered object so that you can use the faster method calls to manipulate it
This short program demonstrates. The only slow part of this code is when the hash is passed to encode to be translated to JSON. The push call doesn't use the tied interface and remains fast
use strict;
use warnings;
use Hash::Ordered;
use JSON::XS;
my $json = JSON::XS->new->pretty;
tie my %h, 'Hash::Ordered';
my $oh = tied %h;
$oh->push( result => { counter => 123 }, number => { num => 55 } );
print $json->encode(\%h), "\n";
output
{
"result" : {
"counter" : 123
},
"number" : {
"num" : 55
}
}
Use the Hash::Ordered tied interface:
my $json = JSON::XS->new;
tie my %hash, "Hash::Ordered";
$hash{'result'} = { 'counter' => "123" };
$hash{'number1'} = { 'num' => '1' };
$hash{'number2'} = { 'num' => '2' };
$hash{'number3'} = { 'num' => '3' };
$hash{'last'} = { 'num' => 'last' };
$json->pretty(1);
my $jsondata = $json->encode(\%hash);
And the JSON data you get is:
{
"result" : {
"counter" : "123"
},
"number1" : {
"num" : "1"
},
"number2" : {
"num" : "2"
},
"number3" : {
"num" : "3"
},
"last" : {
"num" : "last"
}
}
The examples above work fine, but for multidimensional hashes there is an additional step needed to keep the order.
use Hash::Ordered;
use JSON::XS;
use Data::Dumper;
use strict;
use warnings;
sub ordered_hash_ref {
tie my %hash, 'Hash::Ordered';
my $oh = tied %hash;
$oh->push(#_);
return \%hash;
};
my $json = JSON::XS->new->pretty;
tie my %h, 'Hash::Ordered';
my $oh = tied %h;
$oh->push(result => ordered_hash_ref(counter => 123, z => 5, s => 8), address => ordered_hash_ref(Vorname => 'Max',
Nachname => 'Mustermann', Strasse => 'Feldweg', Hausnummer => 45));
my $pretty = $json->encode(\%h);
print Dumper $pretty;
Output
{
"result" : {
"counter" : 123,
"z" : 5,
"s" : 8
},
"address" : {
"Vorname" : "Max",
"Nachname" : "Mustermann",
"Strasse" : "Feldweg",
"Hausnummer" : 45
}
}

How to create key value JSON response in perl script with MYSQL data

I have a perl script which does some query and return response in JSON format. But the return JSON is only array of values.
This is Perl Script
my $sql_query = "SELECT * from table_categories";
my $statement = $db_handle->prepare ($sql_query) or die "Couldn't prepare query '$sql_query': $DBI::errstr\n";
$statement->execute() or die "SQL Error: $DBI::errstr\n";
my #loop_data = ();
while (my #data = $statement->fetchrow_array())
{
push(#loop_data, #data);
}
my $json;
$json->{"entries"} = \#loop_data;
my $json_text = to_json($json);
print $json_text;
and response is like this
{
"entries": [
[
"1",
"Salt and Sugar",
"/images/salt_sugar.png",
"7"
],
[
"2",
"Tea and Coffee",
"/images/tea_and_coffee.png",
"6"
],
[
"3",
"Spice and Pickles",
"/images/categories/spice_pickles.png",
"7"
],
[
"4",
"Pooja Needs",
"/images/categories/pooja-needs.png",
"9"
],
[
"5",
"Dry Fruits",
"/images/categories/dry_fruits.png",
"7"
]
]
}
But I want the response like this:
{
"entries": [
[
"id": "1",
"name": "Salt and Sugar",
"image": "/images/salt_sugar.png",
"rank": "7"
],
[
"id": "2",
"name": "Tea and Coffee",
"image": "/images/tea_and_coffee.png",
"rank": "6"
]
]
}
How to achieve that?
What changes i need to do in PERL script?
Please help.
If you specifically don't have need to loop over every row, I'd prefer to use selectall_arrayref with slicing:
my $json;
my $sql_query = "SELECT * FROM table_categories";
$json->{entries} = $db_handle->selectall_arrayref( $sql_query, {Slice => {} } );
my $json_text = to_json($json);
print $json_text;
So you get keys too and code is more compact and clear.
You can use fetchrow_hashref instead of fetchrow_array to receive a hashref where the column names are used for the property names using the code below.
my #loop_data = ();
while (my $hashref = $statement->fetchrow_hashref())
{
push(#loop_data,$hashref);
}
More information: https://metacpan.org/pod/DBI#fetchrow_hashref