Perl code for multiple JSON messages parsing - json

I am trying to write a perl code to parse multiple JSON messages. The perl code that I have written only parses the values if the JSON file contains only one json message. But it fails when there are multiple messages in that file. It throws error: "Undefined subroutine &Carp::shortmess_heavy". The JSON file is in the following format:
{
"/test/test1/test2/test3/supertest4" : [],
"/test/test1/test2/test3/supertest2" : [
{
"tag1" : "",
"tag2" : true,
"tag3" : [
{
"status" : "TRUE",
"name" : "DEF",
"age" : "28",
"sex" : "f"
},
{
"status" : "FALSE",
"name" : "PQR",
"age" : "39",
"sex" : "f"
}
],
"tag4" : "FAILED",
"tag5" : "/test/test1/test2/test3/supertest2/test02",
"tag6" : ""
}
],
"/test/test1/test2/test3/supertest1" : [
{
"tag1" : "",
"tag2" : false,
"tag3" : [
{
"status" : "TRUE",
"name" : "ABC",
"age" : "21",
"sex" : "m"
},
{
"status" : "FALSE",
"name" : "XYZ",
"age" : "34",
"sex" : "f"
}
],
"tag4" : "PASSED",
"tag5" : "/test/test1/test2/test3/supertest1/test01",
"tag6" : ""
}
],
"/test/test1/test2/test3/supertest6" : []
}
My perl code to parse a single JSON message:
use strict;
use warnings;
use Data::Dumper;
use JSON;
use JSON qw( decode_json );
my $json_file = "tmp1.json";
my $json;
open (my $fh, '<', $json_file) or die "can not open file $json_file";
{ local $/; $json = <$fh>; }
close($fh);
my $decoded = decode_json($json);
print "TAG4 = " . $decoded->{'tag4'} . "\n";
print "TAg5 = " . $decoded->{'tag5'} . "\n";
my #tag3 = #{ $decoded->{'tag3'} };
foreach my $tg3 ( #tag3 ) {
print "Name = ". $tg3->{"name"} . "\n";
print "Status = ". $tg3->{"status"} . "\n";
print "Age = ". $tg3->{"age"} . "\n";
}

To parse multiple JSON objects, use JSON::XS/Cpanel::JSON::XS's incremental parsing.
my $json = '{"foo":"bar"} ["baz"] true 1.23';
my #results = JSON::XS->new->allow_nonref->incr_parse($json);
use Data::Dumper;
print Dumper \#results;
Output:
$VAR1 = [
{
'foo' => 'bar'
},
[
'baz'
],
bless( do{\(my $o = 1)}, 'JSON::PP::Boolean' ),
'1.23'
];

Related

Trying to convert complex JSON to CSV with Powershell

I'm fairly new to PowerShell and I am trying to convert this JSON file to CSV.
Using some simple code like this:
$sourceFilePath = "Records.json"
$destinationFilePath = "Records.csv"
((Get-Content -Path $sourceFilePath -Raw) | ConvertFrom-Json) | Export-CSV $destinationFilePath -NoTypeInformation
Results in:
Because I am getting System.Object[] I think I need to do some kind of expansion or "ForEach" of the contact_ids, site_ids, first name aliases, and aliases. I've done some searching and I can't seem to wrap my head around the examples I've been given. Also, "first name aliases" has spaces in it that add to the complexity.
I don't have any control over the source JSON. Here it is as an example:
[
{
"mdm id" : "947b2a12-3aac-480a-a0bf-626aff106f48",
"first name" : "Claude",
"last name" : "Bones",
"contact_ids" : [
"CTP70499"
],
"site_ids" : [
"5015"
],
"first name aliases" : [
"Claude"
],
"aliases" : [
"Claude Bones"
],
"createdDate" : "2020-06-03T19:59:08Z",
"updatedDate" : "2020-06-03T20:48:27Z",
"veevaID" : "V0B000000000569"
},
{
"mdm id" : "bce21b05-0b28-4ebb-a34d-761c1a397821",
"first name" : "Daniel",
"last name" : "Smith",
"contact_ids" : [
"CTP699"
],
"site_ids" : [
"5015"
],
"first name aliases" : [
"Dan",
"Danial",
"Danne",
"Danny",
"Daniel"
],
"aliases" : [
"Daniel Smith"
],
"createdDate" : "2020-06-03T19:59:08Z",
"updatedDate" : "2020-06-03T20:48:27Z",
"veevaID" : "V0B000000000566"
}
]
The values that are arrays will need to be combined into a single value. Regardless of how many values it may have, if it's designated as an array [] in json, it will need to be manipulated. There are several articles and custom functions written on the internet. Your example could be handled with this bit of code.
$JSONdata = #'
[
{
"mdm id" : "947b2a12-3aac-480a-a0bf-626aff106f48",
"first name" : "Claude",
"last name" : "Bones",
"contact_ids" : [
"CTP70499"
],
"site_ids" : [
"5015"
],
"first name aliases" : [
"Claude"
],
"aliases" : [
"Claude Bones"
],
"createdDate" : "2020-06-03T19:59:08Z",
"updatedDate" : "2020-06-03T20:48:27Z",
"veevaID" : "V0B000000000569"
},
{
"mdm id" : "bce21b05-0b28-4ebb-a34d-761c1a397821",
"first name" : "Daniel",
"last name" : "Smith",
"contact_ids" : [
"CTP699"
],
"site_ids" : [
"5015"
],
"first name aliases" : [
"Dan",
"Danial",
"Danne",
"Danny",
"Daniel"
],
"aliases" : [
"Daniel Smith"
],
"createdDate" : "2020-06-03T19:59:08Z",
"updatedDate" : "2020-06-03T20:48:27Z",
"veevaID" : "V0B000000000566"
}
]
'# | ConvertFrom-Json
$JSONdata | foreach {
$record = [ordered]#{}
foreach($property in $_.psobject.Properties)
{
if($property.value -is [string])
{
$record.Add($property.name,$property.value)
}
else
{
$record.Add($property.name,($property.value -join ', '))
}
}
[PSCustomObject]$record
} | ConvertTo-Csv -NoTypeInformation
Output
"mdm id","first name","last name","contact_ids","site_ids","first name aliases","aliases","createdDate","updatedDate","veevaID"
"947b2a12-3aac-480a-a0bf-626aff106f48","Claude","Bones","CTP70499","5015","Claude","Claude Bones","2020-06-03T19:59:08Z","2020-06-03T20:48:27Z","V0B000000000569"
"bce21b05-0b28-4ebb-a34d-761c1a397821","Daniel","Smith","CTP699","5015","Dan, Danial, Danne, Danny, Daniel","Daniel Smith","2020-06-03T19:59:08Z","2020-06-03T20:48:27Z","V0B000000000566"
Simply change it to Export-Csv

Invoke Parameter into a CloudFormation Powershell script

it is possible to invoke a parameter into a PowerShell script running inside userdata?
I'm trying to assign a password, ADuser, and domain, in order to change the local computer name and join the server into the domain.
I can add the self input during the stack creation, but I don't know how to use the info inside userdata, is there any Ref that it can be used?
I'm able to do this using all the information inside userdata, but I don't want to save the stack with our domain information and credentials.
"Parameters" : {
"VMName":{
"Description":"Name of the EC2 Windows instance",
"Type":"String"
},
"DomainUser":{
"Description":"Name of Service/User domain account to be used to join the EC2 instance into CX domain",
"Type" : "String",
"MinLength" : "3",
"MaxLength" : "25",
"AllowedPattern" : "[a-zA-Z0-9]+\\..+"
},
"DomainCredential":{
"Description":"Password of the Service/User domain account to be used to join the EC2 instance into CX domain",
"Type" : "String",
"MinLength" : "8",
"MaxLength" : "32",
"AllowedPattern" : "(?=^.{6,255}$)((?=.*\\d)(?=.*[A-Z])(?=.*[a-z])|(?=.*\\d)(?=.*[^A-Za-z0-9])(?=.*[a-z])|(?=.*[^A-Za-z0-9])(?=.*[A-Z])(?=.*[a-z])|(?=.*\\d)(?=.*[A-Z])(?=.*[^A-Za-z0-9]))^.*",
"NoEcho" : "True"
},
"Resources" : {
"EC2InstanceOne":{
"Type":"AWS::EC2::Instance",
"DeletionPolicy" : "Retain",
"Properties":{
"InstanceType":{ "Ref" : "InstanceType" },
"SubnetId": { "Ref" : "MySubnetVM1" },
"SecurityGroupIds":[ { "Ref" : "SGUtilized" } ],
"SecurityGroupIds":[ { "Ref" : "SGUtilized2" } ],
"IamInstanceProfile" : { "Ref" : "RoleName" },
"KeyName": { "Ref" : "ServerKeyName" },
"ImageId":{ "Ref" : "AMIUtilized" },
"BlockDeviceMappings" : [
{
"DeviceName" : "/dev/sda1",
"Ebs" : {
"VolumeType" : "standard",
"DeleteOnTermination" : "false",
"VolumeSize" : "50"
}
}
],
"UserData" : { "Fn::Base64" : { "Fn::Join" : [ "", [
"<script>\n",
"PowerShell -Command \"& {$password = 'variable from parameter here' | ConvertTo-SecureString -asPlainText -Force ; $username = 'variable from parameter here'' ; $credential = New-Object System.Management.Automation.PSCredential($username,$password) ; Rename-Computer -NewName 'variable from parameter here'' -DomainCredential $credential}\" \n",
"PowerShell -Command \"& {$domain='variable from parameter here';$password = 'variable from parameter here'' | ConvertTo-SecureString -asPlainText -Force ;$username = 'variable from parameter here'' ; $credential = New-Object System.Management.Automation.PSCredential($username,$password) ; Add-Computer -DomainName $domain -Credential $credential}\" \n",
"PowerShell -Command \"& {Restart-Computer}\" \n",
"</script>"
]
]
}
}
}
Thanks, best regards.
you can to use Fn::Sub like this:
{
"Fn::Sub":
"PowerShell -Command \"& {$domain=${VMName};$password = ${DomainCredential}' | ConvertTo-SecureString -asPlainText -Force ;$username = ${DomainUser}' ; $credential = New-Object System.Management.Automation.PSCredential($username,$password) ; Add-Computer -DomainName $domain -Credential $credential}\" \n"
}
here's a yaml sample :
UserData:
Fn::Base64:
!Sub |
echo ${ParamPassword} | tee - | passwd ec2-user

Wrong hash filling to JSON, Perl

So, i'm making a programm fetching the data as hash from PostgreSQL, which uses a couple JSONs as parameter file and output datafile respectively. And i had some problem with fetching what it shouldn't fetch. Here's parameter json:
{
"queries": [
{
"table_name" : "t1",
"subqueries": [
{
"query_id" : "t1_1",
"query": [
.....some sql query
],
"to_hash" : {
"target_by" : "type_id", // key to index by
"keys" : [
{
"source" : "name", // key in hash from db
"target" : "name" // key in new hash
},
{
"source" : "r",
"target" : "r"
}
]
}
},
{
"query_id" : "t1_2",
"query": [
.....some sql query
],
"to_hash" : {
"target_by" : "type_id",
"keys" : [
{
"source" : "m",
"target" : "m"
}
]
}
}
]
}
]
}
....and here's a perl subroutine:
my $fname = "query_params.json";
my $q_data_raw;
{
local $/;
open(my $fh, "<:encoding(UTF-8)", $fname) or oops("$fname: $!");
$q_data_raw = <$fh>;
close($fh);
}
my $q_data = JSON->new->utf8->decode($q_data_raw);
my %result;
sub blabla {
my $data = shift;
my($tab, $i) = ($data->{table_name}, 0);
if ($data->{subqueries} ne "false"){
my %res_hash;
my #res_arr;
my $q_id;
foreach my $sq (#{$data->{subqueries}}){
my $query = "";
$q_id = $sq->{query_id};
print "\n";
print "$q_id\n";
for(#{$sq->{query}}){
$query .= "$_\n";
}
my $t_by = $sq->{to_hash}{target_by};
my $q_hash = $db_connection->prepare($query);
$q_hash->execute() or die( "Unable to get: " . $db_connection->errstr);
while(my $res = $q_hash->fetchrow_hashref()) {
# print Dumper $res; #print #1
for(#{$sq->{to_hash}->{keys}}){
# print "\nkey:\t" . $_->{target} . "\nvalue:\t".$res->{$_->{source}}; #print #2
$res_hash{$q_id}{$res->{$t_by}}{$_->{target}} = $res->{$_->{source}};
}
$res_hash{$q_id}{$res->{$t_by}}{__id} = $res->{$t_by};
# print Dumper %res_hash; #print #3
}
push #res_arr, $res_hash{$q_id};
# print Dumper $res_hash{$q_id}; #print #4
# print Dumper #res_arr; print #5
$result{$tab}{$q_id} = \#res_arr;
$q_hash->finish();
}
}
}
for (#{$q_data->{queries}}){ // hash from parameter json
blabla($_);
}
my $json = JSON->new->pretty->encode(\%result);
# to json file
....and there's what i get:
{
"t1" : {
"t1_1" : [
{
//type_id_1_* - from first query
"type_id_1_1" : {
"r" : "4746",
"__id" : "type_id_1_1",
"name" : "blablabla"
},
"type_id_1_2" : {
"r" : "7338",
"__id" : "type_id_1_2",
"name" : "nbmnbcxv"
},
....
},
{
//type_id_2_* - from second query
"type_id_2_1" : {
"m" : "6",
"__id" : "type_id_2_1"
},
"type_id_2_2" : {
"m" : "3",
"__id" : "type_id_2_2"
},
............
}
],
"t1_2" : [
{
"type_id_1_1" : {
"r" : "4746",
"__id" : "type_id_1_1",
"name" : "blablabla"
},
"type_id_1_2" : {
"r" : "7338",
"__id" : "type_id_1_2",
"name" : "nbmnbcxv"
},
....
},
{
"type_id_2_1" : {
"m" : "6",
"__id" : "type_id_2_1"
},
"type_id_2_2" : {
"m" : "3",
"__id" : "type_id_2_2"
},
............
}
]
}
}
Somehow, it fetches the queries from other subqueries, what i don't want. And loop seems to be ok, probably. What am i doing wrong?
Well, seems like i initialized #res_arr and #res_hash in wrong level - gotta be in foreach. The output is kinda what i need. At least i don't have duplicates.
Gotta be sleep more X(
....
my $q_id;
foreach my $sq (#{$data->{subqueries}}){
my %res_hash;
my #res_arr;
my $query = "";
$q_id = $sq->{query_id};
print "\n";
print "$q_id\n";
..........
}

Is it possible to parse output of MongoDB query as JSON document?

First of all - I cannot use perl MongoDB driver, so I'm interacting with MongoDB via IPC::Run. Now I'd like to get output from MongoDB as a hash ref.
Here is the code:
#!/usr/bin/env perl
use strict;
use warnings;
use JSON::XS;
use Try::Tiny;
use IPC::Run 'run';
use Data::Dumper;
my #cmd = ('/opt/mongo/bin/mongo', '127.0.0.1:27117/service_discovery', '--quiet', '-u', 'test', '-p', 'test', '--eval', 'db.sit.find().forEach(function(x){printjson(x)})');
my $out;
run \#cmd, '>>', \$out;
my $coder = JSON::XS->new->ascii->pretty->allow_nonref;
my $dec = try {my $output = $coder->decode($out)} catch {undef};
print Dumper (\%$dec);
It is not working now, %$dec is empty.
Here is the output of MongoDB query (value of $out):
{
"_id" : ObjectId("5696787eb8e5e87534777c82"),
"hostname" : "lab7n1",
"services" : [
{
"port" : 9000,
"name" : "ss-rest"
},
{
"port" : 9001,
"name" : "ss-rest"
},
{
"port" : 8060,
"name" : "websockets"
},
{
"port" : 8061,
"name" : "websockets"
}
]
}
{
"_id" : ObjectId("56967ab2b8e5e87534777c83"),
"hostname" : "lab7n2",
"services" : [
{
"port" : 8030,
"name" : "cloud-rest for batch"
},
{
"port" : 8031,
"name" : "cloud-rest for batch"
},
{
"port" : 8010,
"name" : "cloud-rest for bespoke"
},
{
"port" : 8011,
"name" : "cloud-rest for bespoke"
}
]
}
What should I do to make parser treat this output as legitimate JSON?
As suggested by #Matt i used incr_parse method and omitted _id field in output.

How to create key value JSON response in perl script with MYSQL data

I have a perl script which does some query and return response in JSON format. But the return JSON is only array of values.
This is Perl Script
my $sql_query = "SELECT * from table_categories";
my $statement = $db_handle->prepare ($sql_query) or die "Couldn't prepare query '$sql_query': $DBI::errstr\n";
$statement->execute() or die "SQL Error: $DBI::errstr\n";
my #loop_data = ();
while (my #data = $statement->fetchrow_array())
{
push(#loop_data, #data);
}
my $json;
$json->{"entries"} = \#loop_data;
my $json_text = to_json($json);
print $json_text;
and response is like this
{
"entries": [
[
"1",
"Salt and Sugar",
"/images/salt_sugar.png",
"7"
],
[
"2",
"Tea and Coffee",
"/images/tea_and_coffee.png",
"6"
],
[
"3",
"Spice and Pickles",
"/images/categories/spice_pickles.png",
"7"
],
[
"4",
"Pooja Needs",
"/images/categories/pooja-needs.png",
"9"
],
[
"5",
"Dry Fruits",
"/images/categories/dry_fruits.png",
"7"
]
]
}
But I want the response like this:
{
"entries": [
[
"id": "1",
"name": "Salt and Sugar",
"image": "/images/salt_sugar.png",
"rank": "7"
],
[
"id": "2",
"name": "Tea and Coffee",
"image": "/images/tea_and_coffee.png",
"rank": "6"
]
]
}
How to achieve that?
What changes i need to do in PERL script?
Please help.
If you specifically don't have need to loop over every row, I'd prefer to use selectall_arrayref with slicing:
my $json;
my $sql_query = "SELECT * FROM table_categories";
$json->{entries} = $db_handle->selectall_arrayref( $sql_query, {Slice => {} } );
my $json_text = to_json($json);
print $json_text;
So you get keys too and code is more compact and clear.
You can use fetchrow_hashref instead of fetchrow_array to receive a hashref where the column names are used for the property names using the code below.
my #loop_data = ();
while (my $hashref = $statement->fetchrow_hashref())
{
push(#loop_data,$hashref);
}
More information: https://metacpan.org/pod/DBI#fetchrow_hashref