I have the following code for exporting all the items in one of my pods to json. The thing is I don't need all the 130 columns in the json file, but only about 20. Since this will be done for about 150 items I thought I could save some loading time by not printing out all the fields, but I do not know how to do this. For example I only want to print the column value named 'title' for all items in the pod. My code is attached bellow.
<?php
$pods = pods('name', array('orderby' => 'name asc', 'limit' => -1));
$all_companies = $pods->export_data();
if ( !empty( $all_companies ) ) {
die(json_encode($all_companies);
}else{
die(json_encode(array('error' => 'No cars found.')));
}
?>
I thought about doing something like this:
if ( 0 < $all_companies->total() ) {
while ($all_companies->fetch()) {
$json .= $all_companies->field('title');
}
$json = rtrim($json, ",");
$json .= '}}';
}
echo $json;
But it doesn't work and also the code becomes very long.
I'd make an array of the names of the twenty fields you want then build an array of those fields for each item, by doing a foreach of those field names passed to Pods::field() inside the while loop. Like this:
$pods = pods('name', array('orderby' => 'name asc', 'limit' => -1));
$fields = array( 'field_1', 'field_2' );
if ( $pods->total() > 0 ) {
while ( $pods->fetch() ) {
foreach ( $fields as $field ) {
$json[ $pods->id() ] = $pods->field( $field );
}
}
$json = json_encode( $json );
}
Alternatively, you could hack the /pods/<pod> endpoint of our JSON API to accept a list of fields to return as the body of the request. Wouldn't be hard to do, make sure to submit a pull request if you make it work.
Related
I have a CSV file, which contains data like below:
I want parse data from above csv file and store it in a hash initially. So my hash dumper %hash would look like this:
$VAR1 = {
'1' => {
'Name' => 'Name1',
'Time' => '7/2/2020 11:00'
'Cell' => 'NCell1',
'PMR' => '1001',
'ISD' => 'ISDVAL1',
'PCO' => 'PCOVAL1'
},
'2' => {
'Name' => 'Name2',
'Time' => '7/3/2020 13:10',
'Cell' => 'NCell2',
'PMR' => '1002',
'PCO' => 'PCOVAL2',
'MKR' => 'MKRVAL2',
'STD' => 'STDVAL2'
},
'3' => {
'Name' => 'Name3',
'Time' => '7/4/2020 20:15',
'Cell' => 'NCell3',
'PMR' => '1003',
'ISD' => 'ISDVAL3',
'MKR' => 'MKRVAL3'
},
};
Script is below:
#!/usr/bin/perl
use strict;
use warnings;
use Text::CSV;
use Data::Dumper;
my %hash;
my $csv = Text::CSV->new ({ binary => 1, auto_diag => 1 });
open my $fh, "<:encoding(utf8)", "input_file.csv" or die "input_file.csv: $!";
while (my $row = $csv->getline ($fh)) {
my #fields = #$row;
$hash{$fields[0]}{"Time"} = $fields[1];
$hash{$fields[0]}{"Name"} = $fields[2];
$hash{$fields[0]}{"Cell"} = $fields[3];
}
close $fh;
print Dumper(\%hash);
Here id is an key element in each line and based on the data value each data should be stored in respective names of an id.
Problem here is, till column D (Cell) I am able to parse data in above script and there after column D there won't be a header line and it will be like column E will act as header and column F is the value for the particular header's particular id. Similar condition goes to rest of the data values until end. And in middle we can see some values also will be missing. For example there is No MKR value for id 1.
How can I parse these data and store it in hash, so that my hash would look like above. TIA.
Changes made to the script posted was to remove the header line so that it does not form part of the result and added a for loop to set the reset of the data.
Test Data Used:
id,Time,Name,Cell,,,,,
1,7/2/2020 11:00,Name1,NCell1,PMR,1001,ISD,ISDVAL1
2,7/3/2020 13:10,Name2,NCell3,PMR,1002,PCO,PCOVAL2,MKR,MKRVAL2
Updated Script: (This was the first version suggest using the improved version in the edit)
#!/usr/bin/perl
use strict;
use warnings;
use Text::CSV;
use Data::Dumper;
my %hash;
my $csv = Text::CSV->new ({ binary => 1, auto_diag => 1 });
open my $fh, "<:encoding(utf8)", "input_file.csv" or die "input_file.csv: $!";
my $headers = $csv->getline ($fh);
while (my $row = $csv->getline ($fh)) {
$hash{$row->[0]}{Time} = $row->[1];
$hash{$row->[0]}{Name} = $row->[2];
$hash{$row->[0]}{Cell} = $row->[3];
for (my $i = 4; $i < scalar (#{$row}); $i += 2) {
$hash{$row->[0]}{$row->[$i]} = $row->[$i + 1];
}
}
close $fh;
print Dumper(\%hash);
Output:
$VAR1 = {
'2' => {
'MKR' => 'MKRVAL2',
'Name' => 'Name2',
'PCO' => 'PCOVAL2',
'Cell' => 'NCell3',
'Time' => '7/3/2020 13:10',
'PMR' => '1002'
},
'1' => {
'Name' => 'Name1',
'ISD' => 'ISDVAL1',
'Cell' => 'NCell1',
'Time' => '7/2/2020 11:00',
'PMR' => '1001'
}
};
Edit:
Thanks to comment from #choroba here is an improved version of the script setting the hash with all the additional row values first and then adding the first values Time Name Cell using the header line read from the file.
#!/usr/bin/perl
use strict;
use warnings;
use Text::CSV;
use Data::Dumper;
my %hash;
my $csv = Text::CSV->new ({ binary => 1, auto_diag => 1 });
open my $fh, "<:encoding(utf8)", "input_file.csv" or die "input_file.csv: $!";
my $headers = $csv->getline ($fh);
while (my $row = $csv->getline ($fh)) {
$hash{$row->[0]} = { #$row[4 .. $#$row] };
#{$hash{$row->[0]}}{#$headers[1, 2, 3]} = #$row[1, 2, 3];
}
close $fh;
print Dumper(\%hash);
There are some Text::CSV features that you can use to make this a bit simpler. There's a lot of readability to gain by removing density in the loop.
First, you can set the column names for missing header values. I don't know what those columns represent so I've called them K1, V1, and so on. You can substitute better names for them. How I do that isn't as important is that I do that. I'm using v5.26 because I'm using postfix dereferencing:
use v5.26;
my $headers = $csv->getline($fh);
my #kv_range = 1 .. 4;
$headers->#[4..11] = map { ("K$_", "V$_") } #kv_range;
$csv->column_names( $headers );
If I knew the names, I could use those instead of numbers. I merely change the stuff in #kv_range:
my #kv_range = qw(machine test regression ice_cream);
And, when the data file changes, I handle all of that here. When it's outside the loop, there's much less to miss.
Now that I have all columns named, I use getline_hr to get back a hash reference of the line. The keys are the column names I just set. This does a lot of the work for you already. You have to handle the pairs at the end, but that's going to be easy too:
my %Grand;
while( my $row = $csv->getline_hr($fh) ) {
foreach ( #kv_range ) {
no warnings 'uninitialized';
$row->{ delete $row->{"K$_"} } = delete $row->{"V$_"};
}
$Grand{ $row->{id} } = $row;
delete $row->#{ 'id', '' };
}
Now to handle the pairs at the end: I want to take the value in the column K1 and make it a key, then take the value in V1 and make that the value. At the same time, I need to remove those K1 and V1 columns. delete has the nice behavior in that it returns the value for the key you deleted. This way doesn't require any sort of pointer math or knowledge about positions. Those things might change and I've handled all of that before I got this far:
$row->{ delete $row->{"K$_"} } = delete $row->{"V$_"};
You could also do this in a couple steps if that statement is too much for you:
my( $key, $value ) = delete $row->#{ "K$_", "V$_" };
$row->{$key} = $value;
I'd leave the id column in there, but if you don't want it, get rid of it. Also, that step with the deletes might have made some empty string keys for the cells that had no values. Instead of guarding against that and making the foreach more complicated, I let it happen and get rid of it at the end:
delete $row->#{ 'id', '' };
Altogether, it looks like this. It's doing the same thing as Piet Bosch's answer, but I've pushed a lot of the complexity back into the module as well as doing a little pre-loop work:
use v5.26;
use strict;
use warnings;
use Data::Dumper;
use Text::CSV;
my $csv = Text::CSV->new({
binary => 1,
auto_diag => 1
});
open my $fh, "<:encoding(utf8)", "input_file.csv"
or die "input_file.csv: $!";
my $headers = $csv->getline($fh);
my #kv_range = 1 .. 4;
$headers->#[4..11] = map { ("K$_", "V$_") } #kv_range;
$csv->column_names( $headers );
my %Grand;
while( my $row = $csv->getline_hr($fh) ) {
foreach ( #kv_range ) {
no warnings 'uninitialized';
$row->{ delete $row->{"K$_"} } = delete $row->{"V$_"};
}
$Grand{ $row->{id} } = $row;
delete $row->#{ 'id', '' };
}
say Dumper( \%Grand );
And the output looks like this:
$VAR1 = {
'2' => {
'PMR' => '1002',
'PCO' => 'PCOVAL2',
'MKR' => 'MKRVAL2',
'Name' => 'Name2',
'Time' => '7/3/2020 13:10',
'Cell' => 'NCell3'
},
'1' => {
'Cell' => 'NCell1',
'Time' => '7/2/2020 11:00',
'ISD' => 'ISDVAL1',
'PMR' => '1001',
'Name' => 'Name1'
}
};
I want to upload a .csv file into a database but in 3 different table. I mean like in the .csv file have three columns then I want to insert each column into different table. How can I do it? Can anyone give any example or idea to do this.
Here is an example if you wish to upload csv file in db
public function upload(Request $request){ //upload csv
$file = $request->file('file');
$csvData = file_get_contents($file);
$rows = array_map("str_getcsv", explode("\n", $csvData));
//CSV headers
$header = array_shift($rows);
$escapedHeader=[];
//to converting lowercase and remove spaces
foreach ($header as $key => $value) {
$lheader=strtolower($value);
$escapedItem=preg_replace('/[^a-z]/', '', $lheader);
array_push($escapedHeader, $escapedItem);
}
//storing data to database
foreach($rows as $row) {
if (count($header) != count($row)) {
continue;
}
//This will generate a associate array with headers.
$row = array_combine($escapedHeader, $row);//dd($row);
// if there are 3 tables named - student, course, grade
Student::create([
'fname' => $row['firstname'],
'lname' => $row['lastname'],
]);
Course::create([
'course' => $row['course'],
]);
Grade::create([
'grade' => $row['score'],
]);
}
Session::flash('message', 'CSV file imported!');
return response()->json('success');
}
I'm having trouble processing the returned results from a DB SQL Mapper into a recognizable json encoded array.
function apiCheckSupplyId() {
/*refer to the model Xrefs*/
$supply_id = $this->f3->get('GET.supply_id');
$xref = new Xrefs($this->tongpodb);
$supply = $xref->getBySupplyId( $supply_id );
if ( count( $supply ) == 0 ) {
$this->logger->write('no xref found for supply_id=' .$supply_id);
$supply = array( array('id'=>0) );
echo json_encode( $supply );
} else {
$json = array();
foreach ($supply as $row){
$item = array();
foreach($row as $key => $value){
$item[$key] = $value;
}
array_push($json, $item);
}
$this->logger->write('xref found for supply_id=' .$supply_id.json_encode( $json ) );
echo json_encode( $json );
}
}
This is the method I am using but it seems very clunky to me. Is there a better way?
Assuming the getBySupplyId returns an array of Xref mappers, you could simplify the whole thing like this:
function apiCheckSupplyId() {
$supply_id = $this->f3->get('GET.supply_id');
$xref = new Xrefs($this->tongpodb);
$xrefs = $xref->getBySupplyId($supply_id);
echo json_encode(array_map([$xref,'cast'],$xrefs));
$this->logger->write(sprintf('%d xrefs found for supply_id=%d',count($xrefs),$supply_id));
}
Explanation:
The $xrefs variable contains an array of mappers. Each mapper being an object, you have to cast it to an array before encoding it to JSON. This can be done in one line by mapping the $xref->cast() method to each record: array_map([$xref,'cast'],$xrefs).
If you're not confident with that syntax, you can loop through each record and cast it:
$cast=[];
foreach ($xrefs as $x)
$cast[]=$x->cast();
echo json_encode($cast);
The result is the same.
The advantage of using cast() other just reading each value (as you're doing in your original script) is that it includes virtual fields as well.
Im trying to parse out JSON output from the server using Perl. The connection and downloading of the REST data is ok, I just need help parsing the returned data. Here is the snippet for my code:
my $response = HTTP::Tiny->new->get($SERVER_ADDR);
if ($response->{success})
{
my $html = $response->{content};
#LINES = split /\n/, $html;
chomp(#LINES);
print("Lines: '#LINES'\n"); # ZZZ
my $decoded_json = decode_json( $html );
print Dumper $decoded_json;
}
else
{
print "Failed: $response->{status} $response->{reasons}";
}
And here is the results:
Lines: '{"players":[{"currentlyOnline":false,"timePlayed":160317,"name":"MarisaG","lastPlayed":1474208741470}]}'
$VAR1 = {
'players' => [
{
'currentlyOnline' => bless( do{\(my $o = 0)}, 'JSON::PP::Boolean' ),
'timePlayed' => 160317,
'lastPlayed' => '1474208741470',
'name' => 'MarisaG'
}
]
};
There will be multiple entries under the "players" for each player logged in right now. Any tips?
I'm not really sure what you're asking. You have successfully parsed the JSON by by calling decode_json(). You now have a data structure in $decoded_json. Your call to Dumper() shows the structure of that data. It's a hash reference with a single key, players. The value associated with that key is an array reference. Each element in the referenced array is another hash.
So, for example, you could print all of the players' names with code like this.
foreach (#{ $decoded_json->{players} }) {
say $_->{name};
}
I have a configuration file which is in XML format. I need to parse the XML and convert to JSON. I'm able to convert it with XML2JSON module of perl. But the problem is, it is not maintaining the order of XML elements. I strictly need the elements in order otherwise I cannot configure
My XML file is something like this. I have to configure an IP address and set that IP as a gateway to certain route.
<Config>
<ip>
<address>1.1.1.1</address>
<netmask>255.255.255.0</netmask>
</ip>
<route>
<network>20.20.20.0</network>
<netmask>55.255.255.0</netmask>
<gateway>1.1.1.1</gateway>
</route>
</Config>
This is my perl code to convert to JSON
my $file = 'config.xml';
use Data::Dumper;
open my $fh, '<',$file or die;
$/ = undef;
my $data = <$fh>;
my $XML = $data;
my $XML2JSON = XML::XML2JSON->new();
my $Obj = $XML2JSON->xml2obj($XML);
print Dumper($Obj);
The output I'm getting is,
$VAR1 = {'Config' => {'route' => {'netmask' => {'$t' => '55.255.255.0'},'gateway' => {'$t' => '1.1.1.1'},'network' => {'$t' => '20.20.20.0'}},'ip' => {'netmask' => {'$t' => '255.255.255.0'},'address' => {'$t' => '1.1.1.1'}}},'#encoding' => 'UTF-8','#version' => '1.0'};
I have a script which reads the json object and configure..
But it fails as it first tries to set gateway ip address to a route where the ip address is not yet configured and add then add ip address.
I strictly want key ip to come first and then route for proper configuration without error. Like this I have many dependencies where order of keys is a must.
Is there any way I can tackle this problem? I tried almost all modules of XML parsing like XML::Simple,Twig::XML,XML::Parser. But nothing helped..
Here's a program that I hacked together that uses XML::Parser to parse some XML data and generate the equivalent JSON in the same order. It ignores any attributes, processing instructions etc. and requires that every XML element must contain either a list of child elements or a text node. Mixing text and elements won't work, and this isn't checked except that the program will die trying to dereference a string
It's intended to be a framework for you to enhance as you require, but works fine as it stands with the XML data you show in your question
use strict;
use warnings 'all';
use XML::Parser;
my $parser = XML::Parser->new(Handlers => {
Start => \&start_tag,
End => \&end_tag,
Char => \&text,
});
my $struct;
my #stack;
$parser->parsefile('config.xml');
print_json($struct->[1]);
sub start_tag {
my $expat = shift;
my ($tag, %attr) = #_;
my $elem = [ $tag => [] ];
if ( $struct ) {
my $content = $stack[-1][1];
push #{ $content }, $elem;
}
else {
$struct = $elem;
}
push #stack, $elem;
}
sub end_tag {
my $expat = shift;
my ($elem) = #_;
die "$elem <=> $stack[-1][0]" unless $stack[-1][0] eq $elem;
for my $content ( $stack[-1][1] ) {
$content = "#$content" unless grep ref, #$content;
}
pop #stack;
}
sub text {
my $expat = shift;
my ($string) = #_;
return unless $string =~ /\S/;
$string =~ s/\A\s+//;
$string =~ s/\s+\z//;
push #{ $stack[-1][1] }, $string;
}
sub print_json {
my ($data, $indent, $comma) = (#_, 0, '');
print "{\n";
for my $i ( 0 .. $#$data ) {
# Note that $data, $indent and $comma are overridden here
# to reflect the inner context
#
my $elem = $data->[$i];
my $comma = $i < $#$data ? ',' : '';
my ($tag, $data) = #$elem;
my $indent = $indent + 1;
printf qq{%s"%s" : }, ' ' x $indent, $tag;
if ( ref $data ) {
print_json($data, $indent, $comma);
}
else {
printf qq{"%s"%s\n}, $data, $comma;
}
}
# $indent and $comma (and $data) are restored here
#
printf "%s}%s\n", ' ' x $indent, $comma;
}
output
{
"ip" : {
"address" : "1.1.1.1",
"netmask" : "255.255.255.0"
},
"route" : {
"network" : "20.20.20.0",
"netmask" : "55.255.255.0",
"gateway" : "1.1.1.1"
}
}
The problem isn't so much to do with XML parsing, but because perl hashes are not ordered. So when you 'write' some JSON... it can be any order.
The way to avoid this is to apply a sort function to your JSON.
You can do this by using sort_by to explicitly sort:
#!/usr/bin/env perl
use strict;
use warnings;
use XML::Twig;
use JSON::PP;
use Data::Dumper;
sub order_nodes {
my %rank_of = ( ip => 0, route => 1, address => 2, network => 3, netmask => 4, gateway => 5 );
print "$JSON::PP::a <=> $JSON::PP::b\n";
return $rank_of{$JSON::PP::a} <=> $rank_of{$JSON::PP::b};
}
my $twig = XML::Twig -> parse (\*DATA);
my $json = JSON::PP -> new;
$json ->sort_by ( \&order_nodes );
print $json -> encode( $twig -> simplify );
__DATA__
<Config>
<ip>
<address>1.1.1.1</address>
<netmask>255.255.255.0</netmask>
</ip>
<route>
<network>20.20.20.0</network>
<netmask>55.255.255.0</netmask>
<gateway>1.1.1.1</gateway>
</route>
</Config>
In some scenarios, setting canonical can help, as that sets ordering to lexical order. (And means your JSON output would be consistently ordered). This doesn't apply to your case.
You could build the node ordering via XML::Twig, either by an xpath expression, or by using twig_handlers. I gave it a quick go, but got slightly unstuck in figuring out how you'd 'tell' how to figure out ordering based on getting address/netmask and then network/netmask/gateway.
As a simple example you could:
my $count = 0;
foreach my $node ( $twig -> get_xpath ( './*' ) ) {
$rank_of{$node->tag} = $count++ unless $rank_of{$node->tag};
}
print Dumper \%rank_of;
This will ensure ip and route are always the right way around. However it doesn't order the subkeys.
That actually gets a bit more complicated, as you'd need to recurse... and then decide how to handle 'collisions' (like netmask - address comes before, but how does it sort compared to network).
Or alternatively:
my $count = 0;
foreach my $node ( $twig->get_xpath('.//*') ) {
$rank_of{ $node->tag } = $count++ unless $rank_of{ $node->tag };
}
This walks all the nodes, and puts them in order. It doesn't quite work, because netmask appears in both stanzas though.
You get:
{"ip":{"address":"1.1.1.1","netmask":"255.255.255.0"},"route":{"netmask":"55.255.255.0","network":"20.20.20.0","gateway":"1.1.1.1"}}
I couldn't figure out a neat way of collapsing both lists.