I'm implementing right now some kind of mock for my function. And i have a little problem here. Depending on situation, i could get different kind of json as well as a different kind of hash of it. It can be a simple hash with empty values of keys or hash of array of hashes with empty or not empty values.
my %ch1 = (
"a" => "",
"b" => "",
"c" => ""
);
my %ch2 = (
"tab" => [
{
"a" => 11,
"b" => 22,
"c" => 33
},
{
"a" => 44,
"b" => 55,
"c" => 66
}
]
);
I need to make a function that checks both types of hash, counts empty values and compares with ammount of keys of hash.
This one kinda works for the first hash, but i don't know how to make it work for both hashes without hardcoding.
my $tck = 0;
for (keys %ch1){
if ($ch1{$_} eq ""){
print "'$ch1{$_}'\n";
$tck++;
}
}
if ($tck == scalar keys %ch1){
# do something
}
Any suggestions?
You could use Data::Visitor::Callback to do that. It's a pretty straight-forward implementation, as long as there are no other things that contain empty strings in your data structure.
The module visits each item in a data structure and calls user-defined callbacks on those items. It will do that for every ref and every value in those refs.
use strict;
use warnings;
use Data::Visitor::Callback;
my %ch1 = (
"a" => "",
"b" => "",
"c" => ""
);
my $empty_strings;
my $v = Data::Visitor::Callback->new(
value => sub {
++$empty_strings if $_ eq q{}; # q{} is like '' but easier to read
},
);
$v->visit( \%ch1 );
print $empty_strings;
This will output 3, as there are three empty strings in the input hash. Note that it wants a hash reference, not the hash itself.
You can just as well pass in a more complex data structure. The layout does not really matter. I've added an empty string to your second example to show that it works.
use strict;
use warnings;
use Data::Visitor::Callback;
my $empty_strings;
my $v = Data::Visitor::Callback->new(
value => sub {
++$empty_strings if $_ eq q{};
},
);
my %ch2 = (
"tab" => [
{
"a" => 11,
"b" => 22,
"c" => 33
},
{
"a" => '',
"b" => 55,
"c" => 66
}
]
);
$v->visit( \%ch2 );
print $empty_strings;
In this case, the output is 1.
Because there is no easy way to distinguish if a value it looks at is a key or a value, the following things would also be counted with this implementation. So it's not perfect, but should work for the type of data you showed.
my %fail = (
"" => "foo", # one
"b" => [ "", "" ], # two, three
);
This data structure would yield a $empty_strings count of 3.
I am not confident that I understand the problem correctly, but assuming those are the only two types of data structures your program needs to be able to handle, here is one way of doing something with them:
#!/usr/bin/env perl
use strict;
use warnings;
use List::MoreUtils qw( all none );
my %ch1 = (
"a" => "",
"b" => "",
"c" => ""
);
my %ch2 = (
"tab" => [
{
"a" => 11,
"b" => 22,
"c" => 33
},
{
"a" => 44,
"b" => 55,
"c" => 66
}
]
);
use YAML::XS;
for my $h ( \(%ch1, %ch2) ) {
print Dump n_keys_empty_values( $h );
}
sub n_keys_empty_values {
my $h = shift;
if ( all { ref } values %$h ) {
return [ map { my $v = $_; map count_empty_values( $_ ), #$v } values %$h ]
}
elsif ( none { ref } values %$h ){
return [ count_empty_values( $h ) ];
}
else {
die "Unexpected data structure\n";
}
}
sub count_empty_values {
my $h = shift;
[ scalar keys %$h, scalar grep $_ eq '', values %$h ];
}
Output:
---
- - 3
- 3
---
- - 3
- 0
- - 3
- 0
The return value of n_keys_empty_values is a reference to an array of array references. The size of the outer array corresponds to the number of inner hashes passed. count_empty_values takes a reference to a hash and counts the number of keys and number of values which are empty strings.
Write your function in that way, that it iterates over the argument list. Then you can pass either a single hash \%ch1 or a list of hashes #{$ch2{tab}} to the function.
#! /usr/bin/perl
use strict;
use warnings;
my %ch1 = (
"a" => "",
"b" => "",
"c" => ""
);
my %ch2 = (
"tab" => [
{
"a" => 11,
"b" => 22,
"c" => 33
},
{
"a" => 44,
"b" => 55,
"c" => 66
}
]
);
my %ch3 = (
"tab" => [
{
"a" => '',
"b" => '',
"c" => ''
},
{
"a" => '',
"b" => '',
"c" => ''
}
]
);
sub fun
{
for (#_) {
my %ch = %{$_};
my $tck = 0;
for (keys %ch){
if ($ch{$_} eq ""){
print "'$ch{$_}'\n";
$tck++;
}
}
if ($tck == scalar keys %ch){
print "do something\n";
}
}
}
fun (\%ch1);
fun (#{$ch2{tab}});
fun (#{$ch3{tab}});
I'm doing a disk space report that uses File::Find to collect cumulative sizing in a directory tree.
What I get (easily) from File::Find is the directory name.
e.g.:
/path/to/user/username/subdir/anothersubdir/etc
I'm running File::Find to collect sizes beneath:
/path/to/user/username
And build a cumulative size report of the directory and each of the subdirectories.
What I've currently got is:
while ( $dir_tree ) {
%results{$dir_tree} += $blocks * $block_size;
my #path_arr = split ( "/", $dir_tree );
pop ( #path_arr );
$dir_tree = join ( "/", #path_arr );
}
(And yes, I know that's not very nice.).
The purpose of doing this is so when I stat each file, I add it's size to the current node and each parent node in the tree.
This is sufficient to generate:
username,300M
username/documents,150M
username/documents/excel,50M
username/documents/word,40M
username/work,70M
username/fish,50M,
username/some_other_stuff,30M
But I'd like to now turn that in to JSON more like this:
{
"name" : "username",
"size" : "307200",
"children" : [
{
"name" : "documents",
"size" : "153750",
"children" : [
{
"name" : "excel",
"size" : "51200"
},
{
"name" : "word",
"size" : "81920"
}
]
}
]
}
That's because I'm intending to do a D3 visualisation of this structure - loosely based on D3 Zoomable Circle Pack
So my question is this - what is the neatest way to collate my data such that I can have cumulative (and ideally non cumulative) sizing information, but populating a hash hierarchically.
I was thinking in terms of a 'cursor' approach (and using File::Spec this time):
use File::Spec;
my $data;
my $cursor = \$data;
foreach my $element ( File::Spec -> splitdir ( $File::Find::dir ) ) {
$cursor -> {size} += $blocks * $block_size;
$cursor = $cursor -> {$element}
}
Although... that's not quite creating the data structure I'm looking for, not least because we basically have to search by hash key to do the 'rolling up' part of the process.
Is there a better way of accomplishing this?
Edit - more complete example of what I have already:
#!/usr/bin/env perl
use strict;
use warnings;
use File::Find;
use Data::Dumper;
my $block_size = 1024;
sub collate_sizes {
my ( $results_ref, $starting_path ) = #_;
$starting_path =~ s,/\w+$,/,;
if ( -f $File::Find::name ) {
print "$File::Find::name isafile\n";
my ($dev, $ino, $mode, $nlink, $uid,
$gid, $rdev, $size, $atime, $mtime,
$ctime, $blksize, $blocks
) = stat($File::Find::name);
my $dir_tree = $File::Find::dir;
$dir_tree =~ s|^$starting_path||g;
while ($dir_tree) {
print "Updating $dir_tree\n";
$$results_ref{$dir_tree} += $blocks * $block_size;
my #path_arr = split( "/", $dir_tree );
pop(#path_arr);
$dir_tree = join( "/", #path_arr );
}
}
}
my #users = qw ( user1 user2 );
foreach my $user (#users) {
my $path = "/home/$user";
print $path;
my %results;
File::Find::find(
{ wanted => sub { \&collate_sizes( \%results, $path ) },
no_chdir => 1
},
$path
);
print Dumper \%results;
#would print this to a file in the homedir - to STDOUT for convenience
foreach my $key ( sort { $results{$b} <=> $results{$a} } keys %results ) {
print "$key => $results{$key}\n";
}
}
And yes - I know this isn't portable, and does a few somewhat nasty things. Part of what I'm doing here is trying to improve on that. (But currently it's a Unix based homedir structure, so that's fine).
If you do your own dir scanning instead of using File::Find, you naturally get the right structure.
sub _scan {
my ($qfn, $fn) = #_;
my $node = { name => $fn };
lstat($qfn)
or die $!;
my $size = -s _;
my $is_dir = -d _;
if ($is_dir) {
my #child_fns = do {
opendir(my $dh, $qfn)
or die $!;
grep !/^\.\.?\z/, readdir($dh);
};
my #children;
for my $child_fn (#child_fns) {
my $child_node = _scan("$qfn/$child_fn", $child_fn);
$size += $child_node->{size};
push #children, $child_node;
}
$node->{children} = \#children;
}
$node->{size} = $size;
return $node;
}
Rest of the code:
#!/usr/bin/perl
use strict;
use warnings;
no warnings 'recursion';
use File::Basename qw( basename );
use JSON qw( encode_json );
...
sub scan { _scan($_[0], basename($_[0])) }
print(encode_json(scan($ARGV[0] // '.')));
In the end, I have done it like this:
In the File::Find wanted sub collate_sizes:
my $cursor = $data;
foreach my $element (
File::Spec->splitdir( $File::Find::dir =~ s/^$starting_path//r ) )
{
$cursor->{$element}->{name} = $element;
$cursor->{$element}->{size} += $blocks * $block_size;
$cursor = $cursor->{$element}->{children} //= {};
}
To generate a hash of nested directory names. (The name subelement is probably redundant, but whatever).
And then post process it with (using JSON):
my $json_structure = {
'name' => $user,
'size' => $data->{$user}->{size},
'children' => [],
};
process_data_to_json( $json_structure, $data->{$user}->{children} );
open( my $json_out, '>', "homedir.json" ) or die $!;
print {$json_out} to_json( $json_structure, { pretty => 1 } );
close($json_out);
sub process_data_to_json {
my ( $json_cursor, $data_cursor ) = #_;
if ( ref $data_cursor eq "HASH" ) {
print "Traversing $key\n";
my $newelt = {
'name' => $key,
'size' => $data_cursor->{$key}->{size},
};
push( #{ $json_cursor->{children} }, $newelt );
process_data_to_json( $newelt, $data_cursor->{$key}->{children} );
}
}
I have a large excel document (2000+ lines) that uses the cells to specify a tree structure and I would like to parse that into a .json file. The excel-exported .csv document is formatted as follows, where in the excel file a comma would be an empty cell:
Layer1category1,,,,,
,Layer2category,,,,
...
,,,,Layer5category1,
,,,,,item1
,,,,,item2
,,,,Layer5category2,
,,,,,item1
,,,Layer4category2,,
...
Layer1category2,,,,,
...
Layer1category8,,,,, // this is the last category in the uppermost layer
In summary, Layer n categories are prefaced with n-1 commas and followed by 6-n commas, and rows prefaced with 5 commas are the final layer, which is in the format of a string and has many fields other than its name.
I would like this to be converted to a .json file similar to the following. I use "name" because aside from a name each field is also tied to a lot of statistics that also needs to go into the json file.
{"name" : "Layer1category1",
"children": [
{"name" : "Layer2category1",
"children" : [
{"name" : "Layer3category1"
"children" : [
...
{"name" : "Layer5category1",
"children" : [{"name" : "item1"}, {"name" : "item2"}],}
{"name" : "Layer5category2",
"children" : [{"name" : "item1"}],}
{"name" : "Layer4category2",
"children" : [
...
]}
"name" : "Layer1category2",
"children" : [ ... ]
}
Does anyone have any suggestions for how I can approach this? The csv to json converters I have found do not support multi-layered structures. Thanks!
I faced with the same issue and wrote simple php script:
Input
Level I,Level II,Level III,Level IV,Level V,Level VI,Level VII,Level VIII,,,,,,,,,,,,,,,,,,
Role Profile,,,,,,,,,,,,,,,,,,,,,,,,,
,Development,,,,,,,,,,,,,,,,,,,,,,,,
,,Security,,,,,,,,,,,,,,,,,,,,,,,
,,Cloud,,,,,,,,,,,,,,,,,,,,,,,
,,,Cloud technologies,,,,,,,,,,,,,,,,,,,,,,
,,,,IaaS,,,,,,,,,,,,,,,,,,,,,
,,,,,Amazon Web Service (AWS),,,,,,,,,,,,,,,,,,,,
,,,,,Microsoft Azure,,,,,,,,,,,,,,,,,,,,
,,,,,Google Compute Engine (GCE),,,,,,,,,,,,,,,,,,,,
,,,,,OpenStack,,,,,,,,,,,,,,,,,,,,
Output
{
"Role Profile":{
"Development":{
"Security":{},
"Cloud":{
"Cloud technologies":{
"IaaS":{
"Amazon Web Service (AWS)":{},
"Microsoft Azure":{},
"Google Compute Engine (GCE)":{},
"OpenStack":{}
}
}
}
}
}
}
Code
<?php
$fn = "skills.csv"; // input file name
$hasHeader = true; // input file has header, we will skip first line
//
function appendItem( &$r, $lvl, $item ) {
if ( $lvl ) {
$lvl--;
end( $r );
appendItem( $r[key($r)], $lvl, $item );
} else {
$r[$item] = array();
}
}
$r = array();
if ( ( $handle = fopen( $fn, "r" ) ) !== FALSE ) {
$header = true;
while ( ( $data = fgetcsv( $handle, 1000, "," ) ) !== FALSE ) {
if ( $header and $hasHeader ) {
$header = false;
} else {
$lvl = 0;
foreach( $data as $dv ) {
$v = trim( $dv );
if ( $v ) {
appendItem( $r, $lvl, $v );
break;
}
$lvl++;
}
}
}
}
echo json_encode( $r );
?>
I am trying to create a JSON object that lists maps associated to a particular user, but haven't ever worked with nested JSON objects. This is what I want:
{
"success":"list of users maps",
"maps":[
{
"id":"1",
"name":"Home to LE",
"date_created":"1366559121"
},
{
"id":"2",
"name":"Test 1",
"date_created":"1366735066"
}
]
}
with this perl code:
my $maps = [];
for (my $x = 0; $x < $sth->rows; $x++) {
my ($id, $name, $date) = $sth->fetchrow_array();
my $map = qq{{"id":"$id","name":"$name","date_created":"$date"}};
push $maps, $map;
}
my $j = JSON::XS->new->utf8;
my $output = $j->encode({
"success"=>"list of users maps",
"maps"=>$maps
});
But the output I am getting is:
{
"success":"list of users maps",
"maps":[
"{\"id\":\"1\",\"name\":\"Home to LE\",\"date_created\":\"1366559121\"}",
"{\"id\":\"2\",\"name\":\"Test 1\",\"date_created\":\"1366735066\"}"
]
}
So when I process it in my Javascript, the data.maps[x].id is undefined. I am pretty sure that the JSON being output is incorrectly formatted.
Can anyone help me fix it?
It's undefined because what you have at data.maps[x] is not an object, but a string. Since a string has no property called id, you're getting undefined. I'd probably do something like this (if I couldn't change the perl script):
var mapData = JSON.parse(data.maps[x]);
//do stuff with mapData.id
But the better thing to do, is to make sure that it doesn't encode it as a string, but as proper JSON.
This part in your perl script:
my $map = qq{{"id":"$id","name":"$name","date_created":"$date"}};
Is simply making a quoted string out of all that data. Instead, what you want is an actual perl hash that can be translated into a JSON map/associative-array. So try this:
my $map = {
"id" => "$id",
"name" => "$name",
"date_created" => "$date"
};
push $maps, $map;
This way you actually have a perl hash (instead of just a string) that will get translated into proper JSON.
As an example, I wrote some test code:
use strict;
use JSON::XS;
my $maps = [];
push $maps, { id => 1, blah => 2 };
push $maps, { id => 3, blah => 2 };
my $j = JSON::XS->new->utf8->pretty(1);
my $output = $j->encode({
success => "list of blah",
maps => $maps
});
print $output;
When you run this, you get:
{
"success" : "list of blah",
"maps" : [
{
"blah" : 2,
"id" : 1
},
{
"blah" : 2,
"id" : 3
}
]
}