I'm using Chart module to generate charts in PNG format from CSV data:
It works well, the charts look okay, but I get warnings for the undef values (there are 3 such values at the end of the above diagram):
# ~/txv3.pl "./L*TXV3*.csv" > /var/www/html/x.html
Generating chart: L_B17_C0_TXV3LIN_PA3_TI1_CI1
Use of uninitialized value $label in length at /usr/share/perl5/vendor_perl/Chart/Base.pm line 3477, <> line 69.
Use of uninitialized value in subroutine entry at /usr/share/perl5/vendor_perl/Chart/Base.pm line 3478, <> line 69.
Use of uninitialized value $label in length at /usr/share/perl5/vendor_perl/Chart/Base.pm line 3477, <> line 69.
Use of uninitialized value in subroutine entry at /usr/share/perl5/vendor_perl/Chart/Base.pm line 3478, <> line 69.
Use of uninitialized value $label in length at /usr/share/perl5/vendor_perl/Chart/Base.pm line 3477, <> line 69.
Use of uninitialized value in subroutine entry at /usr/share/perl5/vendor_perl/Chart/Base.pm line 3478, <> line 69.
I need to get rid of these warnings as they are useless here and they make the log of my Hudson-job unreadable.
So I've tried (using perl 5.10.1 on CentOS 6.4 / 64 bit):
#!/usr/bin/perl -w
use strict;
....
$pwrPng->set(%pwrOptions);
$biasPng->set(%biasOptions);
my $pwrPngFile = File::Spec->catfile(PNG_DIR, "${csv}_PWR.png");
my $biasPngFile = File::Spec->catfile(PNG_DIR, "${csv}_BIAS.png");
{
no warnings;
$pwrPng->png($pwrPngFile, $pwrData);
$biasPng->png($biasPngFile, $biasData);
}
But the warnings are still printed.
Any suggestions please?
Usually it is best not to ignore warnings.
Why don't you just handle the undef values first, before charting?
Either replace them with something sensible, or skip plotting those rows:
data.csv
RGI,BIAS,LABEL
20,130,"1146346307 #20"
21,135,"1146346307 #21"
22,140,
--
use Scalar::Util qw( looks_like_number );
my $fname = "data.csv";
open $fh, "<$fname"
or die "Unable to open $fname : $!";
my $data = [];
while (<$fh>) {
chomp;
my ($rgi, $bias, $label) = split /,/; # Better to use Text::CSV
next unless looks_like_number($rgi);
next unless looks_like_number($bias);
$label ||= "Unknown Row $."; # Rownum
# Create whatever structure you need.
push #$data, { rgi => $rgi, bias => $bias, label => $label };
}
# Now draw chart
In your Hudson-job, install a handler for the warn signal that filters warnings so the ones you know about won't show up.
BEGIN {
$SIG{'__WARN__'} = sub { my $w = shift; warn $w if $w !~ m|/Chart/Base.pm| };
}
Related
I am trying to parse a given CSV File, stream on regular base.
My requirement is to Access the Data via ColumName (Header).
The ColumNames are not given in row 1. The ColumNames are given in row 2.
The CSV does have 100 rows but I only need 2 data rows to import.
The separator is a tab.
The following script works for header at row 1 and for all rows in the file
I failed to modify it to header at row 2 and to use only 2 rows or a number of rows.
script:
#!/usr/bin/perl
use strict;
use warnings;
use Tie::Handle::CSV;
use Data::Dumper;
my $file = "data.csv";
my $fh = Tie::Handle::CSV->new ($file, header => 1, sep_char => "\t");
my $hfh = Tie::Handle::CSV->new ($file, header => 0, sep_char => "\t");
my $line = <$hfh>;
my $myheader;
while (my $csv_line = <$fh>)
{
foreach(#{$line})
{
if ( $_ ne "" )
{
print $_ . "=" . $csv_line->{$_} . "\n" ;
}
}
}
The Data.csv could look like:
This is a silly sentence on the first line
Name GivenName Birthdate Number
Meier hans 18.03.1999 1
Frank Thomas 27.1.1974 2
Karl Franz 1.1.2000 3
Here could be something silly again
Thanks for any hint.
best regards
Use Text::CSV_XS instead of Tie::Handle::CSV (Which depends on the module so you have it installed already), read and throw away the first line, use the second line to set column names, and then read the rest of the data:
#!/usr/bin/env perl
use warnings;
use strict;
use feature qw/say/;
use Text::CSV_XS;
my $csv = Text::CSV_XS->new({ sep => ",", # Using CSV because TSV doesn't play well with SO formatting
binary => 1});
# Read and discard the first line
$_ = <DATA>;
# Use the next line as the header and set column names
$csv->column_names($csv->getline(*DATA));
# Read some rows and access columns by name instead of position
my $nr = 0;
while (my $record = $csv->getline_hr(*DATA)) {
last if ++$nr == 4;
say "Row $nr: $record->{GivenName} was born on $record->{Birthdate}";
}
__DATA__
This is a silly sentence on the first line
Name,GivenName,Birthdate,Number
Meier,hans,18.03.1999,1
Frank,Thomas,27.1.1974,2
Karl,Franz,1.1.2000,3
Here could be something silly again
Tie::Handle::CSV accepts a filehandle instead of a filename. You can skip the first line by reading one line from it before you pass the filehandle to Tie::Handle::CSV:
use strict;
use warnings;
use Tie::Handle::CSV;
use Data::Dumper;
my $file = "data.csv";
open (my $infile, '<',$file) or die "can't open file $file: $!\n";
<$infile>; # skip first line
my $hfh = Tie::Handle::CSV->new ($infile, header => 1, sep_char => "\t");
my #csv;
my $num_lines = 3;
while ($num_lines--){
my $line = <$hfh>;
push #csv, $line;
}
print Dumper \#csv;
thanks to you both.
To clarify more detail my requirements.
The original Data File does have maybe 100 Colums with dynamic unknown Names for me.
I will create a list of Colums/Attribute from a other Service for which this script should provide the data content of some rows.
Request is in Terms of the data example:
Please provide all Names and all Birthdates of the first 25 Rows.
The next Request could be all Names and Givennames of the first 10 rows.
That means from the content of 100 Columns I have to provide the content for two, four, five Columns only.
The output I use (foreach), is only to test the Access by ColumName to the content of rows.
I mixed up your solution and stayed with Tie::Handle::CSV.
At the moment I have to use the two filehandles- Maybe you have a hint to be more effective.
#!/usr/bin/perl
use strict;
use warnings;
use Tie::Handle::CSV;
use Data::Dumper;
my $file = "data.csv";
open (my $infile, '<',$file) or die "can't open file $file: $!\n";
open (my $secfile, '<',$file) or die "can't open file $file: $!\n";
<$infile>; # skip first line
<$secfile>;
my $fh = Tie::Handle::CSV->new ($secfile, header => 1, sep_char => "\t");
my $hfh = Tie::Handle::CSV->new ($infile, header => 0, sep_char => "\t");
my $line = <$hfh>;
my $numberoflines = 2 ;
while ($numberoflines-- )
{
my $csv_line = <$fh> ;
foreach(#{$line})
{
if ( $_ ne "" )
{
print $_ . "=" . $csv_line->{$_} . "\n" ;
}
}
}
thanks got it running with "keys %$csv_line". I was not using because of missing knowlegde. ;-)
#!/usr/bin/perl
use strict;
use warnings;
use Tie::Handle::CSV;
my $file = "data.csv";
open (my $secfile, '<',$file) or die "can't open file $file: $!\n";
<$secfile>;
my $fh = Tie::Handle::CSV->new ($secfile, header => 1, sep_char => "\t");
my $numberoflines = 3 ;
while ($numberoflines-- )
{
my $csv_line = <$fh> ;
my #Columns = keys %{ $csv_line } ;
foreach (#Columns )
{
if ( $_ ne "" )
{
print $_ . "=" . $csv_line->{$_} . "\n" ;
}
}
print "-----------\n"
}
On last question:
The File I Read will be filled and modified by an other program.
What can I do to detect the File violation in case it makes a problem.
And I dont what the my script dies.
Thanks
regards
I am working with the Text::CSV library of Perl to import data from a CSV file, using the functional interface. The data is stored in an array of hashes, and the problem is that when the script tries to access those elements/keys, they are uninitialized (or undefined).
Using the library Dumper, it is possible to see that the array and the hashes are not empty, in fact, they are correctly filled with the data of the CSV file.
With this small piece of code, I get the following output:
my $array = csv(
in => $csv_file,
headers => 'auto');
foreach my $mem (#{$array}) {
print Dumper $mem;
foreach (keys $mem) {
print $mem{$_};
}
}
Last part of the output:
$VAR1 = {
'Column' => '16',
'Width' => '13',
'Type' => 'RAM',
'Depth' => '4096'
};
Use of uninitialized value in print at ** line 81.
Use of uninitialized value in print at ** line 81.
Use of uninitialized value in print at ** line 81.
Use of uninitialized value in print at ** line 81.
This happens with all the elements of the array. Is this problem related to the encoding, or I am just simply accessing the elements in a incorrect way?
$mem is a reference to a hash, but you keep trying to use it directly as a hash. Change your code to:
foreach (keys %$mem) {
print $mem->{$_};
}
There is a slight complication in that in some versions of perl, 'keys $mem' was allowed directly as an experimental feature, which later got removed. In any case, adding
use warnings;
use strict;
would likely have given you some helpful clues as to what was happening.
When I run your code on my version of Perl (5.24), I get this error:
Experimental keys on scalar is now forbidden at ... line ...
This points to the line:
foreach (keys $mem) {
You should dereference the hash ref:
use warnings;
use strict;
use Data::Dumper;
use Text::CSV qw( csv );
my $csv_file="data.csv";
my $array = csv(
in => $csv_file,
headers => 'auto');
foreach my $mem (#{$array}) {
print Dumper($mem);
foreach (keys %{ $mem }) {
print $mem->{$_}, "\n";
}
}
For an example I am reading file this where after some words I check how many "you" are present.
Good
morning
to
you
May
every
step
you
make
you
be
filled
you
with
happiness
love
you
and
peace
you
Code I wrote:
use warnings;
use strict;
my $log1_file = "log.log";
my $you_count = 0;
my $you_make_count = 0;
my $you_love_count = 0;
my $point ;
open(IN1, "<$log1_file" ) or die "Could not open file $log1_file: $!";
while (my $line = <IN1>) {
$point =$.;
print "$. main while\n";
my #fields = split' ',$line;
if ($fields[0] eq "Good") {
print "$. after good_if\n";
good_check();
print "$. after good_call\n";
seek (IN1,$point,0);
#$. = $point;
print "$. \n";
}
elsif ($fields[0] eq "make") {
print "$. after make_if\n";
make_check();
#$. = $point;
seek (IN1,$point,0);
}
elsif ($fields[0] eq "love") {
print "$. after love_if\n";
love_check();
#$. = $point;
seek (IN1,$point,0);
}
}
print "$you_count\n";
print "$you_make_count\n";
print "$you_love_count\n";
close IN1;
sub love_check{
while (my $line = <IN1>)
my #fields = split' ',$line;
if ($fields[0] eq "you") {
$you_love_count++;
}
}
}
sub make_check{
while (my $line = <IN1>) {
my #fields = split' ',$line;
if ($fields[0] eq "you") {
$you_make_count++;
}
}
}
sub good_check{
while (my $line = <IN1>) {
my #fields = split' ',$line;
if ($fields[0] eq "you") {
$you_count++;
}
}
}
If I use seek (IN1,$point,0); to point back to location I am getting output like below:
1 main while
1 after good_if
20 after good_call
20
21 main while
22 main while
23 main while
24 main while
25 main while
26 main while
27 main while
28 main while
29 main while
29 after make_if
41 main while
42 main while
43 main while
44 main while
44 after make_if
56 main while
Use of uninitialized value $fields[0] in string eq at check.pl line 15, <IN1> line 56.
Use of uninitialized value $fields[0] in string eq at check.pl line 25, <IN1> line 56.
Use of uninitialized value $fields[0] in string eq at check.pl line 33, <IN1> line 56.
57 main while
58 main while
59 main while
60 main while
61 main while
62 main while
63 main while
63 after love_if
68 main while
69 main while
70 main while
70 after love_if
75 main while
76 main while
76 after love_if
81 main while
82 main while
82 after love_if
87 main while
Use of uninitialized value $fields[0] in string eq at check.pl line 15, <IN1> line 87.
Use of uninitialized value $fields[0] in string eq at check.pl line 25, <IN1> line 87.
Use of uninitialized value $fields[0] in string eq at check.pl line 33, <IN1> line 87.
88 main while
89 main while
90 main while
91 main while
6
8
8
The "you" values at final are correct but not getting lines numbers as normal.
And if I use $. = $point; only first sub is working fine.
Can any one tell me best way to point back same location?
This question very much seems as though an XY-problem. Or homework. The logic of counting "you"s connected to certain key words seems arbitrary. The "you"s following "Good", for example, will contain all the "you"s in the other words combined.
As I assume this is some sort of learning exercise, I will comment your code, then come with a suggested solution.
open(IN1, "<$log1_file" ) or die "Could not open file $log1_file: $!";
Always use the three argument open with explicit open mode to avoid code injection. Use a lexical file handle (my $fh) instead of a global bareword (IN1). It should look like this:
open my $fh, "<", $log1_file or die "Could not open '$log1_file': $!";
This split is unnecessary
my #fields = split' ',$line;
Since you only have one word on each line, all this does is remove the newline on the end (because splitting on ' ' is a special case). If you want to remove the newline, you should use chomp, like this: chomp($line)
Using seek and tell to navigate around your file is likely the wrong solution. While you can make it work, there are better solutions.
Using three almost identical subroutines to do the exact same thing (almost) is bad practice, IMO. Using global variables inside the subroutines is also not a good thing. What you should seek is encapsulation: Feed the information you need to the subroutine, and return the values you want afterwards. For example:
my #file = <$fh> # slurp the file into an array
....
if (/^Good$/) {
$good_count += you_check($line_number);
} elsif (/^make$/) {
$make_count += you_check($line_number);
} ....etc
sub you_check {
my $line_number = shift;
my $count = 0;
for my $line ($file[$line_number] .. $file[$#file]) {
$count++ if $line =~ /^you$/;
}
return $count;
}
Assuming we keep #file unchanged, the you_check() function can be used without worrying that using it will alter something else.
With that said, if I was to solve this task, I would use a hash. It will allow you to dynamically determine the keyword, and add new keywords without having to add a lot of new code.
use strict;
use warnings;
use Data::Dumper;
my %count;
my $key;
while (<>) {
if (/(Good|make|love)/) {
$key = $1;
}
if (/you/) {
$count{$key}++ if $key;
}
}
print Dumper \%count;
Use it like this on the command line:
$ count.pl log.log
The output when used with your sample data is:
$VAR1 = {
'love' => 2,
'Good' => 2,
'make' => 2
};
If you still wanted to maintain the rule that the first word count contains all other words, you can keep track of which word comes first and then just add the counts afterwards. Using a hash for this count is scalable for as many words as you like to keep track of.
Files are streams of bytes. They don't understand lines at all. The line-end character is just another character in the stream.
So when the documentation for seek() talks about a "FILEHANDLE's position", it means the number of bytes through the file, not the number of lines. So seeking to a line number is never going to have the result that you want.
Instead, you should use the tell() function to find out which position you are at in the file and then use that value as the parameter that you send to seek().
It would work something like this:
# Lexical filehandle and three-arg open()
open my $in_fh, '<', $log1_file or die ...;
my $pos = tell $in_fh;
while (my $line = <$in_fh>) {
# Do various stuff.
# If you want to go back to the start of the line.
seek $in_fh, $pos, 0;
# Get the pos at the end of the current line
$pos = tell $in_fh;
}
Update: If you want to be able to go to any line in a file, then one approach might be to read through the file once and call tell() at the start of every line. You can then store those values in an array which you can use to get to the start of any line.
So I have encountered a problem while programming with PERL. I use a foreach loop to get some data out of the hash, so it has to loop through it.
The Code:
foreach $title (keys %FilterSPRINTHASH) {
$openSP = $FilterSPRINTHASH{$title}{openSP};
$estSP = $FilterSPRINTHASH{$title}{estSP};
$line = "'$title':{'openSP' : $openSP, 'estSP' : $estSP}\n";
print $outfile "$line\n";
}
The thing is, that I am creating a seperate File with the PERL's writting to a file expression, which will be a JSONP text (later used for HTML).
Back to the problem:
As JSONP requires comma's "," after every line that is not the last one, i had to put a comma at the end of line, however when the last line comes in, I have to remove the comma.
I have tried with CHOP function, but not sure where to put it, since if I put it at the end of foreach, it will just chop the comma in $line, but this wont chop it in the new file I created.
I have also tried with while (<>) statement, with no success.
Any ideas appreaciated.
BR
Using JSON module is far less error prone; no need to reinvent the wheel
use JSON;
print $outfile encode_json(\%FilterSPRINTHASH), "\n";
You can check if it is the last iteration of the loop, then remove the comma from line.
So something like
my $count = keys %FilterSPRINTHASH; #Get number of keys (scalar context)
my $loop_count = 1; #Use a variable to count number of iteration
foreach $title (keys %FilterSPRINTHASH){
$openSP = $FilterSPRINTHASH{$title}{openSP};
$estSP = $FilterSPRINTHASH{$title}{estSP};
$line = "'$title':{'openSP' : $openSP, 'estSP' : $estSP}\n";
if($loop_count == $count){
#this is the last iteration, so remove the comma from line
$line =~ s/,+$//;
}
print $outfile "$line\n";
$loop_count++;
}
i would approach this by storing your output in an array and then joining that with the line separators you wish:
my #output; # storage for output
foreach $title (keys %FilterSPRINTHASH) {
# create each line
my $line = sprintf "'%s':{'openSP' : %s, 'estSP' : %s}", $title, $FilterSPRINTHASH{$title}{openSP}, $FilterSPRINTHASH{$title}{estSP};
# and put it in the output container
push #output, $line;
}
# join all outputlines with comma and newline and then output
print $outfile (join ",\n", #output);
I'm trying to follow the example in the synopsis of HTML::PrettyPrinter. I corrected the typo to create a FileHandle:
my $fh = new FileHandle ">E:\\test.html";
Now the file gets created but I'm getting another error:
Can't call method "isa" on an undefined value at C:/Strawberry/perl/site/lib/HTML/PrettyPrinter.pm line 414.
Here is the code I have so far:
use HTML::TreeBuilder;
# generate a HTML syntax tree
my $tree = new HTML::TreeBuilder;
$tree->parse_file("E:\\file.html");
# modify the tree if you want
use HTML::PrettyPrinter;
my $hpp = new HTML::PrettyPrinter ('linelength' => 130,'quote_attr' => 1);
# configure
$hpp->set_force_nl(1,qw(body head)); # for tags
$hpp->set_force_nl(1,qw(#SECTIONS)); # as above
$hpp->set_nl_inside(0,'default!'); # for all tags
# format the source
my $linearray_ref = $hpp->format($tree);
print #$linearray_ref;
# alternative: print directly to filehandle
use FileHandle;
my $fh = new FileHandle ">E:\\test.html";
if (defined $fh) {
$hpp->select($fh);
$hpp->format();
undef $fh;
$hpp->select(undef),
}
This line is causing the error:
$hpp->format();
HTML::PrettyPrinter::format attempts to call isa on the first argument:
411 sub format {
412 my ($self, $element, $indent, $lar) = #_;
413 # $lar = line array ref
414 confess "Need an HTML::Element" unless $element->isa('HTML::Element');
...
Which causes the error you're getting if it is undef. Passing $tree (which isa HTML::Element) as the first argument populates the file correctly:
$hpp->format($tree);