Perl converting time based on text match - json

I am using Perl to read in variables from a json file and handle them accordingly. The spot I need help with is when I read a time in from the file that could look like the following:
"StartTime":"2015-07-08T03:38:08Z",
"EndTime":"2015-07-10T03:38:08Z"
This is easy to handle, however here is the tricky part:
"StartTime":"now-10",
"EndTime":"now+10"
I have a function which gets these variables from the json file and checks if the string contains the word "now". But after that, I'm not sure what to do. I'm trying to convert "now" to localtime(time), but it's getting ugly fast. Here is my code:
my $_StartTime = getFromJson("StartTime");
my $_EndTime = getFromJson("EndTime");
if($_StartTime =~ /now/) {
(my $sec, my $min, my $hour, my $mday, my $mon, my $year, my $wday, my $yday, my $isdst) = localtime(time);
my $now = sprintf("%04d-%02d-%02dT%02d:%02d:%02dZ", $year+1900, $mon+1, $mday, $hour, $min, $sec);
}
# end time is handled the same way
Am I on the right track? And if so, how can I add the "+/-10" after the "now" in the file? (Note: assume the +/-10 always refers to hours)

There are lots of good modules on the CPAN that could help in this instance. You don't need to use them but it's worth knowing about them nonetheless.
Firstly, JSON might make your life easier when parsing the JSON files as it has easy methods for converting the JSON into native Perl structures.
Secondly, the DateTime family of modules might make it easier to parse and manipulate the dates. Specifically, instead of using sprintf, you could use DateTime::Format::ISO8601 to parse the date:
my $dt = DateTime::Format::ISO8601->parse_datetime( $_StartTime );
DateTime has methods for accessing the day, year, month and so on. These are documented on the main module page.
You could then keep your special case for the now input and do something like:
# work out if it's addition or subtraction and grab the amount
# then use the appropriate DateTime function:
my $dt = DateTime->now()->add( seconds => 10 );
# or
my $dt = DateTime->now()->subtract( seconds => 10 );

Using POSIX::strftime will make your life easier.
use POSIX 'strftime';
my #test_times = qw[now+10 now now-10];
foreach my $start_time (#test_times) {
if (my ($adjust) = $start_time =~ /^now([-+]\d+)?/) {
$adjust //= 0;
$adjust *= 60 * 60; # Convert hours to seconds
my $time = strftime '%Y-%m-%dT%H:%M:%SZ', gmtime(time + $adjust);
say $time;
}
}
Thinking about it further, I think I'd prefer to use Time::Piece. The principle is almost identical.
use Time::Piece;
my #test_times = qw[now+10 now now-10];
foreach my $start_time (#test_times) {
if (my ($adjust) = $start_time =~ /^now([-+]\d+)?/) {
$adjust //= 0;
$adjust *= 60 * 60; # Convert hours to seconds
my $time = gmtime(time + $adjust);
say $time->strftime('%Y-%m-%dT%H:%M:%SZ');
}
}

I would change this to:
my $_StartTime = getFromJson("StartTime");
my $_EndTime = getFromJson("EndTime");
if($_StartTime =~ s/now//) {
my $time = time;
if ($_StartTime =~ /^([-+]?)([0-9]+)/) {
my ($sign, $number) = ($1, $2);
$time += ($sign eq '-' ? -1 : 1) * $number * 3_600;
}
(my $sec, my $min, my $hour, my $mday, my $mon, my $year, my $wday, my $yday, my $isdst) = localtime($rime);
$_StartTime = sprintf("%04d-%02d-%02dT%02d:%02d:%02dZ", $year+1900, $mon+1, $mday, $hour, $min, $sec);
}

You give little information about the format of the original data, and what result you want from this. I assume the code you show is to convert the times formatted with now to one that you recognize so that you can go on from there. But it's best to handle both formats in one place to generate the same final result regardless of the input
This program uses an imaginary JSON data structure and processes all elements inside it. The core is the use of the Time::Piece module, which will parse and format times for you and do date/time arithmetic
I have encapsulated the code that processes both sorts of time values in a subroutine convert_time which returns a Time::Piece object. The code just uses the module's own stringify method to make the value readable, but you can generate any form of string you want using the object's methods
use strict;
use warnings 'all';
use feature 'say';
use JSON 'from_json';
use Time::Piece;
use Time::Seconds 'ONE_HOUR';
my $json = <<END;
[
{
"StartTime": "2015-07-08T03:38:08Z",
"EndTime": "2015-07-10T03:38:08Z"
},
{
"StartTime": "now-10",
"EndTime": "now+10"
}
]
END
my $data = from_json($json);
for my $item ( #$data ) {
for my $key ( keys %$item ) {
my $time = $item->{$key};
say "$key $time";
my $ans = convert_time($time);
print $ans, "\n\n";
}
}
sub convert_time {
my ($time) = #_;
if ( $time =~ /now([+-]\d+)/ ) {
return localtime() + $1 * ONE_HOUR;
}
else {
return Time::Piece->strptime($time, '%Y-%m-%dT%H:%M:%SZ');
}
}
output
StartTime 2015-07-08T03:38:08Z
Wed Jul 8 03:38:08 2015
EndTime 2015-07-10T03:38:08Z
Fri Jul 10 03:38:08 2015
StartTime now-10
Wed Jan 6 05:57:04 2016
EndTime now+10
Thu Jan 7 01:57:04 2016

Related

my input file date format changes on daily basis

I need help please, my $inputfile changes on daily basis which gets generated and store under /tmp directory. File format date as follows.
/tmp
570572 Sep 13 21:02 sessions_record_2021-09-13_210052.csv
570788 Sep 14 09:01 sessions_record_2021-09-14_090041.csv
I'm not sure how to pick it up as an input file instead of hardcoded in my script
#!/usr/bin/perl
use strict; use warnings;
use Tie::Array::CSV;
use Data::Dumper;
use Date::Parse;
use POSIX qw(strftime);
my $hours = 1;
my $timenow = time;
my $inputfile = "sessions_record_2021-09-14_090041.csv";
tie my #sessions_record, 'Tie::Array::CSV', $inputfile, {
tie_file => { recsep => "\r\n" },
text_csv => { binary => 1 }
};
tie my #incidentidlist, 'Tie::Array::CSV', 'incidentidlist.csv';
#incidentidlist = map {
([$$_[4] =~ /\A([^\s]+)/, $$_[4], $$_[18], ($timenow -
str2time($$_[18])) / 60 / 60])
} grep {
$$_[0] =~ /^ServiceINC/ && ($timenow - str2time($$_[18])) / 60 / 60 > $hours
} #sessions_record;
Perl sort function on glob will produce sorted array and you interested in last element which can be addressed with index -1.
use strict;
use warnings;
use feature 'say';
my $in_file = (sort glob('/tmp/sessions_record_*.csv'))[-1];
say $in_file;
If you interested in today's file localtime can be an assistance to form a filename $fname.
use strict;
use warnings;
use feature 'say';
my($mask,$fname);
my($mday,$mon,$year) = (localtime)[3..5];
$year += 1900;
$mon += 1;
$mask = sprintf('/tmp/sessions_record_%4d-%02d-%02d_*.csv', $year, $mon, $mday);
$fname = (glob($mask))[0];
say 'File: ' . $fname;
say '-' x 45;
open my $fh, '<', $fname
or die "Couldn't open $fname";
print while <$fh>;
close $fh;
You can use opendir to open a directory and readdir to read it. For each file accessed you can check if it has the correct format (as per simbabque's comment) and add it to an array.
Then you can sort your array.
Due to the naming convention the latest file will always sort as the 'largest' value in your sort.
You can red more about sorting (if you need to) at https://www.perltutorial.org/perl-sort/

Using Text::CSV on a String Containing Quotes

I have pored over this site (and others) trying to glean the answer for this but have been unsuccessful.
use Text::CSV;
my $csv = Text::CSV->new ( { binary => 1, auto_diag => 1 } );
$line = q(data="a=1,b=2",c=3);
my $csvParse = $csv->parse($line);
my #fields = $csv->fields();
for my $field (#fields) {
print "FIELD ==> $field\n";
}
Here's the output:
# CSV_XS ERROR: 2034 - EIF - Loose unescaped quote # rec 0 pos 6 field 1
FIELD ==>
I am expecting 2 array elements:
data="a=1,b=2"
c=3
What am I missing?
You may get away with using Text::ParseWords. Since you are not using real csv, it may be fine. Example:
use strict;
use warnings;
use Data::Dumper;
use Text::ParseWords;
my $line = q(data="a=1,b=2",c=3);
my #fields = quotewords(',', 1, $line);
print Dumper \#fields;
This will print
$VAR1 = [
'data="a=1,b=2"',
'c=3'
];
As you requested. You may want to test further on your data.
Your input data isn't "standard" CSV, at least not the kind that Text::CSV expects and not the kind that things like Excel produce. An entire field has to be quoted or not at all. The "standard" encoding of that would be "data=""a=1,b=2""",c=3 (which you can see by asking Text::CSV to print your expected data using say).
If you pass the allow_loose_quotes option to the Text::CSV constructor, it won't error on your input, but it won't consider the quotes to be "protecting" the comma, so you will get three fields, namely data="a=1, b=2" and c=3.

Can I use Text::CSV_XS to parse a csv-format string without writing it to disk?

I am getting a "csv file" from a vendor (using their API), but what they do is just spew the whole thing into their response. It wouldn't be a significant problem except that, of course, some of those pesky humans entered the data and put in "features" like line breaks. What I am doing now is creating a file for the raw data and then reopening it to read the data:
open RAW, ">", "$rawfile" or die "ERROR: Could not open $rawfile for write: $! \n";
print RAW $response->content;
close RAW;
my $csv = Text::CSV_XS->new({ binary=>1,always_quote=>1,eol=>$/ });
open my $fh, "<", "$rawfile" or die "ERROR: Could not open $rawfile for read: $! \n";
while ( $line = $csv->getline ($fh) ) { ...
Somehow this seems ... inelegant. It seems that I ought to be able to just read the data from the $response->content (multiline string) as if it were a file. But I'm drawing a total blank on how do this.
A pointer would be greatly appreciated.
Thanks,
Paul
You could use a string filehandle:
my $data = $response->content;
open my $fh, "<", \$data or croak "unable to open string filehandle : $!";
my $csv = Text::CSV_XS->new({ binary=>1,always_quote=>1,eol=>$/ });
while ( $line = $csv->getline ($fh) ) { ... }
Yes, you can use Text::CSV_XS on a string, via its functional interface
use warnings;
use strict;
use feature 'say';
use Text::CSV_XS qw(csv); # must use _XS version
my $csv = qq(a,line\nand,another);
my $aoa = csv(in => \$csv)
or die Text::CSV->error_diag;
say "#$_" for #aoa;
Note that this indeed needs Text::CSV_XS (normally Text::CSV works but not with this).
I don't know why this isn't available in the OO interface (or perhaps is but is not documented).
While the above parses the string directly as asked, one can also lessen the "inelegant" aspect in your example by writing content directly to a file as it's acquired, what most libraries support like with :content_file option in LWP::UserAgent::get method.
Let me also note that most of the time you want the library to decode content, so for LWP::UA to use decoded_content (see HTTP::Response).
I cooked up this example with Mojo::UserAgent. For the CSV input I used various data sets from the NYC Open Data. This is also going to appear in the next update for Mojo Web Clients.
I build the request without making the request right away, and that gives me the transaction object, $tx. I can then replace the read event so I can immediately send the lines into Text::CSV_XS:
#!perl
use v5.10;
use Mojo::UserAgent;
my $ua = Mojo::UserAgent->new;
my $url = ...;
my $tx = $ua->build_tx( GET => $url );
$tx->res->content->unsubscribe('read')->on(read => sub {
state $csv = do {
require Text::CSV_XS;
Text::CSV_XS->new;
};
state $buffer;
state $reader = do {
open my $r, '<:encoding(UTF-8)', \$buffer;
$r;
};
my ($content, $bytes) = #_;
$buffer .= $bytes;
while (my $row = $csv->getline($reader) ) {
say join ':', $row->#[2,4];
}
});
$tx = $ua->start($tx);
That's not as nice as I'd like it to be because all the data still show up in the buffer. This is slightly more appealing, but it's fragile in the ways I note in the comments. I'm too lazy at the moment to make it any better because that gets hairy very quickly as you figure out when you have enough data to process a record. My particular code isn't as important as the idea that you can do whatever you like as the transactor reads data and passes it into the content handler:
use v5.10;
use strict;
use warnings;
use feature qw(signatures);
no warnings qw(experimental::signatures);
use Mojo::UserAgent;
my $ua = Mojo::UserAgent->new;
my $url = ...;
my $tx = $ua->build_tx( GET => $url );
$tx->res->content
->unsubscribe('read')
->on( read => process_bytes_factory() );
$tx = $ua->start($tx);
sub process_bytes_factory {
return sub ( $content, $bytes ) {
state $csv = do {
require Text::CSV_XS;
Text::CSV_XS->new( { decode_utf8 => 1 } );
};
state $buffer = '';
state $line_no = 0;
$buffer .= $bytes;
# fragile if the entire content does not end in a
# newline (or whatever the line ending is)
my $last_line_incomplete = $buffer !~ /\n\z/;
# will not work if the format allows embedded newlines
my #lines = split /\n/, $buffer;
$buffer = pop #lines if $last_line_incomplete;
foreach my $line ( #lines ) {
my $status = $csv->parse($line);
my #row = $csv->fields;
say join ':', $line_no++, #row[2,4];
}
};
}

Calculations cannot be performed by passed value from two different subroutines to a new subroutine: Perl

I used an anonymous hash to pass value from two different subroutines to a new subroutine. But, now I'm not able to perform calculations using the passed variables.
use warnings;
use strict;
use feature 'say';
use DBI;
use autodie;
use Data::Dumper;
use CGI;
print "Enter sequence";
my $seq = <STDIN>;
chomp $seq;
$len = length $seq;
my $f = nuc($seq);
perc({ len => $len });
sub nuc {
my ($c) = #_;
chomp $c;
my $len = length $c;
for (my $i = 0; $i< = $len; $i++) {
my $seq2 = substr($c, $i, 1);
$nuc=$nuc . $seq2;
chomp $nuc;
}
my $l = perc({nuc => $nuc});
}
sub perc {
my $params = shift;
my $k = $params->{nuc};
my $w = $params->{len};
my $db = "hnf1a";
my $user = "root";
my $password = "";
my $host = "localhost";
my $dbh = DBI->connect("DBI:mysql:database=$db:$host",$user,$password);
my $sth = $dbh->prepare('SELECT COUNT(*) FROM mody where nm = ?');
for (1..100) {
$sth->execute(int(rand(10)));
}
chomp (my $input = $k);
my #num = split /':'/, $input;
for my $num(#num) {
say "rows matching input nuc <$num>:";
$sth->execute($num);
my $count = $sth->fetchrow_array;
say "$count";
$u += $count;
}
}
$h = $u / $w;
print $h;
I passed the variables : $nuc and $len to the last subroutine 'perc' by declaring an anonymous hash.
When I use these variables to perform calculations I don't get a proper answer.
For the above division performed I got a statement as 'Illegal division'.
Please help me out. Thanks in advance.
You are making two separate calls to perc, each with only one of the required values in the hash. You can't do that: the subroutine won't "remember" a value passed to it across separate calls unless you write the code to do that
You need to collect all the values and pass them in a single call to perc
There are rather a lot of misunderstandings here. Let's go through your code.
use CGI;
Using CGI.pm is a bit dated, but it's not a terrible idea if you're writing a CGI program. But this isn't a CGI program, so this isn't necessary.
print "Enter sequence";
my $seq = <STDIN>;
chomp $seq;
$len = length $seq;
my $f = nuc($seq);
This looks OK. You prompt the user, get some input, remove the newline from the end of the input, get the length of the input and then pass your input into nuc().
So, let's look at nuc() - which could probably have a better name!
sub nuc {
my ($c) = #_;
chomp $c;
my $len = length $c;
for (my $i = 0; $i< = $len; $i++) {
my $seq2 = substr($c, $i, 1);
$nuc=$nuc . $seq2;
chomp $nuc;
}
my $l = perc({nuc => $nuc});
}
You get the parameter that has been passed in and remove the newline from the end of it (which does nothing as this is $seq which has already had its newline removed). You then get the length of this string (again!)
Then it gets very strange. Firstly, there's a syntax error (< = should be <=). Then you use a C-style for loop together with substr() too... well, basically you just copy $c to $nuc in a really inefficient manner. So this subroutine could be written as:
sub nuc {
my ($c) = #_;
$nuc = $c;
my $l = perc({ nuc => $nuc });
}
Oh, and I don't know why you chomp($nuc) each time round the loop.
Two more strange things. Firstly, you don't declare $nuc anywhere, and you have use strict in your code. Which means that this code doesn't even compile. (Please don't waste our time with code that doesn't compile!) And secondly, you don't explicitly return a value from nuc(), but you store the return value in $f. Because of the way Perl works, this subroutine will return the value in $l. But it's best to be explicit.
Then there's your perc() subroutine.
sub perc {
my $params = shift;
my $k = $params->{nuc};
my $w = $params->{len};
my $db = "hnf1a";
my $user = "root";
my $password = "";
my $host = "localhost";
my $dbh = DBI->connect("DBI:mysql:database=$db:$host",$user,$password);
my $sth = $dbh->prepare('SELECT COUNT(*) FROM mody where nm = ?');
for (1..100) {
$sth->execute(int(rand(10)));
}
chomp (my $input = $k);
my #num = split /':'/, $input;
for my $num(#num) {
say "rows matching input nuc <$num>:";
$sth->execute($num);
my $count = $sth->fetchrow_array;
say "$count";
$u += $count;
}
}
You get the hash ref which is passed in an store that in $params. You then extract the nuc and len values from that hash and store them in variables called $k and $w (you really need to improve your variable and subroutine names!) But each call to perc only has one of those values set - so only one of your two variables get a value, the other will be undef.
So then you connect to the database. And you run a select query a hundred times passing in random integers between 0 and 9. And ignore the value returned from the select statement. Which is bizarre and pointless.
Eventually, you start doing something with one of your input parameters, $k (the other, $w, is completely ignored). You copy it into another scalar variable before splitting it into an array. You then run the same SQL select statement once for each element in that array and add the number you get back to the running total in $u. And $u is another variable that you never declare, so (once again) this code doesn't compile.
Outside of your subroutines, you then do some simple maths with $u (an undeclared variable) and $w (a variable that was declared in a different scope) and store the result in $h (another undeclared variable).
I really don't understand what this code is supposed to do. And, to be honest, I don't think you do too. If you're at school, then you need to go back to your teacher and say that you have no idea what you are doing. If you're in a job, you need to tell your boss that you're not the right person for this task.
Either way, if you want to be a programmer, you need to go right back to the start and cover the very basics again.

perl json map create issue

i'm new in perl (i don't know almost nothing :-))
I had a script which worked on my local computer but not in server :-(
It looks like:
my $json = JSON->new;
my $json_map = [];
for (my $i = 0; $i <= $#commitlist; $i++) {
my %co = %{$commitlist[$i]};
...
push $json_map, {esc_html($co{'id'})=> {author=>esc_html($co{'author'}),pubDate=>$cd{'rfc2822'},link=>$co_url, title=>esc_html($co{'title'})}};
}
my $output = $json->encode($json_map);
print $output . "\n";
It worked and result was like this:
[{"id":{"author":"johny","title":"some title","link":"http://127.0.0.1","pubDate":"Fri, 14 Mar 2014 12:31:17 +0000"}}]
But now I have on server following problem (there is perl 5.8.8 version, but i would like to fix it in the script):
Type of arg 1 to push must be array (not private variable) at XX line XX, near "};"
When you do a push you need to de-reference the first argument (i.e. the array you're pushing things into). Like this:
push #$json_map, ...
Hope this helps.