Extract day of week using perl and mysql date format - mysql

I am looping through a range of consecutive dates. I need to find out which are weekends so they can be discarded. One method would be to determine the day of week via perl (what my script is written in) or query with each pass through the loop. There will never be more than 30 dates, usually 5 or less to check. Would it be more efficient to load up Date::Time and use perl or run a Query without needing modules? If perl is the best method I could use a little help extracting the day of week from the YYYY-MM-DD format. I'd be ok with number 1-7 or shortname mon-sun.
2012-05-01
2012-05-02
2012-05-03
2012-05-04
2012-05-05
2012-05-06
not sure if this is possible but perhaps a more suitable solution would be to write a query (since I know the start and end and they are consecutive) to count days between x and y where dayofweek NOT IN(6,7)

See DateTime::Format::Strptime and DateTime.
use 5.010;
use strictures;
use DateTime::Format::Strptime qw();
my $parser = DateTime::Format::Strptime->new(pattern => '%F');
for my $date (qw(
2012-05-01
2012-05-02
2012-05-03
2012-05-04
2012-05-05
2012-05-06
)) {
my $dow = $parser->parse_datetime($date)->day_of_week;
say "$date is a weekend day" if 6 == $dow or 7 == $dow;
}

MySQL has a day of week function you can use directly.

The most obvious solution is to use the Time::Piece module, which has been a core module of Perl since v5.9 and so probably doesn't need installing on your system.
The wday method returns a numeric day of week where 1 == Sunday, so for the weekend you are looking for values of 7 (Saturday) or 1. This can be adjusted so that Saturday is represented by zero instead (and Sunday by 1) by writing
my $dow = $tp->wday % 7;
after which the test for a weekend is simply $dow < 2.
Here is some code to demonstrate.
use strict;
use warnings;
use Time::Piece;
while (<DATA>) {
chomp;
my $tp = Time::Piece->strptime($_, '%Y-%m-%d');
my $dow = $tp->wday % 7;
print $_;
print " (weekend)" if $dow < 2;
print "\n";
}
__DATA__
2012-05-01
2012-05-02
2012-05-03
2012-05-04
2012-05-05
2012-05-06
output
2012-05-01
2012-05-02
2012-05-03
2012-05-04
2012-05-05 (weekend)
2012-05-06 (weekend)

You could use the core Time::Local module and then compute the weekday using localtime. Weekday 0 corresponds to Sunday, and 6 is Saturday.
#! /usr/bin/env perl
use strict;
use warnings;
use Time::Local;
my #dates = qw(
2012-05-01
2012-05-02
2012-05-03
2012-05-04
2012-05-05
2012-05-06
);
my #days = qw/ Sun Mon Tue Wed Thu Fri Sat /;
foreach my $date (#dates) {
my($yyyy,$mm,$dd) = split /-/, $date;
my $time_t = timelocal 0, 0, 0, $dd, $mm-1, $yyyy-1900;
my $wday = (localtime $time_t)[6];
my $weekend = ($wday == 0 || $wday == 6) ? " *" : "";
print "$date: $days[$wday] ($wday)$weekend\n";
}
Output:
2012-05-01: Tue (2)
2012-05-02: Wed (3)
2012-05-03: Thu (4)
2012-05-04: Fri (5)
2012-05-05: Sat (6) *
2012-05-06: Sun (0) *
For fun, you could go Swiss Army Chainsaw and scrape the output of the cal utility.
#! /usr/bin/env perl
use strict;
use warnings;
use 5.10.0; # for smart matching
sub weekday_type {
my($date) = #_;
die "$0: unexpected date '$date'"
unless my($yyyy,$mm,$dd) =
$date =~ /^([0-9]{1,4})-([0-9]{1,2})-([0-9]{1,2})$/;
my $calendar = `cal -m $mm $yyyy`;
die "$0: cal -m $mm $yyyy failed" if $?;
for (split /\n/, $calendar) {
if (/^ \s* [0-9]{1,2} (?: \s+ [0-9]{1,2})* \s*$/x) {
my #dates = split;
my #weekend = splice #dates, #dates > 1 ? -2 : -1;
return "weekend" if ($dd+0) ~~ #weekend;
}
}
"weekday";
}
Use it as in
my #dates = qw(
2012-05-01
2012-05-02
2012-05-03
2012-05-04
2012-05-05
2012-05-06
);
my #days = qw/ Sun Mon Tue Wed Thu Fri Sat /;
foreach my $date (#dates) {
my $type = weekday_type $date;
print "$date: $type\n";
}
Output:
2012-05-01: weekday
2012-05-02: weekday
2012-05-03: weekday
2012-05-04: weekday
2012-05-05: weekend
2012-05-06: weekend
I don’t recommend doing it this way in production.

use Date::Simple qw/date/;
use Date::Range;
my ( $start, $end ) = ( date('2012-05-02'), date('2012-05-16') );
my $range = Date::Range->new( $start, $end );
my #all_dates = $range->dates;
foreach my $d (#all_dates) {
my $date = Date::Simple->new($d);
print $date->day_of_week." ".$d."<br />";
}
#--OUTPUT--#
3 2012-05-02
4 2012-05-03
5 2012-05-04
6 2012-05-05
0 2012-05-06
1 2012-05-07
2 2012-05-08
3 2012-05-09
4 2012-05-10
5 2012-05-11
6 2012-05-12
0 2012-05-13
1 2012-05-14
2 2012-05-15
3 2012-05-16

Related

How can I use Perl's format function?

I'm learning Perl formatting by following the tutorial at Perl - Formats, but when I type their example into my IDE:
format EMPLOYEE =
===================================
#<<<<<<<<<<<<<<<<<<<<<< #<<
$name $age
######.##
$salary
===================================
.
select(STDOUT);
$~ = EMPLOYEE;
#n = ("Ali", "Raza", "Jaffer");
#a = (20,30, 40);
#s = (2000.00, 2500.00, 4000.000);
$i = 0;
foreach (#n) {
$name = $_;
$age = $a[$i];
$salary = $s[$i++];
write;
}
I get the error:
Scalar found where operator expected at .\qb.pl line 7, near "$name $age"
(Missing operator before $age?)
syntax error at .\qb.pl line 7, near "$name $age"
Execution of .\qb.pl aborted due to compilation errors.
I'm using Perl 5, version 30, Subversion 2 (v5.30.2) built for MSWin32-x64-multi-thread.
When I run your code on 5.24, I see this warning message:
Use of comma-less variable list is deprecated at...
which points to the $name $age line as well.
When I enable diagnostics, I get this explanation:
(D deprecated) The values you give to a format should be
separated by commas, not just aligned on a line.
When I add the comma as follows:
$name,$age
the warning goes away, and I get this output:
===================================
Ali 20
2000.00
===================================
===================================
Raza 30
2500.00
===================================
===================================
Jaffer 40
4000.00
===================================
This warning became an error in 5.28, according to perl5280delta.
Your first mistake was in thinking that formats were a useful tool that would solve your problem. Formats have been largely ignored for almost as long as I've been using Perl. You'll probably find that using Perl6::Form is a better approach.
Your second mistake was in thinking that Tutorials Point was a good place to get information about anything. The tutorials there are written by people who seem to know next to nothing about their subject and (as you have seen here) the examples are riddled with typos that makes them next to useless.
If you're determined to use Perl formats, then the perlform manual page would be the best place to start.
Perhaps the following code sample is easier to read. The output is also easier to comprehend at a quick glance:
use strict;
use warnings;
my($name, $age, $salary);
while(<DATA>) {
($name, $age, $salary) = split;
write;
}
format STDOUT_TOP =
Employee Age Salary
----------------------------------------
.
format STDOUT =
#<<<<<<<<<<<<<<<<<<<<<< #<< ######.##
$name, $age, $salary
.
__DATA__
Ali 20 2000.00
Raza 30 2500.00
Jaffer 40 4000.00
Output
Employee Age Salary
----------------------------------------
Ali 20 2000.00
Raza 30 2500.00
Jaffer 40 4000.00
There are many ways to provide input data. One of them is a hash to keep related data.
use strict;
use warnings;
my %employees = (
Ali => { age => 20, salary => 2000.00},
Raza => { age => 30, salary => 2500.00},
Jaffer => { age => 40, salary => 4000.00}
);
my($name, $age, $salary);
for $name (sort keys %employees) {
($age, $salary) = #{$employees{$name}}{qw/age salary/};
write;
}
format STDOUT_TOP =
Employee Age Salary
----------------------------------------
.
format STDOUT =
#<<<<<<<<<<<<<<<<<<<<<< #<< ######.##
$name, $age, $salary
.
Output
Employee Age Salary
----------------------------------------
Ali 20 2000.00
Jaffer 40 4000.00
Raza 30 2500.00

Group records by day including days in between that contain no records in MySQL

I have to display how many votes were issued on each day (from the first vote, until the last vote) users voted for poll 2.
My current query works fine; but I'm having trouble when a specific day doesn't have any votes in it. For example, for poll 2, this should be the result:
May, 11 2017 = 1
May, 12 2017 = 0
May, 13 2017 = 0
May, 14 2017 = 0
May, 15 2017 = 0
May, 16 2017 = 0
May, 17 2017 = 1
May, 18 2017 = 0
May, 19 2017 = 0
May, 20 2017 = 2
...but instead I'm getting this:
May, 11 2017 = 1
May, 17 2017 = 1
May, 20 2017 = 2
So, what I need is that all days that contain no records (between the first and last vote) also appear in the results. This is my current query:
SELECT DATE(poll_vote.date_insert) AS date_insert,
COUNT(poll_vote.id_vote) AS q
FROM poll_vote WHERE poll_vote.id_poll = 2
GROUP BY DATE(date_insert) ORDER BY date_insert
Here's the SQL Fiddle with the example data. Thanks!
As suggested by #Strawberry, I ended up writing a solution in php instead of mysql. For anyone interested, here's the code.
This solution only returns 0 in 1 day in between blank dates because that's what I needed. Example:
May, 11 2017 = 1
May, 16 2017 = 0
May, 17 2017 = 1
May, 19 2017 = 0
May, 20 2017 = 2
That allows me to display a trending line chart just like this one:
$sql = 'SELECT DATE(poll_vote.date_insert) AS date_insert, COUNT(poll_vote.id_vote) AS q FROM poll_vote WHERE poll_vote.id_poll = :id_poll GROUP BY DATE(date_insert) ORDER BY date_insert';
$stmt = cnn()->prepare($sql);
if($id_poll) $stmt->bindValue(':id_poll', $id_poll, PDO::PARAM_INT);
$stmt->execute();
$data = $stmt->fetchAll();
# insert blank dates between existing dates
$hotness = array();
foreach($data as $item) {
if(!$temp) {
$temp = $item['date_insert'];
} else {
$date_past = new DateTime($temp);
$date_now = new DateTime($item['date_insert']);
$diff = $date_now->diff($date_past)->format("%a");
if($diff > 1) {
$date_new = new DateTime($item['date_insert']);
$date_new->modify('-1 day');
$hotness[] = array(
'date_insert' => $date_new->format('Y-m-d'),
'q' => 0
);
}
}
$hotness[] = array(
'date_insert' => $item['date_insert'],
'q' => $item['q']
);
}
# final result
print_r($hotness);

How HTML works in awk command in shell scripting?

I have a script called "main.ksh" which returns "output.txt" file and I am sending that file via mail (list contains 50+ records, I just give 3 records for example).
mail output I am getting is: (10 cols)
DATE FEED FILE_NAME JOB_NAME SCHEDULED TIME SIZE COUNT STATUS
Dec 17 INVEST iai guxmow080 TUE-SAT 02:03 0.4248 4031 On_Time
Dec 17 SECURITIES amltxn gdcpl3392 TUE-SAT 02:03 0.0015 9 Delayed
Dec 17 CONNECTED amlbene gdcpl3392 TUE-SAT 02:03 0.0001 1 No_Records
output with perfect coloring: (6 cols only)
DATE FEED FILE_NAME JOB_NAME SCHEDULED TIME SIZE COUNT STATUS
Dec 17 INVEST iai guxmow080 On_Time(green color)
Dec 17 SECURITIES amltxn gdcpl3392 Delayed(red color)
Dec 17 CONNECTED amlbene gdcpl3392 No_Records(yellow color)
I am implementing coloring for Delayed, On_Time and No_Records field and I wrote below script which gives me bottom output.
awk 'BEGIN {
print "<html>" \
"<body bgcolor=\"#333\" text=\"#f3f3f3\">" \
"<pre>"
}
NR == 1 { print $0 }
NR > 1 {
if ($NF == "Delayed") color="red"
else if ($NF == "On_time") color="green"
else if ($NF == "No_records") color="yellow"
else color="#003abc"
Dummy=$0
sub("[^ ]+$","",Dummy)
print Dummy "<span style=\"color:" color (bold ? ";font-weight:bold" : "")(size ? ";font-size:size" : "") (italic ? ";font-style:italic" : "") "\">" $NF "</span>"
}
END {
print "</pre>" \
"</body>" \
"</html>"
}
' output.txt > output.html
There are 4 columns are skipped automatically.
| date | feed_names | file_names | job_names | scheduled_time| timestamp| size| count| status |
Dec 19 ISS_BENEFICIAL_OWNERS_FEED amlcpbo_iss_20161219.txt gdcpl3392_uxmow080_ori_isz_dat WEEK_DAYS 00:03 9.3734 34758 On_Time
Dec 19 ISS_INVESTORS_FEED amlinvest_iss_20161219.txt gdcpl3392_uxmow080_ori_isz_dat WEEK_DAYS 00:01 0.0283 82 On_Time
Dec 19 ISS_TRANSACTIONS_FEED amltran_iss_1_20161219.txt gdcpl3392_uxmow080_ori_isz_dat WEEK_DAYS 00:12 14.022 36532 DELAYED
Dec 19 ISS_TRANSACTIONS_FEED amltran_iss_5_20161219.txt gdcpl3392_uxmow080_ori_isz_dat WEEK_DAYS 00:23 0.0010 3 DELAYED
Dec 19 IBS_CUSTOMER_FEED ibscust_aml_***_20161219.txt gdcpl3392_uxmow080_ori_sfp_ibc WEEK_DAYS (11 _out_of_11) -NA- ARRIVED
Dec 19 IBS_DDA_NOSTRO_ACCOUNT_FEED ibsacct_aml_***_20161219.txt gdcpl3392_uxmow080_ori_sfp_ibc WEEK_DAYS (44 _out_of_44) -NA- ARRIVED
Dec 19 GP__TRANSACTIONS_FEED amltrans__20161219.txt gdcpl3392_uxmow080_ori_sfp_glo WEEK_DAYS (3 _out_of_30) -NA- ARRIVED
But when I am trying to print in a sequential order by using below command
awk '{printf("%-5s%s\t%-33s%-35s%-39s%s\t%s%-3s\t%s\t%s\n", $1,$2,$3,$4,$5,$6,$7,$8,$9,$10)}' output.txt, I am getting the output in a sequential format
but 4 cols are skipped. Kindly suggest!!!
| date | feed_names | file_names | job_names | scheduled_time| timestamp| size| count| status |
Dec 19 ISS_BENEFICIAL_OWNERS_FEED amlcpbo_iss_20161219.txt gdcpl3392_uxmow080_ori_isz_dat On_Time
Dec 19 ISS_INVESTORS_FEED amlinvest_iss_20161219.txt gdcpl3392_uxmow080_ori_isz_dat On_Time
Dec 19 ISS_TRANSACTIONS_FEED amltran_iss_1_20161219.txt gdcpl3392_uxmow080_ori_isz_dat DELAYED
Dec 19 ISS_TRANSACTIONS_FEED amltran_iss_5_20161219.txt gdcpl3392_uxmow080_ori_isz_dat DELAYED
Dec 19 IBS_CUSTOMER_FEED ibscust_aml_***_20161219.txt gdcpl3392_uxmow080_ori_sfp_ibc ARRIVED
Dec 19 IBS_DDA_NOSTRO_ACCOUNT_FEED ibsacct_aml_***_20161219.txt gdcpl3392_uxmow080_ori_sfp_ibc ARRIVED
Dec 19 GP__TRANSACTIONS_FEED amltrans__20161219.txt gdcpl3392_uxmow080_ori_sfp_glo YET_TO_RECEIVE

Get date without offset

I'm using MySQL to store my data. I'm recorded dates with different offset according to each timezone. What is important on those dates are only "effective" time. Here is an example:
date1 = "Thu, 27 Oct 2016 07:00:00 CEST +02:00"
# I want to get 700
date2 = "Thu, 27 Oct 2016 22:00:00 CEST +02:00"
# I want to get 2200
I would like to get these values to compare them with my current time. Then if it's currently "2016-11-17 10:12:00 +0100", I would like to compare on a where clause:
"1012" > "2200"
# or
"1012" > "700"
Here is a fake record value:
# my_table
# | id | my_hour
# | XX | "Thu, 27 Oct 2016 07:00:00 CEST +02:00"
# | XY | "Thu, 27 Oct 2016 22:00:00 CEST +11:00"
Here is my code:
hour = Time.zone.now
# "2016-11-18 21:10:00 +0100"
where("CAST(DATE_FORMAT(my_hour,'%H%i') AS UNSIGNED) < ?", hour.strftime("%H%M").to_i)
# => Get XX record?
Use strftime to format date in Rails.
hour = Time.zone.now
MyTable.where("cast(strftime('%H%M', my_hour) as int) < ?", hour.strftime("%H%M").to_i)
here MyTable is your Model, my_hour is your date column and hour is current time.

Timestamp to Epoch in a CSV file with GAWK

Looking to convert human readable timestamps to epoch/Unix time within a CSV file using GAWK in preparation for loading into a MySQL DB.
Data Example:
{null};2013-11-26;Text & Device;Location;/file/path/to/;Tuesday, November 26 12:17 PM;1;1385845647
Looking to take column 6, Tuesday, November 26 12:17 PM, and convert to epoch time for storage. All times shown will be in EST format. I realize AWK is the tool for this, but can't quite seem to structure the command. Currently have:
cat FILE_IN.CSV | awk 'BEGIN {FS=OFS=";"}{$6=strftime("%s")} {print}'
However this returns:
{null};2013-11-26;Text & Device;Location;/file/path/to/;1385848848;1;1385845647
Presumably, this means I'm calling the current epoch time (1385848848 was current epoch at time of execution) and not asking strftime to convert the string; but I can't imagine another way to doing this.
What is the proper syntax for gawk/strftime to convert an existing timestamp to epoch?
Edit: This question seems loosely related to How do I use output from awk in another command?
$ cat file
{null};2013-11-26;Text & Device;Location;/file/path/to/;Tuesday, November 26 12:17 PM;1;1385845647
$ gawk 'BEGIN{FS=OFS=";"} {gsub(/-/," ",$2); $2=mktime($2" 0 0 0")}1' file
{null};1385445600;Text & Device;Location;/file/path/to/;Tuesday, November 26 12:17 PM;1;1385845647
Here's how to generally convert a date from any format to seconds since the epoch using your current format as an example and with comments to show the conversion process step by step:
$ cat tst.awk
function cvttime(t, a) {
split(t,a,/[,: ]+/)
# 2013 Tuesday, November 26 10:17 PM
# =>
# a[1] = "2013"
# a[2] = "Tuesday"
# a[3] = "November"
# a[4] = "26"
# a[5] = "10"
# a[6] = "17"
# a[7] = "PM"
if ( (a[7] == "PM") && (a[5] < 12) ) {
a[5] += 12
}
# => a[5] = "22"
a[3] = substr(a[3],1,3)
# => a[3] = "Nov"
match("JanFebMarAprMayJunJulAugSepOctNovDec",a[3])
a[3] = (RSTART+2)/3
# => a[3] = 11
return( mktime(a[1]" "a[3]" "a[4]" "a[5]" "a[6]" 0") )
}
BEGIN {
mdt ="Tuesday, November 26 10:17 PM"
secs = cvttime(2013" "mdt)
dt = strftime("%Y-%m-%d %H:%M:%S",secs)
print mdt ORS "\t-> " secs ORS "\t\t-> " dt
}
$ awk -f tst.awk
Tuesday, November 26 10:17 PM
-> 1385525820
-> 2013-11-26 22:17:00
I'm sure you can modify that for the current problem.
Also, if you don't have gawk you can write the cvttime() function as (borrowing #sputnik's date command string):
$ cat tst2.awk
function cvttime(t, cmd,secs) {
cmd = "date -d \"" t "\" '+%s'"
cmd | getline secs
close(cmd)
return secs
}
BEGIN {
mdt ="Tuesday, November 26 10:17 PM"
secs = cvttime(mdt)
dt = strftime("%Y-%m-%d %H:%M:%S",secs)
print mdt ORS "\t-> " secs ORS "\t\t-> " dt
}
$
$ awk -f tst2.awk
Tuesday, November 26 10:17 PM
-> 1385525820
-> 2013-11-26 22:17:00
I left srtftime() in there just to show that the secs was correct - replace with date as you see fit.
For the non-gawk version, you just need to figure out how to get the year into the input month/date/time string in a way that date understands if that maters to you - shouldn't be hard.
You can convert date to epoch with this snippet :
$ date -d 'Tuesday, November 26 12:17 PM' +%s
1385464620
So finally :
awk -F";" '{system("date -d \""$6"\" '+%s'")}' file
Thanks #Keiron for the snippet.