I'm triyng to format the first column of a CSV which is a unix timestamp in milliseconds to a format like this command:
date -d #$( echo "($line_date + 500) / 1000" | bc)
where $line_date is something like 1487693882310
And my file has this information:
1487152859086,,,,,,localhost.localdomain,ServerUpDown,ServerUp,,,,,,, ,,,,
1487613634268,,,,,,localhost.localdomain,ServerUpDown,ServerUp,,,,,,, ,,,,
1487614351573,,,,,,spadmin,logout,,,,,,,, ,,,,
1487614500536,,,,,,System,run,Perform Maintenance,,,,,,, ,,,,
I would like it to be like this:
mié feb 15 11:00:59 CET 2017,,,,,,localhost.localdomain,ServerUpDown,ServerUp,,,,,,, ,,,,
lun feb 20 19:00:34 CET 2017,,,,,,localhost.localdomain,ServerUpDown,ServerUp,,,,,,, ,,,,
lun feb 20 19:12:32 CET 2017,,,,,,spadmin,logout,,,,,,,, ,,,,
lun feb 20 19:15:01 CET 2017,,,,,,System,run,Perform Maintenance,,,,,,, ,,,,
I've tried this but it didn't work:
awk 'BEGIN{FS=OFS=","}{$1=`date -d #$( echo "($date_now + 500) / 1000" | bc)\`}1' file.csv
Any help will be much apreciated.
Thank you very much in advanced.
Kind regards.
Héctor
One way is to leave the CSV line intact and prepend it with the parsed timestamp as the first column.
Something like:
gawk -F, '{ printf "%s.%03u,",strftime("%Y-%m-%dT%H:%M:%S", $1/1000),$1%1000; print }' file.csv
Outputs:
2017-02-15T10:00:59.086,1487152859086,,,,,,localhost.localdomain,ServerUpDown,ServerUp,,,,,,, ,,,,
2017-02-20T18:00:34.268,1487613634268,,,,,,localhost.localdomain,ServerUpDown,ServerUp,,,,,,, ,,,,
2017-02-20T18:12:31.573,1487614351573,,,,,,spadmin,logout,,,,,,,, ,,,,
2017-02-20T18:15:00.536,1487614500536,,,,,,System,run,Perform Maintenance,,,,,,, ,,,,
Or you can rebuild the first field and then print the whole record like this:
echo 1487152859086,,,,,,localhost.localdomain,ServerUpDown,ServerUp,,,,,,, ,,,, | awk 'BEGIN{OFS=FS=","}{$1=strftime("%a %b %d %H:%M:%S %Z %Y", $1)}1'
You'll get this:
Fri Dec 13 14:45:52 CSTM 1901,,,,,,localhost.localdomain,ServerUpDown,ServerUp,,,,,,, ,,,,
Related
How can Miller 5.6.2 convert a local date and time to UTC using an expression simpler than the following?
$ printf "time1\n2019-06-13 05:54 PM\n" | mlr --csv put '
$time1=sec2gmt(
localtime2sec(
strftime(
strptime($time1, "%Y-%m-%d %H:%M %p"),
"%Y-%m-%d %H:%M:%S")));'
time1
2019-06-13T10:54:00Z
Note that my local time zone in June is EDT or -04:00:
$ date --date='2019-06-13 05:54 PM' '+%Y-%m-%dT%H:%M:%S %Z'
2019-06-13T17:54:00 EDT
$ date --date='2019-06-13 05:54 PM' '+%Y-%m-%dT%H:%M:%S%z'
2019-06-13T17:54:00-0400
I found two similar expressions, both of which use strptime_local() instead of strptime() to parse the local date and time string and convert it to seconds since the Epoch in UTC (GMT):
$ printf "time1\n2019-06-13 05:54 PM\n" | mlr --csv put '
$time1=strftime(
strptime_local($time1, "%Y-%m-%d %H:%M %p"),
"%Y-%m-%dT%H:%M:%SZ");'
time1
2019-06-13T10:54:00Z
$ printf "time1\n2019-06-13 05:54 PM\n" | mlr --csv put '
$time1=sec2gmt(strptime_local($time1, "%Y-%m-%d %H:%M %p"));'
time1
2019-06-13T10:54:00Z
Both strftime() and sec2gmt() assume an argument of seconds since the Epoch in UTC.
Function strptime() assumes an input date and time string in UTC and ignores a time zone in the input string:
$ printf "time1\n2019-06-13 05:54 PM\n" | mlr --csv put '
$time1=strptime($time1, "%Y-%m-%d %H:%M %p");'
time1
1560405240.000000
$ printf "time1\n2019-06-13 05:54 PM EDT\n" | mlr --csv put '
$time1=strptime($time1, "%Y-%m-%d %H:%M %p %Z");'
time1
1560405240.000000
Function strptime_local() also ignores a time zone in the input date and time string, but assumes the string is in the local time zone and converts it to UTC:
$ printf "time1\n2019-06-13 05:54 PM\n" | mlr --csv put '
$time1=strptime_local($time1, "%Y-%m-%d %H:%M %p");'
time1
1560423240.000000
$ printf "time1\n2019-06-13 05:54 PM EDT\n" | mlr --csv put '
$time1=strptime_local($time1, "%Y-%m-%d %H:%M %p %Z");'
time1
1560423240.000000
$ printf "time1\n2019-06-13 05:54 PM EST\n" | mlr --csv put '
$time1=strptime_local($time1, "%Y-%m-%d %H:%M %p %Z");'
time1
1560423240.000000
$ printf "time1\n2019-06-13 05:54 PM AUT\n" | mlr --csv put '
$time1=strptime_local($time1, "%Y-%m-%d %H:%M %p %Z");'
time1
1560423240.000000
I have a script called "main.ksh" which returns "output.txt" file and I am sending that file via mail (list contains 50+ records, I just give 3 records for example).
mail output I am getting is: (10 cols)
DATE FEED FILE_NAME JOB_NAME SCHEDULED TIME SIZE COUNT STATUS
Dec 17 INVEST iai guxmow080 TUE-SAT 02:03 0.4248 4031 On_Time
Dec 17 SECURITIES amltxn gdcpl3392 TUE-SAT 02:03 0.0015 9 Delayed
Dec 17 CONNECTED amlbene gdcpl3392 TUE-SAT 02:03 0.0001 1 No_Records
output with perfect coloring: (6 cols only)
DATE FEED FILE_NAME JOB_NAME SCHEDULED TIME SIZE COUNT STATUS
Dec 17 INVEST iai guxmow080 On_Time(green color)
Dec 17 SECURITIES amltxn gdcpl3392 Delayed(red color)
Dec 17 CONNECTED amlbene gdcpl3392 No_Records(yellow color)
I am implementing coloring for Delayed, On_Time and No_Records field and I wrote below script which gives me bottom output.
awk 'BEGIN {
print "<html>" \
"<body bgcolor=\"#333\" text=\"#f3f3f3\">" \
"<pre>"
}
NR == 1 { print $0 }
NR > 1 {
if ($NF == "Delayed") color="red"
else if ($NF == "On_time") color="green"
else if ($NF == "No_records") color="yellow"
else color="#003abc"
Dummy=$0
sub("[^ ]+$","",Dummy)
print Dummy "<span style=\"color:" color (bold ? ";font-weight:bold" : "")(size ? ";font-size:size" : "") (italic ? ";font-style:italic" : "") "\">" $NF "</span>"
}
END {
print "</pre>" \
"</body>" \
"</html>"
}
' output.txt > output.html
There are 4 columns are skipped automatically.
| date | feed_names | file_names | job_names | scheduled_time| timestamp| size| count| status |
Dec 19 ISS_BENEFICIAL_OWNERS_FEED amlcpbo_iss_20161219.txt gdcpl3392_uxmow080_ori_isz_dat WEEK_DAYS 00:03 9.3734 34758 On_Time
Dec 19 ISS_INVESTORS_FEED amlinvest_iss_20161219.txt gdcpl3392_uxmow080_ori_isz_dat WEEK_DAYS 00:01 0.0283 82 On_Time
Dec 19 ISS_TRANSACTIONS_FEED amltran_iss_1_20161219.txt gdcpl3392_uxmow080_ori_isz_dat WEEK_DAYS 00:12 14.022 36532 DELAYED
Dec 19 ISS_TRANSACTIONS_FEED amltran_iss_5_20161219.txt gdcpl3392_uxmow080_ori_isz_dat WEEK_DAYS 00:23 0.0010 3 DELAYED
Dec 19 IBS_CUSTOMER_FEED ibscust_aml_***_20161219.txt gdcpl3392_uxmow080_ori_sfp_ibc WEEK_DAYS (11 _out_of_11) -NA- ARRIVED
Dec 19 IBS_DDA_NOSTRO_ACCOUNT_FEED ibsacct_aml_***_20161219.txt gdcpl3392_uxmow080_ori_sfp_ibc WEEK_DAYS (44 _out_of_44) -NA- ARRIVED
Dec 19 GP__TRANSACTIONS_FEED amltrans__20161219.txt gdcpl3392_uxmow080_ori_sfp_glo WEEK_DAYS (3 _out_of_30) -NA- ARRIVED
But when I am trying to print in a sequential order by using below command
awk '{printf("%-5s%s\t%-33s%-35s%-39s%s\t%s%-3s\t%s\t%s\n", $1,$2,$3,$4,$5,$6,$7,$8,$9,$10)}' output.txt, I am getting the output in a sequential format
but 4 cols are skipped. Kindly suggest!!!
| date | feed_names | file_names | job_names | scheduled_time| timestamp| size| count| status |
Dec 19 ISS_BENEFICIAL_OWNERS_FEED amlcpbo_iss_20161219.txt gdcpl3392_uxmow080_ori_isz_dat On_Time
Dec 19 ISS_INVESTORS_FEED amlinvest_iss_20161219.txt gdcpl3392_uxmow080_ori_isz_dat On_Time
Dec 19 ISS_TRANSACTIONS_FEED amltran_iss_1_20161219.txt gdcpl3392_uxmow080_ori_isz_dat DELAYED
Dec 19 ISS_TRANSACTIONS_FEED amltran_iss_5_20161219.txt gdcpl3392_uxmow080_ori_isz_dat DELAYED
Dec 19 IBS_CUSTOMER_FEED ibscust_aml_***_20161219.txt gdcpl3392_uxmow080_ori_sfp_ibc ARRIVED
Dec 19 IBS_DDA_NOSTRO_ACCOUNT_FEED ibsacct_aml_***_20161219.txt gdcpl3392_uxmow080_ori_sfp_ibc ARRIVED
Dec 19 GP__TRANSACTIONS_FEED amltrans__20161219.txt gdcpl3392_uxmow080_ori_sfp_glo YET_TO_RECEIVE
I'm using MySQL to store my data. I'm recorded dates with different offset according to each timezone. What is important on those dates are only "effective" time. Here is an example:
date1 = "Thu, 27 Oct 2016 07:00:00 CEST +02:00"
# I want to get 700
date2 = "Thu, 27 Oct 2016 22:00:00 CEST +02:00"
# I want to get 2200
I would like to get these values to compare them with my current time. Then if it's currently "2016-11-17 10:12:00 +0100", I would like to compare on a where clause:
"1012" > "2200"
# or
"1012" > "700"
Here is a fake record value:
# my_table
# | id | my_hour
# | XX | "Thu, 27 Oct 2016 07:00:00 CEST +02:00"
# | XY | "Thu, 27 Oct 2016 22:00:00 CEST +11:00"
Here is my code:
hour = Time.zone.now
# "2016-11-18 21:10:00 +0100"
where("CAST(DATE_FORMAT(my_hour,'%H%i') AS UNSIGNED) < ?", hour.strftime("%H%M").to_i)
# => Get XX record?
Use strftime to format date in Rails.
hour = Time.zone.now
MyTable.where("cast(strftime('%H%M', my_hour) as int) < ?", hour.strftime("%H%M").to_i)
here MyTable is your Model, my_hour is your date column and hour is current time.
I am trying to convert a date/time of the following format YYYYDDMMHHmm to seconds since epoch in Tcl.
I've tried to do this with the following, but it isn't working:
clock scan 201403251850 -format %Y%m%d%H%M
For the above, I am trying to convert 6:50PM on March 25th, 2014 to seconds since epoc.
How can I achieve this?
Chris
I tested this on Tcl 8.3.3, so should work with 8.0: The regex may need some tweaking to suit the pre-8.1 regexp engine.
proc scan_datetime {datetime} {
# expect a datetime string like "6:50PM on March 25th, 2014"
regexp {^([0-9]+):([0-9]{2})([AP]M) on ([[:alpha:]]+) ([0-9]{1,2}).., ([0-9]{4})} $datetime -> hr min ampm mon day year
if {$ampm == "PM"} then {incr hr 12} elseif {$hr == 12} then {set hr 0}
clock scan "$day $mon $year $hr:$min"
}
puts [clock format [scan_datetime "6:50PM on March 25th, 2014"]]
puts [clock format [scan_datetime "12:34AM on February 1st, 2012"]]
Tue Mar 25 18:50:00 EDT 2014
Wed Feb 01 00:34:00 EST 2012
If the above regular expression doesn't work in 8.0, try this:
proc scan_datetime {datetime} {
set d {[0-9]}
set a {[A-Za-z]}
regexp "^($d$d?):($d$d)(\[AP]M) on ($a+) ($d$d?).., ($d$d$d$d)" $datetime -> hr min ampm mon day year
if {$ampm == "PM"} then {incr hr 12} elseif {$hr == 12} then {set hr 0}
clock scan "$day $mon $year $hr:$min"
}
Specifically for the format YYYYmmddHHMM:
tcl8.3.3 % set t 201403251452
201403251452
tcl8.3.3 % set d {[0-9]}
[0-9]
tcl8.3.3 % regsub "($d$d$d$d)($d$d)($d$d)($d$d)($d$d)" $t {\2/\3/\1 \4:\5} tt
1
tcl8.3.3 % clock scan $tt
1395773520
tcl8.3.3 % clock format [clock scan $tt]
Tue Mar 25 14:52:00 EDT 2014
Looking to convert human readable timestamps to epoch/Unix time within a CSV file using GAWK in preparation for loading into a MySQL DB.
Data Example:
{null};2013-11-26;Text & Device;Location;/file/path/to/;Tuesday, November 26 12:17 PM;1;1385845647
Looking to take column 6, Tuesday, November 26 12:17 PM, and convert to epoch time for storage. All times shown will be in EST format. I realize AWK is the tool for this, but can't quite seem to structure the command. Currently have:
cat FILE_IN.CSV | awk 'BEGIN {FS=OFS=";"}{$6=strftime("%s")} {print}'
However this returns:
{null};2013-11-26;Text & Device;Location;/file/path/to/;1385848848;1;1385845647
Presumably, this means I'm calling the current epoch time (1385848848 was current epoch at time of execution) and not asking strftime to convert the string; but I can't imagine another way to doing this.
What is the proper syntax for gawk/strftime to convert an existing timestamp to epoch?
Edit: This question seems loosely related to How do I use output from awk in another command?
$ cat file
{null};2013-11-26;Text & Device;Location;/file/path/to/;Tuesday, November 26 12:17 PM;1;1385845647
$ gawk 'BEGIN{FS=OFS=";"} {gsub(/-/," ",$2); $2=mktime($2" 0 0 0")}1' file
{null};1385445600;Text & Device;Location;/file/path/to/;Tuesday, November 26 12:17 PM;1;1385845647
Here's how to generally convert a date from any format to seconds since the epoch using your current format as an example and with comments to show the conversion process step by step:
$ cat tst.awk
function cvttime(t, a) {
split(t,a,/[,: ]+/)
# 2013 Tuesday, November 26 10:17 PM
# =>
# a[1] = "2013"
# a[2] = "Tuesday"
# a[3] = "November"
# a[4] = "26"
# a[5] = "10"
# a[6] = "17"
# a[7] = "PM"
if ( (a[7] == "PM") && (a[5] < 12) ) {
a[5] += 12
}
# => a[5] = "22"
a[3] = substr(a[3],1,3)
# => a[3] = "Nov"
match("JanFebMarAprMayJunJulAugSepOctNovDec",a[3])
a[3] = (RSTART+2)/3
# => a[3] = 11
return( mktime(a[1]" "a[3]" "a[4]" "a[5]" "a[6]" 0") )
}
BEGIN {
mdt ="Tuesday, November 26 10:17 PM"
secs = cvttime(2013" "mdt)
dt = strftime("%Y-%m-%d %H:%M:%S",secs)
print mdt ORS "\t-> " secs ORS "\t\t-> " dt
}
$ awk -f tst.awk
Tuesday, November 26 10:17 PM
-> 1385525820
-> 2013-11-26 22:17:00
I'm sure you can modify that for the current problem.
Also, if you don't have gawk you can write the cvttime() function as (borrowing #sputnik's date command string):
$ cat tst2.awk
function cvttime(t, cmd,secs) {
cmd = "date -d \"" t "\" '+%s'"
cmd | getline secs
close(cmd)
return secs
}
BEGIN {
mdt ="Tuesday, November 26 10:17 PM"
secs = cvttime(mdt)
dt = strftime("%Y-%m-%d %H:%M:%S",secs)
print mdt ORS "\t-> " secs ORS "\t\t-> " dt
}
$
$ awk -f tst2.awk
Tuesday, November 26 10:17 PM
-> 1385525820
-> 2013-11-26 22:17:00
I left srtftime() in there just to show that the secs was correct - replace with date as you see fit.
For the non-gawk version, you just need to figure out how to get the year into the input month/date/time string in a way that date understands if that maters to you - shouldn't be hard.
You can convert date to epoch with this snippet :
$ date -d 'Tuesday, November 26 12:17 PM' +%s
1385464620
So finally :
awk -F";" '{system("date -d \""$6"\" '+%s'")}' file
Thanks #Keiron for the snippet.