I have a log file which contains timestamp column. The timestamp is in unix epoch time format.
I want to create a partition based on a timestamp with partitions year, month and day.
So far I have done this but it is throwing an error.
PARSE ERROR cannot recognize input '(' in column type
Here is my code.
from (
from raw_data
MAP ${PREFIX}raw_data.line
USING 's3://scripts/clean.py'
AS (timestamp STRING, name STRING)
) map_out
INSERT OVERWRITE TABLE date_base_data_temp PARTITION(year(timestamp), month(timestamp)), day(timestamp)))
select map_out.name;
Oof, that looks ugly. Try using this function in Hive:
SELECT from_unixtime(unix_timestamp) as new_timestamp from raw_data ...
Or if timestamp is in ms instead of seconds:
SELECT from_unixtime(unix_timestamp DIV 1000) as new_timestamp from raw_data ...
That converts a unix timestamp into a YYYY-MM-DD HH:MM:SS format, then you can use the following functions to get the year, month, and day:
SELECT year(new_timestamp) as year, month(new_timestamp) as month, day(new_timestamp) as day ...
With more recent releases of Hive and SparkSQL, data type of date and type casting options are available. Following should work in Hive as well as Spark SQL
SELECT cast(from_unixtime(epoch_datetime) as date) from myHiveTable
If you need to convert the date in custom format, use this:
select date_format(from_unixtime(epoch_datetime),'yyyyMM') as formatted_date from myHiveTable;
which will return the date as yearMonth e.g. 201708
Adding this query to the list where the timestamp needs to be converted to date string yyyy-MM-dd for a string partition:
hive> select date_format(from_unixtime(epoch_datetime), 'yyyy-MM-dd') as day from table_name limit 20;
-- If required, remove the millis precision for timestamps
hive> select date_format(from_unixtime(cast(epoch_datetime/1000 as bigint)), 'yyyy-MM-dd') as day from table_name limit 20;
select order_id, date_format(from_unixtime(order_date/1000),'yyy-MM-dd') as order_date ,order_customer_id,order_status
from orders
or if you see any error on the same , try to use
select order_id, date_format(from_unixtime(order_date DIV 1000),'yyy-MM-dd') as order_date ,order_customer_id,order_status
from orders
Related
I am trying to convert string '2022-12-28T22:28:43.260781049Z' to datetime format.
I have such query:
SELECT date(str_to_date('2022-12-28T22:28:43.260781049Z','%Y-%m-%d')) as date,
hour(str_to_date('2022-12-28T22:28:43.260781049Z',"%H:%M:%S")) as hour
FROM transaction
And such output:
date
time
'2022-12-28'
NULL
How to get time as well?
You can directly use a CAST on your string value to TIMESTAMP, then extract the date and the time with the hononimous DATE and TIME MySQL functions.
SELECT DATE(CAST(timestamp_ AS DATETIME)) AS date_,
TIME(CAST(timestamp_ AS DATETIME)) AS time_
FROM transactions;
Check the demo here.
Use timestamp instead of str_to_date:
SELECT hour(timestamp('2022-12-28T22:28:43.260781049Z')) as hour
I am writing the code in java which use DATE columns from mySQL database.
I would like to get average from DATE column but:
-avg(date_col) returns not date yyyy-mm-dd but some kind "weird" number,
I read a lot of stack topics "get average from date" but everyone uses timestamp or datetime (not date)... and all i tried were not working for me.
To sum up...
I have to get average from DATE column, which returns me "yyyy-mm-dd" because my program is parsing it in the following etap.
How can i get it?
Convert the date to unixtime, take the average and convert back to datetime:
select cast(from_unixtime(avg(unix_timestamp(dt))) as date)
from data;
Unixtime is simply seconds elapsed from 1970-01-01, so it is an integer where you can calculate the average from.
DATE_FORMAT with UNIX-Timestamp will do the trick
SELECT DATE_FORMAT(FROM_UNIXTIME(AVG(UNIX_TIMESTAMP(dt))) , '%Y-%m-%d')
FROM data;
Thsi convert time intp 2019-11-30 .
But there are more Format options
How to insert this date format on my table: 30/12/2018?
If it's impossible, then how can i turn this date format: 2018-12-30 12:10:00 to 30/12/2018 on a echo?
Store the date/time in the native format (i.e. as a datetime or date). Then, use date_format() to convert it to the format you want on output:
select date_format(datecol, '%d/%m/%Y')
You should proceed as follows :
ensure that the field where you store the date is of type datetime or date
use function STR_TO_DATE to convert strings to dates before writing to database
use function DATE_FORMAT to format the datetime values to the relevant format when reading form database.
Here is a small example of CREATE/INSERT/SELECT :
CREATE TABLE mytable (
mydate datetime
);
INSERT INTO mytable
VALUES (STR_TO_DATE('30/12/2018', '%d/%m/%Y'));
SELECT DATE_FORMAT(mydate, '%d/%m/%Y')
FROM mytable;
I need to filter all dates which greater than, say 01 january 2011.
select * from table_name where date > '01/01/2011';
the problem is that date field store int values, here is an example:
1339011098
1336717439
1339010538
How to convert the date field on the sql query (from the int format to date format), I need to convert it to a valid date so that I can compare it towards the above date.
Thanx.
You're going the wrong direction. Rather than converting potentially millions of records for the compare, try converting your target date, which you only need to do once. Those look like unix timestamps, so the resulting query should look like this:
SELECT * FROM `Table_name` WHERE date > unix_timestamp('01/01/2011')
Or, if you can control this, try using the ISO date format, which avoids confusion with european date formats for dates like 3/2/13:
SELECT * FROM `Table_name` WHERE date > unix_timestamp('2011-01-01')
You can use UNIX_TIMESTAMP()
select *
from table_name
where date > unix_timestamp('2011-01-01')
Or conversely use FROM_UNIXTIME()
select *
from table_name
where FROM_UNIXTIME(date, "%Y-%m-%d") > '2011-01-01'
First, you should not store date values as integers and if it's under your control your goal should be to fix the database and any queries that insert an integer value for that column instead of date.
The two date functions that you need to use with the current setup are UNIX_TIMESTAMP(), which accepts a date value and returns the epoch timestamp integer and FROM_UNIXTIME() which accepts an epoch timestamp integer and returns a date value.
For your example, you could use either:
SELECT * FROM table_name WHERE FROM_UNIXTIME(date_field) > '01/01/2011';
or
SELECT * FROM table_name WHERE date_field > UNIX_TIMESTAMP('01/01/2011');
But I would advise using FROM_UNIXTIME as a general rule as this would simplify more sophisticated queries such as:
SELECT * FROM table_name WHERE
FROM_UNIXTIME(date_field)
BETWEEN '01/01/2013' AND '04/01/2013';
Essentially, until your date field is actually storing values that are date types, your queries should covert the field values with FROM_UNIXTIME.
SELECT * FROM table ORDER BY timestamp;
When i'm trying to order the records by its timestamp i'm getting the following order
08/20/2012 02:09:39 PM
08/20/2012 03:19:08 PM
08/20/2012 09:04:24 AM
08/20/2012 09:05:25 AM
How to change the query so that the records are ordered from AM to PM?
The problem is that your string representation of the timestamp is not in canonical format, that is, sorting the string value does not sort in timestamp order.
To get the rows sorted in order, you can convert the character representation of the value into a DATETIME or TIMESTAMP datatype, or at least into a character representation in a canonical format (e.g. 'YYYY-MM-DD hh:mm:ss' with a 24 hour clock).
The STR_TO_DATE function is useful for converting a string representation in a known format into DATETIME:
SELECT * FROM table
ORDER BY STR_TO_DATE(`timestamp`,'%m/%d/%Y %h:%i:%s %p')
http://dev.mysql.com/doc/refman/5.1/en/date-and-time-functions.html#function_date-format
http://dev.mysql.com/doc/refman/5.1/en/date-and-time-functions.html#function_str-to-date
You will want to use STR_TO_DATE()
select *
from dates
order by STR_TO_DATE(dt,'%m/%d/%Y %h:%i:%s') desc
See SQL Fiddle with demo
just add the DESC:
SELECT * FROM table ORDER BY timestamp DESC;