I ingested csv table to a Hive table but the date is being shown as integer value. Is there a way to convert Integer value (stored as string) to date in Hive?
When I do this
select cast(day_id2 as date) from table1
...I get null values:
Can someone tell me an elegant way to convert integer values (stored as string) into date values.
Related
I am trying to insert data from an UTF-8 encoded CSV file into Redshift database but I get the error when attempting to insert timestamp into a column which has timestamp data type.
Here's a sample CSV:
employeeId,employeeDept,employeeName,shiftStartTime,shiftEndTime,onPremises
KL214691,John Smith,operations,2023-01-17 09:01:34,2023-01-17 16:52:41,1
KL214692,Samantha Kennedy,operations,2023-01-17 08:31:54,2023-01-17 16:09:10,1
Here's a sample table DDL:
create table historical_metrics_agent_status_time_on_status
(
employeeid varchar(10),
employeename varchar(100),
employeedept varchar(50),
shiftstarttime timestamp encode az64,
shiftendtime timestamp encode az64,
onpremises boolean,
importdatetime timestamp encode az64
)
sortkey (employeeid);
The error message shows that there's an invalid digit - on position 4 in column shiftstarttime which has raw field value 2023-01-17 09:01:34. It looks like it's not reading timestamp from CSV file properly. Is there something I'm missing in CSV?
Check stl_load_errors for the exact row that is failing. My guess is that one of the VARCHAR columns has a comma (,) in it and is throwing off the alignment of the CSV to table columns. Like if one of the names is entered as “Smith, Joe”.
I am using load data syntax to load a csv file to a table.The file is same format as hive accepts. But still after load data is issued, Last 2 columns returns null on select.
1750,651,'2013-03-11','2013-03-17'
1751,652,'2013-03-18','2013-03-24'
1752,653,'2013-03-25','2013-03-31'
1753,654,'2013-04-01','2013-04-07'
create table dattable(
DATANUM INT,
ENTRYNUM BIGINT,
START_DATE DATE,
END_DATE DATE )
ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' ;
LOAD DATA LOCAL INPATH '/path/dtatable.csv' OVERWRITE INTO TABLE dattable ;
Select returns NULL values for the last 2 cols
Other question was what if the date format is different than YYYY-MM-DD. is it possible to make hive identify the format? (Because right now i am modifying the csv file format to accept by hive)
Answer to your 2nd question:
You will need an additional temporary table to read your input file, and then you can do date conversions in your insert select statements.In your temporary table store date fields as string. Ex.
create table dattable_ext(
DATANUM INT,
ENTRYNUM BIGINT,
START_DATE String,
END_DATE String)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';
Load data into temporary table
LOAD DATA LOCAL INPATH '/path/dtatable.csv' OVERWRITE INTO TABLE dattable_ext;
Insert from temporary table to the managed table.
insert into table dattable select DATANUM, ENTRYNUM,
from_unixtime(unix_timestamp(START_DATE,'yyyy/MM/dd'),'yyyy-MM-dd'),
from_unixtime(unix_timestamp(END_DATE,'yyyy/MM/dd'),'yyyy-MM-dd') from dattable_ext;
You can replace date format in unix_timestamp function with your input date format.
LasySimpleSerDe (default) does not work with quoted CSV. Use CSVSerDe:
create table dattable(
DATANUM INT,
ENTRYNUM BIGINT,
START_DATE DATE,
END_DATE DATE )
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
WITH SERDEPROPERTIES (
"separatorChar" = ",",
"quoteChar" = "'"
)
STORED AS TEXTFILE;
Also read this: CSVSerDe treats all columns to be of type String
Define you date columns as string and apply conversion in select.
I have extracted the modification time of a file using the struct stat structure:
long modtime = image_stats.st_mtime;
This returns 1508240128.
Now, I wish to store this value into a MySQL table which has datatype as datetime.
If I store it directly, it fails saying it is not a datetime format.
How do I store it?
You can use FROM_UNIXTIME to convert a timestamp into a DATETIME
Query
SELECT FROM_UNIXTIME(1508240128);
Result
FROM_UNIXTIME(1508240128)
---------------------------
2017-10-17 13:35:28
as insert query
Query
INSERT INTO
[table]
(datetime_column)
VALUES
(FROM_UNIXTIME(1508240128))
I have 5000+ dates in the following format.
00-00-0000
And the column data type is varchar. If I change the column type then all my rows are put to 00-00-0000 rather than converting the string literal to a date.
Is it possible to change all 50000+ rows to datetime and also the column data type? What would be the best way to do this?
Create a temporary DATE column and update it using STR_TO_DATE function:
UPDATE mytable
SET temp_date = STR_TO_DATE(varchar_date, '%m-%d-%Y')
Then drop the varchar date column and rename the temp date column.
I have a table in, MySQL 5.5, having an INT column which is actually a 8-digit number representing a date value in the format YYYYMMDD
How can I convert this column into a DATE type so as for easy compare with other DATE column in other table?
Use mysql STR_TO_DATE
The STR_TO_DATE() converts the str string into a date value based on the fmt format string.
SELECT STR_TO_DATE('20141012','%Y%m%d') FROM `table`