I am creating a table in Hive, running a mapper transformation and then saving a table. I want to partition the table based on when I ran the Hive query.
I create the table:
CREATE EXTERNAL TABLE IF NOT EXISTS testtable (
test_test STRING
) PARTITIONED BY (time STRING)
LOCATION 'loc/table'
;
Then run the transformation and save the table while trying this:
FROM (
MAP
one.test_test
USING
'python job.py'
AS test1
FROM
one
) test_step
INSERT OVERWRITE TABLE testtable PARTITION (time=unix_timestamp())
SELECT CAST ( test_step.test1 AS STRING ) AS test_test
;
However, when I do the
time=unix_timestamp()
, I get an exception. How would I go about doing this?
Thanks
I think it should work if you use dynamic partitioning (https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-DynamicPartitionInserts). The partition field is just another column in the table, so if you have a value for the column in your query, then Hive will automatically put it in the right partition. So your statement would look something like this
FROM (
MAP
one.test_test
USING
'python job.py'
AS test1
FROM
one
) test_step
INSERT OVERWRITE TABLE testtable PARTITION (time)
SELECT CAST ( test_step.test1 AS STRING ) AS test_test,
unix_timestamp() as time
;
Doing it like this might create a lot of partitions though, as the value of unix_timestamp() will change during execution of the query. It would be better to use an extra statement to create the partition first and then insert.
EDIT: To add a partition beforehand you'd need to set the timestamp you want somehow, e.g. a parameter for the script. Then
ALTER TABLE testtable ADD PARTITION (time=your_timestamp_here);
This would go before your original query where you replace unix_timestamp() with your_timestamp_here (which of course would be a valid unix timestamp).
Related
My error is no more spool space when I create a table with the qualify number_rows,
the goal is to get the first 100 samples by key,
each key is composed by the following fields: (top_typ_vision, instid, don_gener3, don_gener4,rg_no, lieu_stkph_cd,id_sect_base_resp)
When I execute the select, the code works very well, once I add the create I get the error no more spool space
thank you !!
```sql
create multiset table mdc_cobalt_det as (
sel
top_typ_vision,
instid,
type_enr as type_obj_ofs,
don_gener1,
don_gener2,
don_gener3,
don_gener4,
rg_no,
lieu_stkph_cd,
id_sect_base_resp
from PROD_V_CTRL_ANOMALIE
qualify row_number () over (partition by top_typ_vision,
instid,
don_gener3,
don_gener4,
rg_no, lieu_stkph_cd,
id_sect_base_resp order by rg_no ) <= 100)
with data
primary index (top_typ_vision, rg_no, don_gener3, don_gener4, lieu_stkph_cd, id_sect_base_resp);
I suggest you to:
Collect statistics on your input table and try to run it again;
Create this mdc_cobalt_det table as a NOPI table and check data distribution in fields chosen to be your primary index.
I am getting an date/time error while creating table in my sql ? how can i fix this?
create table ibi.stpl_toll_day_fare_fy_ex as
select
cast(date_time as date) as calendar_date,
I get this error
Error code :1292.incorrect datatime value.
Please check if the value in the date_time field of the existing table has the right data (if may have non relevant value to date)
From clause missing. You can query as below:
Using temporary table for demo, you can use your table
create temporary table yourtable(date_time date);
insert into yourtable(date_time) values (now());
create table stpl_toll_day_fare_fy_ex as
select
cast(date_time as date) as calendar_date from yourtable;
Select * from stpl_toll_day_fare_fy_ex;
I stored date as a
varchar(256) latin1_swedish_ci
which shows up as: 11/22/2012
Now I wanted to convert the existing data and column to a DATE
What would be the easiest way to do this in SQL
thanks
To be on the safe side I personally would add a new column, Transfer the data and then delete
update `table` set `new_col` = str_to_date( `old_col`, '%m/%d/%Y' ) ;
Check the data is OK before you delete the column.
Edit to convert the existing data use:
STR_TO_DATE(t.datestring, '%d/%m/%Y')
Source: Convert String to Date
You may need to create a temp table to hold your data for conversion:
CREATE [TEMPORARY] TABLE [IF NOT EXISTS] tbl_name
(create_definition,...)
[table_options]
[partition_options]
Then you can truncate the table:
TRUNCATE [TABLE] tbl_name
Then you can alter the column datatype:
ALTER TABLE tablename MODIFY columnname DATE;
Then you can reload from the temp table:
INSERT INTO table
SELECT temptable.column1, temptable.column2 ... temptable.columnN
FROM temptable;
Use the STR_TO_DATE function.
Make a new column that is the correct datatype, move the strings from the old column into the new one with the STR_TO_DATE function, and then delete the old column.
UPDATE table SET new_col = STR_TO_DATE(old_col);
MySQL STR_TO_DATE Reference.
I am looking for a way to change the datatype of a column. Currently, in my database, the date columns types were defined as varchar and I need to convert them back to the date type.
Any idea how to do it?
You will need to adapt this based your your exact table structure but something like;
CREATE TABLE temp (startdate varchar(255), stuff varchar(255));
INSERT INTO temp
SELECT startdate,stuff
FROM mytable;
TRUNCATE TABLE mytable;
ALTER TABLE mytable ALTER COLUMN startdate DATETIME NOT NULL;
INSERT INTO mytable
SELECT CAST(startdate AS DATETIME), stuff FROM temp;
DROP TABLE temp;
First, create the new column with type data
Next, run update query, to populate the new column with the value of the old one, applying any conversion if needed
Next, drop the old column
Finally, rename the new column to the old one
Create a new DATE column with a temporary name
Populate the new column with an UPDATE query that makes use of STR_TO_DATE()
If everything's right, remove the VARCHAR column and rename the DATE column.
Mysql default date format is : YYYY-MM-DD . If your try to insert the date otherwise, as you actually did, the date will be inserted with these values : 000-00-00, giving you a hint to the acceptable date format for mySql.
Wanna share this for SQL server users. For me this method is much convenient and safer.
In your Table create new column "NewDate" (temporarily or name whatever you want).
Make sure no invalid Datetime format in the Table you want to convert. Try those formats here: https://www.w3schools.com/sql/trysqlserver.asp?filename=trysql_func_sqlserver_cast3 <--- you need to check thoroughly otherwise there would be an error executing the command below.
Execute this command:
UPDATE myTable
SET NewDate = CAST(OldDate AS datetime)
WHERE (OldDate <> '') AND (OldDate IS NOT NULL) --to make sure you cast only what is needed otherwise there would be an error.
You can now delete the old column i.e. "OldDate".
Finally you can drag and drop the new table you've created to the slot where you just deleted the old column in the table design.
If the field of your column is VARCHAR and stored date as DD-MM-YYYY then we have to convert the date in YYYY-MM-DD format by following PHP code.
$cd = array();
$cd1 = array();
$cdf = array();
$getdata = mysqli_query($link,"SELECT columnname FROM tablename");
while($row=mysqli_fetch_array($getdata))
{
$cd = $row['columnname'];
$cd1 = strtotime($cd);
$cdf = date('Y-m-d',$cd1);
mysqli_query($link,"UPDATE tablename SET columnname =
REPLACE(columnname,'$cd','$cdf')");
}
After running this PHP code, in your MySQL table change the datatype of your column to 'DATE'.
It works for me without losing or truncate data.
I have to change the values in my table and create an temporary table from that. But not with UPDATE, since I have to keep the original table.
for example a table like Table(id, date), I have to create a temp table by changing the date values. The ones which are NULL must be CURRENT_DATE().
How can I manage that??
Create temporary table with (id,date)
then
INSERT INTO tempoaryTable (id,date)
SELECT id, IFNULL(date,CURRENT_DATE())
FROM yourTable
You can use INSERT-SELECT form to copy all values from original table to temporary, setting corresponding field to CURRENT_DATE().
Syntax might be off as I don't now mysql
insert into temp_table
select id,coalesce(date,current_date) from mytable