Doing CSV file import to SQL database/table.
There is a date column which comes over as "1899-12-30 00:00:00.000".
Date column is Nullable on the database side.
Not sure why it's still showing 1899..
I used different derived expressions and nothing seem to be working.
I want to see Null instead of "1899-12-30 00:00:00.000" in the table/database.
Pls. advice
I tried to retain Null values as Nulls in the file on Flat file source.
That resolved the issue. Not sure how I overlooked this option..
This is an option:
First just load the data into database. Don't worry about 1899-12-30 00:00:00.000.
Add a SQL Task and set the value of DateColumn to NULL wherever its "1899-12-30 00:00:00.000". I'm pretty sure you don't have actual 1899-12-30 00:00:00.000 values in your database, so you don't need to worry.
I was using a Script Component transformation task, and to make my Date column NULL I had to explicitly set its value à la
Row.ColumnName_IsNull = true;
otheriwse it defaulted to 12/30/1899.
Related
phpMyAdmin - I need to manually change datetime value in a column.
current value - 2018-11-03 20:47:18
desired value - 2018-11-02 20:41:50
After editing the cell and pressing Enter the result is:
2018-11-02 20:41:50.000000
I tried multiple times - seems there is no way - the nulls are always added.
The column is datetime type, without default values.
Any help?
The zeros at the end are added automatically and make no difference. PHP for example inserts DATETIME without them, but the phpmyadmin editor does. They wont affect the value itself.
If you don't wat to see the zeros, execute the following query:
UPDATE table SET column = "2018-08-20 00:00:00" WHERE id = 999
I have a MySql table with a String 2014-02-21 16:53:01 stored in a varchar column. I need to use Talend Data Integration to convert it into a date for calculations in the format dd/mm/yyyy hh:mm:ss and to store it in that format in another MySql as a date column after the calculations.
I have a talend tmap component but I get parsing errors and not sure how to solve it and the general sequence of steps needed.
In the tmap I have a variable with the expression
row1.date !=null ? TalendDate.parseDate("dd/mm/yyyy hh:mm:ss",row1.date):null
Which I hope will load the variable wiIs that the best way to
Convert the string in the source table to a date in the format I need?
How do you handle null dates as I need to get the difference between 2 dates but handle the situation where 1 or both may be blank or null
How do I get the destination table to store the date in the format I need? I have the Date Pattern supplied in the destination tmap schema but is that enough if the date format is yyyy-mm-dd ?
Any Talend experts able to help a novice out?
For better control on null check I recommend you to use below function.
!Relational.ISNULL(row1.date) && !"".equalsIgnoreCase(row1.date) ?TalendDate.parseDate("MM/dd/yyyy HH:mm",row1.date):(null or default date);
if you are doing any operation on date? then don`t supply null as a result because you may get error on next component.
always pass default value so that you can check in next component and ignore it or store it based on business need.
You can try below condition which will take care of blank string also;-
row1.date !=null && !"".equalsIgnoreCase(row1.date)? TalendDate.parseDate("MM/dd/yyyy HH:mm",row1.date) : null
Thanks for the help. I decided to configure the schema for the database input to identify the dates as the date datatype instead of string which got Talend to do the work of converting
The pointers on validating the date were useful as well. Thanks for the help UmeshR
I've got a table setup which has populated data. In column "date", I have dates in the following format:
yyyymmdd i.e. 20131110
I have created a new field and called it newdate with the text format.
Then, I open up the SQL window and put the following in
UPDATE wl_daily
SET
newdate = UNIX_TIMESTAMP(date)
For some reason, it is running correctly, however it only outputs NULL to all the rows. Also, the column name is blank for some reason
Any suggestions?
That's because your field in a string and you're trying to add timestamp to it which is not a string. You need to use a valid datetime field like timestamp for this to work.
Advice: don't store dates and times as strings. Store them in their native format. It makes working with dates and times much easier.
While John Cronde's answer is correct - it doesnt help your situtation
UNIX_TIMESTAMP(STR_TO_DATE(`date`, '%Y%m%d'))
will do the conversion for example
SELECT UNIX_TIMESTAMP(STR_TO_DATE('20131111', '%Y%m%d'))
returns
unix_timestamp(STR_TO_DATE('20131111', '%Y%m%d'))
---------------------------------------------------
1384128000
You should only use this to convert your columns to the date specific columns. Converting each time you need a number will add load and slow down the query if used in production
The destination table has a column named updated and is set as:
Name: updated
Type: timestamp
Attributes: on update CURRENT_TIMESTAMP
Null: No
Default: CURRENT_TIMESTAMP
Extra: ON UPDATE CURRENT_TIMESTAMP
The source data is in a CSV, where the updated field is blank for every row, e.g.
id,item_name,updated,quantity
1,car,,4
2,truck,,5
3,chair,,5
After importing using PHPMyAdmin, I expect to see the updated column filled with the current date/time when the import was executed, however, I get all 0000-00-00 00:00:00 instead.
MySQL version: 5.5.30. This result is the same with MyISAM and InnoDB.
That is because the field is supplied. It is supplied as empty string, which is something different than NULL or default.
Empty strings will be converted to 0 when cast to a numeric value. And timestamp 0 formatted as a date is 0000-00-00 00:00:00.
You can either run UPDATE table SET updated=NOW() after importing (if you always import a complete set) or remove the entire column from the CSV.
How do you do a SET updated in phpmyadmin?
With the column described above, assign each value of the update column in the CSV as "NULL". The date of csv import is then added to mySQL.
One strange thing I noticed was errors in defining in the column. Even with an empty table, it was telling me that the default value for TIMESTAMP could not be CURRENT_TIME. To fix this, I banged my head on the table three times, changed the type to datetime, saved, then re-edited, and was allowed to change the type to TIMESTAMP.
The below steps works for setting timestamps to NULL.
1. Make the default value of the column - updated as NULL (anyhow, you want the timestamp to be empty)
2. In csv file type NULL instead of keeping the field blank
This method can also be used for auto increments
I have an ssis package that imports a view from one database into different db and places the data into a table. The view has all datatypes set to varchar which I cannot change. Some of these varchars are really datetime values. The view has set all NULL values to N/A and i want to replace the N/A with db null. The table I am importing the data into has the date columns set to datetime rather than varchar and I cannot insert N/A into it. This is my first ssis package and I'm having some issues with getting an expression setup to accomplish this. What would the string replace expression look like to make this happen? Thanks.
I was going about this the wrong way. The soultion is just to use as sql query to update the data rather than an expression.
I haven't tested but would something like this work in a derived column?
REPLACE("N/A", [yourfieldname] , NULL(DT_WSTR, 30) )
(adjust the string length to the current stringlength)