Im trying to add a date field to a JSON record and insert it to mongo using PutMongo processor (I succedded to do so using PutMongoRecord but when Im using PutMongo processor the field type is string and not date because there is no use of the JsonTreeReader)...I need the PutMongo processor to insert this field as a date in order to put a TTL index on it...
Currently I used JoltTransform and some variations of UpdateRecord but I only managed to convert the date to a timestamp which doesn't help me...
Is there a way to convert a string or a timestamp to a date field and insert it to mongo as a date field with the PutMongo processor?
Thanks in advance :)
I would recommend you stick to PutMongoRecord. You haven't said why you prefer to use PutMongo, but if you need to add some fields to enable that TTL index it's as simple as adding some optional attributes in your schema and calling UpdateRecord to put them in there before doing the put.
Related
Problem
I need to query out a date value for use in some ETL processing. However, the data is stored in MySQL as a VARCHAR column in values like 1/1/19.
What I've tried
A simple CAST(myDateColumn as DATE) fails as I get values like 0001-01-19 returned back.
Question
Am I able to pass a custom date format string into the CAST call somehow to tell it how to parse out the date parts? If not, could a SUBSTRING type of function help here?
As discussed in the comments above, you can use the STR_TO_DATE() function to parse a string into a proper YYYY-MM-DD date.
I used a wrapper between mongodb and postgres.
Now, I got my tables and my data and I can access to it.
now, when I listed my table, I found that the field date that is under an object is not type date or timestamp,
let me explain more with short scree,
When I use postgeSQL, my object is converted to json and I got all the fields, but no more dates. but numbers,like in the screen shot
and I can't alter the type of the field date.
what should I do?
I have a MySql table with a String 2014-02-21 16:53:01 stored in a varchar column. I need to use Talend Data Integration to convert it into a date for calculations in the format dd/mm/yyyy hh:mm:ss and to store it in that format in another MySql as a date column after the calculations.
I have a talend tmap component but I get parsing errors and not sure how to solve it and the general sequence of steps needed.
In the tmap I have a variable with the expression
row1.date !=null ? TalendDate.parseDate("dd/mm/yyyy hh:mm:ss",row1.date):null
Which I hope will load the variable wiIs that the best way to
Convert the string in the source table to a date in the format I need?
How do you handle null dates as I need to get the difference between 2 dates but handle the situation where 1 or both may be blank or null
How do I get the destination table to store the date in the format I need? I have the Date Pattern supplied in the destination tmap schema but is that enough if the date format is yyyy-mm-dd ?
Any Talend experts able to help a novice out?
For better control on null check I recommend you to use below function.
!Relational.ISNULL(row1.date) && !"".equalsIgnoreCase(row1.date) ?TalendDate.parseDate("MM/dd/yyyy HH:mm",row1.date):(null or default date);
if you are doing any operation on date? then don`t supply null as a result because you may get error on next component.
always pass default value so that you can check in next component and ignore it or store it based on business need.
You can try below condition which will take care of blank string also;-
row1.date !=null && !"".equalsIgnoreCase(row1.date)? TalendDate.parseDate("MM/dd/yyyy HH:mm",row1.date) : null
Thanks for the help. I decided to configure the schema for the database input to identify the dates as the date datatype instead of string which got Talend to do the work of converting
The pointers on validating the date were useful as well. Thanks for the help UmeshR
I'm trying to input a specific date type into my mysql database.
The format is dd-'month'-yy where month is the abbreviated month. For example, I would like to put in, 12-DEC-78 which would be December 12, 1978.
I've tried inputting that style with just the DATE data type, but it just reformats to 0's- like 0000-00-00.
I've looked all over the MySQL page itself and have had no luck with finding a data type that supports this format at table creation. Is there a type I'm overlooking or a setting to switch over to this method?
You will need to convert that date into a MySQL friendly format. STR_TO_DATE() will do that for you:
STR_TO_DATE('12-DEC-78', "%d-%b-%y")
You may want to try this:
Change the DATE type to char(9), name "olddate" This allows you to insert '12-Dec-78' format.
After you have inserted the CSV, create a new column with DATE type, name "newdate"
UPDATE table SET newdate=STR_TO_DATE(oldate, "%d-%b-%y")
Remove olddate column
I've got a table setup which has populated data. In column "date", I have dates in the following format:
yyyymmdd i.e. 20131110
I have created a new field and called it newdate with the text format.
Then, I open up the SQL window and put the following in
UPDATE wl_daily
SET
newdate = UNIX_TIMESTAMP(date)
For some reason, it is running correctly, however it only outputs NULL to all the rows. Also, the column name is blank for some reason
Any suggestions?
That's because your field in a string and you're trying to add timestamp to it which is not a string. You need to use a valid datetime field like timestamp for this to work.
Advice: don't store dates and times as strings. Store them in their native format. It makes working with dates and times much easier.
While John Cronde's answer is correct - it doesnt help your situtation
UNIX_TIMESTAMP(STR_TO_DATE(`date`, '%Y%m%d'))
will do the conversion for example
SELECT UNIX_TIMESTAMP(STR_TO_DATE('20131111', '%Y%m%d'))
returns
unix_timestamp(STR_TO_DATE('20131111', '%Y%m%d'))
---------------------------------------------------
1384128000
You should only use this to convert your columns to the date specific columns. Converting each time you need a number will add load and slow down the query if used in production