I am trying to convert a text field in an XML file with a value of 2014-04-01T00:00:00-04:00 to a date via a data conversion component.
The full error message is
[Data Conversion [2]] Error: Data conversion failed while converting column "BKG_DATE" (417) to column "Converted_BKG_DATE" (26). The conversion returned status value 2 and status text "The value could not be converted because of a potential loss of data.".
The conversion is from a DT_WSTR to a database time [DT_DBTIME].
Please note I only need the date value from the xml field value.
Can anyone help me to resolve this.
Instead of the Data Conversion component, try the Derived Column transformation, and derive the new column from an expression that builds the date by getting the appropriate substrings from the text column.
If you need only the date value, you need to convert to DT_DBDATE, which is a date structure that consists of year, month, and day. DT_DBTIME is a time structure that consists of hour, minute, and second, and so is unsuitable.
See MSDN > Integration Services Data Types: https://msdn.microsoft.com/en-gb/library/ms141036.aspx
To convert from date-time-with-offset values in this format 2014-04-01T00:00:00-04:00 to DT_DBDATE, you can use a Derived Column transformation with the following expression. In this example, the incoming values are in a column called RawDateTime.
(DT_DBDATE)LEFT(RawDateTime,10)
We can test it using a data viewer.
In my case, the date came in this format:
2021-03-31T16:26:26Z
I found that the only date type that natively supported that format was DT_DBTIMESTAMPOFFSET. This worked direct on my input file configuration -- I did not require a Derived Column transform.
Related
I am pulling a date from a JSON array called closeDate - using a subquery with data->>'closeDate' as PortalCloseDate which returns a string with the date format YYYY-MM-DD. I am trying to convert it to a date. When I return the column I have tried cast(cls.PortalCloseDate as date) and get the error message: ERROR: invalid input syntax for type date: "2020-12-09"
I have also tried to_date(cls.PortalCloseDate,'YYYY-MM-DD') and to_date(replace(cls.PortalCloseDate,'-',''),'YYYYMMDD') but get the error message: ERROR: invalid value "2" for "YYYY" Detail: Value must be an integer.
I need this to be a date data type since I am using calculations to find the time between two dates. Any help is appreciated.
edit: I exported to csv and there are a few dates with a different format than the rest. dates in csv For the majority of the dates they were automatically put in a date type but for the row with the date 2020-12-09 it was in a general format. I have attached a screenshot of the csv
I've got string price with comma at the end and I have to put it into decimal(10,2).
I tried to convert it into numeric with the same precision and scale but still gives me error of potential data loose.
Error: 0xC02020C5 at Przychód, Data Conversion 2: Data conversion failed while converting column "pr_kwota" (46) to column "Copy of pr_kwota" (10). The conversion returned status value 2 and status text "The value could not be converted because of a potential loss of data.".
Data from flat file(str 50)
What i tried to do:
You may need to use the transformation "Derived Column" instead of "Data Conversion" and probably the issue is the space as well as the comma so you can use this function inside the Derived Column transformation:
(DT_NUMERIC,10,2)(REPLACE(REPLACE(pr_kwota," ",""),",","."))
I have to send data from csv into SQL DB.
Problem starts when I try to convert data into Int. It wasnt my idea and I really cant do much with this datatype. When I'm trying to achieve this problem pop up:
Data Conversion 2: Data conversion failed while converting column
"pr_czas" (387) to column "C pr_dCz_id" (14). The conversion returned
status value 2 and status text "The value could not be converted
because of a potential loss of data.".
Tried already to ignore this problem but then another problems came up so there is no other way than solving this.
I have to convert this data from csv file which is str 50 into int 4
It must be int4. One of the requirements Dont know what t odo.
This is data I'm trying to put into int4. Look on pr_czas
This is data's datatype
Before I tried to do same thing with just DD.MM.YYYY but got same result...
Given an input column named [pr_czas] that contain string values that look like 31.01.2020 00:00 which appears to be a formatted date time represented in the format "DD.mm.YYYY HH:MM", I would like to express that as a whole number DDMMYYHHMM
Add a derived column to your data flow and call this new_pr_czas
The logic I'm going to use is a series of REPLACE statements and cast the final result to an integer. Replace the period, replace the colon and the space - all with nothing
(DT_I8)REPLACE(REPLACE(REPLACE([pr_czas], ".", ""), ":", ""), " ", "")
This is an easy case but things to note.
An integer/int32/I4 has a maximum value of 2 billion.
310120200000 is too large to fit into that space so you would need to make that an bigint/int64/I8. If I remember your previous question, you were having troubles with a lookup task so this data type mismatch might hurt you there.
The other thing to be aware of is that leading zeros will be dropped when converted to a number because they are not significant. If you need to retain the leading zeros, then you're working with string data type. This is an advantage to working with the ISO standard but if your data expects DD, then far be it for me to say otherwise.
If you need to slice your date into another format, then you'll want to have a few derived columns. The first one will generate a string column for each piece of pr_czas - year, month, day, hour and minute. You'll use the substring method for this and findstring to find the period space and colon.
The next data flow will be used to put those string pieces back into the new format and cast that to I8. Why? Because you can't debug doing it all in one shot but you can put a data viewer between two derived columns to figure out where a slice went awry.
I am having JSON docs with date in them, and I am trying to create a range index. When I do the ISO8601 format like this
2015-11-02T00:00:00-0600
I get the following error
IndexerEnv::putRangeIndex: XDMP-RANGEINDEX: Range index error: date "2015-11-02T00:00:00-0600": XDMP-LEXVAL: Invalid lexical value "2015-11-02T00:00:00-0600"
I thought it needs to be in UTC time and also needs a 'Z' in the end indicating Zulu time zone, so I forced for testing the time to be as follows
2015-11-02T00:00:00Z
I still get the same error..
IndexerEnv::putRangeIndex: XDMP-RANGEINDEX: Range index error: date "2015-11-02T00:00:00Z": XDMP-LEXVAL: Invalid lexical value "2015-11-02T00:00:00Z"
What is wrong here ? and also another question, does Marklogic understand dates only when we add 'Z' (Zulu timezone) in the date ?
Thanks
Is your range index created with the proper data type?
The error tells me you have created a range index of datatype date and your trying to push a dateTime into a date range.
Try changing the definition of the range index from type date to type dateTime.
HTH,
Peter
Thank you all, found my error..
For types other than xs:string, I am not supposed to add the collation parameter to the sort-order.. Changed this to
<sort-order type="xs:dateTime" collation="http://marklogic.com/collation/" direction="ascending">
<field name="concept_createdDate"/>
</sort-order>
to
<sort-order type="xs:dateTime" direction="ascending">
<field name="concept_createdDate"/>
</sort-order>
and all worked
I am doing Data Integration project to create a Data Warehouse in mySql. I have a couple of Csv/flat files to be ingested into Informatica Cloud which will populate the destination table in mySql database (The destination table is already created).
The Csv files have columns related to datetime but they are in String format (MM/DD/YYYY HH:MM) and I am not able to pass them through Informatica Cloud to the mySql datatbase tables. The column in mySql Database has format YYYY-MM-DD HH:MM:SS as the data type is datetime.
I tried different strategies of keeping the destination columns datatype as datetime and sometime varchar.
I got Following errors on different attempts with different changes.
*Transformation [Expr_DSS_0010LY0I000000000002_1] had an error evaluating output column [SystemModstamp_OUT]. Error message is [<<Expression Error>> [TO_DATE]: invalid string for converting to Date... t:TO_DATE(u:TO_CHAR(t:TO_DATE(u:'5/30/2013 12:26',u:'mm/dd/yyyy hh:mi'),u:'yyyy-mm-dd hh:mm'),u:'MM/DD/YYYY HH24:MI:SS')].
Transformation [Expr_DSS_0010LY0I000000000002_1] had an error evaluating output column [CreatedDate_OUT]. Error message is [<> [TO_DATE]: invalid string for converting to Date... t:TO_DATE(u:'6/24/2008 18:23',u:'mm/dd/yyyy hh24:mi:ss')].
Transformation [Expr_DSS_0010LY0I000000000002_1] had an error evaluating output column [CreatedDate_OUT]. Error message is [<> [TO_DATE]: invalid string for converting to Date... t:TO_DATE(u:TO_CHAR(t:TO_DATE(u:'6/24/2008 18:23',u:'mm/dd/yyyy hh24:mi'),u:'YYYY-MM-DD hh:mm:ss'),u:'MM/DD/YYYY HH24:MI')].*
I also tried the documentation related to Informatica Cloud (Csv datetime) but didn't get any concrete solution.
I will be really glad if someone can help.
Thanks in advance.
A single TO_DATE should work
TO_DATE(input_date_field,'mm/dd/yyyy hh24:mi')
I just finished solving the exactly same issue myself.
There is no good way to do this, I will explain what I did:
Basically I treated the date columns as string in power center. i.e.
I changed the target definition only in PC to varchar for all the dates column.
SQ, expects everything is string for their attributes.
Where as the physical definition is still datetime in mysql DB.
For me this worked. hope it works for you too. Good luck !!