I am exporting F&O D365 data to ADLS in CSV format. Now, I am trying to read the CSV stored in ADLS and copy into Azure Synapse dedicated SQL pool table using Azure data factory. However, I can create the pipeline and it's working for few tables without any issue. But it's failing for one table (salesline) because of mismatch in number of column.
Below is the CSV format sample, there is no column name(header) in CSV because it's exported from F&O system and column name stored in salesline.CDM.json file.
5653064010,,,"2022-06-03T20:07:38.7122447Z",5653064010,"B775-92"
5653064011,,,"2022-06-03T20:07:38.7122447Z",5653064011,"Small Parcel"
5653064012,,,"2022-06-03T20:07:38.7122447Z",5653064012,"somedata"
5653064013,,,"2022-06-03T20:07:38.7122447Z",5653064013,"someotherdata",,,,test1, test2
5653064014,,,"2022-06-03T20:07:38.7122447Z",5653064014,"parcel"
5653064016,,,"2022-06-03T20:07:38.7122447Z",5653064016,"B775-92",,,,,,test3
I have created ADF pipeline using copy data activity to copy the data from ADLS(CSV) to Synapse SQL table however I am getting below error.
Operation on target Copy_hs1 failed: ErrorCode=DelimitedTextMoreColumnsThanDefined,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error found when processing 'Csv/Tsv Format Text' source 'SALESLINE_00001.csv' with row number 4: found more columns than expected column count 6.,Source=Microsoft.DataTransfer.Common,'
Column mapping looks like below- Because CSV first row has 6 column so it's appearing 6 only while importing schema.
I have repro’d with your sample data and got the same error while copying the file using the copy data activity.
Alternatively, I have tried to copy the file using data flow and was able to load the data without any errors.
Source file:
Data flow:
Source dataset: only the first 6 columns are read as the first row contains only 6 columns in the file.
Source transformation: connect source dataset in source transformation.
Source preview:
Sink transformation: Connect sink to synapse dataset.
Settings:
Mappings:
Sink output:
After running the data flow, data is loaded to the sink synapse table.
Change my csv to xlsx help me to solve this problem in Copy Activity ADF.
1.From Copy data settings set "Fault Tolerance" = "Skip Incompatible rows"
skip incompatible rows
2.From Dataset connection settings set Escape character to Double quotes
Escape character
I've created a table in SQL Server 2016 with this definition:
CREATE TABLE Stage.Test (f1 DATE NULL);
INSERT INTO Stage.Test (f1) VALUES ('1/1/2000');
Notice the f1 column uses the DATE data type.
Then I created a data flow task in SQL Server Data Tools (VS 2019) to load data from that table. I created an OLEDB Source Component and set the source table to Stage.Test. But when I examine the data type of the "f1" column (in both the 'External Column' and 'Output Column' columns), it says it's a Unicode string:
Why is it choosing a Unicode string instead of DT_DATE?
I haven't seen this exact example when it comes to date fields but SSIS converts data automatically when it has detected the field to be of a certain type. Perhaps it's the '/' in your date field that does it. We do not use this date format over in these parts of the world so I've never had the problem. You can especially see this when you import excel files with SSIS. I usually have this problem with strings where unicode strings can sometimes become non-unicode strings.
A way to fix could be to:
edit the sql query to explicitly cast the field as date
add a conversion step in the data flow after the source (like a derived column getting the parts of the string in the right order)
try to change the output datatype by right clicking on the source in the data flow and using the advanced editor and then edit the output column datatype:
SSIS Source Advanced Editor Output
I'm not sure if that 3 would work with this date format issue as I do not have experience with the format myself but it is working fine for my unicode/non-unicode problem.
I've to import some data from a CSV file formatted with Excel and in UTF-8 encoding.
The error thrown at me when importing the data refers to fields with blank spaces that should be set to null:
incorrect integer value '' for column 'name_column' at row 1
I tried to change the format coding to set null/NULL/none instead of blank spaces in the Excel fields, but the result is the same.
I cannot import the data by query because LOAD DATA is not supported by my MySQL version.
Ok so I am trying to import data from a CSV file into mySQL database table. The table is called serial_code, however when I try to upload the CSV file I have an error message. I have tried taking the column names out, also adding NULL to the last column EngineSerialCode and I have also viewed the CSV in a text editor and it shows the columns correctly with , comma.
Invalid column count in CSV input on line 1.
Version of phpMyAdmin and mySQL
Database Fields
CSV Fields I want to import
Ok so I found the issue and it was quite frustrating. I had a serial code in my data that had a "," comma instead of a "-". Also I noticed that my CSV file EngineSerialCode was placed in column E and not D as I only have 4 columns in my database. From the image column C is overlapping column D hence my mistake.
I have a csv file from a legacy dbase dbf file. The data contains a few columns which have number values with hyphens. Like this '661-'. I am trying to import the csv into MySql using 'Import External Data' in MySql Yog. The issue is, the columns that have values with hyphens are getting imported as decimals resulting in '-661.0000'.
This is odd as the column format in the csv (via MS Excel) is 'general' not 'number' and I am trying to import these values into varchar fields. Seems MySql is ignoring the settings.
Has anyone faced something like this or have any suggestions on how I can get the data in as a string not a decimal?
Thanks
ANSWER - sorry to be answering my own question but I did solve it with some group input.
The file needs to be saved with all fields (that you want to be treated as string) in quotes. MS Excel DOES NOT seem to have an option for this. Apache Open Office does. Open the file in AOO and save as text.csv. From there you can edit the filters and set all cells to be in quotes. Problem solved.