I am trying to copy data from a flat file to a SQL table using SSIS.
I have a Data Flow Task where I have created a Flat File Source pointing to the csv file and an OLE DB Destination pointing to the table I want the data in.
The problem I am facing is when I run the package, I get only 2621 rows copied to the SQL destination table, where I have about 1,70,000 records in the csv. Not sure why this is happening.
Thanks in advance.
This could be a number of things. This is what comes to mind:
The connection string to your flat file is overwritten by a variable expression or a package configuration. Check SSIS -> Package configurations or the Expressions properties on your connection manager.
The DataRowsToSkip property on your flat file connection manager is set to a value.
The meta data definition of your flat file is incorrectly configured in your connection manager. See properties such as Format, Row delimiter, Column delimiter, etc. Use the preview function to see the output.
The error output on your flat file source is set to Ignore failure, meaning that lines which SSIS cannot process (due to, e.g., incompatible data types) are ignored without warning.
Related
I am working in SQL Server 2008 and BIDS (SSIS). I have a simple data flow task in which I'm importing a flat file into an OLE DB destination. On the OLE DB destination, I'm getting an error message, stating "cannot convert between unicode and non-unicode string data types".
Now, I know one solution method is to put a data conversion in between the flat file and the OLE DB destination. However, my concern is why this is even happening in the first place. In the connection manager for the given flat file, all columns are string (DT_STR) data types, and the Unicode option is unchecked. Similarly, all columns in the destination table (upon the inspection of the metadata in SSMS) are varchar data types. So, there is no nvarchar-to-varchar mapping going on at all.
Why is SSIS thinking that I have a unicode-to-nonunicode mapping happening? And, is there an easier way to resolve this than inserting a data conversion step for the columns that have this problem?
[Since I don't seem to be allowed to comment on the question, I'm having to put my question here.]
Have you checked the table you're trying to insert the data into to see if the columns in the table are varchar or nvarchar? The SSIS metadata could be out of sync with the database table.
I want to process value "excel file location" (column) from Table and SSIS should refer column value path and load that excel data into destination table.
Purpose of this task is In future we just update the file location in the table no need to change the source details in ssis.
Thanks in Advance.
Use an Execute SQL Task in the Control Flow to obtain the Excel file path from the database, and put it into an SSIS variable. Then, in your Data Flow, use the variable for your data source file path.
I am importing an Excel table from an ftp site to SQL using SSIS. The destination table is going to be used to calculate good and bad records based on another SQL database. Here is my problem. The Excel file is name RTW_032613_ABC_123.xls. This file name is a concatenation of a number of fields. I cannot recreate it based on the fields in the table, so I need to retain it and pass it to the new table in SQL. I have a parameter #FileName that I am using to loop through the files in the ftp folder. What I would like to do is either combine the import of data from the Excel file with the file name or insert the file name in each record after the import. I am calling the SSIS procedure from another stored procedure in SQL. I tried adding a SQL data flow task but I am not seeing where I add the insert statement on either the SQL Server Compact Destination or SQL Server Destination.
I am over my head with SSIS on this one. The key is that the parameter that I need is available in SSIS but I really need to get it passed on to my SQL table.
TIA
If I'm reading your question right, you have an SSIS package with a variable containing the filename and you want to save the filename with each row that you are sending to your SQL table? If so:
Add a derived column to the data flow, making a new column and referencing the variable in the expression
Include that new column in the mapping for the destination of your data flow, sending the filename to whichever column you would like to save that data in.
No need for a seperate SQL task.
SQL 2008 : Using SQL SSIS Package, I am exporting Excel Sheet to Text File and then Exporting to SQL Database. I decided to follow two steps of Export process as because of datatype mismatch between SQL Database and some of Excel Columns. When there is a mismatch in the Datatype, SSIS Package stops the Export Process but results as SSIS Package ran successfully.
To avoid partial import of Excel sheet, i decided to follow two steps of export. When different datatype of Excel Sheet goes to Text File, it reflect as String. When i am exporting back from Text file to SQL Database,it goes as STRING and again failing to export due to datatype mismatch.
Major Issue : If there is a mismatch in the Excel Sheet at nth row, Export stops at the nth row. Still, we get the output as Package Successful.
How to fix the datatype mismatch of Excel Data Export to SQL Database ?
You could approach it slightly differently as follows: read the excel spreadsheet in directly to one or more staging table(s) but force all columns to be VARCHAR/NVARCHAR columns. This will then enable you to read all rows into SQL without errors. Then create a second DFT to read from the staging table(s) into your correctly defined table(s)(correctly defined in terms of field type/size). You can build error logging into the second DFT to pump erroneous rows into an error table for later attention.
I am trying to create an SSIS package for integrating between MSSQL and MYSQL. I have no prior experience of working with Bids or SSIS and following the instructions from here.
I added the OLE DB Source, Lookup, Conditional Split, OLE DB Destination and OLE DB Command components to the Data Flow and configured the connection managers and column mappings upto the Conditional Split component.
From here, I am facing two problems -
1) After configuring the OLE DB Destination, it shows error symbol on the component that says could not convert between unicode and non unicode string datatypes. To solve this, I tried to insert a Data Conversion Component between the Conditional Split and the Destination and configured it for the problematic column. But that doesnt seem to help
2) While configuring the OLE DB Command, the right hand side column in Column mappings tab shows zero columns. I have added the Sql command with question marks so i guess it should be showing columns named "Param_0", "Param_1" etc if i am not wrong. I even tried to add them manually from the input and output properties tab but then it shows the warning, external columns for OLE DB command are out of sync with data source
What am I missing here ?
Thanks
The way you describe your first problem, it sounds like it should work. Here are a couple of things to check.
The data conversion component creates a new column for the converted data. Make sure you are referring to it in your following transformations and destination.
Right-click on the Data Conversion component and select Advanced Editor. Select the Input and Output Properties tab in the Advanced Editor. Expand the Data Conversion Output branch of the tree-view and select your new column. Ensure that the Data Type Properties show the data type that you want to convert too. If these values are not right then something is not right with the setup in the component.
For your second problem, the issue can frequently be caused by an error with the SqlCommand value. First, make sure the Connection Manager is correct on the Connection Manager tab. Switch to the Column Mappings tab. Near the bottom of the form, there may be a warning message that indicates that the SQL statement cannot be prepared. In other words, SSIS can't figure out what the statement is supposed to do. Address any problems with the SQL statement and switch back to the Column Mappings tab. The columns will appear once the SQL statement can be parsed.
If you want to avoid the conversion issues then change your destination table column types from char/varchar to nchar/nvarchar. I'm pretty sure you will need to use an ADO connector for mysql source and destinations, you should be able to read data from the mysql source and write to the mssql database w/o using anything other than source and destination components.