I am trying to create an SSIS package for integrating between MSSQL and MYSQL. I have no prior experience of working with Bids or SSIS and following the instructions from here.
I added the OLE DB Source, Lookup, Conditional Split, OLE DB Destination and OLE DB Command components to the Data Flow and configured the connection managers and column mappings upto the Conditional Split component.
From here, I am facing two problems -
1) After configuring the OLE DB Destination, it shows error symbol on the component that says could not convert between unicode and non unicode string datatypes. To solve this, I tried to insert a Data Conversion Component between the Conditional Split and the Destination and configured it for the problematic column. But that doesnt seem to help
2) While configuring the OLE DB Command, the right hand side column in Column mappings tab shows zero columns. I have added the Sql command with question marks so i guess it should be showing columns named "Param_0", "Param_1" etc if i am not wrong. I even tried to add them manually from the input and output properties tab but then it shows the warning, external columns for OLE DB command are out of sync with data source
What am I missing here ?
Thanks
The way you describe your first problem, it sounds like it should work. Here are a couple of things to check.
The data conversion component creates a new column for the converted data. Make sure you are referring to it in your following transformations and destination.
Right-click on the Data Conversion component and select Advanced Editor. Select the Input and Output Properties tab in the Advanced Editor. Expand the Data Conversion Output branch of the tree-view and select your new column. Ensure that the Data Type Properties show the data type that you want to convert too. If these values are not right then something is not right with the setup in the component.
For your second problem, the issue can frequently be caused by an error with the SqlCommand value. First, make sure the Connection Manager is correct on the Connection Manager tab. Switch to the Column Mappings tab. Near the bottom of the form, there may be a warning message that indicates that the SQL statement cannot be prepared. In other words, SSIS can't figure out what the statement is supposed to do. Address any problems with the SQL statement and switch back to the Column Mappings tab. The columns will appear once the SQL statement can be parsed.
If you want to avoid the conversion issues then change your destination table column types from char/varchar to nchar/nvarchar. I'm pretty sure you will need to use an ADO connector for mysql source and destinations, you should be able to read data from the mysql source and write to the mssql database w/o using anything other than source and destination components.
Related
We have a problem reading and loading some columns from AS/400 via OLE DB Provider for DB2 in SSIS 2015, the project is in SQL Server 2016.
We're migrating the project from an old server that is working with SQL Server 2008.
In the new one, when I preview the data in the Data Flow OLE DB Source, some columns have "System.Byte[]" as a value.
And I got the message "The metadata of the following output columns does not match thr metadata of the external columns with which the output columns are associated".
The external columns are read as DT_BYTES, while on the old server they were read as DT_STR.
The output columns are also expecting a DT_STR.
SSIS changed the metadata of the external columns, thus, making them unreadable.
I tried casting them as string, nothing works.
I tried setting the "ValidateExternalMetadata" to False, nothing.
The crazy thing is, with some other tables that have similar data types it reads them well, they have exactly the same data types as the failed ones.
There's no problem with the AS/400 itself as the old SSIS 2008 reads everything well.
So, I'm sure the problem is either with OL DB2 Provider, SSIS 2015 or SQL Server 2016.
I can't force it to read the external columns as STR with the Advanced Editor.
I also tried converting the data types in the output columns to DT_BYTES to match it, but still it didn't work.
It can't even read the columns from the beginning.
I tried a whole new package with no query, just reading the table as it is from the source, still the same problem.
I changed the Code Page to match the oiginal (1256), still nothing, then I changed it to match the new one (1252), still no success.
I'm really stuck and tried everything I could.
Any help please?
I finally solved it!
The solution is to open the connection manager, go to "All" tab, and match the Binary Codepage value (which is 0 by default) to the Host CCSID value.
I also set up the DBMS Platform to "DB2/AS400".
I am implementing a SSIS package and currently trying to do the following.
Truncate the destination table
Fetch the data by executing the stored procedure and insert it into the destination table.
I have created an Execute SQL task to address step 1 and dataflow with oledb source and oledb destination to address the second point. It been working successfully so far but isn't working for one my stored procedure that uses temp tables.
When I edit the oledb source and click the preview button, I get the error no column returned
I know that SSIS has an issue with generating column while executing stored procedures that depend on temp tables. I have converted the stored proc to use temporary table variables and its now able to return columns in SSIS when I do a preview. The only downside is that the stored procedure is taking longer time to execute. Its taking 1 hour 15 mins as compared to 15 mins while using temp tables.
I did see a suggestion to use SET FMTONLY before executing the stored procedure as an alternate solution to changing to temp table variables but that didn't seem to work as I am getting syntax or permission denied error.
Could somebody tell me a solution to my problem which does not compromise on the performance.
Sounds like you've already read all the approaches to using Temp tables in SSIS, including the IF 1=0... trick? If you haven't seen that one yet, google it.
You say that using Table Variables causes your stored procedure to take about 5 times longer than using Temp Tables. The most likely reason for that is that you are indexing your temp tables but not your table variables. If you didn't know that table variables can be indexed, they can. You might try that.
Finally, a solution that you haven't mentioned is that you can replace your temporary table with a real table that gets truncated when you're done using it.
Short comment:
Try EXEC WITH RESULT SETS and specify the metadata yourself for a proc with temp tables; or use the Script Component as a source and specify the Output columns yourself.
Long comment:
Technically speaking, it is the driver/database you are using in SSIS that would decide the behavior when working with temp tables.
Metadata is an important factor when using SSIS's pipeline components. By metadata, I mean the names of the columns, their data types etc that a pipeline component uses. When designing a data flow, someone/something should provide this metadata to the components that require it.
In most cases, SSIS automatically retreives the metadata. Components that do not connect to a external data source, like Conditional Split etc, get their metadata from the other components they are connected to. For the pipeline components that connect to a external data source (like Oledb source, oledb destination, Lookup etc.), SSIS provides a mechanism to get this metadata without human involvement. This mechanism involves the driver connecting to the database and retrieving the metadata of the output. If the driver/database is capable of returning the metadata, then that metadata is used. If the driver/database is incapable, then you get the errors you are seeing. The rest of my comments are based on the assumption that you are using a SQL Server database in your question.
When working with a SQL Server database in SSIS, typically, we use the native client drivers provided by Microsoft. When trying to get the metadata, these drivers try to get the metadata without actually executing the SQL Statement (actual execution can have side effects; and also, might take more than a few seconds/minutes/hours; and you dont want side effects and long wait times during package design time.) So to get the metadata, the driver relies on the metadata of the actual objects used in the sql command. If the command uses a physical table or view, SQL Server already has the metadata available and can supply it to the driver. If it is a temp table, SQL Server does not have the metadata until it can create the temp table. If using FMT ONLY option, you can use it in such a way to create the temp tables, but avoid any heavy processing/side affects and thus be able to retrieve metadata without penalties. Post 2012, these native client drivers rely on some newer functionality to retrieve metadata than the drivers before 2012. In 2012 and after, the driver uses the sp_describe_first_result_set proc to retrieve metadata. So, whether you can get metadata or not is determined by the ability of the sp_describe_first_result_set proc.
So while SSIS can automatically get the metadata (because of the driver/database), it does not automatically get the metadata in some cases (again because of the driver/database). In cases involving the second scenario, some other process (typically a human) can help the driver infer metadata or provide the metadata to the component directly.
To help the driver, in case of SQL Server 2012 and after, you can use the WITH RESULTSETS clause to specify the output metadata. When this clause is present, the driver will use it and doesnt try to query the metadata from system objects; and thus avoid the error which you would otherwise get. If you are using the drivers that came with SQL Server 2008, you can use FMT ONLY. This option is at the driver/database level.
Another option could be to use a Script Component as the Source and in the Output columns, you can specify the columns/metadata. SSIS would not try to retrieve metadata from the datasource in this case, but would rely on the definitions you provided in the Output section of the Script Component.
As you can see, both options involve a human (or some other process) specifying the metadata instead of SSIS trying to retrieve the metadata in an automated fashion. I would prefer the first option if working with SQL Server and the second option if working with databases like MySql.
I am working in SQL Server 2008 and BIDS (SSIS). I have a simple data flow task in which I'm importing a flat file into an OLE DB destination. On the OLE DB destination, I'm getting an error message, stating "cannot convert between unicode and non-unicode string data types".
Now, I know one solution method is to put a data conversion in between the flat file and the OLE DB destination. However, my concern is why this is even happening in the first place. In the connection manager for the given flat file, all columns are string (DT_STR) data types, and the Unicode option is unchecked. Similarly, all columns in the destination table (upon the inspection of the metadata in SSMS) are varchar data types. So, there is no nvarchar-to-varchar mapping going on at all.
Why is SSIS thinking that I have a unicode-to-nonunicode mapping happening? And, is there an easier way to resolve this than inserting a data conversion step for the columns that have this problem?
[Since I don't seem to be allowed to comment on the question, I'm having to put my question here.]
Have you checked the table you're trying to insert the data into to see if the columns in the table are varchar or nvarchar? The SSIS metadata could be out of sync with the database table.
I have a simple DataFlow with two objects the source which is a mdb file and the destination which is an MSSQL database.
The idea is to migrate the data from one to another.
The problem is that the data is extracted from an Access query, and one column has ~1000 characters, and in SSIS in advanced properties the external column has the default 255 length so when i execute the task it tries to truncate it. To disable the throw error on truncate is not an option, and modifying the Length of the external column cannot be done, it throws and error regarding the metadata.
First of all can anyone explain WHY?
Second of all i need a resolution and i need it fast because it's kinda driving me crazy.
This kind of problem occours, because the ssis task "guesses" the length of the column by inspecting the first 100(afaik) rows. So if all rows from 1 to 100 have a length of 10 and the row 101 has the legnth of 11, the task will fail, because the length was "guessed" to 10.
Modifying throws an error, because you have validateExternalMetadata set to true. To solve this problem, go to advanced options of your import task (access) and set the value to false.
This means, the task will accept modified values you entered without checking it.
Did you try to SSIS Import and Export Wizard to import the data, from within the BI development environment? That is the easiest way with MsAccess as this not only imports the data but also saves the package. If you get an error during the import ( using the wizard), please post it, as this helps in further investigation. Also, as #stb suggested, try having the first record over 1000 characters.
Access supports queries which are the equivalent to views in MSSQL.
The column size is defined not by looking at a few results but by the default column length of the column data type.
I created another table with the desired data types and before the data flow i've put in the package 2 sql scripts: one to delete all the data in the table and one to execute the query against the table, as to treat it as a temporary table.
Then the actual data flow is executed against this pseudo-temporary table.
This solved my problem.
When I use ADO.net source in DataFlow to read Blob Column and pass it to Script Component to do further validations - need script compoment to do further validations on each column to generate master / child error records master (for each row) and child (for each error column). This works fine.
As I need to parameterize my source, I can't use ADO.net and instead need to use the OLEDB Source which supports parameters. When I use this OLEDB source, the script component doesnt recognise the BLOB data being passed by OLEDB source. It reports datatype problems i.e., convering nonunicode to unicode.
How can this be done.
Regards
Can you confirm what your source database is (SQL Server, Oracle, etc).
I had the same problem using the 'Oracle OLEDB provider for Oracle' data source. The provider seems to convert every varcahr into an nvarchar. I solved this by adding a 'data conversion' component, and explicitly converting all nvarchar columns to varchar here.
The new columns are incuded in the output of this compnent, so you can link them to the fields on your spreadsheet.