I've successfully created a BIML script on BIDS 2008 with BIDS Helper 1.6.6.0 which automates the creation of SSIS packages to import data from an Oracle database (11g Enterprise Edition Release 11.2.0.3.0 - 64bit) into SQL Server 2008 R2. I am having an issue at package run-time which causes the package to fail at Data Flow valdation with:
Warning: The external columns for component "Source" (1) are out of synchronization with the data source columns. The external column "LIMIT_AMOUNT" needs to be updated.
The external column "LIMIT_BASE_AMOUNT" needs to be updated.
The external column "GROSS_BASE_AMOUNT" needs to be updated.
Error: The OLE DB provider used by the OLE DB adapter cannot convert between types "DT_BYTES" and "DT_NUMERIC" for "LIMIT_AMOUNT".
Error: The OLE DB provider used by the OLE DB adapter cannot convert between types "DT_BYTES" and "DT_NUMERIC" for "LIMIT_BASE_AMOUNT".
Error: The OLE DB provider used by the OLE DB adapter cannot convert between types "DT_BYTES" and "DT_NUMERIC" for "GROSS_BASE_AMOUNT".
Error: There were errors during task validation.
Upon inspection, it appears that the metadata for NUMBER columns without scale and precision in Oracle are mapped to DT_BYTES in the generated SSIS. The description of the above object (a view) in Oracle is as follows:
Name Null Type
--------------------- ---- ------------
ID NUMBER(12)
CURRENCY VARCHAR2(3)
LIMIT_AMOUNT NUMBER
LIMIT_BASE_AMOUNT NUMBER
GROSS_BASE_AMOUNT NUMBER
STATUS VARCHAR2(15)
Checking in all_tab_columns shows the three NUMBER columns as having a DATA_LENGTH of 22 and NULL DATA_PRECISION and DATA_SCALE.
COLUMN_ID COLUMN_NAME DATA_TYPE DATA_LENGTH DATA_PRECISION DATA_SCALE
---------- ---------------------- ------------- ----------- -------------- ----------
1 ID NUMBER 22 12 0
2 CURRENCY VARCHAR2 3
3 LIMIT_AMOUNT NUMBER 22
4 LIMIT_BASE_AMOUNT NUMBER 22
5 GROSS_BASE_AMOUNT NUMBER 22
6 STATUS VARCHAR2 15
The Oracle documentation states that this is the equivalent of a float
Specify a floating-point number using the following form:
NUMBER
The absence of precision and scale designators specifies the maximum range and precision for an Oracle number.
The workaround so far was to implement a custom SELECT which casts these fields to the desired type, but that's not very elegant or maintainable. I would like to understand why BIML seems to get the data type mapping wrong, whereas SSIS is able to determine that the metadata is wrong when the package is first opened after it has been created –I get a pop-up in BIDS stating that
The metadata of the following output columns does not match the metadata of the external columns with which the output columns are associated:
Output "Output": "LIMIT_AMOUNT", "LIMIT_BASE_AMOUNT", "GROSS_EXP_BASE_AMOUNT"
Do you want to replace the metadata of the output columns with the metadata of the external columns?
EDIT: Adding relevant Biml details regarding connections & dataflow
<#
string OraConnectionStr = #"Provider=OraOLEDB.Oracle;Data Source=(In-line TNS);User Id=redacted;Password=redacted;Persist Security Info=True;";
string StagingConnectionStr = "Data Source=SVR;Initial Catalog=DB;Integrated Security=SSPI;Provider=SQLNCLI10;";
#>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Connections>
<Connection Name="<#=StagingConnectionName#>"
ConnectionString="<#=StagingConnectionStr#>" />
<Connection Name="<#=OraConnectionName#>"
ConnectionString="<#=OraConnectionStr#>" />
</Connections>
<Packages>
<!-- Assume object stagingTables is populated and methods have been defined -->
<# foreach (DataRow row in stagingTables.Rows) { #>
<Package Name="<#= GetChildPackageName(row) #>"
ConstraintMode="Linear" AutoCreateConfigurationsType="None">
<Dataflow Name="<#=GetStagingTableDescriptiveName(row)#>" >
<Tasks>
<Transformations>
<OleDbSource Name="Source - <#=GetStagingTableDescriptiveName(row)#>"
ConnectionName="<#=OraConnectionName#>"
AlwaysUseDefaultCodePage="true"
DefaultCodePage="1252">
<DirectInput>SELECT * FROM <#GetOracleObjectName(row)#></DirectInput>
</OleDbSource>
<OleDbDestination Name="Destination - <#=GetStagingTableDescriptiveName(row)#>"
ConnectionName="<#=DataLoadConnectionName#>">
<ExternalTableOutput Table="<#= GetStagingTableObjectName(row) #>" />
</OleDbDestination>
</Transformations>
</Dataflow>
</Tasks>
</Package>
<# } #>
</Packages
</Biml>
Thanks in advance.
Related
I am creating an SSIS package that will run each month. This particular stored procedure needs to run for one week at a time since the data returned is very large.
I have set up my stored procedure to with two parameters: #StartDT and #EndDT. I created two SSIS variables: StartDT and Wk1EndDT (I'll create the other start and end dates for the weeks once I get this one working).
StartDT has this expression:
(DT_DATE)((DT_WSTR, 4)YEAR(DATEADD("mm", -1, GETDATE())) + "-" +RIGHT("0" + (DT_WSTR,2)MONTH(DATEADD("mm", -1, GETDATE())),2)+"-01")
Wk1EndDT has this expression:
DATEADD("DD",7, #[User::StartDT])
I'm using a DataFlow task with a SQL command text of:
EXECUTE dbo.uspUploadWk1 ?,?
When I go to preview the results, I receive the following error message:
There was an error displaying the preview.
No value given for one or more required parameters. (Microsoft SQL Server Native Client 11.0)
I have the parameters set like this:
I am not sure why this isn't working. I've searched all over and have not found an answer. I am using Visual Studio 2015.
Assuming an OLE DB Connection Manager, the Mappings tab should be using a zero based ordinal system on the Parameters column. Yes, it defaults to naming them as Parameter0, Parameter1, etc but for an OLE DB connection manager, you'll use the ordinal position of the question marks, ?, starting at zero.
For ODBC, it becomes a 1 based counting but still uses ? as the parameter place holder.
ADO.NET uses named parameters so we'd match EXECUTE dbo.uspUploadWk1 #Parameter0, #Parameter1 but the ADO.NET source component doesn't support parameterization
Reference on parameters and mapping for Execute SQL Task but the syntax remains the same for Data Flow Task components
When I load numeric column directly to a money column, OLEDB component is truncating the value in it, but TSQL and ADO.NET components are rounding as expected.
Source value : 2.081250
Vat1: map directly ,
Vat2: map after converting to money data type
Result:
VAT1 VAT2 CONN
2.0813 2.0813 TSQL
2.0813 2.0813 ADO
2.0812 2.0813 OLEDB
2.0812 : numeric -> money directly with OLEDB
Do you have any idea? Is it default behavior of OLEDB component?
Regards,
Mustafa
Test Case:
-- Source
CREATE TABLE TEST_NUMERIC(VAT1 NUMERIC(38,6), VAT2 NUMERIC(38,6))
INSERT INTO TEST_NUMERIC VALUES( 2.081250, 2.081250)
-- Destination
CREATE TABLE TEST_MONEY(VAT1 MONEY, VAT2 MONEY, CONN NVARCHAR(20))
-- Test 1 - With TSQL
INSERT INTO TEST_MONEY
SELECT VAT1, CAST(VAT2 AS MONEY), 'TSQL' FROM TEST_NUMERIC
-- Test2 - With SSIS, ADO NET and OLE DB components:
ADO.Net Source:
SELECT VAT1, CAST(VAT2 AS MONEY) VAT2, N'ADO' AS CONN
FROM TEST_NUMERIC
Ado .Net Destination: TEST_MONEY
**OLE DB Source:**
SELECT VAT1, CAST(VAT2 AS MONEY) VAT2, N'OLEDB' AS CONN
FROM TEST_NUMERIC
OLEDB Destination: TEST_MONEY
SSIS Package:
The maximum scale for DT_CY (SSIS data type for money) is four and OLE DB connections do support this data type, which by the data in your example (2.081250), would be 2.0812. Since you're doing the casting within SQL Server for the V2 column, this is rounded up to 2.0813 before SSIS processes it. The ADO equivalent of money would be decimal (reference here). So while you can add the money column in your ADO.NET destination and map to the the numeric data type of your source, the destination money data type is still interpreted as decimal, which preserves the longer scale, and in this case would round up to the 2.0813 value that you're seeing here.
I know there are 3 types of parameter in 'Parameter mapping' - Input Parameter, Output Parameter and Return Parameter. I understand how to use Input and Output parameter. But when I try to set the parameter type as 'Return Parameter', it doesn't work. Below is my SQL Server stored procedure.
ALTER Procedure [dbo].[spRandomReturn]
As
Begin
Return Convert(int, rand() * 10)
End
In SSIS Execute SQL task, I have set
connection type: OLE DB
parameter mapping: variable name: User::#random (I set SSIS a User parameter in SSIS: random INT32), Direction: ReturnValue, Type: Numeric, Parameter Name: #random
SQL statement:
Declare #r int = #random EXEC #r = spRandomReturn
I created a return parameter in SSIS, but it doesn't work and throws error.
Since you're using OLE DB Connection Manager, you need to use the ? to indicate where parameters are.
Thus, your query becomes
EXECUTE ? = [dbo].[spRandomReturn]
And within your parameter mapping, you'd have
Reproduction
Biml, the Business Intelligence Markup Language, describes the platform for business intelligence. Here, we're going to use it to describe the ETL. BIDS Helper, is a free add on for Visual Studio/BIDS/SSDT that addresses a host of shortcomings with it. Specifically, we're going to use the ability to transform a Biml file describing ETL into an SSIS package. This has the added benefit of providing you a mechanism for being able to generate exactly the solution I'm describing versus clicking through many tedious dialogue boxes.
You can see in the following bit of XML, I create a connection called CM_OLE and this points to localhost\dev2014 at tempdb. You would need to modify this to reflect your environment.
I create a package named so_28419264. This package contains 2 variables. One is Query which contains the first bit of code. The second is ReturnValue which we will use to capture the return value on the Mapping tab. I initialize this one to -1 as the provided stored procedure would never generate a negative value.
I add two Tasks, both Execute SQL Tasks. The second one does nothing, it simply serves as a point for me to put a breakpoint on. The first Execute SQL Task is where we invoke our Stored Procedure and assign the results into our variable
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Connections>
<OleDbConnection Name="CM_OLE" ConnectionString="Data Source=localhost\dev2014;Initial Catalog=tempdb;Provider=SQLNCLI10.1;Integrated Security=SSPI;Auto Translate=False;" />
</Connections>
<Packages>
<Package ConstraintMode="Linear" Name="so_28419264">
<Variables>
<Variable DataType="String" Name="Query">EXECUTE ? = [dbo].[spRandomReturn];</Variable>
<Variable DataType="Int32" Name="ReturnValue">-1</Variable>
</Variables>
<Tasks>
<ExecuteSQL ConnectionName="CM_OLE" Name="SQL Demonstrate Return Value">
<VariableInput VariableName="User.Query" />
<Parameters>
<Parameter DataType="Int32" VariableName="User.ReturnValue" Name="0" Direction="ReturnValue" />
</Parameters>
</ExecuteSQL>
<ExecuteSQL ConnectionName="CM_OLE" Name="Put Breakpoint on me">
<DirectInput>SELECT 1;</DirectInput>
</ExecuteSQL>
</Tasks>
</Package>
</Packages>
</Biml>
Results
It works
I have an SSIS package where I need to get the date the package last ran from an ADO NET Source then assign it to a variable so what I can use it in a query for another ADO NET Source. I can't find an example on the Googles that actually works. I'm running VS 2012 and connecting to a SQL Server 2012 instance. If there is more information needed let me know.
Create a variable #User::LastRanDate.
Create an Execute SQL task.
Set the ConnectionType property to ADO.NET.
Set the Connection property to your ADO.NET connection.
Set the SQLStatement property to the statement which will return the date you want. Make sure the first column returned is the date.
Set the ResultSet property to Single row.
On the Result Set tab of the Task editor, hit Add and set the Result Name value to 0 and the Variable Name value to #User::LastRanDate. (ADO.NET result sets are returned as indexed arrays.)
Upon completion of the Task, #User::LastRanDate will now be set to whatever the query returned and you can use it to build up your query for your other ADO.NET source.
Working with parameterized queries in an ADO.NET Data Source in SSIS is not as easy as an OLE DB one. Basically, you're going to have to write the query with the expression language and pray your source doesn't lend itself to sql injection.
How to Pass parameter in ADO.NET Source SSIS
how to pass parameters to an ado.net source in ssis?
I created a package with 3 variables as shown below
Package
Variables
I have LastRunDate as a DateTime and a QueryAdo as a string. This evaluated as an Expression with the expression being "SELECT RD.* FROM dbo.RunData AS RD WHERE RD.InsertDate > '" + (DT_WSTR, 25) #[User::LastRunDate] + "';"
Execute SQL Task
I create an Execute sql task that uses a query and is set to return a single row. I assign this value into my SSIS Variable.
In my results tab, I assign the zeroeth column to my variable LastRunDate
Data flow
Note there is an expression here. On the ADO.NET source, I originally used SELECT RD.* FROM dbo.RunData AS RD to get my meta data set.
After I was happy with my data flow, I then went to the control flow and substituted my Query variable in as the expression on the ADO.NET Source component (see the referenced questions).
Try it, try it, you will see
I used the following script to build out my demo environment
create table dbo.RussJohnson
(
LastRunDate datetime NOT NULL
);
create table dbo.RunData
(
SomeValue int NOT NULL
, InsertDate datetime NOT NULL
);
insert into dbo.RussJohnson
SELECT '2014-08-01' AS LastRunDate
INSERT INTO
dbo.RunData
(
SomeValue
, InsertDate
)
SELECT
D.rc AS Somevalue
, dateadd(d, D.rc, '2014-07-30') AS InsertDate
FROM
(
SELECT TOP 15 ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) AS rc
FROM sys.all_columns AS SC
) D;
Since I have BIDS Helper installed, I used the following Biml to generate this package as described. For those playing along at home, you will need to edit the third line so that the ADO.NET connection manager is pointing to a valid server and database.
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Connections>
<AdoNetConnection Name="CM_ADO_DB" ConnectionString="Data Source=localhost\dev2014;Integrated Security=SSPI;Connect Timeout=30;Database=tempdb;" Provider="SQL" />
</Connections>
<Packages>
<Package Name="so_25125838" ConstraintMode="Linear">
<Variables>
<Variable DataType="DateTime" Name="LastRunDate" >2014-01-01</Variable>
<Variable DataType="Int32" Name="RowCountOriginal" >0</Variable>
<Variable DataType="String" Name="QueryAdo" EvaluateAsExpression="true">"SELECT RD.* FROM dbo.RunData AS RD WHERE RD.InsertDate > '" + (DT_WSTR, 25) #[User::LastRunDate] + "';"</Variable>
</Variables>
<Tasks>
<ExecuteSQL
Name="SQL GetLastRunDate"
ConnectionName="CM_ADO_DB"
ResultSet="SingleRow"
>
<DirectInput>SELECT MAX(RJ.LastRunDate) AS LastRunDate FROM dbo.RussJohnson AS RJ;</DirectInput>
<Results>
<Result Name="0" VariableName="User.LastRunDate" />
</Results>
</ExecuteSQL>
<Dataflow Name="DFT POC">
<Transformations>
<AdoNetSource Name="ADO_SRC Get New Data" ConnectionName="CM_ADO_DB">
<DirectInput>SELECT RD.* FROM dbo.RunData AS RD</DirectInput>
</AdoNetSource>
<RowCount Name="CNT Original rows" VariableName="User.RowCountOriginal" />
</Transformations>
<Expressions>
<Expression ExternalProperty="[ADO_SRC Get New Data].[SqlCommand]">#[User::QueryAdo]</Expression>
</Expressions>
</Dataflow>
</Tasks>
</Package>
</Packages>
</Biml>
I have a ton Visual FoxPro db files that I am trying to import into an empty SQL 2008 Express database. When I run through the SQL Import and Export Wizard everything seems to communicate fine. When I get to the mappings section I can click on preview and see the data in the selected FP table. When I click on Edit Mappings or Next I get:
===================================
Column information for the source and destination data could not be retrieved.
"eqr_sellers" -> [dbo].[eqr_sellers]:
- Cannot find column -1.
(SQL Server Import and Export Wizard)
===================================
Cannot find column -1. (System.Data)
------------------------------
Program Location:
at System.Data.DataColumnCollection.get_Item(Int32 index)
at Microsoft.DataTransformationServices.Controls.ProviderInfos.MetadataLoader.LoadColumnsFromTable(IDbConnection myConnection, String[] strRestrictions)
at Microsoft.SqlServer.Dts.DtsWizard.OLEDBHelpers.LoadColumnsFromTable(MetadataLoader metadataLoader, IDbConnection myConnection, String[] strRestrictions, DataSourceInfo dsi)
at Microsoft.SqlServer.Dts.DtsWizard.TransformInfo.PopulateDbSourceColumnInfoFromDB(IDbConnection mySourceConnection)
at Microsoft.SqlServer.Dts.DtsWizard.TransformInfo.PopulateDbSourceColumnInfo(IDbConnection mySourceConnection, ColumnInfoCollection& sourceColInfos)
Any insight would be appreciated.
What are the data types? Auto-Increment Integer fields are not supported by the ODBC connectors I have used in the past.