I am creating an SSIS package that will run each month. This particular stored procedure needs to run for one week at a time since the data returned is very large.
I have set up my stored procedure to with two parameters: #StartDT and #EndDT. I created two SSIS variables: StartDT and Wk1EndDT (I'll create the other start and end dates for the weeks once I get this one working).
StartDT has this expression:
(DT_DATE)((DT_WSTR, 4)YEAR(DATEADD("mm", -1, GETDATE())) + "-" +RIGHT("0" + (DT_WSTR,2)MONTH(DATEADD("mm", -1, GETDATE())),2)+"-01")
Wk1EndDT has this expression:
DATEADD("DD",7, #[User::StartDT])
I'm using a DataFlow task with a SQL command text of:
EXECUTE dbo.uspUploadWk1 ?,?
When I go to preview the results, I receive the following error message:
There was an error displaying the preview.
No value given for one or more required parameters. (Microsoft SQL Server Native Client 11.0)
I have the parameters set like this:
I am not sure why this isn't working. I've searched all over and have not found an answer. I am using Visual Studio 2015.
Assuming an OLE DB Connection Manager, the Mappings tab should be using a zero based ordinal system on the Parameters column. Yes, it defaults to naming them as Parameter0, Parameter1, etc but for an OLE DB connection manager, you'll use the ordinal position of the question marks, ?, starting at zero.
For ODBC, it becomes a 1 based counting but still uses ? as the parameter place holder.
ADO.NET uses named parameters so we'd match EXECUTE dbo.uspUploadWk1 #Parameter0, #Parameter1 but the ADO.NET source component doesn't support parameterization
Reference on parameters and mapping for Execute SQL Task but the syntax remains the same for Data Flow Task components
Related
I have made the ssis package in which i made the data flow for incremental data. Source and destination server ip's are different. Below you can find the flow diagram of my packageControl flow diagram
Data flow diagram
the package is working fine .
In the Execute SQl task :- it controls the log table and start the incremental task
query which i used is :-
insert into audit_log (
Packagename,
process_date,
start_datetime,
end_datetime,
Record_processed,
status
)values('CRM-TO-TRANSORGDB',null,GETDATE(),null,null,null);
select MAX(ID) as ID,MAX(process_date) as proc_date from audit_log where Packagename ='CRM-TO-TRANSORGDB' ;
store the ID and proc_date in the variable.
in the Execute SQl task 1:- it just update the log table.
UPDATE audit_log
SET
process_date=?,
end_datetime = GETDATE(),
status='SUCCESS'
record_processed=?
WHERE (packagename = 'CRM-TO-TRANSORGDB') AND ID=? ;
this is the query we have used to update the log table.
In the Data flow simple fetching the all the records and put in into the destination table.
this all i have done .
But my question are:-
1) How to compare the total no. of row counts from the source table to destination table in ssis package.
2) if its doesn't matches than it will restart my task automatically.
#thomas as per your instruction i have done the following thing:
1) i have made the Execute SQl Task for source and destination .
2) and Add the Execute Package task and added the condition for not matching the count.
and added the expression for check row_count_src!= row_count_dest
and in Source_table_count i have used the below query:
select count(SubOrderID) as row_count_src from fact_suborder_journey
WHERE Suborderdate between '2016-06-01' and GETDATE()-1 ;
in dest_table_count i have used the below query:
select count(SubOrderID) as row_count_dest from fact_suborder_journey
WHERE Suborderdate between '2016-06-01' and GETDATE()-1 ;
i have added the two variable as int64 in ths ssis package. and map in the result set below you can find the pic what i have done.
but After done all this this i am getting this error:
[Execute SQL Task] Error: An error occurred while assigning a value to variable "row_count_src": "The type of the value being assigned to variable "User::row_count_src" differs from the current variable type. Variables may not change type during execution. Variable types are strict, except for variables of type Object.
".
I havent tested this completely but you might be able to do something like this. This creates a loop of your packages and will executes as long as your count variables are different from each other.
What have i done?
First i have a DataFlow Task which moves data from source to
destination.
Then i have an Execute SQL task which basically counts all rows from
TableA and maps it to variable count1 eg. Source table
Then i have an Execute SQL task which basically counts all rows from
TableB and maps it to variable count2 eg. Destination Table
Then i create an Execute Package task where i reference it too it
self. Then i make a precedence constraint with an expression saying
Count1 != count2.
Because if they are different you want to restart the task. If they
are equal the last task Execute Package task will never be executed.
Hope that is something like that?
If I understand your challenge correctly...
In the data flow task, use a RowCount transformation between source
and destination to capture the rows written to the destination. This
will be stored in a variable.
In the control flow, get the max row counts available from the log table and store that a variable.
Create an execute package tasks that executes this same package and put a precedence constraint before if that compares if variable from Step1 <> variable in Step2.
I am a bit new to SSIS and given a task to send mail to particular stores based on Purchase Orders -> PONumber.
The steps should be as follows:
1)Take a XML file from a particular folder
2)Get the PONumber from that file
3)Write a query to fetch all the store email addresses for PONumbers
4)Send a mail to particular restaurant
Below screenshot is a package I had created. The only thing I am getting an issue is the Execute SQL Task , not sure what is the exact cause?
Could you please help on how can I debug this ? This was working fine before, but suddenly it started showing errors.
IMAGE1
IMAGE5
Execute SQL task is expecting results from the query, but is not getting any. Maybe you could use SQL Server profiler to catch exact SQL that is executed on SQL Server. Then you can use that SQL in query window to troubleshoot what it returns or why it is not not giving any results.
Edit.
With your current additional information interesting place is "parameter mapping" page, which you did not include. You should link SSIS variable to query parameter in there as Matt explained. SSIS does NOT link your variables in SSIS and query automatically even if they have the same names.
#dvlpr is correct your problem is you are getting NO results when Execute SQL Task 1 needs a single result.
The code you pasted is a little unclear as to which code is where but I will assume the first part is the code you use in SSIS Execute Task and the latter is an example in SSMS. If that is the case the problem is you are assigning the variable with a value of 0 in the script itself which I assume there is no PONUMBER that is 0:
Declare #POID as Varchar(50)
Set #POID = 0
WHERE (BizTalk_POA_HEADER.PONUMBER = #POID)
If you want to pass in the PONUMBER from your first dataflow task you need to load that to a variable and then use the variable in your Execute SQL task and made sure you setup parameter mapping correctly when doing so. here is one SO question on parameters that will help How to pass variable as a parameter in Execute SQL Task SSIS? And here is use of an expression task in a Data Flow task to set the variables value SSIS set result set from data flow to variable (note use the non-accepted answer that it was added later and was for 2012+ while the original was for 2008)
Next unless you are guaranteed only 1 result you will also need to add TOP 1 to your select statement because if you get more than 1 result you will get a different error again.
EDIT Per all of the comments:
So the configuration looks like you are using an ADO.NET connection which allows you to use named paramaters. There are restrictions if you don use that (https://msdn.microsoft.com/en-us/library/cc280502.aspx). The parameter mapping looks correct, and the result set should be fine. As far as your Error I don't know because you haven't posted the exact error so I cannot know what is the problem. If you use ADO.Net with your current Execute SQL Task configuration in the images you do have a couple of problems. 1 you are trying to declare the variable that you want to pass as a parameter that doesn't work, you need to remove that DECLARE statement. I suspect all you really need to do is modify your SQL Input to be:
SELECT DISTINCT BizTalk_POA_HEADER.PONUMBER, FAN_Suppliers.SupplierName,
FAN_Company_Details.CompanyName, FAN_Company_Details.[PrimaryEmail],
BizTalk_POA_HEADER.[DeliveryDate]
FROM BizTalk_POA_HEADER INNER JOIN
FAN_PO_Details ON BizTalk_POA_HEADER.PONUMBER =
CONCAT('PO',FAN_PO_Details.PoNumber) INNER JOIN
FAN_PO ON FAN_PO_Details.PurchaseOrderID = FAN_PO.PurchaseOrderID
INNER JOIN FAN_SupplierDetails ON FAN_PO.SupplierDetailsID =
FAN_SupplierDetails.SuppliersDetailsID INNER JOIN
FAN_Suppliers ON FAN_SupplierDetails.SupplierID = FAN_Suppliers.SupplierID
INNER JOIN FAN_Company_Details ON FAN_PO.CompanyID =
FAN_Company_Details.CompanyDetailsID
WHERE (BizTalk_POA_HEADER.PONUMBER = #POID)
Just get rid of the declare #POID and SET = 0 for a couple of reasons 1 because it is redundant when you have setup parameter mapping, 2 SSIS doesn't like it and will throw an error, 3 because you are setting a value of 0 to it which means it would always be 0.....
I need to get last processed date of SSAS cube in SSIS and save it into a variable.
I've tried a "Execute SQL task":
SELECT LAST_DATA_UPDATE as LAST_DT FROM $system.mdschema_cubes
WHERE CUBE_NAME = 'CubeName'
It works ok in MSSQL management studio MDX query window but in SSIS it says: Unsupported data type on result set binding.
Then I've tried:
WITH MEMBER [Measures].[LastProcessed] AS ASSP.GetCubeLastProcessedDate() SELECT [Measures].[LastProcessed] ON 0 FROM [CubeName]
And it says '[ASSP].[GetCubeLastProcessedDate]' function does not exist.
Any ideas how to do this?
Thank you
A linked server might be your best option;
Create the linked server with the following, changing as appropriate:
EXEC master.dbo.sp_addlinkedserver
#server = N'LINKED_SERVER_OLAP_TEST', --Change to a suitable name
#srvproduct='', --Creates the productname as blank
#provider=N'MSOLAP', --Analysis Services
#datasrc=N'localhost', --Change to your datasource
#catalog=N'TESTCUBE' --Change to set the default cube
Change the data source of your Execute SQL Task to make sure it is pointing to any of the databases where the linked server is hosted, I.E. don't use an analysis service datasource use a standard OLE DB. Then have the following in your execute SQL task (Changing as appropriate).
SELECT *
FROM OpenQuery(LINKED_SERVER_OLAP_TEST,'SELECT LAST_DATA_UPDATE as LAST_DT FROM $system.mdschema_cubes
WHERE CUBE_NAME = ''CUBENAME''')
Set the variable to be DATETIME and the result set to be single row.
There may well be other ways to do this, however I have always found this method the most straight forward.
So here we have an error I keep getting in my SSIS package but I can't see what is wrong with the statement. I have even tried another sql statement from a project that works and it still raises the error.
The system is VS 2005 running 64 bit debugger, on XP machine. The project has amongst other things a script task then a sql task, the script task outputs the month value to a variable (Dts.Variables("monthName").Value = month), which I then use to create dynamic table name in SQL statement. I haven't got to the excel sheet bit yet as I am trying to get the sql task stage working.
So i have a variable at package level called SQLTableCreate, and in that I have the properties set to:
Evaluate as Expression = true
Expression = "Create Table "+ #[user::monthName]+"(Column1 DATETIME,Column2 NVARCHAR(255),Column3 NVARCHAR(255),Column4 NVARCHAR(255),Column5 NVARCHAR(255),Column6 NVARCHAR(255),Column7 NVARCHAR(255),Column8 NVARCHAR(255),Column9 NVARCHAR(255),Column10 NVARCHAR(255))"
And when I build the package I get:
Nonfatal errors occurred while saving the package:
Error at Package: The variable "user::monthName" was not found in the Variables collection. The variable might not exist in the correct scope.
Error at Package: Attempt to parse the expression ""Create Table "+ #[user::MonthName]+"(Column1 DATETIME,Column2 NVARCHAR(255),Column3 NVARCHAR(255),Column4 NVARCHAR(255),Column5 NVARCHAR(255),Column6 NVARCHAR(255),Column7 NVARCHAR(255),Column8 NVARCHAR(255),Column9 NVARCHAR(255),Column10 NVARCHAR(255))"
" failed and returned error code 0xC00470A6. The expression cannot be parsed. It might contain invalid elements or it might not be well-formed. There may also be an out-of-memory error.Error at Package: The expression for variable "SQLTableCreate" failed evaluation. There was an error in the expression.
There is also a default SQL statement for the variable SQLTableCreate, which uses the current excel connection manager table name. When I put my dynamic statement in the expression section of properties it fills the value and valuetype property of the SQLTableCreate variable with the message:
The expression for variable "SQLTableCreate" failed evaluation. There was an error in the expression.
It's exactly as the error says
The variable "user::monthName" was not found in the Variables collection
Things in SSIS are case sensitive and Variables are one of those things. Make your expression
"Create Table "+ #[User::monthName]+"(Column1 DATETIME,Column2 NVARCHAR(255),Column3 NVARCHAR(255),Column4 NVARCHAR(255),Column5 NVARCHAR(255),Column6 NVARCHAR(255),Column7 NVARCHAR(255),Column8 NVARCHAR(255),Column9 NVARCHAR(255),Column10 NVARCHAR(255))"
Also, I hope this table design is just a sample and not real. Lack of column names and strong data types is technical debt you don't need to incur at this stage.
I am trying to write ETL that collects data from many identical server into a central repository. What I'm trying to do is write one package with source address, user id and password as parameters and execute the package once per server to be copied.
Is this doable? How can I use parameters to create a source?
I meant to ask how to parametrize the connection manager (is that even a real word?), not where to store the connection parameters. The answer is simple:
Create package parameters for Server, Database, User ID and Password
Create a connection manager as part of defining a data flow component
once a connection is defined, right-click on the connection manager at the bottom of the package design screen and select "Parametrize".
Select "ServerName" in the property drop-down
Select "Use existing parameter" or create new parameter if skipped step 1
If using existing parameter, select it from the drop down
Click OK to save (gotta do it after each parameter)
Repeat steps 4-7 for the rest of the parameters
You can store parameters in a table. Query the table with a sql task and store the results in a object variable. You can then use this variable in a for loop. Use expressions in SSIS to change values of your connection during each loop iteration.
Several books outline this method. Here is a code example.
Here are some steps - hopefully I didn't miss anything. You mention a server "Address", but I'm not sure exactly what you are trying to do. This example queries multiple sql servers.
You create the variables, SQL_RS with type of object, SRV_Conn with type of string. This holds my servername. In the execute SQL task, I have a query which returns the names of sql servers I want to query. Then set the following properties:
SELECT RTRIM(Server) AS servername
FROM ServerList_SSIS
WHERE (Server IS NOT NULL)
and coalesce(exclude,'0') <> 'True'
and IsOracle is Null
Execute SQL Task > General > ResultSet = "Full Result Set"
Execute SQL Task > Result Set Tab "Result Set Name = 0", Variable Name = "User::SQL_RS"
So we have a list of server names in the SQL_RS variable now.
ForEach > Collection > Enumerator = "Foreach ADO Enumerator"
ForEach > Collection > Enumerator Configuration > ADO Object source Variable = User::SQL_RS
This maps the first column of the SQL_RS object to the SRV_Conn variable, so each iteration of the loop will result in a new value in this variable.
ForEach > Variable Mappings > Variable = User::SRV_Conn, Index = 0
Inside the ForEach are some other sql execs, performing queries on sql databases, so I need to change the ServerName of my 'MultiServer' connection. I have another connection for the initial query that got me the list of servers to query. Making the connection dynamic is done in properties of the connection - right-click the connection > properties. Click the ellipses to the right of the expressions.
Connection > Properties > Expressions > Property = ServerName, Expression = "#[User::SRV_Conn]"
Note: The index value of 0 for the variable mapping works for Native OLEDB\SQL Server Native Client. If you're using another db provider, you may need to use other index types - this makes setup more confusing.
OLEDB = 0,1
ADO.NET = #Varname
ADO = #Param1, Param2
ODBC = 1,2
Full listing here.