In my work place's SQL Server 2008 set up, we have set up a database table called DATE_RANGE_CONTROL with two columns: startDate and endDate of data type DateTime. Our SSIS packages extract data from an Oracle Database for a set date range so most of our SQL code in the DB Source Editor takes the form of:
SELECT COLUMN1,COLUMN2,COLUMN3,TRANSACTION_DATE
FROM SOURCE_DATABASE_TABLE
WHERE TRANSACTION_DATE BETWEEN startDate AND endDate
What I want to do is to set the startDate and endDate dynamically so that at the time of execution, the query gets the values from the DATE_RANGE_CONTROL table. I have looked at using the SQL Command from variable as an option but I cannot figure out how to feed the values of startDate and endDate from the DATE_RANGE_CONTROL table to the DB Source SQL command.
I have done something similar in the SQL SERVER 2000 DTS packages where I was setting the SQLCommand property of the data pump dynamically using the ACTIVEX script task. How do i do it in SSIS?
After spending a considerable amount of time on Google and looking through the Microsoft SQL Server 2008 SSIS documentation, I have finally cracked it. So proud of myself. The solution is thus:
In SQL Server Business Intelligence Development Studio, create a new SSIS package.
Create 3 variables, I called mine startDate, endDate and strDataSourceSQL. Make sure they all have a scope level of Package and ensure thay have Data Types of Int32, Int32 and String respectively..
On the Control Flow tab, drag a Script Task from the toolbox to the Control Flow pane. Right click it and choose edit from the pop up menu.
On the Script page, click on the ellipsis to the right of ReadWriteVariables. Tick the variables you created in step 1 i.e. user::startDate, user::endtDate and user::strDataSourceSQL. You can rename your script task on the General page. It is good practice.
On the Script page,click on the Edit Script... button to reveal a visual studio IDE pane with some pre-written code.
Make the following changes. At the top of the page, you will notice a few lines the import namespaces and classes to use. Edit the lines of code and make sure you have the following lines:
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Data.OleDb;
Go to the menu Project > Add Reference. On the .NET tab, make sure you select Microsoft.SqlServer.DTSPipelineWrap and Microsoft.SqlServer.DTSRuntimeWrap Credit to DotNetMonster for the tip.
In the public void Main() method, you will see a comment that says // TODO: Add your code hereWe will replace that line or add the following lines of code. By the way, I am using OLEDB for my data source and data destination.
//Declare variables
Microsoft.SqlServer.Dts.Runtime.Wrapper.IDTSConnectionManagerDatabaseParameters100 cmParams;
System.Data.OleDb.OleDbConnection oleDbConn;
ConnectionManager cm;
Int32 strStartDate;
Int32 strEndDate;
String startDateSQL = "SELECT StartDate FROM DATE_RANGE_CONTROL";
String endDateSQL = "SELECT EndDate FROM DATE_RANGE_CONTROL";
//Declare and assign Connection manager
cm = Dts.Connections["YOUR_OLEDB_CONNECTION_MANAGER_NAME"];
//Set Connection paramters
cmParams = cm.InnerObject as Microsoft.SqlServer.Dts.Runtime.Wrapper.IDTSConnectionManagerDatabaseParameters100;
//Instantiate the Oledb connection
oleDbConn = cmParams.GetConnectionForSchema() as OleDbConnection;
Assign the strStartDate and strEndDate variables, the result of running a function that we will display later. All it does provide an SQL command to the OLEDB connection.
strStartDate = (int)assignSQLCommand(startDateSQL,oleDbConn).ExecuteScalar();
strEndDate = (int)assignSQLCommand(endDateSQL, oleDbConn).ExecuteScalar();
//Assign the startDate and endDate variables the value of the SQL resultset
Dts.Variables["startDate"].Value = strStartDate;
Dts.Variables["endDate"].Value = strEndDate;
Declare a variable called dataSourceSQL of type string and assign your DataSource SQL code to it. Replace the date range values with the values of the variables above. For me, the code is as follows:
String dataSourceSQL = "SELECT COLUMN1,COLUMN2,COLUMN3,TRANSACTION_DATE
FROM SOURCE_DATABASE_TABLE
WHERE TRANSACTION_DATE BETWEEN '"+Dts.Variables["startDate"].Value +"' AND '"+Dts.Variables["endDate"].Value+"'";
Then assign the value of dataSourceSQL to your strDataSourceSQL package variable
Dts.Variables["strDataSourceSQL"].Value = dataSourceSQL ;
Outside the curly brackets of the public void Main() method, add the following function we discussed earlier.
public OleDbCommand assignSQLCommand(String SqlCommand, OleDbConnection oledbconn)
{
return new System.Data.OleDb.OleDbCommand(SqlCommand, oledbconn);
}
Go to the menu Build > Build [script name]. Save it and close the script page. Finally, all that is left is to create a Data Source task. Go to the Data Flow tab, drag an OLE DB DB Source to the pane. Right click on the OLE DB Source and choose edit. On the Connections Manager page, select your OLE DB connection manager and on the Data access mode choose SQL command from variable. In the variable name select user::strDataSourceSQL. As long as you set up your script task to run before your data flow, the job is done!
This is how I could do that.
In the OLE DB Source, enter your sql statement and enter ? as the parameter.
click [Parameters...] button and pick your variable from the list
Related
There are 4 Connection strings with different SQL Servers (which I set up in SSIS Connection Managers section):
Database name is same in all the servers:
SERVER DATABASE
dbTestServer dbFees (Main Server and Database)
dbTestServer1 dbFees1
dbTestServer2 dbFees1
dbTestServer3 dbFees1
dbTestServer is the OLEDB Source and other Servers are OLEDB Destination that needs to be updated everytime we run package.
Now, I want to take data from dbTestServer-dbFees and copy to all the other databases. I created a Dataflow task to copy data from dbTestServer to dbTestServer1.
But I need to put this data flow task inside ForEach Loop container to change the connection/Server dynamically so that it will work like:
First run- By default OLEDB Source is set to dbTestServer and OLEDB Destination is set to dbServer1 and data is copied from dbFees to dbFees1.
Second run- OLEDB Source is set to dbTestServer and OLEDB Destination is set to dbServer2 and data is copied from dbFees to dbFees1
Third run- OLEDB Source is set to dbTestServer and OLEDB Destination is set to dbServer3 and data is copied from dbFees to dbFees1.
I need step by step solution as I am new to SSIS packages and I tried multiple solutions but NOTHING worked so far!
Appreciate your help!
Thank you
I suggest using FOR LOOP.
My logic is to increment variable on each loop and create an expression with a connection string and a number of iteration.
1st step is to create Connection Manager with server name dbTestServer1 and database name dbFees1
2nd step will be to add a connection manager to OLE DB Destination
3rd step is to create 2 variables: ConnString and Iteration.
For Iteration default value set to 1, because you need dbTestServer 1
ConnString you need to set like your initial connection string, just on place 1 in dbTestServer1 to set (DT_STR, 1, 65001)#[User::Iteration].
Like on next 2 pictures:
When you set variables, you need to set expression in OLE DB Connection Manager.
From drop-down select connection string and type #[User::ConnString].
And finally set FOR LOOP like on picture
NOTE: I can't test package because I don't have server names like you, but this is logic of how to solve your problem. And this is only solution for what you asked, you must create whole package on your own.
For main server and database, just add one OLE DB Source with static names for server name and database name.
And you don't need script task if you using my logic.
Here is the code I have used to dynamically change connection server/database inside C# Script task in SSIS:
Variables I pass to the C# Script task under ReadOnlyVariables:
(set these up in your Variables inside SSIS)
User::DatabaseListOnThisLoop_ConnectionString
User::DatabaseListOnThisLoop_DatabaseName
This is the name of the connection string I am dynamically change that is in my ConnectionMangers in SSIS:
SourceServerDBForClassification_Dynamic
FULL SCRIPT from my C# Script task inside SSIS. As long as you setup the variables and put the 2 in above in the ReadOnly section of the script task, you should be able to just copy/paste the entire code below into your C# Script task.
NOTE: The Namespace may give you an issue so may want to keep the one that is generated in your code when adding the script task.
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms; // dont think this is needed, I used this for message box for some testing, but leaving here just in case
namespace ST_f8d6dad17af541bbb0010c9fce3ccbb0
{
[Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
{
public void Main()
{
// get connection string from variable
string ServerConnectionStringOnThisLoop = Dts.Variables["DatabaseListOnThisLoop_ConnectionString"].Value.ToString();
string DatabaseOnThisLoop = Dts.Variables["DatabaseListOnThisLoop_DatabaseName"].Value.ToString();
// this could change depend on what type of connection you are using for provider and other settings
string DynamicConnectionString = "Data Source=" + ServerConnectionStringOnThisLoop + ";Initial Catalog=" + DatabaseOnThisLoop + ";Provider=SQLNCLI11.1;Integrated Security=SSPI;";
// Add the OLE DB connection manager set to existing connection
ConnectionManager SourceServerDBForClassification_Dynamic = Dts.Connections["SourceServerDBForClassification_Dynamic"];
// now set the dynamic connection above to the connection string passed in from SSIS package
SourceServerDBForClassification_Dynamic.ConnectionString = DynamicConnectionString;
// now set the package connection to the one we just created from using the variable from the SSIS package
Dts.Connections["SourceServerDBForClassification_Dynamic"].ConnectionString = SourceServerDBForClassification_Dynamic.ConnectionString;
Dts.TaskResult = (int)ScriptResults.Success;
}
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
}
}
I have tried making a dataset using the dataset tool to link it on the design side however that hasn't worked as it keeps disappearing and this is the other way it does not display any values on the datagrid view i am unsure why and i am fairly new to this side of vb so if you could explain it as well that would be great. Thanks in advance.
Imports MySql.Data.MySqlClient
Public Class Search
Private Async Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
Try
If TextBox1.Text = "" Then 'this acts as a simple presence check on the textbox
Else
Dim val = "name"
If RadioButton1.Checked = True Then 'This changes the type of search i do as it filters which column the query looks in
val = "type"
End If
Await getDataSet(TextBox1.Text) ' waits for infomation to be retrieved
End If
Catch ex As System.Exception
System.Windows.Forms.MessageBox.Show(ex.Message) 'Catches any errors
End Try
End Sub
Async Function getDataSet(partname As String) As Task(Of DataSet) 'This retrieves the values that matches the users input
Return Await Task.Factory.StartNew(
Function()
Dim connectionString = "server=localhost; userid=root; password=; database=partstest1; CharSet=utf8;" 'These are the login details for the database in the form of a connection string
Dim commandText = "SELECT ID, Benchpoint, Name, Type, BrandID FROM `parts` WHERE `name` Like '%" & TextBox1.Text & "%';"
Using connDB = New MySqlConnection(connectionString), objCmd = New MySqlCommand(), objAdpt = New MySqlDataAdapter()
connDB.Open()
objCmd.Connection = connDB
objCmd.CommandText = commandText
objCmd.CommandType = CommandType.Text 'These lines specify the command i am using and execute it
objAdpt.SelectCommand = objCmd
Dim objDs = New DataSet()
objAdpt.Fill(objDs) 'Puts all of the values into a dataset
PartsDataGridView.DataSource = objDs.Tables(0) 'This shows the datasource and displays it
Console.WriteLine(objDs)
Return objDs
End Using
End Function)
End Function
End Class
Install MySQL/connector/etc
Install MySQL for Visual Studio (needed to get it to show as a datasource in VS) - https://dev.mysql.com/downloads/windows/visualstudio/
Add a new dataset to your project
Double click the dataset, right click the surface of it, choose Add >> TableAdapter
Choose existing connection or New Connection if you have no existing. Choose MySQL Database (install MySQL for Visual Studio from step 2 if this is absent)
Enter server details, choose to save the password, back in the wizard choose Yes, include sensitive.. and click Next, choose Yes, save the connection string, click Next, choose Use SQL Statements, Next, enter a suitable query that selects based on the primary key such as SELECT * FROM yourtable WHERE id = #id, Next, call the methods FillById and GetDataById to distinct them from other queries you may add later like FillByCity or GetDataByStatus. At the end of this wizard you should have a datatable and tableadapter that looks like your database table:
Switch to the form designer, and also ensure that the Data Sources window is visible (click View Menu >> Other Windows >> *Data Sources**), expand all the nodes in the Data Sources panel
Drag the node representing your table (mine is called person) out of the datasources window and drop it on the form. Several things will appear:
a datagridview already bound to the dataset's datatable showing person data,
a toolstrip with a textbox for inputting the ID (remember you made a query that takes a parameter),
another toolstrip with navigators and a save button),
a dataset (needed to store the data),
a bindingsource (acts as a link between the dataset's datatable and the grid, knows what current record is showing, allows sorting and filtering),
a tableadapter (an enhanced data adapter that will pull data from the db into the dataset and send back any changes),
a tableadapter manager (a device that knows what order to run updates in for hierarchical data)
Also, as a demonstration of how it works, drag all the nodes UNDERNEATH person onto the form also - you can see they have different icons. They will re-use most of the components on the form but you'll get textboxes, datetimepickers, checkboxes etc that are bound to the bindingsource and will hence show the current item from the underlying list (the datagridview shows them all, with an indicator of which row is current)
Run the project, type an ID into the textbox, hit **FillById*, write some new info into another row at the bottom - it will commit to the underlying dataset/datatable when you change row in the DGV, note that flicking back and forth between the rows in the DGV causes the textboxes to change as the notion of Current row is updated (navigating the grid, or clicking the arrows in the navigator changes the bindingsource .Current property, and all controls bind through the bindingsource
hit save - now go look in your db in MySQL Workbench - the new record is there. You've made a full databound UI, and not written a line of code - all the code is there in the FormX.vb if you want to see it - written for you by the designer. Take a look
Note that it still says -4 (in my UI) for the temp id; the program didn't download the new ID that was assigned by the DB. To make it do this, youre supposed to be able to go back to your dataset, right click the tableadapter, click Advanced Options, and tick on Refresh the datatable - this will cause the adapter to update any local IDs after it saves the row. IDs can cascade update if they are part of a datarelation between two datetables. In MySQL this doesn't work (I'm reporting a bug to them now), but you can make it work by manually editing the XML file representing the dataset (right click the DataSet in solution explorer, choose Open With.. Choose XML Editor, locate the insert statement eg:
INSERT INTO `person` (`Name`, `Birthdate`, `Salary`) VALUES (#p1, #p2, #p3)
Add another statement after it so it looks like:
INSERT INTO `person` (`Name`, `Birthdate`, `Salary`) VALUES (#p1, #p2, #p3)
;SELECT `Id`, `Name`, `Birthdate`, `Salary` FROM `person` WHERE `Id` = last_insert_id()
Now, saving the table will cause the UI to refresh with the ID values calculated by the auto increment in the DB
I have a strange situation. I have an SSIS package and this package takes connection string from a table in SQL server something like
(Data Source=XYZ;Initial Catalog=Mail;Provider=SQLNCLI11.1;Integrated Security=SSPI;Auto Translate=False;) and there are many more connection string like this.
It uses this connection string to insert records to the desired SQL server. But, whenever a server is not reachable the SSIS package fails and stops execution.
I want to know how we can continue SSIS package if a particular server fails so that it doesn't effect other servers. Any help or comment is appreciated.
I assume the current package looks something like
Execute SQL Task (Get master list of connection strings)
ForEach (recordset) Enumerator (Assign current connection string to Variable)
Execute SQL Task (Inserts into table)
For each record you find, you assign that to a variable which is then used to drive the ConnectionString property of an OLE DB Connection manager.
Assuming that approximates the problem, you would need to add a precursor step to #3 which tests the validity of the connection string/manager. In SSIS, this mostly commonly be implemented through a Script Task. Rather than deal with Failing the Script Task, I'd also create an SSIS variable called IsConnectionValid and the result of the Script Task will be to set that to True or False depending on the state of the connection.
Psuedologic
Assumes Read only Collection is our variable #[User::ConnectionString]
Assumes Read/Write collection is our variable #[User::IsConnectionValid]
Assumes I can code without opening a text editor but the logic is sound
// re-initialize our state to false
Dts.Variables["User::IsConnectionValid"].Value = false;
// Access the current connection string
string cs = Dts.Variables["User::ConnectionString"].Value.ToString();
try
{
using (System.Data.SqlClient.SqlConnection conn = new System.Data.SqlClient.SqlConnection(cs))
{
conn.Open();
// If the connection didn't blow chow, then it's good *right now*
// The server could fall over in the next instant but this is the best we can do
Dts.Variables["User::IsConnectionValid"].Value = true;
}
}
catch(Exception ex)
{
// Swallow the error
;
}
Now that we have this script task running and safely evaluating our connection, then the last step is to change the Precedence Constraint between this new step 3 (script task) and the old step 3 (Execute SQL Task). Double click the connecting line and change it from the current default of OnSuccess to PrecedentAndConstraint (name approximate)
The constraint is simply #[User::IsConnectionValid] and it remains an AND conditional. This means that it will only give the Execute SQL Task the signal to fire if the variable is true and the preceding task didn't fail.
I am very new to MS Access, forgive me for this simple question but I am very confused with my current problem.
So I want to run a VBA function after a table receives an update on one of its fields. What I have done is:
Create a Macro named Macro_update under CREATE->Macro, with action RunCode, and its argument is the VBA function I wish to run. The function has no bug.
Select my table, and under Table->After Update, I wrote
IF [Old].[status]=0 And [status]=1 THEN
RunDataMacro
MacroName Macro_update
But after I update my status field in my table nothing happened... I am suspicious of the fact that in step 2 my action is RunDataMacro, but I am actually running a Macro (is there a difference?)... any help is appreciated!
You can use a Data Macro to get it working locally for now. This means that the table will need to be stored in an Access database.
If your web service is not actually using the Access Runtime to interface with the access database container, then the data macros may not fire correctly nor as intended. Your mileage may vary.
If you later migrate your database to a SQL server (MySQL, Microsoft SQL, PostgreSQL) then your data macros will need to be implemented natively on the SQL server as a Trigger.
For now, I'm writing some instructions below to demonstrate how to call a VBA function from a Data Macro locally within a single Access database:
Create the VBA Function This will be the function that you want to call from the data Macro.
Create this in a Module, not in a Form or Class Module.
This has to be a function and cannot be a sub
Code:
Public Function VBAFunction(OldValue As String, NewValue As String) As String
Debug.Print "Old: " & Chr(34) & OldValue & Chr(34);
Debug.Print vbTab;
Debug.Print "New: " & Chr(34) & NewValue & Chr(34)
VBAFunction = "Worked"
End Function
Create the Data Macro (Going to be more descriptive here since people get lost here easy)
Open the Table (i.e. TestTable) in Design View
Find the correct Ribbon
In table design view, there is a contextual ribbon called Design.
On that ribbon, there is an option called Create Data Macros
Click on Create Data Macros and select After Update
The Macro Designer window should now open
Choose SetLocalVar from the Add New Action combo box
A SetLocalVar section appears.
In this section, I see Name and Expression
Set Name to an arbitrary value, such as: MyLocalVar
Set Expression to the following
Be sure to type the = sign, which will result in two equal signs being shown
Expression Text:
=VBAFunction([Old].[FieldName],[FieldName])
Save the Data Macro and Close the Macro Designer.
Save the Table and Close the Table
Test It: Create an Update Query
Next you will need to create an Update Query that performs an update on the Table that houses the Data Macro you just created.
To test this, you can just update a string field in all records to one value.
UPDATE [TestTable] SET [TestText] = "Test"
Run the query
Press Control + G to bring up the Immediate Window. You will notice that the Data Macro fired for every updated record.
I am running a SSIS package to load say a million rows from a flat file, which uses a script task for complex transformations and a SQL Server table destination. I am trying to figure out the best way (well, ANY way at this stage) to write out to a different table the row count (probably in multiples of 1000 to be more efficient) DURING the data flow processing. This is so that I can determine the percentage of progress throughout a task that might take a few minutes, simply by querying the table periodically.
I can't seem to add any SQL task into the flow, so I'm guessing the only way is to connect to the SQL database inside the .NET script. This seems painful and I'm not even sure it is possible. Is there another more elegant way? I've seen reference to "Rows Read" performance counter but not sure where I access this in SSIS and still not sure how to write it to a SQL table during the Data Flow processing.
Any suggestions appreciated.
Glenn
there are two easy options here:
Option 1: use the built-in logging with SSIS and watch the on progress event. this can be configured to log to several different outputs including relational database and flat files
See more Here
Option 2: you could add a SSIS script component that could fire off notifications to an external system like a database table
I recently solved this in a slightly different manner, which I find superior to using scripting and opening separate connections in code to DBs:
In the source query or a transform shape, add a row count (incremental)
In a conditional branching, use a modulo expression (%) to branch whenever the number is a multiple of for example 1000, but this could be configurable or based on source data (for example 0.0% to 100.0% of the data)
Create a log connection manager and use a destination. Control the batching sizes so that rows are immediately committed to the target table.
Why not write a .NET application and you can integrate into that to get information as to where the SSIS package is at.
Basically everything that is sent to the console you can get, and there are event handlers you can attach to to get information while it is processing the package.
Here is a link that may help you to go with this approach:
http://www.programminghelp.com/database/sqlserver/sql-server-integration-services-calling-ssis-package-in-c/
OK, had some success at last.... added a call to the following sub in the script component:
Sub UpdateLoadLog(ByVal Load_ID As Int32, ByVal Row_Count As Int32, ByVal Row_Percent As Int32, ByVal connstr As String)
Dim dbconn As OleDbConnection
Dim Sql As String
Dim dbcomm As OleDbCommand
dbconn = New OleDbConnection(connstr)
dbconn.Open()
Sql = "update myTable set rows_processed = " & Row_Count & ", rows_processed_percent = " & Row_Percent & " where load_id = " & Load_ID & " and load_log_type = 'SSIS'"
dbcomm = New OleDbCommand(Sql, dbconn)
dbcomm.ExecuteNonQuery()
dbconn.Close()
dbconn = Nothing
dbcomm = Nothing
End Sub
This gets executed every 1000 rows, and successfully updates the table. The row already existed as it gets created in the control flow at the start of the package, and updated again in the control flow at the very end with final rowcount and 100%.
Thanks for all your suggestions guys.
Is the application consuming the row count a .net application? When it comes to sharing information between applications there are a lot of accepted practices. May be you should take a look in to them. And for your particular case, if it is .net application that consumes this row number for calculating progress, may be you can store the information some place else other than a DB table, like file system, web service, windows environment variables, log (like windows events log), etc are some that came to my mind now. I think updating a windows environment variable with row count form with in your script component will be a good enough solution. Just like using a global variable to share data between two functions inside a program. :)