Update local database all modified data to server database - mysql

I'm facing a problem when I want to update data from local database to server data, replacing everything that has been modified at local database. I know it might be simple but I got no idea about this, so any help will be appreciate.
In my situation, I want to use a button to upload all modified data to
the server database. Now I'm just using 2 databases at same server to do
testing.
Private Sub btnUp_Click(sender As System.Object, e As System.EventArgs) Handles btnUp.Click
localconn.ConnectionString = lctext
serverconn.ConnectionString = sctext
Try
localconn.Open()
serverconn.Open()
Dim localcmd As New OdbcCommand("select a.acc_id as localid, a.acc_brcid, a.smartcardid, a.acc_created, a.acc_modified as localmodified, b.acd_firstname, b.acd_ic, b.acd_oldic, b.acd_race, b.acd_dob, b.acd_rescity, b.acd_resaddr1, b.acd_telmobile, b.acd_email, b.acd_telwork, b.acd_modified, b.acd_accid from nsk_account a inner join nsk_accountdetail b on a.acc_id = b.acd_accid", localconn)
Dim servercmd As New OdbcCommand("select c.acc_id, c.acc_brcid, a.smartcardid, c.acc_created, c.acc_modified, d.acd_firstname, d.acd_ic, d.acd_oldic, d.acd_race, d.acd_dob, d.acd_rescity, d.acd_resaddr1, d.acd_telmobile, d.acd_email, d.acd_telwork, d.acd_modified, d.acd_accid from nsk_account c inner join nsk_accountdetail d on c.acc_id = d.acd_accid", serverconn)
localcmd.CommandType = CommandType.Text
Dim rdr As OdbcDataReader = localcmd.ExecuteReader()
Dim thedatatable As DataTable = rdr.GetSchemaTable()
'localcmd.Parameters.Add("#localid", OdbcType.Int, "a.acc_id")
'localcmd.Parameters.Add("#localmodified", OdbcType.DateTime, "b.acd_modified")
Dim localid As String
Dim localmodi As String
localcmd.Parameters.AddWithValue("localid", localid)
localcmd.Parameters.AddWithValue("localmodified", localmodi)
For Each localid In thedatatable.Rows
Dim calldata As New OdbcCommand("SELECT acc_modified from nsk_account where acc_id ='" + localid + "'", serverconn)
Dim reader As OdbcDataReader = calldata.ExecuteReader
txtSDate.Text = reader("acc_modified").ToString
If localmodi <= txtSDate.Text Then
'do nothing, proceed to next data
Else
Dim ACCoverwrite As New OdbcCommand("Update nsk_account SET smartcardid = #mykad, acc_created = #created, acc_modified = #modify WHERE acc_id ='" + localid + "'", serverconn)
Dim DEToverwrite As New OdbcCommand("Update nsk_accountdetail SET acd_firstname = #name, acd_ic = #newic, acd_oldic = #oldic, acd_race = #race, acd_dob = #dob, acd_rescity = #city, acd_resaddr1 = #address, acd_telmobile = #phone, acd_email = #email, acd_telwork = #language, acd_modified = #detmodify WHERE acd_accid ='" + localid + "'", serverconn)
ACCoverwrite.ExecuteNonQuery()
DEToverwrite.ExecuteNonQuery()
End If
Next
MessageBox.Show("Upload success", "Error", MessageBoxButtons.OK, MessageBoxIcon.Warning)
Catch ex As Exception
MessageBox.Show(ex.Message, "Error", MessageBoxButtons.OK, MessageBoxIcon.Warning)
Finally
localconn.Close()
serverconn.Close()
End Try
End Sub
any comment or suggestion will be appreciate.

I hope you mean table by table. I didn't read your code much but you got the idea - you need 2 connections but here where there are 2 distinct ways of doing it.
Way #1 - you can use when amounts of data (how to say it better? - not huge). You can load a DataTable object with data from server and update changed records. You can use DataAdapter and issue CommitChanges - all changed/new rows will be written to server.
NOTE: you need a mechanism that will reliably able to tell which rows are new and modified on your local DB. Are you OK if your PK in local DB will be different than on the server? You need to answer these questions. May be you need a special mechanism for PK locally. For example, add rows using negative PK integers, which will tell you that these rows are new. And use "ModifiedDate", which together with PK will tell if the row needs updating.
Way #2 - use anytime, even with larger amount of data. Take a local row and examine it. If it is new - insert, if it is existing and "DateModified" changed - do update. There are variations of how to do it. You can use SQL MERGE statement, etc.
But these are two major ways - direct row insert/update and disconnected update/mass commit.
Also, you can do it in bulk, using a transaction - update some rows-commit, and start new transaction. This will help if the application being used as you updating it.
I hope these ideas help. If you do what you do, where you have
For Each localid In thedatatable.Rows
I am not sure what localid is. It should be
' prepare command before loop
sql = "Select * From Table where ID = #1"
' you will create parameter for #1 with value coming from
' row("ID")
Dim cmd As New .....
cmd.Parameters.Add(. . . . )
For Each row As DataRow In thedatatable.Rows
cmd.Parameters(0).Value = row("ID") ' prepare command upfront and only change the value
using reader as IDataReader = cmd.ExecuteReader(. . . . )
If Not reader.Read() Then
' This row is not found in DB - do appropriate action
Continue For
Else
' here check if the date matches and issue update
' Better yet - fill some object
End if
end using
' if you fill object with data from your row -here you can verify if
' update needed and issue it
. . . . . .
Next

Related

VB.Net MySQL Connector. Bulk insert stopped working

I've been using the code below to populate tables in a MySQL database for a few years and it's been working fine.
Imports MySql.Data.MySqlClient
Public sub Thing(dt as datatable)
Try
'dt is datatable object populated from MSsqlServer
Dim MySQLcmdSelect As MySqlCommand = New MySqlCommand("Select * From " & tableName) With {
.Connection = MySQLcnn
}
Dim MySQLad As New MySqlDataAdapter(MySQLcmdSelect)
Dim MySQLcmdBuilder As New MySqlCommandBuilder(MySQLad)
Dim MySQLcmd As MySqlCommand = MySQLcmdBuilder.GetInsertCommand()
MySQLcmd.Connection = MySQLcnn
MySQLad.InsertCommand = MySQLcmd
MySQLad.Update(dt)
Catch ex As Exception
Debug.Print(ex.Message)
Console.WriteLine("Error when populating " + tableName + ": " + ex.Message + vbCrLf + ex.InnerException.ToString)
WriteLog(ex.Message)
End Try
End Sub
The problem is, it's stopped working!
The connector seems to be working ok as I can drop tables and create them, but this will not let me bulk insert the contents of the datatable 'dt' into the MySQL table.
It doesn't throw an exception, it just passes over the .update(dt) line as if it were a Debug.Print() line when there's 1000s of rows in dt.
Effectively I'm selectively extracting data from tables in MS SQLserver and bulk uploading them to tables with the same name in MySQL.
a) is this the best way to do it, and
b) why has this suddenly stopped working?
According to phpMyAdmin its Server version: 5.6.51 - MySQL Community Server (GPL)
EDIT: Further info... all the SQL commands to drop existing tables and create tables work. It is only the .update(dt) that doesn't work.
Possible problems with your code could be that the schemas do not match exactly, auto increment columns, or the connection could have been disposed elsewhere (it should be declared local to the method where it is used). Another problem is that dt has had AcceptChanges called or LoadOptions were set incorrectly.
Private ConnStr As String = "Your connection string"
Public Sub Thing(dt As DataTable, tableName As String)
'check the datatable - if it is ok then this check is probably not necessary
Dim changesDT = dt.GetChanges()
If changesDT.Rows.Count < 1 Then
MessageBox.Show("No changes to update")
Exit Sub
End If
Using cn As New MySqlConnection(ConnStr),
MySQLad As New MySqlDataAdapter("Select * From " & tableName, cn)
cn.open() ' added by OP
Dim MySQLcmdBuilder As New MySqlCommandBuilder(MySQLad)
MySQLad.Update(dt)
End Using
End Sub
Perhaps you have "0000-00-00"? If so, a setting has changed the default -- new versions of MySQL disallow zero dates. Read about sql_modes.
Still can't get the bulk upload to work, but we have redesigned this so that data is inserted into MySQL at the same time we store it in the MS-SQL database and have removed the need to bulk upload.

How to use MySQL command inside another MySQLReader while loop in VB.net

I have a project list in a table in MYSQL, and each project has different tables. So in my code, first I'm getting the list of projects, and from each project I'm getting data from respective tables.
conn.Open()
cmdl_readuser = New MySqlCommand("SELECT * FROM projectlist where status = 'Active'", conn)
rl_readuser = cmdl_readuser.ExecuteReader()
While rl_readuser.Read()
Projectname = rl_readuser.GetString("ProjectName")
Space_Remove(Projectname)
cmd_listview = New MySqlCommand("SELECT `Process`,AVG(`Risk Factor`) as AvgRisk ,AVG(`Risk Factor After Mitigation`) as AvgRiskafterImp FROM " + sTableName + " ;", conn)
r2_readuser = cmd_listview.ExecuteReader()
While r2_readuser.Read()
ProjectRiskGridView.Rows.Add(r2_readuser("Process").ToString(), r2_readuser("AvgRisk"), r2_readuser("AvgRiskafterImp"))
End While
End While
conn.Close()
I'm getting data from table and put it in DataGridView. Finally, my DataGridView has records as list of Projects, and its respective average risk.
Here when control comes to the second MySQLCommand, it returns back. It doesn't go to next line. I think problem is with 'conn' .

Two users attempt do to the same thing at the same time

I'm writing a program using visual basic 2015 and MySql database and I asked this question somewhere else before but they couldn't help me.
For example we have a table named "users" :
username | coins
user1 | 3
user2 | 5
I want to change the coin value and this code is working fine but in this code : 1- I get the column's value. 2- I add a unit to the value. 3- I put the value in column. So the problem is what if the database is online and two different computers run the program and try to change the value at the same time? For example one of them is trying to give user1 2 coins and the other one is trying to give user1 4 coins. Then they click on add coin at the same time. First computer gets the value and it is 3. Second Computer gets the value and it is 3. First computer add 2 coins (3+2=5) and put 5 in the column. Now second computer add 4 coins (3+4=7) and put 7 in the column. So we have 7 in our column but we should have 9 because 3+2+4=9
So here is the question : Is there a way to add a value directly to a column or is there another way to solve this problem?
MysqlConn = New MySqlConnection
MysqlConn.ConnectionString = "Server='" & TextBox_MYSQL_Host.Text & "';UserID='" & TextBox_MYSQL_Username.Text & "';Password='" & TextBox_MYSQL_Password.Text & "';Database='" & TextBox_MYSQL_Database.Text & "';Character Set=utf8"
Try
MysqlConn.Open()
DataAdptr = New MySqlDataAdapter
DataTable = New DataTable
MysqlComd = New MySqlCommand
With MysqlComd
.Connection = MysqlConn
.CommandText = "Select * FROM users WHERE username ='" & user1.text & "';"
End With
DataAdptr.SelectCommand = MysqlComd
DataAdptr.Fill(DataTable)
If DataTable.Rows.Count = 0 Then
' Error!
Else
Dim gold As Integer = DataTable.Rows(0).Item("coins").ToString()
gold = gold + 1
MysqlComd = New MySqlCommand
MysqlComd.Connection = MysqlConn
MysqlComd.CommandText = "UPDATE users SET coins='" & gold & "' WHERE username='" & user1.Text & "'"
MysqlComd.ExecuteNonQuery()
End If
MysqlConn.Close()
Catch ex As MySqlException
' Connection Error
Finally
MysqlConn.Dispose()
End Try
End Sub
Disregarding the issues with your code as mentioned in the comments, there are two ways to address the issue you're specifically asking about.
Firstly, you can continue to do it the way you are and employ optimistic concurrency. What that does is assume that each user is the only user editing the data and save as though that's the case and then throw an exception if it turns out to not be. Specifically, when a record is saved, the current values in the database are compared with the original data retrieved by that user and the new values will be saved if and only if they match. If they don't, it's up to you to catch the ConcurrencyException that's thrown and do whatever is appropriate. That will usually be retrieving the data again and either asking the user to start editing again or merge the current data with their existing modifications.
The other option is to simply increment what's in the database rather than saving a specific value, e.g.
UPDATE MyTable SET MyColumn = MyColumn + 1 WHERE ID = #ID

How to execute two separate queries from one DBCommand object?

I have the following code that tries to get records from two different tables and then add them to the particular comboboxes. Only the first query works and the second one is ignored.
Try
sqlConn = New MySqlConnection
connStr = New String("Server = localhost; Database = gen_database; Uid = root; Pwd =")
sqlConn.ConnectionString = connStr
myCommand = New MySqlCommand("Select DevCompanyName from developer_name_table; Select DevType from development_type_table")
myCommand.CommandType = CommandType.Text
myCommand.Connection = sqlConn
ComboBox1.Items.Clear()
sqlConn.Open()
MsgBox("Connection Open.")
dR = myCommand.ExecuteReader()
Do While dR.Read()
ComboBox1.Items.Add(dR("DevCompanyName"))
ComboBox2.Items.Add(dR("DevType")) 'Error shows here Could not find specified column in results: DevType
Loop
Catch ex As MySqlException
MsgBox(ex.ToString)
Finally
dR.Close()
sqlConn.Close()
End Try
I can think of another way which is to do it in multiple query but can the code be simplified to something like this?
Using a DBDataReader with 2 queries, only only the first executes because there is no way to signal which table/query each read item is from. Your "double read" loop seems to assume they will be returned at the same time (rather than one query after the other - the same way they are sent to the DBCOmmand); if it did work that way, it would fail whenever there are not the same number of rows in each table.
Using DataTables affords you the chance to simply bind the result to your combos rather than copying data into them:
Dim SQL = "SELECT * FROM Sample; SELECT * FROM Simple"
Dim ds As New DataSet
Using dbcon As New MySqlConnection(MySQLConnStr),
cmd As New MySqlCommand(SQL, dbcon)
dbcon.Open()
Dim da As New MySqlDataAdapter(cmd)
da.Fill(ds)
End Using
' debug results
Console.WriteLine(ds.Tables.Count)
Console.WriteLine(ds.Tables(0).Rows.Count)
Console.WriteLine(ds.Tables(1).Rows.Count)
If I look at the output window, it will print 2 (tables), 10000 (rows in T(0)) and 6 (rows in T(1)). Not all DBProviders have this capability. Access for instance will choke on the SQL string. Other changes to the way your code is composed:
DBConnections and DBCommand objects need to be disposed. They allocate resources, so they need to be created, used and disposed to release those resources.
The Using block does that for us: The target objects are created at the start and closed and disposed at End Using.
The code above stacks or combines 2 such blocks.
The code fills a DataSet from the query, in this case creating 2 tables. Rather than copying the data from one container to another (like a control), you can use a DataTable as the DataSource:
cboDevName.DataSource = ds.Tables(0)
cboDevName.DisplayMember = "DevName" ' column names
cboDevName.ValueMember = "Id"
cboDevType.DataSource = ds.Tables(1)
cboDevType.DisplayMember = "DevType"
cboDevType.ValueMember = "DevCode"
The result will be all the rows from each table appearing in the respective combo control. Typically with this type of thing, you would want the ID/PK and the name which is meaningful to the user in the query. The user sees the friendly name (DisplayMember), which the code can easily access the unique identifier for the selection (ValueMember).
When using bound list controls, rather than using SelectedIndex and the SelectedIndexChanged event, you'd use SelectedValue and SelectedItem to access the actual data.
MSDN: Using Statement (Visual Basic)
You can use .net connector (download here)
Then you can install it and add it to your project (project -> references -> add -> browse).
Finally add import:
Imports MySql.Data.MySqlClient
So you'll be able to use this:
Dim connStr as String = "Server = localhost; Database = gen_database; Uid = root; Pwd ="
Dim SqlStr as String = "Select DevCompanyName from developer_name_table; Select DevType from development_type_table"
Dim ds As DataSet = MySqlHelper.ExecuteDataset(CnStr, SqlStr)
ds will contain two datatables: one for each query

SSIS Script Transformation slows down after 400k records

I have an SSIS Transformation Task that I use as my final destination task to insert data into a SQL Server table. The reason I use the transformation task and not an SQL Server Destination task is because I do not know beforehand what the columns will in the table that we will be inserting into.
In a for each loop container, I am looking for access DB's (in 97 format). The rest of the control flow basically creates a new SQL database and also a table. The access files are what we call "minute" databases where they contain minute information gathered by another process. I need to create a new SQL DB named after the 'minute' db and a table called 'MINUTE' with the columns created based on certain info from the access db. For each of our clients, based on the number of parameters they have at their site, determines the number of columns I need to create in the SQL Minute table.
In the data flow I have two key components: The OLE DB source component (Source - Minute Table) and the Script Transformation task (Destination - Minute Table).
The "Source - Minute Table" gets the data from the access database. The "Destination - Minute Table" transforms the data and inserts it into the appropriate DB and table.
Everything works as it should. I tested it on a DB with 491,000+ records and it took 1 minute. However, I'm testing with one of our larger customers that has over 50 parameters and the access database contains 2+ million records. The package flies until I reach around 477,000 records, and then it pretty much comes to a halt. I can wait 10 minutes, and even longer, until the record count updates, and then continue to wait again.
I've done much research and followed all of the recommendations and guidelines that I have found. My datasource is not sorted. I use SQL command instead of Table, etc in the OLE DB Source. I've changed the values of DefaultBufferMaxRows and DefaultBufferSize many times and get the same results.
Code:
Public Class ScriptMain
Inherits UserComponent
Private conn As SqlConnection
Private cmd As SqlCommand
Private DBName As SqlParameter
Private columnsForInsert As SqlParameter
Private tableValues As SqlParameter
Private numberOfParams As Integer
Private db As String
Private folderPath As String
Private dbConn As String
Private folder As String
Private columnParamIndex As Integer
Private columnDate As DateTime
Private columnMinValue As Double
Private columnStatus As String
Private columnCnt1 As Int16
Private dateAdded As Boolean = False
Private columnStatusCnt As String
Private columnsConstructed As Boolean = False
Private buildValues As StringBuilder
Private columnValues As StringBuilder
Private i As Integer = 0
'This method is called once, before rows begin to be processed in the data flow.
'
'You can remove this method if you don't need to do anything here.
Public Overrides Sub PreExecute()
MyBase.PreExecute()
Try
'Dim dbConnection As String = "Server=(local)\SQLExpress;Database=DataConversion;User ID=sa;Password=sa123;"
'conn = New SqlConnection(dbConnection)
'conn.Open()
'cmd = New SqlCommand("dbo.InsertValues", conn) With {.CommandType = CommandType.StoredProcedure}
'columnsForInsert = New SqlParameter("#Columns", SqlDbType.VarChar, -1) With {.Direction = ParameterDirection.Input}
'cmd.Parameters.Add(columnsForInsert)
'DBName = New SqlParameter("#DBName", SqlDbType.VarChar, -1) With {.Direction = ParameterDirection.Input}
'cmd.Parameters.Add(DBName)
'tableValues = New SqlParameter("#Values", SqlDbType.VarChar, -1) With {.Direction = ParameterDirection.Input}
'cmd.Parameters.Add(tableValues)
db = Variables.varMinFileName.ToString
folder = Variables.varMinFolderName.ToString
folderPath = folder & "\" & db & ".mdb"
dbConn = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" & folderPath
Using SourceDataAdapter As OleDbDataAdapter = New OleDbDataAdapter("SELECT DISTINCT PARAM_INDEX FROM [MINUTE];", dbConn)
Dim SourceDatatable As New DataTable
SourceDataAdapter.Fill(SourceDatatable)
numberOfParams = SourceDatatable.Rows.Count
End Using
'columnValues.Append("dtmTime, ")
buildValues = New StringBuilder
columnValues = New StringBuilder
columnValues.Append("dtmTime, ")
Catch ex As Exception
Dim writer As New StreamWriter("C:\MinuteLog.log", True, System.Text.Encoding.ASCII)
writer.WriteLine(ex.Message)
writer.Close()
writer.Dispose()
Finally
End Try
End Sub
' This method is called after all the rows have passed through this component.
'
' You can delete this method if you don't need to do anything here.
Public Overrides Sub PostExecute()
MyBase.PostExecute()
'
' Add your code here
'
buildValues = Nothing
columnValues = Nothing
End Sub
Public Overrides Sub Input0_ProcessInput(Buffer As Input0Buffer)
While Buffer.NextRow()
Input0_ProcessInputRow(Buffer)
End While
End Sub
'This method is called once for every row that passes through the component from Input0.
Public Overrides Sub Input0_ProcessInputRow(ByVal Row As Input0Buffer)
Dim column As IDTSInputColumn100
Dim rowType As Type = Row.GetType()
Dim columnValue As PropertyInfo
Dim result As Object
Dim rtnValue As String = Variables.varMinFileName.Replace("_", "")
Dim colName As String
Try
For Each column In Me.ComponentMetaData.InputCollection(0).InputColumnCollection
columnValue = rowType.GetProperty(column.Name)
colName = column.Name.ToString
If Not colName.Contains("NULL") Then
'If Not columnValue Is Nothing Then
Select Case column.Name.ToString
Case "PARAM_INDEX"
'result = columnValue.GetValue(Row, Nothing)
result = Row.PARAMINDEX
columnParamIndex = CType(result, Byte)
If columnsConstructed = False And i <= numberOfParams - 1 Then
columnValues.Append(String.Format("VALUE_{0}, STATUS_{0}, ", columnParamIndex.ToString))
End If
Exit Select
Case "dtmTIME"
'result = columnValue.GetValue(Row, Nothing)
result = Row.dtmTIME
columnDate = CType(result, DateTime)
If dateAdded = False Then ' only need to add once since rows are vertical
buildValues.Append("'" & columnDate & "', ")
dateAdded = True
End If
Exit Select
Case "MIN_VALUE"
'result = columnValue.GetValue(Row, Nothing)
result = Row.MINVALUE
columnMinValue = CType(result, Double)
buildValues.Append(columnMinValue & ", ")
Exit Select
Case "MIN_STATUS"
'result = columnValue.GetValue(Row, Nothing)
result = Row.MINSTATUS
columnStatus = CType(result, String)
Exit Select
Case "MIN_CNT_1"
'result = columnValue.GetValue(Row, Nothing)
result = Row.MINCNT1
columnCnt1 = CType(result, Byte)
columnStatusCnt = columnStatus & "010" & columnCnt1.ToString.PadLeft(5, "0"c) & "-----"
buildValues.Append("'" & columnStatusCnt & "', ")
Case Else
Exit Select
End Select
'End If
End If
Next
If i = numberOfParams - 1 Then
If columnsConstructed = False Then
columnValues.Remove(columnValues.Length - 2, 1)
End If
buildValues.Remove(buildValues.Length - 2, 1)
Dim valueResult As String = buildValues.ToString()
SetStoredProc()
cmd.Parameters("#Columns").Value = columnValues.ToString
cmd.Parameters("#DBName").Value = "[" & rtnValue & "].[dbo].[MINUTE]"
cmd.Parameters("#Values").Value = valueResult
cmd.ExecuteNonQuery()
buildValues.Clear()
columnsConstructed = True
dateAdded = False
columnParamIndex = 0
columnMinValue = 0
columnStatus = String.Empty
columnCnt1 = 0
i = 0
conn.Close()
conn.Dispose()
Else
i += 1
End If
Catch ex As Exception
Dim writer As New StreamWriter("C:\MinuteLog.log", True, System.Text.Encoding.ASCII)
writer.WriteLine(ex.Message)
writer.Close()
writer.Dispose()
Finally
'buildValues = Nothing
'columnValues = Nothing
End Try
End Sub
Private Sub SetStoredProc()
Try
Dim dbConnection As String = "Server=(local)\SQLExpress;Database=DataConversion;User ID=sa;Password=sa123;"
conn = New SqlConnection(dbConnection)
conn.Open()
cmd = New SqlCommand("dbo.InsertValues", conn) With {.CommandType = CommandType.StoredProcedure}
columnsForInsert = New SqlParameter("#Columns", SqlDbType.VarChar, -1) With {.Direction = ParameterDirection.Input}
cmd.Parameters.Add(columnsForInsert)
DBName = New SqlParameter("#DBName", SqlDbType.VarChar, -1) With {.Direction = ParameterDirection.Input}
cmd.Parameters.Add(DBName)
tableValues = New SqlParameter("#Values", SqlDbType.VarChar, -1) With {.Direction = ParameterDirection.Input}
cmd.Parameters.Add(tableValues)
Catch ex As Exception
Dim writer As New StreamWriter("C:\MinuteLog.log", True, System.Text.Encoding.ASCII)
writer.WriteLine(ex.Message)
writer.Close()
writer.Dispose()
End Try
End Sub
End Class
Since I can't upload images yet here I've included a blog link I created that includes ample screen shots to help understand the problem mentioned here:
SSIS slows down during transformation task
Any help in determining why my package slows after 400k records and doesn't process all 2+ million records in a reasonable time is much appreciated!
Thanks,
Jimmy
This probably isn't terribly helpful but my guess is you are running out of memory. If SSIS has to page you've had it in my experience.
Can you batch up the work somehow in several smaller runs perhaps?
Full solution can be viewed here on my blog with screenshots - SSIS slowdown solved
In order to get around SSIS slowing down when a large number of records are being transformed and inserted into SQL Server as my destination, I redesigned my SSIS package. Instead of doing an insert in a data transformation task for every record that comes through the buffer, I’ve eliminated it and have used a stored procedure to do a bulk insert. In order to accomplish this, I read in the data from each access DB into a table called “MINUTE” in my SQL Server instance. This minute table has the same schema as the access DB’s and I let SSIS do the heavy lifting of importing all the data into this table. After the data is imported, I execute my stored procedure which transforms the data in this minute table (horizontal records) and does a bulk insert into my new destination MINUTE SQL table (one vertical record.)
The stored procedure that does the bulk insert and transforms the data looks like this:
PROCEDURE [dbo].[InsertMinuteBulk]
-- Add the parameters for the stored procedure here
(#Columns varchar(MAX), #DBName varchar(4000))
AS
BEGIN
DECLARE #SQL varchar(MAX)
SET #SQL =’;WITH Base AS (
SELECT dtmTime,
param_index,
CONVERT(nvarchar(16), MIN_VALUE) AS [VALUE_],
CONVERT(nvarchar(3), MIN_STATUS) + ”000” + LEFT(replicate(”0”,5) + CONVERT(nvarchar(5), MIN_CNT_1),5) + ”—–” AS [STATUS_]
FROM [DataConversion].[dbo].[MINUTE]
)
,norm AS (
SELECT dtmTime, ColName + CONVERT(varchar, param_index) AS ColName, ColValue
FROM Base
UNPIVOT (ColValue FOR ColName IN ([VALUE_], [STATUS_])) AS pvt
)
INSERT INTO ‘ + #DBName + ‘
SELECT *
FROM norm
PIVOT (MIN(ColValue) FOR ColName IN (‘+#Columns+’)) AS pvt’
EXEC (#SQL);
In the Data Flow task, the “Minute Data Source" is an ADO.NET Data Source and feeds the data into my SQL Server destination – "Minute Data Destination".
In the Control Flow, the final task of "Bulk Insert Minute Data" executes the Bulk Insert stored procedure.
The package now runs uninterrupted and is pretty fast considering the size of data that I’m reading, transforming and inserting.
I’ve ran the package as an SSIS job and it took 38 minutes to complete converting 7 months (or 7 minute access DB’s) worth of minute data with over 2 million rows in each access DB.