I've been using the code below to populate tables in a MySQL database for a few years and it's been working fine.
Imports MySql.Data.MySqlClient
Public sub Thing(dt as datatable)
Try
'dt is datatable object populated from MSsqlServer
Dim MySQLcmdSelect As MySqlCommand = New MySqlCommand("Select * From " & tableName) With {
.Connection = MySQLcnn
}
Dim MySQLad As New MySqlDataAdapter(MySQLcmdSelect)
Dim MySQLcmdBuilder As New MySqlCommandBuilder(MySQLad)
Dim MySQLcmd As MySqlCommand = MySQLcmdBuilder.GetInsertCommand()
MySQLcmd.Connection = MySQLcnn
MySQLad.InsertCommand = MySQLcmd
MySQLad.Update(dt)
Catch ex As Exception
Debug.Print(ex.Message)
Console.WriteLine("Error when populating " + tableName + ": " + ex.Message + vbCrLf + ex.InnerException.ToString)
WriteLog(ex.Message)
End Try
End Sub
The problem is, it's stopped working!
The connector seems to be working ok as I can drop tables and create them, but this will not let me bulk insert the contents of the datatable 'dt' into the MySQL table.
It doesn't throw an exception, it just passes over the .update(dt) line as if it were a Debug.Print() line when there's 1000s of rows in dt.
Effectively I'm selectively extracting data from tables in MS SQLserver and bulk uploading them to tables with the same name in MySQL.
a) is this the best way to do it, and
b) why has this suddenly stopped working?
According to phpMyAdmin its Server version: 5.6.51 - MySQL Community Server (GPL)
EDIT: Further info... all the SQL commands to drop existing tables and create tables work. It is only the .update(dt) that doesn't work.
Possible problems with your code could be that the schemas do not match exactly, auto increment columns, or the connection could have been disposed elsewhere (it should be declared local to the method where it is used). Another problem is that dt has had AcceptChanges called or LoadOptions were set incorrectly.
Private ConnStr As String = "Your connection string"
Public Sub Thing(dt As DataTable, tableName As String)
'check the datatable - if it is ok then this check is probably not necessary
Dim changesDT = dt.GetChanges()
If changesDT.Rows.Count < 1 Then
MessageBox.Show("No changes to update")
Exit Sub
End If
Using cn As New MySqlConnection(ConnStr),
MySQLad As New MySqlDataAdapter("Select * From " & tableName, cn)
cn.open() ' added by OP
Dim MySQLcmdBuilder As New MySqlCommandBuilder(MySQLad)
MySQLad.Update(dt)
End Using
End Sub
Perhaps you have "0000-00-00"? If so, a setting has changed the default -- new versions of MySQL disallow zero dates. Read about sql_modes.
Still can't get the bulk upload to work, but we have redesigned this so that data is inserted into MySQL at the same time we store it in the MS-SQL database and have removed the need to bulk upload.
I have a project list in a table in MYSQL, and each project has different tables. So in my code, first I'm getting the list of projects, and from each project I'm getting data from respective tables.
conn.Open()
cmdl_readuser = New MySqlCommand("SELECT * FROM projectlist where status = 'Active'", conn)
rl_readuser = cmdl_readuser.ExecuteReader()
While rl_readuser.Read()
Projectname = rl_readuser.GetString("ProjectName")
Space_Remove(Projectname)
cmd_listview = New MySqlCommand("SELECT `Process`,AVG(`Risk Factor`) as AvgRisk ,AVG(`Risk Factor After Mitigation`) as AvgRiskafterImp FROM " + sTableName + " ;", conn)
r2_readuser = cmd_listview.ExecuteReader()
While r2_readuser.Read()
ProjectRiskGridView.Rows.Add(r2_readuser("Process").ToString(), r2_readuser("AvgRisk"), r2_readuser("AvgRiskafterImp"))
End While
End While
conn.Close()
I'm getting data from table and put it in DataGridView. Finally, my DataGridView has records as list of Projects, and its respective average risk.
Here when control comes to the second MySQLCommand, it returns back. It doesn't go to next line. I think problem is with 'conn' .
I'm writing a program using visual basic 2015 and MySql database and I asked this question somewhere else before but they couldn't help me.
For example we have a table named "users" :
username | coins
user1 | 3
user2 | 5
I want to change the coin value and this code is working fine but in this code : 1- I get the column's value. 2- I add a unit to the value. 3- I put the value in column. So the problem is what if the database is online and two different computers run the program and try to change the value at the same time? For example one of them is trying to give user1 2 coins and the other one is trying to give user1 4 coins. Then they click on add coin at the same time. First computer gets the value and it is 3. Second Computer gets the value and it is 3. First computer add 2 coins (3+2=5) and put 5 in the column. Now second computer add 4 coins (3+4=7) and put 7 in the column. So we have 7 in our column but we should have 9 because 3+2+4=9
So here is the question : Is there a way to add a value directly to a column or is there another way to solve this problem?
MysqlConn = New MySqlConnection
MysqlConn.ConnectionString = "Server='" & TextBox_MYSQL_Host.Text & "';UserID='" & TextBox_MYSQL_Username.Text & "';Password='" & TextBox_MYSQL_Password.Text & "';Database='" & TextBox_MYSQL_Database.Text & "';Character Set=utf8"
Try
MysqlConn.Open()
DataAdptr = New MySqlDataAdapter
DataTable = New DataTable
MysqlComd = New MySqlCommand
With MysqlComd
.Connection = MysqlConn
.CommandText = "Select * FROM users WHERE username ='" & user1.text & "';"
End With
DataAdptr.SelectCommand = MysqlComd
DataAdptr.Fill(DataTable)
If DataTable.Rows.Count = 0 Then
' Error!
Else
Dim gold As Integer = DataTable.Rows(0).Item("coins").ToString()
gold = gold + 1
MysqlComd = New MySqlCommand
MysqlComd.Connection = MysqlConn
MysqlComd.CommandText = "UPDATE users SET coins='" & gold & "' WHERE username='" & user1.Text & "'"
MysqlComd.ExecuteNonQuery()
End If
MysqlConn.Close()
Catch ex As MySqlException
' Connection Error
Finally
MysqlConn.Dispose()
End Try
End Sub
Disregarding the issues with your code as mentioned in the comments, there are two ways to address the issue you're specifically asking about.
Firstly, you can continue to do it the way you are and employ optimistic concurrency. What that does is assume that each user is the only user editing the data and save as though that's the case and then throw an exception if it turns out to not be. Specifically, when a record is saved, the current values in the database are compared with the original data retrieved by that user and the new values will be saved if and only if they match. If they don't, it's up to you to catch the ConcurrencyException that's thrown and do whatever is appropriate. That will usually be retrieving the data again and either asking the user to start editing again or merge the current data with their existing modifications.
The other option is to simply increment what's in the database rather than saving a specific value, e.g.
UPDATE MyTable SET MyColumn = MyColumn + 1 WHERE ID = #ID
I have the following code that tries to get records from two different tables and then add them to the particular comboboxes. Only the first query works and the second one is ignored.
Try
sqlConn = New MySqlConnection
connStr = New String("Server = localhost; Database = gen_database; Uid = root; Pwd =")
sqlConn.ConnectionString = connStr
myCommand = New MySqlCommand("Select DevCompanyName from developer_name_table; Select DevType from development_type_table")
myCommand.CommandType = CommandType.Text
myCommand.Connection = sqlConn
ComboBox1.Items.Clear()
sqlConn.Open()
MsgBox("Connection Open.")
dR = myCommand.ExecuteReader()
Do While dR.Read()
ComboBox1.Items.Add(dR("DevCompanyName"))
ComboBox2.Items.Add(dR("DevType")) 'Error shows here Could not find specified column in results: DevType
Loop
Catch ex As MySqlException
MsgBox(ex.ToString)
Finally
dR.Close()
sqlConn.Close()
End Try
I can think of another way which is to do it in multiple query but can the code be simplified to something like this?
Using a DBDataReader with 2 queries, only only the first executes because there is no way to signal which table/query each read item is from. Your "double read" loop seems to assume they will be returned at the same time (rather than one query after the other - the same way they are sent to the DBCOmmand); if it did work that way, it would fail whenever there are not the same number of rows in each table.
Using DataTables affords you the chance to simply bind the result to your combos rather than copying data into them:
Dim SQL = "SELECT * FROM Sample; SELECT * FROM Simple"
Dim ds As New DataSet
Using dbcon As New MySqlConnection(MySQLConnStr),
cmd As New MySqlCommand(SQL, dbcon)
dbcon.Open()
Dim da As New MySqlDataAdapter(cmd)
da.Fill(ds)
End Using
' debug results
Console.WriteLine(ds.Tables.Count)
Console.WriteLine(ds.Tables(0).Rows.Count)
Console.WriteLine(ds.Tables(1).Rows.Count)
If I look at the output window, it will print 2 (tables), 10000 (rows in T(0)) and 6 (rows in T(1)). Not all DBProviders have this capability. Access for instance will choke on the SQL string. Other changes to the way your code is composed:
DBConnections and DBCommand objects need to be disposed. They allocate resources, so they need to be created, used and disposed to release those resources.
The Using block does that for us: The target objects are created at the start and closed and disposed at End Using.
The code above stacks or combines 2 such blocks.
The code fills a DataSet from the query, in this case creating 2 tables. Rather than copying the data from one container to another (like a control), you can use a DataTable as the DataSource:
cboDevName.DataSource = ds.Tables(0)
cboDevName.DisplayMember = "DevName" ' column names
cboDevName.ValueMember = "Id"
cboDevType.DataSource = ds.Tables(1)
cboDevType.DisplayMember = "DevType"
cboDevType.ValueMember = "DevCode"
The result will be all the rows from each table appearing in the respective combo control. Typically with this type of thing, you would want the ID/PK and the name which is meaningful to the user in the query. The user sees the friendly name (DisplayMember), which the code can easily access the unique identifier for the selection (ValueMember).
When using bound list controls, rather than using SelectedIndex and the SelectedIndexChanged event, you'd use SelectedValue and SelectedItem to access the actual data.
MSDN: Using Statement (Visual Basic)
You can use .net connector (download here)
Then you can install it and add it to your project (project -> references -> add -> browse).
Finally add import:
Imports MySql.Data.MySqlClient
So you'll be able to use this:
Dim connStr as String = "Server = localhost; Database = gen_database; Uid = root; Pwd ="
Dim SqlStr as String = "Select DevCompanyName from developer_name_table; Select DevType from development_type_table"
Dim ds As DataSet = MySqlHelper.ExecuteDataset(CnStr, SqlStr)
ds will contain two datatables: one for each query
I have an SSIS Transformation Task that I use as my final destination task to insert data into a SQL Server table. The reason I use the transformation task and not an SQL Server Destination task is because I do not know beforehand what the columns will in the table that we will be inserting into.
In a for each loop container, I am looking for access DB's (in 97 format). The rest of the control flow basically creates a new SQL database and also a table. The access files are what we call "minute" databases where they contain minute information gathered by another process. I need to create a new SQL DB named after the 'minute' db and a table called 'MINUTE' with the columns created based on certain info from the access db. For each of our clients, based on the number of parameters they have at their site, determines the number of columns I need to create in the SQL Minute table.
In the data flow I have two key components: The OLE DB source component (Source - Minute Table) and the Script Transformation task (Destination - Minute Table).
The "Source - Minute Table" gets the data from the access database. The "Destination - Minute Table" transforms the data and inserts it into the appropriate DB and table.
Everything works as it should. I tested it on a DB with 491,000+ records and it took 1 minute. However, I'm testing with one of our larger customers that has over 50 parameters and the access database contains 2+ million records. The package flies until I reach around 477,000 records, and then it pretty much comes to a halt. I can wait 10 minutes, and even longer, until the record count updates, and then continue to wait again.
I've done much research and followed all of the recommendations and guidelines that I have found. My datasource is not sorted. I use SQL command instead of Table, etc in the OLE DB Source. I've changed the values of DefaultBufferMaxRows and DefaultBufferSize many times and get the same results.
Code:
Public Class ScriptMain
Inherits UserComponent
Private conn As SqlConnection
Private cmd As SqlCommand
Private DBName As SqlParameter
Private columnsForInsert As SqlParameter
Private tableValues As SqlParameter
Private numberOfParams As Integer
Private db As String
Private folderPath As String
Private dbConn As String
Private folder As String
Private columnParamIndex As Integer
Private columnDate As DateTime
Private columnMinValue As Double
Private columnStatus As String
Private columnCnt1 As Int16
Private dateAdded As Boolean = False
Private columnStatusCnt As String
Private columnsConstructed As Boolean = False
Private buildValues As StringBuilder
Private columnValues As StringBuilder
Private i As Integer = 0
'This method is called once, before rows begin to be processed in the data flow.
'
'You can remove this method if you don't need to do anything here.
Public Overrides Sub PreExecute()
MyBase.PreExecute()
Try
'Dim dbConnection As String = "Server=(local)\SQLExpress;Database=DataConversion;User ID=sa;Password=sa123;"
'conn = New SqlConnection(dbConnection)
'conn.Open()
'cmd = New SqlCommand("dbo.InsertValues", conn) With {.CommandType = CommandType.StoredProcedure}
'columnsForInsert = New SqlParameter("#Columns", SqlDbType.VarChar, -1) With {.Direction = ParameterDirection.Input}
'cmd.Parameters.Add(columnsForInsert)
'DBName = New SqlParameter("#DBName", SqlDbType.VarChar, -1) With {.Direction = ParameterDirection.Input}
'cmd.Parameters.Add(DBName)
'tableValues = New SqlParameter("#Values", SqlDbType.VarChar, -1) With {.Direction = ParameterDirection.Input}
'cmd.Parameters.Add(tableValues)
db = Variables.varMinFileName.ToString
folder = Variables.varMinFolderName.ToString
folderPath = folder & "\" & db & ".mdb"
dbConn = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" & folderPath
Using SourceDataAdapter As OleDbDataAdapter = New OleDbDataAdapter("SELECT DISTINCT PARAM_INDEX FROM [MINUTE];", dbConn)
Dim SourceDatatable As New DataTable
SourceDataAdapter.Fill(SourceDatatable)
numberOfParams = SourceDatatable.Rows.Count
End Using
'columnValues.Append("dtmTime, ")
buildValues = New StringBuilder
columnValues = New StringBuilder
columnValues.Append("dtmTime, ")
Catch ex As Exception
Dim writer As New StreamWriter("C:\MinuteLog.log", True, System.Text.Encoding.ASCII)
writer.WriteLine(ex.Message)
writer.Close()
writer.Dispose()
Finally
End Try
End Sub
' This method is called after all the rows have passed through this component.
'
' You can delete this method if you don't need to do anything here.
Public Overrides Sub PostExecute()
MyBase.PostExecute()
'
' Add your code here
'
buildValues = Nothing
columnValues = Nothing
End Sub
Public Overrides Sub Input0_ProcessInput(Buffer As Input0Buffer)
While Buffer.NextRow()
Input0_ProcessInputRow(Buffer)
End While
End Sub
'This method is called once for every row that passes through the component from Input0.
Public Overrides Sub Input0_ProcessInputRow(ByVal Row As Input0Buffer)
Dim column As IDTSInputColumn100
Dim rowType As Type = Row.GetType()
Dim columnValue As PropertyInfo
Dim result As Object
Dim rtnValue As String = Variables.varMinFileName.Replace("_", "")
Dim colName As String
Try
For Each column In Me.ComponentMetaData.InputCollection(0).InputColumnCollection
columnValue = rowType.GetProperty(column.Name)
colName = column.Name.ToString
If Not colName.Contains("NULL") Then
'If Not columnValue Is Nothing Then
Select Case column.Name.ToString
Case "PARAM_INDEX"
'result = columnValue.GetValue(Row, Nothing)
result = Row.PARAMINDEX
columnParamIndex = CType(result, Byte)
If columnsConstructed = False And i <= numberOfParams - 1 Then
columnValues.Append(String.Format("VALUE_{0}, STATUS_{0}, ", columnParamIndex.ToString))
End If
Exit Select
Case "dtmTIME"
'result = columnValue.GetValue(Row, Nothing)
result = Row.dtmTIME
columnDate = CType(result, DateTime)
If dateAdded = False Then ' only need to add once since rows are vertical
buildValues.Append("'" & columnDate & "', ")
dateAdded = True
End If
Exit Select
Case "MIN_VALUE"
'result = columnValue.GetValue(Row, Nothing)
result = Row.MINVALUE
columnMinValue = CType(result, Double)
buildValues.Append(columnMinValue & ", ")
Exit Select
Case "MIN_STATUS"
'result = columnValue.GetValue(Row, Nothing)
result = Row.MINSTATUS
columnStatus = CType(result, String)
Exit Select
Case "MIN_CNT_1"
'result = columnValue.GetValue(Row, Nothing)
result = Row.MINCNT1
columnCnt1 = CType(result, Byte)
columnStatusCnt = columnStatus & "010" & columnCnt1.ToString.PadLeft(5, "0"c) & "-----"
buildValues.Append("'" & columnStatusCnt & "', ")
Case Else
Exit Select
End Select
'End If
End If
Next
If i = numberOfParams - 1 Then
If columnsConstructed = False Then
columnValues.Remove(columnValues.Length - 2, 1)
End If
buildValues.Remove(buildValues.Length - 2, 1)
Dim valueResult As String = buildValues.ToString()
SetStoredProc()
cmd.Parameters("#Columns").Value = columnValues.ToString
cmd.Parameters("#DBName").Value = "[" & rtnValue & "].[dbo].[MINUTE]"
cmd.Parameters("#Values").Value = valueResult
cmd.ExecuteNonQuery()
buildValues.Clear()
columnsConstructed = True
dateAdded = False
columnParamIndex = 0
columnMinValue = 0
columnStatus = String.Empty
columnCnt1 = 0
i = 0
conn.Close()
conn.Dispose()
Else
i += 1
End If
Catch ex As Exception
Dim writer As New StreamWriter("C:\MinuteLog.log", True, System.Text.Encoding.ASCII)
writer.WriteLine(ex.Message)
writer.Close()
writer.Dispose()
Finally
'buildValues = Nothing
'columnValues = Nothing
End Try
End Sub
Private Sub SetStoredProc()
Try
Dim dbConnection As String = "Server=(local)\SQLExpress;Database=DataConversion;User ID=sa;Password=sa123;"
conn = New SqlConnection(dbConnection)
conn.Open()
cmd = New SqlCommand("dbo.InsertValues", conn) With {.CommandType = CommandType.StoredProcedure}
columnsForInsert = New SqlParameter("#Columns", SqlDbType.VarChar, -1) With {.Direction = ParameterDirection.Input}
cmd.Parameters.Add(columnsForInsert)
DBName = New SqlParameter("#DBName", SqlDbType.VarChar, -1) With {.Direction = ParameterDirection.Input}
cmd.Parameters.Add(DBName)
tableValues = New SqlParameter("#Values", SqlDbType.VarChar, -1) With {.Direction = ParameterDirection.Input}
cmd.Parameters.Add(tableValues)
Catch ex As Exception
Dim writer As New StreamWriter("C:\MinuteLog.log", True, System.Text.Encoding.ASCII)
writer.WriteLine(ex.Message)
writer.Close()
writer.Dispose()
End Try
End Sub
End Class
Since I can't upload images yet here I've included a blog link I created that includes ample screen shots to help understand the problem mentioned here:
SSIS slows down during transformation task
Any help in determining why my package slows after 400k records and doesn't process all 2+ million records in a reasonable time is much appreciated!
Thanks,
Jimmy
This probably isn't terribly helpful but my guess is you are running out of memory. If SSIS has to page you've had it in my experience.
Can you batch up the work somehow in several smaller runs perhaps?
Full solution can be viewed here on my blog with screenshots - SSIS slowdown solved
In order to get around SSIS slowing down when a large number of records are being transformed and inserted into SQL Server as my destination, I redesigned my SSIS package. Instead of doing an insert in a data transformation task for every record that comes through the buffer, I’ve eliminated it and have used a stored procedure to do a bulk insert. In order to accomplish this, I read in the data from each access DB into a table called “MINUTE” in my SQL Server instance. This minute table has the same schema as the access DB’s and I let SSIS do the heavy lifting of importing all the data into this table. After the data is imported, I execute my stored procedure which transforms the data in this minute table (horizontal records) and does a bulk insert into my new destination MINUTE SQL table (one vertical record.)
The stored procedure that does the bulk insert and transforms the data looks like this:
PROCEDURE [dbo].[InsertMinuteBulk]
-- Add the parameters for the stored procedure here
(#Columns varchar(MAX), #DBName varchar(4000))
AS
BEGIN
DECLARE #SQL varchar(MAX)
SET #SQL =’;WITH Base AS (
SELECT dtmTime,
param_index,
CONVERT(nvarchar(16), MIN_VALUE) AS [VALUE_],
CONVERT(nvarchar(3), MIN_STATUS) + ”000” + LEFT(replicate(”0”,5) + CONVERT(nvarchar(5), MIN_CNT_1),5) + ”—–” AS [STATUS_]
FROM [DataConversion].[dbo].[MINUTE]
)
,norm AS (
SELECT dtmTime, ColName + CONVERT(varchar, param_index) AS ColName, ColValue
FROM Base
UNPIVOT (ColValue FOR ColName IN ([VALUE_], [STATUS_])) AS pvt
)
INSERT INTO ‘ + #DBName + ‘
SELECT *
FROM norm
PIVOT (MIN(ColValue) FOR ColName IN (‘+#Columns+’)) AS pvt’
EXEC (#SQL);
In the Data Flow task, the “Minute Data Source" is an ADO.NET Data Source and feeds the data into my SQL Server destination – "Minute Data Destination".
In the Control Flow, the final task of "Bulk Insert Minute Data" executes the Bulk Insert stored procedure.
The package now runs uninterrupted and is pretty fast considering the size of data that I’m reading, transforming and inserting.
I’ve ran the package as an SSIS job and it took 38 minutes to complete converting 7 months (or 7 minute access DB’s) worth of minute data with over 2 million rows in each access DB.