TableDataAdapter.update() and TableDataAdapter.Fill() it take long time - mysql

have table with 200 columns, and when try to fill or update apporx. (15000) rows using the following:-
TableDataAdapter.update(DataTable)
TableDataAdapter.Fill(Datatable)
it take around 8 mins and some cases stacked and not complete the process
then tried to use the below code for select command to fill
Dim My_Trans as SQLTransaction
Dim My_Table as new DataTable
Dim My_SQLDataAdapter as new SqlDataAdapter
Dim My_BindingSource as new BindingSource
conn.Open()
My_Trans = conn.BeginTransaction()
myqry = "Select * From My_Table"
My_SQLDataAdapter = New SqlDataAdapter(myqry, conn)
Try
My_SQLDataAdapter.SelectCommand.Transaction = My_Trans
My_BindingSource.DataSource = My_Table
DataGridView1.DataSource = My_BindingSource '(<----- her is take long time)
My_Trans.Commit()
Catch ex As Exception
My_Trans.Rollback()
End Try
conn.Close()
but still take the same time , even for update command too, when tracking found the time taking in (DataGridView1.DataSource = My_BindingSource)
any advise please how speed up fill and update process?
many thanks in advance

I have tried as follow:-
1- Adding Class
Namespace DataSet1TableAdapters
Partial Public Class Table1TableAdapter
Public Function CreartTransaction() As Data.IDbTransaction
Try
Dim cmd As SqlCommand
Dim oConnection = me.CommandCollection(0).Connection
if oConnection.State = ConnectionState.Open
Else
oConnection.Open()
End If
Dim oTrans As SqlTransaction = oConnection.BeginTransaction()
For Each cmd In me.CommandCollection
cmd.Connection = oConnection
cmd.Transaction = oTrans
Next
Return oTrans
Catch ex As Exception
MsgBox(ex.ToString)
End Try
End Function
End Class
End Namespace
2- then on from added this code to save changes (Delete/Add/Modified):-
Dim oTrans As SqlTransaction = Table1TableAdapter.CreartTransaction
me.Table1TableAdapter.Transaction = oTrans
try
me.Table1TableAdapter.Update(me.DataSet1.Table1) (<----- Here when Have new rows not work and stop process without any error message)
oTrans.Commit
Catch ex As Exception
oTrans.Rollback
End Try
if Modified or deleted rows everything works fine, but if new rows added stacked and stop process without error message,
any explanation why ?
many thanks

Related

Concurrency violation on one of tables

I have weird error, this code bellow works just fine:
Public Function queryMakerSta()
Try
Dim query1 As String = "SELECT * FROM kerametal.dok_sta_d where broj = 1;"
Dim table As New DataTable
Using connection As New MySqlConnection(konekcija)
Using adapter As New MySqlDataAdapter(query1, connection)
Dim cmb As New MySqlCommandBuilder(adapter)
table.Clear()
adapter.Fill(table)
'Aplikacija
table.Rows(0)("tip") = "22"
adapter.Update(table)
End Using
Return True
End Using
Catch ex As Exception
End Try
End Function
But this code does not work and throws exception:
Public Function queryMakerSta()
Try
Dim query1 As String = "SELECT * FROM kerametal.dok_zag_d where broj = 1;"
Dim table As New DataTable
Using connection As New MySqlConnection(konekcija)
Using adapter As New MySqlDataAdapter(query1, connection)
Dim cmb As New MySqlCommandBuilder(adapter)
table.Clear()
adapter.Fill(table)
'Aplikacija
table.Rows(0)("tip") = "22"
adapter.Update(table)
End Using
Return True
End Using
Catch ex As Exception
End Try
End Function
Both tables have indexes, autoincrement, etc. I'm really cracking my head around this one and can't seem to figure out why it does not work.
And both querys return single row!

MySqlDataAdapter SelectCommand Statement

I have the following functional code:
Call ConnSettings()
Dim objDs As New DataSet
Dim Query As String
Query = "the query"
Cmd = New MySqlCommand(Query, MysqlConn)
Dim dAdapter As New MySqlDataAdapter
dAdapter.SelectCommand = Cmd
''Dim dAdapter As New MySqlDataAdapter(Query, MysqlConn)
Try
MysqlConn.Open()
dAdapter.Fill(objDs)
MysqlConn.Close()
Catch ex As Exception
MessageBox.Show(ex.Message)
Finally
MysqlConn.Dispose()
End Try
I have noticed that the code is equally functional if I also use
Dim dAdapter As New MySqlDataAdapter(Query, MysqlConn)
instead of:
Cmd = New MySqlCommand(Query, MysqlConn)
Dim dAdapter As New MySqlDataAdapter
dAdapter.SelectCommand = Cmd
Obviously, I would like to use a single line instead of three. However, I am fairly new to VB and would like to know if there are any issues with doing that.
Let's try to improve your code....
First ConnSettings doesn't initialize a global connection variable but a local one and returns it
Public Function ConnSettings() As MySqlConnection
Dim conn As MySqlConnection
conn = new MySqlConnection(yourConnectionStringHere)
conn.Open()
return conn
End Function
Now the code that needs a MySqlConnection could call this ConnSettings and put the return value in a Using Statement
objDS = new DataSet()
Try
Using conn = ConnSettings()
Using dAdapter = New MySqlDataAdapter(theQuery, conn)
dAdapter.Fill(objDs)
End Using
End Using
Catch(ex as Exception
MessageBox.Show(ex.Message)
End Try
This code puts the connection returned in a Using Statement. When the code flows out of the Using Statement the connection is closed and disposed (same for the MySqlDataAdapter) As you can see there is no need for the Finally clause and the Try/Catch block is present just because you want to give an error message to your user (while this is a common practice there is no really sense to put your user in the unconfortable position to try to understand these technically dense messages, better use a log file and advise your user to send the log to you)
In this context also the code inside the ConnSettings is a bit useless. What you really need is just the connectionstring and you could write the creation of the MySqlConnection directly in the calling code.

Why the performance difference between TableAdapter.Fill data sources

I have a windows form app with a DataGridView populated by a TableAdapter. I'm using the Fill method to update the data for the UI in a looping Async sub like so.
Private Async Sub updateUI()
Dim sw As New Stopwatch
While True
Await Task.Delay(3000)
sw.Restart()
'myTableAdapter.Fill(getDataWithMySQL())
'myTableAdapter.Fill(myDataSet.myTable)
myTableAdapter.Fill(myDataSet.myStoredProc)
logger.Debug(sw.ElapsedMilliseconds)
End While
End Sub
The getDataWithMySQL function is as follows:
Private Function getDataWithMySQL() As myDataSet.myDataTable
Dim connStr As String = My.Settings.myConnectionString
Dim sql As String = "SELECT ... LEFT JOIN ..."
Dim dt As New myDataSet.myDataTable
Using conn As New MySqlConnection(connStr)
Using cmd As New MySqlCommand()
With cmd
.CommandText = sql
.Connection = conn
End With
Try
conn.Open()
Dim sqladapter As New MySqlDataAdapter(cmd)
sqladapter.Fill(dt)
Catch ex As MySqlException
MsgBox(ex.Message)
End Try
End Using
End Using
Return dt
End Function
myDataSet.myTable is the same as myDataSet.myStoredProc except it is created by joining tables in the DataSet designer. Obviously myDataSet.myStoredProc is the same query in a stored procedure in the source database.
So benchmarking the Fill with each method using a Stopwatch I get the following results:
myDataSet.myStoredProc
~750ms
myDataSet.myTable
~550ms
getDataWithMySQL()
<10ms
So my question is, what is causing the performance difference here? I'd prefer to use myDataSet.myTable or myDataSet.myStoredProc but I don't know if it is possible to optimise them in some way so as to match the performance of getDataWithMySQL().

Checking which row from DataGridView exists in MySQL takes too long

I'm executing a query which returns me around 400 records (orders) and I display them in DataGridView. Then I have to check each row and make a green every row which exists in other MySQL database (by ID). I think I'm not doing this optimally. Here is what I am doing:
For Each oRow As DataGridViewRow In dgv.Rows
Dim orderNumber As Integer = oRow.Cells(0).Value
Dim exist As Boolean = mySql.CheckIfExists(shop, orderNumber)
If Not exist Then
Continue For
End If
If exist Then
oRow.DefaultCellStyle.BackColor = Color.LightGreen
oRow.DefaultCellStyle.SelectionBackColor = Color.Green
oRow.DefaultCellStyle.SelectionForeColor = Color.LightCoral
Continue For
End If
Next
Here is CheckIfExists() method in MySQL class:
Public Function CheckIfExists(ByVal shop As String, ByVal orderNumber As Integer) As Boolean
Dim dt As New DataTable
Dim sql As String = "" 'sql query'
Connect(mSOH) 'msoh is a connection string from My.Settings'
Dim sqlCommand As New MySqlCommand
With sqlCommand
.Connection = connection
.CommandText = sql
End With
Try
Dim sqlReader As MySqlDataReader = sqlCommand.ExecuteReader()
While sqlReader.Read()
Return True
End While
Catch ex As MySqlException
Logi.LogInfo(ex)
Catch ex As Exception
Logi.LogInfo(ex)
Finally
Disconnect()
End Try
Return False
End Function
And Connect and Disconnect methods if they are important:
Private Sub Connect(shop As String)
Select Case shop
Case "jablotron"
connection = New MySqlConnection(csJablotron)
Case "bcs"
connection = New MySqlConnection(csBCS)
Case "mSOH"
connection = New MySqlConnection(csmSOH)
Case Else
connection = New MySqlConnection(shop)
End Select
Try
connection.Open()
Catch ex As MySqlException
Logi.LogInfo(ex)
Catch ex As Exception
Logi.LogInfo(ex)
End Try
End Sub
Private Sub Disconnect()
Try
connection.Dispose()
Catch ex As MySqlException
Logi.LogInfo(ex)
Catch ex As Exception
Logi.LogInfo(ex)
End Try
End Sub
So if I try to check each row by this way it takes some time (around <1 second if database is on localhost, and around 30 second if database is on remote server and I try to connect via VPN). Is this way being optimal - checking each row like this? Is it a good approach? Please give me some tip and advice :) I know that code is in VB.NET, but guys from C# can also help me :)
Use this logic: Pre-load your order numbers into lists and cache them. For how long? - this is something for you to decide, based on how volatile the data is. Then check against those lists.
As well, It will be faster if you just parse your grid data by database and build In statements for each of your databases and load all needed records at once. Then only loop through already loaded data. And only relative data will be loaded.
Dim existingOrders as New List(of String)
Dim shopOrders as New List(of String)
/* Collect orders into list */
For Each oRow As DataGridViewRow In dgv.Rows
Dim orderNumber As Integer = oRow.Cells(0).Value
shopOrders.Add(orderNumber)
Next
/* Get and accumulate existing orders from each shop */
For Each shop as string In shops
GetOrdersByShop(shop, shopOrders, existingOrders)
Next
For Each oRow As DataGridViewRow In dgv.Rows
If existingOrders.Contains(oRow.Cells(0).Value) Then
oRow.DefaultCellStyle.BackColor = Color.LightGreen
oRow.DefaultCellStyle.SelectionBackColor = Color.Green
oRow.DefaultCellStyle.SelectionForeColor = Color.LightCoral
End If
Next
/* ---------------------------------------------------- */
Private Sub GetOrdersByShop(shop As string, shopOrders As List(of String),
existingOrders as New List(of String))
/* here, turn shopOrders into In('ord1', 'ord2', . . . )
and after running query add found orders to existingOrders */
End Sub
You can also use dictionary - it will make search work faster than list and you may store which order is in which shop, etc.
First off, a DataGridView is a VIEW object. You should use it as such. For data, you use a data object.
Personally I would do something like this:
Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load
Dim dtMain = SqlReadDatatable(firstConnString, "SELECT * FROM SomeTable")
Dim lstGreenIds = (From row In SqlReadDatatable(secondConnString, "SELECT ID FROM SomeOtherTable")
Select CInt(row("ID"))).ToList
dtMain.Columns.Add(New DataColumn("Exists", GetType(Boolean)))
For Each dtRow As DataRow In (From row In dtMain.Rows Where lstGreenIds.Contains(CInt(row("ID"))))
dtRow("Exists") = True
Next
DataGridView1.DataSource = dtMain
End Sub
Private Function SqlReadDatatable(ByVal connStr As String, ByVal query As String) As DataTable
Dim dt As New DataTable
Using conn As New SqlConnection(connStr) : conn.Open()
Using cmd As New SqlCommand(query, conn)
dt.Load(cmd.ExecuteReader)
End Using
End Using
Return dt
End Function
Private Sub DataGridView1_RowPostPaint(sender As Object, e As DataGridViewRowPostPaintEventArgs) Handles DataGridView1.RowPostPaint
Dim dgv = CType(sender, DataGridView)
Dim dgvRow = dgv.Rows(e.RowIndex)
Dim dtRow = CType(dgvRow.DataBoundItem, DataRow)
If dtRow("Exists") Then
dgvRow.DefaultCellStyle.BackColor = Color.LightGreen
dgvRow.DefaultCellStyle.SelectionBackColor = Color.Green
dgvRow.DefaultCellStyle.SelectionForeColor = Color.LightCoral
End If
End Sub
This cleanly separates data from visualisation. The color is only applied when the row comes into view, no need to iterate over the whole bunch at once.
EDIT: I should also note that the cleanest way to do this would be server-side. In MySQL you can use cross-database queries if both databases are on the same server. If they are on different servers, you can create a FEDERATED TABLE on one of the servers and connect it to the remote server. You can read about those here: http://dev.mysql.com/doc/refman/5.0/en/federated-storage-engine.html . In the case of two databases on the same server, you could just do a join:
SELECT f.*, NOT ISNULL(s.OrderNum) AS Exists
FROM 'firstdb'.'firsttable' f
LEFT JOIN 'seconddb'.'secondtable' s ON f.OrderNum = s.OrderNum
And then use the Exists column in code to color the row.

mysql select statement not filling datatable

I have a MySQL object I am trying to select all data from and insert into a datatable however every time it gets to my first if statement regarding said datatable It gets a possible null value error
dbfile dbserver dbuser and dbpassw all are public variables set on on a module.
variable dimensions
Dim cnn As New MySqlConnection()
Dim cmd As New MySqlCommand
Dim adptr As New MySqlDataAdapter
Dim filltab As DataTable
code
Dim cb As New MySqlConnectionStringBuilder
cb.Database = dbfile
cb.Server = dbserver
cb.UserID = dbuser
cb.Password = dbpassw
Using cnn As New MySqlConnection(cb.ConnectionString)
Using cmd As New MySqlCommand("SELECT * from gpstracking", cnn)
Try
cnn.Open()
adptr = New MySqlDataAdapter(cmd)
adptr.Fill(filltab)
Catch ex As Exception
MsgBox(ex.ToString)
End Try
If filltab.Rows.Count > 0 Then 'this is the row that the error for filltab being a null value hits
pop = 0
For pop As Integer = 0 To filltab.Rows.Count - 1
' WebBrowser1.Document.InvokeScript("AddMarker", New Object() {"Unit: " & filltab.Rows(pop)("unitnumb") & " at " & filltab.Rows(pop)("time"), filltab.Rows(pop)("lat"), table.Rows(pop)("long")})
Next
End If
End Using
End Using
one error on
adptr.Fill(filltab) 'Exception:Thrown: "Value cannot be null." (System.ArgumentNullException)
I also just noticed it is announcing "failure to collect details" on
Catch ex As Exception
MsgBox(ex.ToString)
End Try
I don't know vb syntax, but you have Dim filltab As DataTable without New (like in other dimensions), and I guess MySqlDataAdapter.Fill requires the argument to be not null. Just a guess.
instead of using this code
adptr = New MySqlDataAdapter(cmd)
adptr.Fill(filltab)
try use
filltab.load(cmd.ExecuteReader())