I am running a SSIS package to load say a million rows from a flat file, which uses a script task for complex transformations and a SQL Server table destination. I am trying to figure out the best way (well, ANY way at this stage) to write out to a different table the row count (probably in multiples of 1000 to be more efficient) DURING the data flow processing. This is so that I can determine the percentage of progress throughout a task that might take a few minutes, simply by querying the table periodically.
I can't seem to add any SQL task into the flow, so I'm guessing the only way is to connect to the SQL database inside the .NET script. This seems painful and I'm not even sure it is possible. Is there another more elegant way? I've seen reference to "Rows Read" performance counter but not sure where I access this in SSIS and still not sure how to write it to a SQL table during the Data Flow processing.
Any suggestions appreciated.
Glenn
there are two easy options here:
Option 1: use the built-in logging with SSIS and watch the on progress event. this can be configured to log to several different outputs including relational database and flat files
See more Here
Option 2: you could add a SSIS script component that could fire off notifications to an external system like a database table
I recently solved this in a slightly different manner, which I find superior to using scripting and opening separate connections in code to DBs:
In the source query or a transform shape, add a row count (incremental)
In a conditional branching, use a modulo expression (%) to branch whenever the number is a multiple of for example 1000, but this could be configurable or based on source data (for example 0.0% to 100.0% of the data)
Create a log connection manager and use a destination. Control the batching sizes so that rows are immediately committed to the target table.
Why not write a .NET application and you can integrate into that to get information as to where the SSIS package is at.
Basically everything that is sent to the console you can get, and there are event handlers you can attach to to get information while it is processing the package.
Here is a link that may help you to go with this approach:
http://www.programminghelp.com/database/sqlserver/sql-server-integration-services-calling-ssis-package-in-c/
OK, had some success at last.... added a call to the following sub in the script component:
Sub UpdateLoadLog(ByVal Load_ID As Int32, ByVal Row_Count As Int32, ByVal Row_Percent As Int32, ByVal connstr As String)
Dim dbconn As OleDbConnection
Dim Sql As String
Dim dbcomm As OleDbCommand
dbconn = New OleDbConnection(connstr)
dbconn.Open()
Sql = "update myTable set rows_processed = " & Row_Count & ", rows_processed_percent = " & Row_Percent & " where load_id = " & Load_ID & " and load_log_type = 'SSIS'"
dbcomm = New OleDbCommand(Sql, dbconn)
dbcomm.ExecuteNonQuery()
dbconn.Close()
dbconn = Nothing
dbcomm = Nothing
End Sub
This gets executed every 1000 rows, and successfully updates the table. The row already existed as it gets created in the control flow at the start of the package, and updated again in the control flow at the very end with final rowcount and 100%.
Thanks for all your suggestions guys.
Is the application consuming the row count a .net application? When it comes to sharing information between applications there are a lot of accepted practices. May be you should take a look in to them. And for your particular case, if it is .net application that consumes this row number for calculating progress, may be you can store the information some place else other than a DB table, like file system, web service, windows environment variables, log (like windows events log), etc are some that came to my mind now. I think updating a windows environment variable with row count form with in your script component will be a good enough solution. Just like using a global variable to share data between two functions inside a program. :)
Related
I export my msaccess queries sql code for source versioning. I have been doing this for a while, but just now noticed an inconsistency. The exported sql code was different from the code I would get from navigating to the SQL view in Design Mode. It would be an older version. My export code boils down to this:
Dim db As Database
Dim qr As QueryDef
Dim arq_out As Integer
Set db = CurrentDb
arq_out = FreeFile
arq_name = "Querydefs.sql"
Open destPath & arq_name For Output As arq_out
For Each qr In db.QueryDefs
If Not qr.Name Like "~TMP*" Then
Print #arq_out, qr.Name
Print #arq_out, qr.sql
Print #arq_out, String(70, "-")
End If
Next qr
The current database is one I had not used for more than a year.
You may understand better with this timeline:
Updated a few queries (long time ago). The update consisted of renaming some queries and then adjusting accordingly downstream.
Exported SQL query code with above VBA (today).
Noticed that a certain queryA's sql was weirdly using its dependency's old name.
Opened queryAin design view and it showed the dependency's correct name. Different from what was exported. So I closed without saving.
Ran Compact & Repair before exporting again with the same VBA.
queryA had the correct code exported.
Noticed some queryB in the same database had a similar "obsolete" problem.
Same as 4, 5 and 6.
So it seems MsAccess stores it's SQL somewhere else and is feeding me an old or cached version. I do not even know if I regressed some relevant change due to this wrench in my workflow, as I use it ad-hoc for data analysis as a work limitation.
I tried querying the system tables to see if those held the correct SQL, but ran into the same issue. They would show me the old dependencies and be automatically updated after I opened the design view. I used the following sample query:
SELECT MSysQueries.Name1, MSysObjects.Name
FROM MSysQueries INNER JOIN MSysObjects ON MSysQueries.ObjectId = MSysObjects.Id
WHERE (((MSysQueries.Name1)="Dependency_Old_Name"));
I would appreciate help to either: get the actual sql code from MsAccess or force MsAccess to update all QueryDefs.SQL.
Opening and closing the queries in design view would not be the preferred solution, as it sometimes introduces spurious as Expr1. But it may well be the solution if nothing else is possible.
this is my first time posting a question here, but I almost always get good answers from searching this site. I'm trying to find out if there is a better way to speed up this process I have of writing a table column in Access to a .txt file. Here is the code I have, which works, but it's on the slow side. It takes about 45s to write around 7000 items.
lsFileName2 = "E:\DOI_Inventory2\SyncData\EquipUser.txt"
LiFileNumb = FreeFile
SysCmd acSysCmdSetStatus, "Updating User Equipment List"
Open lsFileName2 For Output As LiFileNumb
With rst
Do While Not .EOF
Write #LiFileNumb, ![EqUserFile]
.MoveNext
Loop
End With
dbs.Close
Close LiFileNumb
I'm fairly new to the IT field so any help would be greatly appreciated. Thanks
Just to add a note, the actual query is fine. I already checked that and it's pretty fast.
Create a query, save it with a name, and then use TransferText to export the query's data to your text file.
So assuming you have a SELECT query named qryExportMe like this which returns your table's column data correctly ...
SELECT EqUserFile
FROM YourTable;
... refer to this TransferText example and adapt it to fit your needs.
DoCmd.TransferText TransferType:=acExportDelim, _
Tablename:="qryExportMe", _
FileName:="E:\DOI_Inventory2\SyncData\EquipUser.txt", _
HasFieldNames:=True
Check the TransferText options at that linked page or from Access' built in help system.
Note you are not required to include the option names. I added them to help you keep track of which is which.
If this approach speeds up your export operation adequately, I think it will be because Access handles the task as a single set-based file write. Your recordset approach required Access to process one row at a time ... and one reason such approaches are called RBAR (row by agonizing row) is because they are often painfully slow.
I'm creating an application which allows you to manage various data. The application is designed to work in a network, and thus in multi-user. For this reason I decided to trust the Datatable.
I have a class created by me for the management of operations MYSQL Database but now I still can not create a streamlined process to send the datatable to MySQL database.
Currently I am so
Dim SQLStm As String
'variable for sql query
Dim SQLManager As New ER.DB.ERMysql
For Each Riga In Datatable.Rows
'example query
SQLStm = "INSERT INTO test(Name,Phone)VALUES(Riga("Name"),Riga("Phone"))"
Try
Dim CMD As New MySqlCommand
CMD.Connection = connection
CMD.CommandText = SQLStm
CMD.ExecuteNonQuery()
Catch ex As Exception
End Try
Next
End Sub
or skim all the rows and gradually sending to the database. There is a better way to accomplish this?
Thanks to all
I would say the best way to do this would be via XML. Convert the datatable to xml format and pass it through to a procedure which accepts an XML. This saves the process from running once per every line within a datatable, and it is all done in one go. The current way you are doing this would not scale well for large data sets, but XML would scale far better.
instead performing a db insert for each row, build the whole query string first and then perform it as one large INSERT command. It's much faster.
i am new to ms-access, and i have started to look at code that works. unfortunately it is completely uncommented.
Private Sub OKbut_Click()
Dim dt As Date
dt = Now()
Set rstOrder = New ADODB.Recordset
rstOrder.Open "tblUsers", CurrentProject.Connection, adOpenStatic, adLockOptimistic
If rstOrder.Supports(adAddNew) Then
With rstOrder
.AddNew
.Fields("title") = title
.Fields("first") = first
.Fields("last") = last
.Fields("gender") = gender
.Fields("date_submitted") = dt
.Update
End With
End If
rstOrder.Close
Set rstOrder = Nothing
Dim rs As DAO.Recordset
Set rs = CurrentDb.OpenRecordset("SELECT id FROM tblUsers WHERE date_submitted=#" & dt & "#")
duser = rs.Fields("id")
rs.Close
Set rs = Nothing
Do While Not user_defined(duser)
DoCmd.OpenForm "define_user_frm", , , , , acDialog
Loop
'Forms(0).user_lst.RowSource = "select * from users where id=" & duser
Me.SetFocus
DoCmd.Close
End Sub
what does the function Now() do?
ADODB.Recordset is just a way to connect to a table?
adOpenStatic, adLockOptimistic what are these?
why are we checking this: if rstOrder.Supports(adAddNew) ?
why do we need to do this: Set rstOrder = Nothing?
what does this do:? Do While Not user_defined(duser)
DoCmd.OpenForm "define_user_frm", , , , , acDialog
Loop
Approaching existing code like this isn't the best way to learn to program. There are plenty of online tutorials that will help you get started and progress in a linear manner. The other advice I'd give is to try things out for yourself. Write a small method that contains "MsgBox Now()" and run it to see what happens. Then run it a couple of seconds later.
I can't over-recommend finding a beginners tutorial.
However that said . . .
Now - returns the computers system date and time, at the moment it's called
Yes - Among other things.
Hints to the query engine. Open Static says you don't want to see rows added after the record set is retrieved, optimistic locking tells the engine which record locking option to use (the other is pessemistic)
Checking this because you can't add records to all resordsets (e.g. complex queries joining several tables)
Not stricly necessary, but better for memory management, otherwise the recordset object (and all associated resources - memory, handles etc) wouldn't be disposed until the end of the method.
Opens an ms access form until the condition "user_defined(duser)" returns true
Hope this helps.
what does the function Now() do?
Now() returns the current system date and time
ADODB.Recordset is just a way to connect to a table?
Yes - more importantly, it is a good way to iterate through records of a dataset individually. Like a cursor in SQL Server. You could do something like
While not rstOrder.EOF
'a.k.a. while there are still records left to iterate through
'Do something
End While
adOpenStatic, adLockOptimistic what are these?
From http://www.dotnetspider.com/forum/16958-In-VB-What-difference-between-adopendynamic.aspx
adOpenForwardOnly - this is the default cursor if no other is specified. This cursor allows only forward movement through a recordset
adOpenKeyset - this cursor supports forwards as well as backwards navigation. It also allows you to update a recordset and all changes will be reflected in other users recordsets. The cursor also supports bookmarking
adOpenDynamic - this cursor supports forward and backward navigation but bookmarks may not be supported (ie Access). Any changes made to data are immediately visible with no need to resynchronise the cursor with the database
adOpenStatic - this cursor uses a static copy of data from the database and therefore no changes to the data are visible and supports forward and backward navigation
why are we checking this: if rstOrder.Supports(adAddNew) ?
It's a way of writing more robust code - i.e. before attempting to add a new record, first check whether the recordset supports the addition of new records.
why do we need to do this: Set rstOrder = Nothing?
Frees the recordset from memory. Not absolutely necessary but again, makes for more robust code.
what does this do:? Do While Not user_defined(duser) DoCmd.OpenForm "define_user_frm", , , , , acDialog Loop
Checks for existence of a user and if it doesn't exist, it opens a form called "define_user_frm" which I assume allows for creating a new user.
what does the function Now() do?
Place the cursor on the word Now in your subroutine, then press the F1 key. Is Access' help not helpful for answering your question?
I'm trying to suggest it should be quicker for you to use the built-in Help feature for a question like that, instead of posting to Stack Overflow and waiting for responses. Then you only need post the questions that Help doesn't answer adequately for you.
I'm very rusty with VB/Access but:
1) Now returns the current date and time
2) Recordset is a set of records - from memory doesn't have to be a table, could be a query.
3) Those are flags to indicate how the table (as it is a table in this case) should be accessed can't remember static but the Lock says that we're going to use optimistic locking i.e. we will basically assume that the record won't be changed by anyone else.
4) Because you can open recordsets in a number of different ways - so it may well be the case that you have a read-only set of data. In the context of the example its notionally redundant but in terms of good practice its a reasonable bit of defensive programming.
5) Probably don't but again its sensible defensive programming and it ensures that all resources associated with the recordset are released. rstOrder is coming from somewhere external to the sub so not leaving it in an open state is sensible.
6) The loop repeats until the function user_defined(duser) returns true. duser being the - erk, that looks horrid - ID returned by the query but that doesn't look safe. DoCmd.Open form is popping up the "define_user_form" as a dialog so... if the user doesn't exist then fire up the dialog to define the user then check again that the user has been defined. That's also a bit questionable (although of course without a bit more context its hard to tell).
Basically VBA in Access is mostly VB6 but with sufficient differences to drive one slightly mad... there's a whole pile of stuff that you can assume to exist in access that you have to be rather more explicit about in VB.OLD
what does the function Now() do?
Returns a Variant (Date) specifying the current date and time according your computer's system date and time.
ADODB.Recordset is just a way to connect to a table?
The ADO Recordset object is used to hold a set of records from a database table. A Recordset object consist of records and columns (fields). It can be used to open tables, queries, custom SQL statements, etc.
adOpenStatic, adLockOptimistic what are these?
adOpenStatic: Provides a static copy of the records (you can't see additions, changes or deletions by other users), but all types of movement are enabled (e.g. .moveNext, .MoveFirst, .moveLast, etc).
adLockOptimistic: Record locking is performed only when you call the Update method.
why are we checking this: if rstOrder.Supports(adAddNew) ?
Checking whether the recordset allows new records to be added.
why do we need to do this: Set rstOrder = Nothing?
When you're completely finished with rstOrder, setting it equal to Nothing removes it from memory. You can think of it as a simple version of garbage collection.
what does this do:? Do While Not user_defined(duser) DoCmd.OpenForm
"define_user_frm", , , , , acDialog
Loop
Assuming user_defined() is a custom function, it loops through the record just added and opens a form that allows you to define the user.
Reference:
http://www.w3schools.com/ado/ado_ref_recordset.asp
what does the function Now() do? - Creates a DateTime for the current date and time the function is being run.
ADODB.Recordset is just a way to connect to a table? This is an object or saying that rstOrder is not a ABDODM.Recordset with a method to open the connection to the DB.
why do we need to do this: Set rstOrder = Nothing? - This is cleaning up the connection so that the computer does not keep the memory and for rstOrder. Other wise after running your computer might slow down or have less memory.
what does this do:? Do While Not user_defined(duser) DoCmd.OpenForm "define_user_frm", , , , , acDialog Loop - This is a loop that will be repeated and will Open an Access form as long as user_defined is not duser.
Now() just returns the current date and time
ADODB.Recordset is in this instance being used as a way to connect to a table. It is generally used to connect to an external (read SQL) dataset. And can be populated from a query or stored procedure. Similar to the access builtin Recordset
These are switches used to set certain options in the connection. I would recommend reading over the MSDN documentation.
Not every recordset supports adding records (such as that returned from a query) this simply checks that before you attempt to add a record. It is part of writing code that won't throw errors.
This is simply a way to clean up the memory. As far as I know VBA is garbage collected but if you have a large recordset this can free up some memory sooner.
I'm not sure about this here. but it looks like it's calling a user defined function and it returns false it opens a form.
I have a program that uses a Microsoft Access database for its back-end. I need to have some VBA code (that calls a web service) execute whenever specific tables/fields are updated by the program. I see this working just like a trigger in SQL Server.
Is it possible to monitor for and act upon changes like this in Access?
Update
The program in question does not run inside of Access (i.e. not a VBA app), it simply uses an MDB file as its back-end storage. Unfortunately I don't have access to the program's code as it is a closed third party application.
This question is old, but the answers are no longer correct. Access 2010 added data macro events that can be run when data is inserted, updated or deleted. The following events are available while using either the table datasheet view or table design view (events are attached directly to table and not through the form macro button):
After Delete Macro Event
After Insert Macro Event
After Update Macro Event
Before Change Macro Event
Before Delete Macro Event
More information is located here:
https://msdn.microsoft.com/en-us/library/office/dn124692.aspx
https://support.office.com/en-us/article/Create-a-data-macro-b1b94bca-4f17-47ad-a66d-f296ef834200
Access the GUI environment vs Jet the database format are separate things.
If you are using an Access database as a backend - it's just the JET functionality you can work with. Access the GUI (which includes VBA) runs on the client machine and there is no automated trigger functionality.
If your program is the only program using the Access file, then it should know when a table is being updated and execute some code in place of a trigger.
Otherwise, you need another application/service running all the time that is checking the access file tables for updates (maybe you have some update_date type of field on your tables?).
When an Access database file gets written to, it's date/time stamp changes. I suppose you could try using a file monitor to detect changes to the file, and then examine the file to see what has changed.
It would help if the Access database has LastModified date/time columns in the tables.
If you are using Jet (i.e. the data is stored in an MDB file back end) then the only places you can run code would be in the After Update Event in a Form. The problem here of course is if the data is changed without using the form then the event will not fire.
If you are using MS Access 2003 then to run a Web Service you can download the Microsoft Office 2003 Web Services Toolkit Click Here to download
If you are stuck in VBA it gets a little rough. One way to go would be to have a form with timer in it (you could have it open invisibly. The timer could check the table, say once a minute (or whatever interval seems suitable) for changes in record count, and verify the table still exists. (code below)
But personally this isn't what I would recommend that you do. Access is notorious for corruption. When used as a simple back end you are fairly safe most of the time, but to have it running a monitor, means the file is always open. This is basically playing Russian Roulette with your database. At minimum I would link to your database from another Access file and monitor the linked tables, that way if your monitor crashes, you don't take the production DB with you. Finally, make sure that you don't query too often, as I'd hate to see you be the sole cause of the website timing out:)
Option Explicit
Private m_lngLstRcrdCnt_c As Long
Private Sub Form_Open(Cancel As Integer)
Const lngOneMinute_c As Long = 60000
Me.TimerInterval = lngOneMinute_c
End Sub
Private Sub Form_Timer()
Const strTblName_c As String = "Foo"
Const strKey_c As String = "MyField1"
Dim rs As DAO.Recordset
Dim lngRcrdCnt As Long
If TableExists(strTblName_c) Then
Set rs = CurrentDb.OpenRecordset("SELECT Count(" & strKey_c & ") FROM " & strTblName_c & ";", dbOpenSnapshot)
If Not rs.EOF Then lngRcrdCnt = Nz(rs.Fields(0&).Value, 0&)
rs.Close
If lngRcrdCnt <> m_lngLstRcrdCnt_c Then
m_lngLstRcrdCnt_c = lngRcrdCnt
'Number of records changed, do something.
End If
Else
'Table is deleted, do something.
m_lngLstRcrdCnt_c = -1
End If
End Sub
Private Function TableExists(ByVal name As String) As Boolean
Dim tdf As DAO.TableDef
On Error Resume Next
Set tdf = CurrentDb.TableDefs(name)
If LenB(tdf.name) Then 'Cheap way to catch broken links.
Set SafeGetTable = tdf
End If
End Function