In my MS Access application I have several forms that are very data intensive (several subforms based on even more tables). My users are complaining that when opening the data across the network the load times are unbearably long.
I have do have a slit front end / back end setup using the excellent autofe application.
One solution I have come up with to the problem is instead of docmd.close when the user clicks the "Save & Close" button I me.visible = false. The user then has the long wait time the first time after the application is loaded but for later loads performance is improved by a noticeable amount.
So far this has been working fairly well. I am just concerned that there may be some hidden gotchas hidden in this strategy that I haven't encountered yet.
My users aren't overly intelligent and I don't use the application myself so I can't expect to get meaningful feedback if something is behaving erratically.
Anyone else employed this strategy successfully or know of a good reason not to do it?
Anyone else employed this strategy successfully or know of a good reason not to do it?
Yes, that strategy is similar to recipe #8.1 Accelerate the Load Time of Forms from the second edition of the Access Cookbook. However that recipe pre-loads a set of forms, with WindowMode:=acHidden, at database startup. So the tradeoff is that database startup takes longer, but subsequent form opens (for the pre-loaded forms) are comparatively fast.
The discussion for that recipe didn't mention any drawbacks for that technique. In limited use, I haven't discovered any. And since it seems to improve your users' experience, I would continue to use it.
Beyond that, I would take a hard look at the amount of data your forms pull from the back-end database. Limit the number of rows retrieved as the Record Sources for the main and subforms. Give the user a method to select a different record or small set of records. Also make sure you use indexing to support Record Source WHERE and ORDER BY clauses. Avoid WHERE conditions that use functions which will force a full table scan to figure out which rows to exclude from the Record Source. Similar considerations apply to combo and list boxes which use saved queries or SELECT statements as their Record Sources; if you can't limit the rows, at least make sure to optimize data retrieval.
At first, just hiding the form is not too bad, I think.
I would dig a bit more on WHY your load times are so long. You mentionned several subforms. Are they all displayed at the same time, or are they in the various pages of a Tab control ?
In the latter case, you could quite easily unbind the subforms that are not visible, and bind them on the PageClick event. That makes a big difference in performance.
EDIT:
Also, a bit out of scope for this question, but good for every performance issue:
-did you double check that the foreign keys in the related tables are properly indexed ?
-make sure the back-end is regularly compacted.
Are you making sure that the data gets refreshed in an appropriate timeframe?
Yes, I've doen the same thing myself in very complex forms which had about 10 or 15 tabs each with a subform. Worked for at least ten years. You had to watch for varous form level values or unbound controls which you assume start as null or zero. But once it's running smoothly it should run just fine. We had to this back in Access 97 days because Access would crash with out of memory errors after the users had opened and closed varous forms thousands of times per day.
Related
I am working on a project, which will be used by around 500 employees in my organization. Currently, it's still in development phase, and very few people(around 10) are using it. I'm using MySQL. I just want to know, what happens if many users are doing front end edits and then save, at the same point of time? Some SELECT queries that I've written do take as long as 6 seconds to execute. As only one query can be executed at any point of time, if already a query is in progress, and another hits the database, will it create problem? If this is a common situation in large scale projects, please let me know how can I handle this. I'm not sure, if I've made myself clear :). Any advice or links will be very helpful.
From technical aspect, no - nothing bad will happen, the database won't go ballistics and die on you, they're made for purposes like concurrent access.
From logical point of view - something bad will happen. If two people edit the same thing at the same time and then post it at the same time - it gets saved to hard drive one after another. The last one to save is the one whose updates will be on the HDD, effectively causing the first person to lose their changes.
You can approach this problem from several angles. Some projects introduce the concept of locking (not table locking but in-app locking). It revolves around marking a record as locked using a boolean column and if anyone tries to access that record for updating, the software says that someone else is editing it. It's something really difficult to implement and for the most time it doesn't work as expected (I think I vaguely remember Joomla! using something like that, it was one of the most annoying features ever).
The other option you have is to save each update as a revision. That way you can keep track on who updated what and when and you never lose any records in case of would-get overwritten. I believe that SO and Wikipedia use that approach and it works really great because you can inspect what two or more people have done and merge their contributions.
Optimistic Concurrency Control
http://en.wikipedia.org/wiki/Optimistic_concurrency_control
Make sure that each record contains date metadata on last changed/modified time, and load that as part of your data object. Then when attempting to commit the row to database, check the last_modified time in the table to ensure that it is the SAME as the one stored in memory for your object. If it matches, commit it, else throw exception.
I have a query that takes about a minute to complete since it deals with a lot of data, but I also want to put the results on a website. The obvious conclusion is to cache it (right?) but the data changes as time goes by and I need a way to automatically remake the cached page maybe every 24 hours.
can someone point me to how to do this?
edit: I want to make a "top 10" type of thing so it's not displaying the page that is the problem but the amount of time it takes for the query to run.
Caching the results of the query with a 24hr TTL (expiry) would probably work fine. Use a fragment cache assuming this is a chunk of the page.
You can setup memcached or redis as stated to store the cache. Another thing you can do is setup a job that warms the cache every 24 hrs (or as desired) so that unlucky user doesn't have to generate the cache for you.
If you know when the cache is expired base on a state or change in your database you can expire the cache based on that. A lot of times I use the created at or updated at fields as part of the cache key to assist in this process.
There is some good stuff in the scaling rails screencasts by envy labs and new relic. http://railslab.newrelic.com/scaling-rails, a little out of date but the principles are still the same.
Also, checkout the caching rails guides. http://guides.rubyonrails.org/caching_with_rails.html
Finally, make sure indexes are setup properly, use thoughtbots post here: http://robots.thoughtbot.com/post/163627511/a-grand-piano-for-your-violin
Typed on my phone so apologies for typos.
Think a little beyond the query. If your goal is to allow the user to view a lot of data, then grab that data as they want it rather than fighting with a monsterous query that's going to overwhelm your UI. The result not only looks better, but is much, much quicker.
My personal trick for this pattern is DataTables. It's a grid that allows you to use Ajaxed queries (which is built in) to get data from your query a "chunk" at a time that the user wants to see. It can sort, page, filter, limit, and even search with some simple additions to the code. It even has a plug-in to export results to excel, pdf, etc.
The biggest thing that Datatables has that others don't is a concept called "pipelining" which allows you to get an amount to show (say 20) plus an additional amount forward and/or backwards. This allows you to still do manageable queries, but not to have to hit the database each time the user hits "next page"
I've got an app dealing with millions of records. One query of all data would be impossible....it would just take too long. Grabbing 25 at a time, however, is lightning fast, no tricks required. Once the datatable was up, I just performance tuned my query, did some indexing where needed, and voila.....great, responsive app.
Here's a simple example:
<table id="example"></table>
$('#example').dataTable( {
"bProcessing": true,
"bServerSide": true,
"sAjaxSource": "/processing/file.php"
} );
Use a cache store that allows auto-expiration after a certain length of time.
Memcached does it, Redis too I guess !
I have an MS Access frontend and MySQL backend.
Everthing runs well however I am trying to avoid the following.
On an Access form, I use it to do read queries.
Once the results are shown on the Access form (excel like format), I can sort A-Z, Z-A etc
The thing is that everytime I do any sorting, I can see on MySQL Administrator that a query (in fact more than one) is being performed. I need to avoid this for obvious reasons, as I user will constantly do sortings and all sort of things with the displayed data.
Since the the data is displayed already in the client side (Access form), I don't undertand the reason for MySQL to be requeried again when doing the sorting. I would expect anything you do with the results of the query will happen "locally".
Any ideas what I am doing and wrong and/or how to avoid this ?
thanks ind advance
That's pretty typical to make a roundtrip back to the server for sorting. I would imagine most applications do not sort from internal memory but hit the database again. It would probably be quicker to sort locally if you only had a handful of rows. But if you are dealing with millions of rows it is going to be much quicker to perform that sort on the db as that is what it is made for.
Why do you not want it to make that roundtrip for sorting?
Is your form bound to the whole table? If so, that's a design error. It should be bound only to a limited recordset. In that case, only a small query is going to be executed server-side.
Secondly, you might want to turn on Jet SHOWPLAN (Google for it) and see what the Jet/ACE db engine is doing with regard to optimization.
Third, it may be hitting the server, but is it a significant hit?
Last of all, you may get the results you want by using disconnected recordsets assigned to your forms instead of bound recordsets. This is a lot more complicated than using the default bound forms, but can be necessary. It does make using Access a lot less pleasant as the fit and finish is not nearly as smooth as with traditional Access bound forms.
Whenever I'm to prepare a long form for the client I always want to split it into separate pages, so the visitor doesn't have to fill it all, but does it in steps.
Something like:
Step 1 > Step 2 > Step 3 > Thank You!
I've never done it for one reason: I don't know how to store the data from separate steps efficiently? By efficiently I mean, how to store it, so when a visitor decides not to finish it at Step 3 all the data is deleted.
I've come up with few ways of how this could be resolved, but I'm just not convinced by any of them:
Storing form data in database
I can imagine a table with columns representing each question, with final column representing a bool value whether the form has been completed or not?
But I would have to do a clean-up of the table every now and then (maybe even every time it gets updated with new data?) and delete all entries with complete = 0.
Store form data in session data.
This on the other hand, does not have to store data in database (depending on how sessions are being handled) and all info would be in Cookie. But what if browser doesn't support cookies or user disabled them (rare, but happens), or if form has file attachments, then this is a no-go.
echo'ing form data from previous page as <input type="hidden"> on the next page
Yes, I'm aware this is a rather stupid idea, but it's an alternative. Poor, but it is.
Option 1 seems to be the best, but I feel it's unnecessary to store temporary data in DB. And what if this becomes a very popular form, with a lot of visitors filling it in? The amount of Updates/Deletes could be massive?
I want to know how you deal with it.
Edit
David asked a good question. What technology I'm using?
I personally use PHP+MySQL, but I feel like it's more generic question. Please share your solutions no matter of what server-side technology you use, as I'm sure the concept can be adapted one way or the other to different technologies.
I think the choice between options 1 and 2 comes down to how much data you are storing. I think in most cases the amount of data you are collecting on a form is going to be fairly small (a few kilobytes). In that case, I think storing it in the session data is the way to go. There's not that much overhead in passing that amount of data back and forth. Also, unless your users are on computers where there is a strict security policy in place, the application should work. If you make the page requirements clear users can decide if they want to proceed or not.
If you are storing large amounts of data for the form then a database would be better so you don't need to pass the data back and forth. However, I think this is going to be a fairly rare situation. Even if the application allows the uploading of files you can save those to a temporary location and only write them to the database once the form is completed. The other situation where you might want to use a database is if your form needs to be able to support the user leaving and coming back at a later time to resume the form.
I agree that option 1 is the best, because it has a few benefits over the other 2:
If the data is persisted, users can come back later and continue the process
Your code base will be much cleaner with incremental saves, and it alleviates the need for 1 massive save operation
Your foot print (each page request) will be lighter than option 3
If you're worried about performance, you can queue the data to be saved, since it's not necessary to save it near-real-time.
Edit to clear up a misconception: The data inside PHP Sessions, by default, are NOT stored in Cookies and are capable of storing a lot of data without too much overhead.
I'd go with number 2, but use the cookie only for identifying the user. The session data should actually be stored on your server and the cookie merely provides a lookup key to the session object that contains all the details.
If the site becomes popular and needs to run on more than a single web server, then your session data will need to be persisted in some kind of database anyway. In that case you would need a database that could handle massive amounts of transactions.
Note: I agree that this is a platform independent question. Stack Overflow users prefer to see code in questions and prefer to give code in answers, so that's why I normally ask what language someone is using.
To be brutally honest, just use the database as in option 1 and stop worring about data volumes. Seriously if your site is that successful that it becomes a problem then you ought be able fund a re-vamp to cope.
There's nothing wrong with taking the POST data from the previous step and adding hidden input elements. Just take all the POST data from the previous page that you care about and get them into the current page's form. This way, you don't have to worry about using persistent storage in any form, whether it's on the client side or the server side.
What are the perceived downsides? That there are a lot of extra elements on the page? Not that the user sees. All you have to do is add an element for each input you ask the user to give (on every page, if you want the user to be able to go back). Besides these elements, which don't give any visual clutter, there's nothing extra.
There's also the fact that all the form data will have to be transmitted on every page load. Sure, but this is probably going to be faster than a lookup in a database, and you don't have to worry about getting rid of stale data.
We're thinking of "growing" a little MS-Access DB with a few tables, forms and queries for multiple users. (Using a different back-end is another, but more long-term option that is unfortunately currently not acceptable.)
Most users will be read-only, but there will be a few (currently one or two) users that have to be able to do changes (while the read-only users are also using the DB). We're not so much concerned about the security aspects, but more about some of the following issues:
How can we make sure that the write-user can make changes to the table data while other users use the data? Do the read-users put locks on tables? Does the write-user have to put locks on the table? Does Access do this for us or do we have to explicitly code this?
Are there any common problems with "MS Access transactions" that we should be aware of?
Can we work on forms, queries etc. while they are being used? How can we "program" without being in the way of the users?
Which settings in MS Access have an influence on how things are handled?
Our background is mostly in Oracle, where is Access different in handling multiple users? Is there such thing as "isolation levels" in Access?
Any tips or pointers to helpful articles would be greatly appreciated.
I find the answers to this question to be problematic, confusing and incomplete, so I'll make an effort to do better.
Q1: How can we make sure that the write-user can make changes to the table data while other users use the data? Do the read-users put locks on tables? Does the write-user have to put locks on the table? Does Access do this for us or do we have to explicitly code this?
Nobody has really answered this in any complete fashion. The information on setting locks in the Access options has nothing to do with read vs. write locking. No Locks vs. All Records vs. Edited Record is how you set the default record locking for WRITES.
No locks means you are using OPTIMISTIC locking, which means you allow multiple users to edit the record and then inform them after the fact if the record has changed since they launched their own edits. Optimistic locking is what you should start with as it requires no coding to implement it, and for small users populations it hardly ever causes a problem.
All Records means that the whole table is locked any time an edit is launched.
Edited Record means that fewer records are locked, but whether or not it's a single record or more than one record depends on whether your database is set up to use record-level locking (first added in Jet 4) or page-level locking. Frankly, I've never thought it worth the trouble to set up record-level locking, as optimistic locking takes care of most of the problems.
One might think that you want to use record-level pessimistic locking, but the fact is that in the vast majority of apps, two users are almost never editing the same record. Now, obviously, certain kinds of apps might be exceptions to that, but if I ran into such an app, I'd likely try to engineer it away by redesigning the schema so that it would be very uncommon for two users to edit the same record (usually by going to some form of transactional editing instead, where changes are made by adding records, rather than editing the existing data).
Now, for your actual question, there are a number of ways to accomplish restricting some users to read-only and granting others write privileges. Jet user-level security was intended for this purpose and works fine insofar as it's "security" for any meaningful definition of the term. In general, as long as you're using a Jet/ACE data store, the best security you're going to get is that provided by Jet ULS. It's crackable, yes, but your users would be committing a firable offense by breaking it, so it might be sufficient.
I would tend to not implement Jet ULS at all and instead just architect the data editing forms such that they checked the user's Windows logon and made the forms read-only or writable depending on which users are supposed to get which access. Whether or not you want to record group membership in a data table, or maintain Windows security groups for this purpose is up to you. You could also use a Jet workgroup file to deal with it, and provide a different system.mdw file for the write users. The read-only users would log on transparently as admin, and those logged on as admin would be granted only read-only access. The write users would log on as some other username (transparently, in the shortcut you provide them for launching the app, supplying no password), and that would be used to set up the forms as read or write.
If you use Jet ULS, it can become really hairy to get it right. It involves locking down all the tables as read-only (or maybe not even that) and then using RWOP queries to provide access to the data. I haven't done but one such app in my 14 years of professional Access development.
To summarize my answers to the parts of your question:
How can we make sure that the write-user can make changes to the table data while other users use the data?
I would recommend doing this in the application, setting forms to read/only or editable at runtime depending on the user logon. The easiest approach is to set your forms to be read-only and change to editable for the write users when they open the form.
Do the read-users put locks on tables?
Not in any meaningful sense. Jet/ACE does have read locks, but they are there only for the purpose of maintaining state for individual views, and for refreshing data for the user. They do not lock out write operations of any kind, though the overhead of tracking them theoretically slows things down. It's not enough to worry about.
Does the write-user have to put locks on the table?
Access in combination with Jet/ACE does this for you automatically, particularly if you choose optimistic locking as your default. The key point here is that Access apps are databound, so as soon as a form is loaded, the record has a read lock, and as soon as the record is edited, whether or not it is write-locked for other users is determined by whether you are using optimistic or pessimistic locking. Again, this is the kind of thing Access takes care of for you with its default behaviors in bound forms. You don't worry about any of it until the point at which you encounter problems.
Does Access do this for us or do we have to explicitly code this?
Basically, other than setting editability at runtime (according to who has write access), there is no coding necessary if you're using optimistic locking. With pessimistic locking, you don't have to code, but you will almost always need to, as you can't just leave the user stuck with the default behaviors and error messages.
Q2: Are there any common problems with "MS Access transactions" that we should be aware of?
Jet/ACE has support for commit/rollback transactions, but it's not clear to me if that's what you mean in this question. In general, I don't use transactions except for maintaining atomicity, e.g., when creating an invoice, or doing any update that involves multiple tables. It works about the way you'd expect it to but is not really necessary for the vast majority of operations in an Access application.
Perhaps one of the issues here (particularly in light of the first question) is that you may not quite grasp that Access is designed for creating apps with data bound to the forms. "Transactions" is a topic of great importance for unbound and stateless apps (e.g., browser-based), but for data bound apps, the editing and saving all happens transparently.
For certain kinds of operations this can be problematic, and occasionally it's appropriate to edit data in Access with unbound forms. But that's very seldom the case, in my experience. It's not that I don't use unbound forms -- I use lots of them for dialogs and the like -- it's just that my apps don't edit data tables with unbound forms. With almost no exceptions, all my apps edit data with bound forms.
Now, unbound forms are actually fairly easy to implement in Access (particularly if you name your editing controls the same as the underlying fields), but going with unbound data editing forms is really missing the point of using Access, which is that the binding is all done for you. And the main drawback of going unbound is that you lose all the record-level form events, such as OnInsert, BeforeUpdate and so forth.
Q3. Can we work on forms, queries etc. while they are being used? How can we "program" without being in the way of the users?
This is one of the questions that's been well-addressed. All multi-user or replicated Access apps should be split, and most single-user apps should be, too. It's good design and also makes the apps more stable, as only the data tables end up being opened by more than one user at a time.
Q4. Which settings in MS Access have an influence on how things are handled?
"Things?" What things?
Q5. Our background is mostly in Oracle, where is Access different in handling multiple users? Is there such thing as "isolation levels" in Access?
I don't know anything specifically about Oracle (none of my clients could afford it even if they wanted to), but asking for a comparison of Access and Oracle betrays a fundamental misunderstanding somewhere along the line.
Access is an application development tool.
Oracle is an industrial strength database server.
Apples and oranges.
Now, of course, Access ships with a default database engine, originally called Jet and now revised and renamed ACE, but there are many levels at which Access the development platform can be entirely decoupled from Jet/ACE, the default database engine.
In this case, you've chosen to use a Jet/ACE back end, which will likely be just fine for small user populations, i.e., under 25. Jet/ACE can also be fine up to 50 or 100, particularly when only a few of the simultaneous users have write permission. While the 255-user limit in Jet/ACE includes both read-only and write users, it's the number of write users that really controls how many simultaneous users you can support, and in your case, you've got an app with mostly read-only users, so it oughtn't be terribly difficult to engineer a good app that has no problems with the back end.
Basically, I think your Oracle background is likely leading you to misunderstand how to develop in Access, where the expected approach is to bind your forms to recordsources that are updated without any need to write code. Now, for efficiency's sake it's a good idea to bind your forms to subsets of records, rather than to whole tables, but even with an entire table in the recordsource behind a data editing form, Access is going to be fairly efficient in editing Jet/ACE tables (the old myth about pulling the whole table across the wire is still out there) as long your data tables are efficiently indexed.
Record locking is something you mostly shouldn't have any cause to worry about, and one of the reasons for that is because of bound editing, where the form knows what's going on in the back end at all times (well, at intervals about a second apart, the default refresh interval). That is, it's not like a web page where you retrieve a copy of the data and then post your edits back to the server in a transaction completely unconnected to the original data retrieval operation. In a bound environment like Access, the locking file on the back-end data file is always going to be keeping track of the fact that someone has the record open for editing. This prevents a user's edits from stomping on someone else's edits, because Access knows the state and informs the user. This all happens without any coding on the part of the developer and is one of the great advantages of the bound editing model (aside from not having to write code to post the edits).
For all those who are experienced database programmers familiar with other platforms who are coming to Access for the first time, I strongly suggest using Access like an end user. Try out all the point and click features. Run the form and report wizards and check out the results that they produce. I can't vouch for all of them as demonstrating good practices, but they definitely demonstrate the default assumptions behind the way Access is intended to be used.
If you find yourself writing a lot of code, then you're likely missing the point of Access.
The first thing to do (if not already done) is to split your database into a front end (with all the forms/reports etc) and a back end(with all the data). The second thing is to setup version control on the front end.
The way I have done that in a lot of my databases is to have the users run a small “jumper” database to open the main database. This jumper does the following things
• Checks to see if the user has the database on their C drive
• If they do not then install and run
• If they do then check what version they have
• If the version numbers do not match then copy down the latest version
• Open the database
This whole checking process normally takes under half a second. Using this model you can do all your development on a separate database then when you are ready to “release” you just put the new mde up onto the network share and the next time the user opens the jumper the latest version is copied down.
There are also other things to think about in multiuser database and it might be worth checking for the common mistakes such as binding a form to a whole table etc
i think Access is a best choice for your case. But you have to split database, see:
http://accessblog.net/2005/07/how-to-split-database-into-be-and-fe.html
•How can we make sure that the write-user can make changes to the table data while other users use the data? Do the read-users put locks on tables? Does the write-user have to put locks on the table? Does Access do this for us or do we have to explicitly code this?
there are no read locks unless you put them explicitly. Just use "No Locks"
•Are there any common problems with "MS Access transactions" that we should be aware of?
should not be problems with 1-2 write users
•Can we work on forms, queries etc. while they are being used? How can we "program" without being in the way of the users?
if you split database - then no problem to work on FE design.
•Which settings in MS Access have an influence on how things are handled?
What do you mean?
•Our background is mostly in Oracle, where is Access different in handling multiple users? Is there such thing as "isolation levels" in Access?
no isolation levels in access.
BTW, you can then later move data to oracle and keep access frontend, if you have lot of users and big database.
Table or record locking is available in Access during data writes. You can control the Default record locking through Tools | Options | Advanced tab:
No Locks
All Records
Edited Record
You can set this on a form's Record Locks or in your DAO/ADO code for specific needs.
Transactions shouldn't be a problem if you use them correctly.
Best practice: Separate your tables from All your other code. Give each user their own copy of the code file and then share the data file on a network server. Work on a 'test' copy of the code (and a link to a test data file) and then update user's individual code files separately. If you need to make data file changes (add tables, columns, etc), you will have to have all users get out of the application to make the changes.
See other answers for Oracle comparison.
The correct way of building client/server Microsoft Access applications where the data is stored in a RDBMS is to use the Linked Table method. This ensures Data Isolation and Concurrency is maintained between the Microsoft Access client application and the RDBMS data with no additional and unnecessary programming logic and code which makes maintenance more difficult, and adds to development time.
see: http://claysql.blogspot.com/2014/08/normal-0-false-false-false-en-us-x-none.html
Access is a great multi-user database. It has lots of built in features to handle the multi-user situation. In fact, it is so very popular because it is such a great multi-user database. There is an upper limit on how many users can all use the database at the same time doing updates and edits - depending on how knowledgeable the developer is about access and how the database has been designed - anywhere from 20 users to approx 50 users. Some access databases can be built to handle up to 50 concurrent users, while many others can handle 20 or 25 concurrent users updating the database. These figures have been observed for databases that have been in use for several or more years and have been discussed many times on the access newsgroups.
I have found the SMB2 protocol introduced in Vista to lock the access databases.
It can be disabled by the following regedit:
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\LanmanServer\Parameters]
"Smb2"=dword:00000000