Related
I've been researching this question over the last few days as I prepare to deliver the first of 3 phases with my first system using a split database. I would like your advice as I haven't found enough info to make a full decision yet.
At the moment I'm working in dev on an unsplit database. When I split it in live I'll take a copy of both parts but what do I do with them for phase 2?
I'm thinking that I'll now make them my dev version after relinking the tables (as I've effectively moved the back end) which would then mean that I no longer work with the unsplit database. Is that the right approach?
When it comes to putting phase 2 live I don't think I have any choice other than manually applying table updates to the live back end (once backed up). For the live front end, do I just replace it with my dev front end and then relink the tables or do I export the changes into the live front end? I guess I could do either depending on the number of objects that are changing/new. Is that right? I'll then take copies and make them my dev versions for phase 3.
Finally in dev I have form and report templates and test forms, reports and queries which are not needed in live so do I remove them for each deployment and then add them all back in to the new dev front end or just put them live? Normally I would take them out but there are a lot of them and I don't know of any quick way to add them back in so what do you do?
Primarily my questions are asked from a risk point of view - what steps best reduce the risk of messing things up in live.
Update:
For those of you who are looking for answers on this, in addition to the fine responses below I have since found the following that might also help:
How to Continuously Develop and Deploy an Access 2010 Database Application
At the moment I'm working in dev on an unsplit database. W
Don't, a VERY bad idea. How this works?
Well, for sure at the start, you are building LOTS of new tables, changing relationships, and building tables at a high rate of development.
so, at this point in time, you can develop un-split.
You will then find after some time the rate (and need) to create new tables, and change the so called "database schema" calms down to a dull roar.
At this point your still developing away - not yet deployed.
So, somewhere around this point? You want to split. You REALLY want to do this.
There is a boatload of reasons for this, but several are:
while a split vs non split is "very similar", they are not the same!
Thus, you can't and don't want to develop code that NOT really tested
as to HOW the code will run in the real world
So, many issues can change or crop up during development that is DIFFERENT when run split. So some commands (such as seek()) don't work, and a few other issues can crop up. You don't want to develop for a whole week, then split and now find 20 hard to fix bugs in your code. So, by developing as split as soon as possible, then ANY and ALL issues that come up will be seen as you develop along, and thus can fix, see, and deal with such issues at THAT POINT in time. Much worse is to write a bunch of code, get ready to deploy, and then find new bugs.
Next up:
Having a split system is great, since say a customer might let you remote into their system. You can pull down a copy of their data, re-link your tables to point from your "test data" to real live production data.
Or, say your developing on site. You might want to test some dangerous delete code, or code that modifies the data. So, you can't risk working on production data, so now you re-link and point to your test back end. So, this setup allows you to test code, but MORE important test on a copy of the database with great ease.
And it also allows you to develop off site. You can take the latest front end for their system, maybe get their latest data file, maybe not. but you can now with ease simple change the database that your applcation runs with.
The other big issue? Say your working on site, and have a test database of theirs on the network folder. You write some code, test for a new report. You find it runs SLOW AS A TURTLE. You check your code, maybe add a few indexes, and boom! - your report now runs great.
If you test un-split, then a boatload of performance issues can crop up, but NOT SEEN during the development cycle. Once again, you don't want to develop for weeks or whatever, split, and NOW find a whole bunch of forms and code runs REALLY slow.
So, the goal, the idea here?
You want to get split as SOON as possible?
How soon?
Well, this is one of those things that only you can know!!!
As I stated, at start of development, sure, start out un-split.
Once the table designs are quite solid, then you can split. You then ALWAYS develop as split (and the above list of reasons why is the VERY short list - there are many more reasons).
Now, the problem of course with split? Say you want to add a new column to a table?
Well, it is MORE work, since now ANY AND ALL changes to the data schema are done in the back end. So, you have to close down the front end (FE), open the back end, and now use the table designer to add that one column. (or maybe change or setup a new relationship between some tables. Or maybe add a new table).
This is a "bit more" work, since now you close down the BE, and open the FE, and now you MUST re-link tables. And if you added new tables to the BE, then you have to add that/those new table links.
Because this "dance" is extra work, that is why you wait as long as possible to split. As I stated, you "just know" when that time has arrived to split. (when table and schema tables changes settles down to a low rate of change). Since the rate of change is now low in regards to table changes, then it not much work nor pain to have to do the above every time you want to change the table structures.
In fact, think of any program you buy? It has a applcation part, and then a data file part. In effect, once you split, you have the same two parts, and in fact in some applications I written, they are allowed to use "different" back ends - not unlike any other applcation in which you launch the applcation, and THEN choose the datafile to work with.
So, what about developing off site? Well, that can be REALLY difficult, since you have your own copy of the FE and the BE.
If you HAVE TO make changes to the BE?
I open up a work document. And if I add a new column to say tblCusotmers?
Then I enter this:
Add new column TaxRate, Currency, to table tblCustomers
So, you build up a "log" of changes. Now, when you travel to the customer site, and want to roll out and deploy the new FE? Well, you have to FIRST open up their BE, and make the above changes to their production BE database.
Now in some cases, where is was not possible for me to be on site? (in fact, I had a automatic update system to automatic roll out a new version of my software - and it would automatic down load from the internet. In this case? I had to write code in the FE on startup that would use VBA code to MAKE the changes to the data tables. This can be REALLY hard to do, but is possible. I just recommend the plane jane word document, and you keep track of your changes.
So, the above is quite much how this development process works.
Since you will have to re-link the tables? Then near everyone has googled for a VBA table re-link routines. You want that, since having such code is MUCH easier then say using the linked table manager each time. And we even often have a table in the FE that saves the BE server location, and on startup will check, and if the location of the files don't match, we launch the re-link code.
that way, you can deploy the applcation to different sites, and have it automatic re-link. Another way is to have a simple text file in the same location as the FE on each computer, and on startup read the text file with the BE location - and re-link if required.
So, the typical process to role out a new FE (which is placed on each work station - do NOT break this rule!!!).
So, I point/relink my front end to the production BE. I then compile down to a accDE, and then deploy that new compiled FE to all the work stations. In fact, I have some code in VBA at start up that compares a version number, and if the version number is lower, then the VBA code will copy down the next FE sitting in a folder.
This might not be a big deal if you have 2-4 users. But, if you have two sites, and each has 35 users, then you want to figure out a automated approach.
However, do not prolong jumping over to the split development cycle, as you really for all practical purposes MUST develop in a split environment. So, for the first part, you can develop un-split. But, once you split - that's it, and from that point on-wards, you are to develop as split. There are boatloads of benefits, but it also really quite much standard approach from a developer point of view.
So, you have to master the linked table manager rite quick, and then VERY much consider adding some re-link code, since you want with great ease to point to a different back end - including at deployment time.
so, as a general rule, you should think of your FE like a .exe program, for a new version roll out, yes, you copy (over write) their existing FE's on each work station. And as noted, in most cases, it should be a compiled accDE, and not a un-compiled accDB for the FE.
For reducing risk:
you should have your development version, a test version for live and the live version.
You are developing on develop
Customer tests the changes on test (with test data)
after that you move to live
For the move from develop to test I create an update/migration script.
In this script are all alterations included that needs to be done on the back-end.
I use the script to create the test version and with this I can check if it is working properly.
In case there are database changes that I can't reflect in my script (either insufficient skill or restrictions from db) I add them to my checklist.
I am using version control to see changes during development and to import modules, queries etc. to the new version.
Updating the front end is done via Import of the latest version (without not needed forms / reports).
One of my sites is a social networking site running on MySQL. I use postal code and country information to geolocate users using a webservice. This webservice also allows you to download all their many tables of information so that you can access it locally. My site has gotten big enough that I wish to do this now.
My question is, should I create a new database on my site for all of this postal code and country information and all its tables, or should I incorporate those tables into my existing database for my social networking site?
What are the pros/cons either way?
When you're talking about scaling and want to know about other databases like NOSQL, you might find this article interesting: http://highscalability.com/blog/2010/12/6/what-the-heck-are-you-actually-using-nosql-for.html
I'd vote in favor of a separate database if you planned to use the data as read-only and put a web service in front of it to access it. Users would search it based on a small handful of parameters (e.g. address info to get lat/lon data).
I'd say put it in the existing database if you planned to JOIN it with other information in your current schema.
it will live on the same disk probably.
so disk space is not an issue.
if you query the tables in a completely separate manner, then no impact on the existing site.
if you query things together, then easier when all in one database.
overall administration of one database vs 2 is easier.
i think it's a no brainer... they go in one db.
I am starting to create an MSAccess database, I have no Access experience - my previous experience is with MySQL and Oracle. Initially I had some difficulty coming to terms with the fact that MSAccess usually stores both the front end application and the Jet Engine database in the same file. It's different from what I'm used to. Plus the database will be shared over a network, and it just makes more sense to split the application from the data.
After some reading, I see that it is possible to store data in one file, and then link to the application elements in another file. Every article I've come across for this deals with splitting the database into two parts, after the database has already been made, and never discusses creating split database applications from the start. Is it because that would be a bad idea? I can't really imagine why, except that I've noticed that Access does not let me keep two database files open at the same time (it automatically closes one). So I am foreseeing a need to constantly to open and re-open files if I go down that route.
There is one practical reason why you might want to start with a single database. If you start with a front and back end file, you'll have to create tables in one database, then set up the link for each table manually.
This is not a big deal, but if you're just starting the system, you can save some busywork by developing the pilot system in one file, then splitting it. My assumption is you'll probably be making a lot of changes to the data structure at the outset; your work will go smoother if you're working in one file.
It is definitely a good idea to split the database before you deploy it to production. I'm not sure why you're having problems opening 2 Access files at once; this is not a restriction of Access.
You can create the two db files separately at the outset. I do that often. I seldom need both open at the same time in the Access interface. I only open the back-end database, which houses the tables, indexes, and relationships, to modify the design of those db objects. And those types of changes are relatively infrequent; most of the development workload is for the front-end db. To modify data in the tables, you can use the table links from the front-end db.
It is not a bad idea. You can have two files open at the same time, either open another Access instance or launch by double-clicking the second file. Make sure you have created a suitable back-end design before you start on the front-end.
It is more efficient to have it all in one file while you're alone to work on it. Once the database design is finalised, then you can split the db.
Splitting the db is usefull during testing as well: it allows you to reset your data to a known state in about 5 sec, just by copying a saved version of the back-end.
I have created an MS Access 2003 application, set up as a split front-end/back-end configuration, with a user group of about five people. The front end .mdb sits on a network file server, and it contains all the queries, forms, reports, and VBA code, plus links to all the tables in the back end .mdb and some links to ODBC data sources like an AS/400. The back end sits on the same network file server, and it just has the table data in it.
This was working well until I "went live" and my handful of users started coming up with enhancement requests, bug reports, etc. I have been rolling out new code by developing/testing in my own copy of the front-end .mdb in another network folder (which is linked to the same back-end .mdb), then posting my completed file in a "come-and-get-it" folder, alerting the users, and they go copy/paste the new front-end file to their own folders on the network. This way, each user can update their front end when they're at a 'stopping point' without having to boot everyone out at once.
I've found that when I'm developing now, sometimes Access becomes extremely slow. Like, when I am developing a form and attempt to click a drop-down on the properties box, the drop-down arrow will push in, but it will take a few seconds before the list of options appears. Or there's tons of lag in selecting & moving controls on a form. Or lots of keyboard lag.
Then, at other times, there's no lag at all.
I'm wondering if it's because I'm linked to the same back end as the other users. I did make a reasonable effort to set up the queries, forms, reports etc. with minimal record locking, if any at all, depending on the need. But I may have missed something, or perhaps there is some other performance issue I need to address.
But I'm wondering if there is an even better way for me to set up my own development back-end .mdb, so I can be testing my code on "safe" data instead of the same live data as the rest of the users. I'm afraid that it's only a matter of time before I corrupt some data, probably at the worst possible moment.
Obviously, I could just set up a separate back-end .mdb and manually reconfigure the table links in the front end every time, using the Linked Table Manager. But I'm hoping there is a more elegant solution than that.
And I'm wondering if there are any other performance issues I should be considering in this multi-user, split database configuration.
EDIT: I should have added that I'm stuck with MS Access (not MS-SQL or any other "real" back end); for more details see my comment to this post.
If all your users are sharing the front end, that's THE WRONG CONFIGURATION.
Each user should have an individual copy of the front end. Sharing a front end is guaranteed to lead to frequent corruption of the shared front end, as well as odd corruptions of forms and modules in the front end.
It's not clear to me how you could be developing in the same copy of the front end that the end users are using, since starting with A2000, that is prohibited (because of the "monolithic save model," where the entire VBA project is stored in a single BLOB field in a single record in one of the system tables).
I really don't think the problems are caused by using the production data (though it's likely not a good idea to develop against production data, as others have said). I think they are caused by poor coding practices and lack of maintainance of your front end code.
turn off COMPILE ON DEMAND in the VBE options.
make sure you require OPTION EXPLICIT.
compile your code frequently, after every few lines of code -- to make this easy, add the COMPILE button to your VBE toolbar (while I'm at it, I also add the CALL STACK button).
periodically make a backup of your front end and decompile and recompile the code. This is accomplished by launching Access with the /decompile switch, opening your front end, closing Access, opening your front end with Access (with the SHIFT key held down to bypass the startup code), then compacting the decompiled front end (with the SHIFT key held down), then compiling the whole project and compacting one last time. You should do this before any major code release.
A few other thoughts:
you don't say if it's a Windows server. Linux servers accessed over SAMBA have exhibited problems in the past (though some people swear by them and say they're vastly faster than Windows servers), and historically Novell servers have needed to have settings tweaked to enable Jet files to be reliably edited. There are also some settings (like OPLOCKS) that can be adjusted on a Windows server to make things work better.
store your Jet MDBs in shares with short paths. \Server\Data\MyProject\MyReallyLongFolderName\Access\Databases\ is going to be much slower reading data than \Server\Databases. This really makes a huge difference.
linked tables store metadata that can become outdated. There are two easy steps and one drastic one to be taken to fix it. First, compact the back end, and then compact the front end. That's the easy one. If that doesn't help, completely delete the links and recreate them from scratch.
you might also consider distributing an MDE to your end users instead of an MDB, as it cannot uncompile (which an MDB can).
see Tony Toews's Performance FAQ for other generalized performance information.
1) Relink Access tables from code
http://www.mvps.org/access/tables/tbl0009.htm
Once I'm ready to publish a new MDE to the users I relink the tables, make the MDE and copy the MDE to the server.
2) I specifically created the free Auto FE Updater utility so that I could make changes to the FE MDE as often as I wanted and be quite confident that the next time someone went to run the app that it would pull in the latest version. For more info on the errors or the Auto FE Updater utility see the free Auto FE Updater utility at http://www.granite.ab.ca/access/autofe.htm at my website to keep the FE on each PC up to date.
3) Now when working on site at a clients I make the updates to the table structure after hours when everyone is out of the system. See HOW TO: Detect User Idle Time or Inactivity in Access 2000 (Q210297) http://support.microsoft.com/?kbid=210297 ACC: How to Detect User Idle Time or Inactivity (Q128814) http://support.microsoft.com/?kbid=128814
However we found that the code which runs on the timer event must be disabled for the programmers. Otherwise weird things start happening when you're editing code.
Also print preview would sometimes not allow the users to run a menu item to export the report to Excel or others. So you had to right click on the Previewed report to get some type of internal focus back on the report so they could then export it. This was also helped by extending the timer to five minutes.
The downside to extending the timer to five minutes was if a person stays in the same form and at the same control for considerable parts of the day, ie someone doing the same inquiries, the routine didn't realize that they had actually done something. I'll be putting in some logic sometime to reset this timer whenever they do something in the program.
4) In reference to another person commenting about scripts and such to update the schema see Compare'Em http://home.gci.net/~mike-noel/CompareEM-LITE/CompareEM.htm. While it has its quirks it does create the VBA code to update tables, fields, indexes and relationships.
Use VBA to unlink and re-link your tables to the new target when switching from dev to prod. It's been to many years for me to remember the syntax--I just know the function was simple to write.
Or use MS-Access to talk to MS-Access through ODBC, or some other data connection that lives outside of the client mdb.
As with all file base databases, you will eventually run into problems with peak usage or when you go over a small magical number somewhere between 2 and 30.
Also, Access tends to corrupt frequently, so backup, compact and repair need to be done on an frequent basis. 3rd party tools used to exist to automate this task.
As far as performance goes, the data is being processed client side, so you might want to use something like netmeter to watch how much data is going over the wire. The same principle about indexing and avoiding table scans apply to file base dbs as well.
Many good suggestions from other people. Here's my 2 millicents worth. My backend data is on server accessed through a Drive mapping. In my case, the Y drive. Production users get the mapping through a login script using active directory. Then the following scenarios are easily done by batch file:
Develop against local computer by doing a subst command in a batch file
run reports against last nights data by pointing Y to the backup server (read only)
run reports against end of month data by pointing to the right directory
test against specialized scenarios by keeping a special directory
In my environment (average 5 simultaneous users, 1000's of rows, not 10,000's.) corruption has occurred, but it's rare and manageable. Only once in the last several years have we resorted to the previous days backup. We use SQL Server for our higher volume stuff, but it's not as convenient to develop against, probably because we don't have a SQL admin on site.
You might also find some of the answers to this question (how to extract schemas from access) to be useful as well. Once you've extracted a schema using one of the techniques that were suggested you gain a whole range of new options like the ability to use source control on the schemas, as well as being able to easily build "clean" testing environments.
Edit to respond to comment:
There's no easy way to source control an Access database in it's native format, but schema files are just text files like any other. Hence, you can check them in and out of the source control software of your choice for easy version control/rollbacks.
Or course, it relies on you having a series of scripts set up to re-build your database from the schema. Once you do, it's normally fairly trivial to create an option/alternative version that rebuilds it in a different location, allowing you to build test environments from any previous committed version of the schema. I hope that clarifies a bit!
If you want to update the back end MDB schema automatically when you release a new FE to the clients then see Compare'Em http://home.gci.net/~mike-noel/CompareEM-LITE/CompareEM.htm will happily generate the VBA code need to recreate an MDB. Or the code to create the differences between two MDBs so you can do a version upgrade of the already existing BE MDB. It's a bit quirky but works.
I use it all the time.
You need to understand that a shared mdb file for the data is not a robust solution. Microsoft would suggest that SQL Server or some other server based database would be a far better solution and would allow you to use the same access front end. The migration wizard would help you make the changeover if you wanted to go that way.
As another uses pointed out, corruption will occur. It is simply a question of how often, not if.
To understand the performance issues you need to understand that to the server the mdb file with the data in it is simply that, a file. Since no code runs on the server, the server does not understand transactions, record locking etc. It simply knows that there is a file that a bunch of people are trying to read and write simultaniously.
With a database system such as SQL Server, Oracle, DB2. MySQL etc. the database program runs on the server and looks to the server like a single program accessing the database file. It is the database program (running on the server) that handles record locking, transactions, concurrency, logging, data backup/recovery and all the other nice things one wants from a database.
Since a database program designed to run on the server is designed to do that and only that, it can do it far better and more efficently that a program like Access reading an writing a shared file (mdb).
There are two rules for developing against live data
The first rule is . . . never develop
against live data. Not ever.
The second rule is . . .never develop
against live data. Not ever.
You can programatically change the bindings for linked tables, so you can write a macro to change your links when you're deploying a new version.
The application is slow because it's MS Access, and it doesn't like many concurrent users (where many is any number > 1).
I have been asked by a friend to help him assess a number of quotes for porting a desktop application based on MS access and VBA to a web based app. The application seems to have a relatively large amount of business logic coded into the VBA.
My question is very specific - are there any good tools or resources out there which could assist the porting from access, rather than doing a complete re-write?
The end technology used for the web app does not matter hugely, but would ideally be as mainstream as possible.
You may explore the possibilities offered by Sharepoint. It may help you get the data accessible online but how well will that work depends also on how much VBA code was used in the Access application.
There are some tools around that pretend they can convert MS Access to PHP/ASP websites like DB Forms, but I haven't tried them and they usually only convert the visible part of the app and not the queries and VBA.
They can be helpful to get started though.
Random thoughts
The VBA tends to be the biggest issue.
Moving to ASP.Net will take time and for that you are faced with difficult choices:
transfer all code to the ASP.NET to just get it working
rethink the structure and do a proper ASP.Net implementation from scratch.
I'd prefer the first one: just try has hard as possible to get results fast.
Use SSMA to move the data to SQL Server (unless you want to keep Access as the backend).
Make the forms look the same as on your existing application (or at least have the same function), port the VBA to VB.Net (or C# if you feel like it) form by form, module by module and test that they work as you go along.
Don't try to refactor or make things better at this stage, the point is to 'slap' the old code on the new 'system' and make it bark as it used to, not better, not worse.
Only then can you start refactoring and improving using the new tools at your disposal.
I'm saying all this assuming that there was nothing terribly wrong with the old app and that it just needed to be ported for online consumption.
If the old app was defective and wasn't fulfilling its role, then more emphasis should be placed on re-thinking which parts should be translated and which one should be reworked.
At any rate, you need to have a detailed action plan and a review of the current code and functionalities and try to limit as much as possible your expectations for the first version of the new system: avoid letting everyone input their wishes or your project will become horrendously difficult.
Concentrate on the minimum needed to achieve a certain level of functionality that will satisfy your users, then build on that.
There may be some tools to some of the basic stuff, like to upsize to a different database or maybe the look and text boxes of the forms, but converting what sounds like a lot of VBA code, not so sure.
Is this an intranet/local network type of web app or are you putting it out on the internet? Security will become a major difference between this and your Access app.
Make sure they understand Access/VBA so you can maintain the business logic that has been over the life of the Access app.
Convince your friend to stop/slow any development on the Access app to prevent the company from aiming at a moving target. This may not be realistic, but really needs to be considered.
Is there a reason why hosting the app on Windows Terminal Server would not suffice? This means zero changes to the app, no reprogramming cost and no danger of losing crucial business logic. If you use the Citrix extensions, you can run it in a web browser (though I guess that only works with IE -- I've never used them). But the RDP client comes in versions for Mac and Linux as well as Windows, so you can basically support anybody as long as they install the RDP client for their OS.
Yes, it's more installation on the client end, but it's a helluva lot cheaper and easier on the development and avoids the problem of losing important things coded into the Access app.
Of course, supporting large user populations on WTS/Citrix can get expensive and if the Access app is in need of re-engineering, anyway, it can change the balance. But it's something that you should consider. It's really easy to set up WTS, in fact, and provisioning a server for it basically a matter of adding RAM and Internet bandwidth (though RDP is really efficient to begin with).
One key mistake many people make when trying to run an Access app on WTS:
YOU MUST SPLIT THE DATABASE (front with forms/reports/etc., back end with data tables only), and each user must have their own copy of the front end (stored in user profile on the WTS, or in a folder on your WTS server's data partition with appropriate permissions assigned to the user groups authorized to use the app). Tony Toews's front-end updater is very useful in this context, and explicitly engineered to work in a Terminal Server environment.