I've been researching this question over the last few days as I prepare to deliver the first of 3 phases with my first system using a split database. I would like your advice as I haven't found enough info to make a full decision yet.
At the moment I'm working in dev on an unsplit database. When I split it in live I'll take a copy of both parts but what do I do with them for phase 2?
I'm thinking that I'll now make them my dev version after relinking the tables (as I've effectively moved the back end) which would then mean that I no longer work with the unsplit database. Is that the right approach?
When it comes to putting phase 2 live I don't think I have any choice other than manually applying table updates to the live back end (once backed up). For the live front end, do I just replace it with my dev front end and then relink the tables or do I export the changes into the live front end? I guess I could do either depending on the number of objects that are changing/new. Is that right? I'll then take copies and make them my dev versions for phase 3.
Finally in dev I have form and report templates and test forms, reports and queries which are not needed in live so do I remove them for each deployment and then add them all back in to the new dev front end or just put them live? Normally I would take them out but there are a lot of them and I don't know of any quick way to add them back in so what do you do?
Primarily my questions are asked from a risk point of view - what steps best reduce the risk of messing things up in live.
Update:
For those of you who are looking for answers on this, in addition to the fine responses below I have since found the following that might also help:
How to Continuously Develop and Deploy an Access 2010 Database Application
At the moment I'm working in dev on an unsplit database. W
Don't, a VERY bad idea. How this works?
Well, for sure at the start, you are building LOTS of new tables, changing relationships, and building tables at a high rate of development.
so, at this point in time, you can develop un-split.
You will then find after some time the rate (and need) to create new tables, and change the so called "database schema" calms down to a dull roar.
At this point your still developing away - not yet deployed.
So, somewhere around this point? You want to split. You REALLY want to do this.
There is a boatload of reasons for this, but several are:
while a split vs non split is "very similar", they are not the same!
Thus, you can't and don't want to develop code that NOT really tested
as to HOW the code will run in the real world
So, many issues can change or crop up during development that is DIFFERENT when run split. So some commands (such as seek()) don't work, and a few other issues can crop up. You don't want to develop for a whole week, then split and now find 20 hard to fix bugs in your code. So, by developing as split as soon as possible, then ANY and ALL issues that come up will be seen as you develop along, and thus can fix, see, and deal with such issues at THAT POINT in time. Much worse is to write a bunch of code, get ready to deploy, and then find new bugs.
Next up:
Having a split system is great, since say a customer might let you remote into their system. You can pull down a copy of their data, re-link your tables to point from your "test data" to real live production data.
Or, say your developing on site. You might want to test some dangerous delete code, or code that modifies the data. So, you can't risk working on production data, so now you re-link and point to your test back end. So, this setup allows you to test code, but MORE important test on a copy of the database with great ease.
And it also allows you to develop off site. You can take the latest front end for their system, maybe get their latest data file, maybe not. but you can now with ease simple change the database that your applcation runs with.
The other big issue? Say your working on site, and have a test database of theirs on the network folder. You write some code, test for a new report. You find it runs SLOW AS A TURTLE. You check your code, maybe add a few indexes, and boom! - your report now runs great.
If you test un-split, then a boatload of performance issues can crop up, but NOT SEEN during the development cycle. Once again, you don't want to develop for weeks or whatever, split, and NOW find a whole bunch of forms and code runs REALLY slow.
So, the goal, the idea here?
You want to get split as SOON as possible?
How soon?
Well, this is one of those things that only you can know!!!
As I stated, at start of development, sure, start out un-split.
Once the table designs are quite solid, then you can split. You then ALWAYS develop as split (and the above list of reasons why is the VERY short list - there are many more reasons).
Now, the problem of course with split? Say you want to add a new column to a table?
Well, it is MORE work, since now ANY AND ALL changes to the data schema are done in the back end. So, you have to close down the front end (FE), open the back end, and now use the table designer to add that one column. (or maybe change or setup a new relationship between some tables. Or maybe add a new table).
This is a "bit more" work, since now you close down the BE, and open the FE, and now you MUST re-link tables. And if you added new tables to the BE, then you have to add that/those new table links.
Because this "dance" is extra work, that is why you wait as long as possible to split. As I stated, you "just know" when that time has arrived to split. (when table and schema tables changes settles down to a low rate of change). Since the rate of change is now low in regards to table changes, then it not much work nor pain to have to do the above every time you want to change the table structures.
In fact, think of any program you buy? It has a applcation part, and then a data file part. In effect, once you split, you have the same two parts, and in fact in some applications I written, they are allowed to use "different" back ends - not unlike any other applcation in which you launch the applcation, and THEN choose the datafile to work with.
So, what about developing off site? Well, that can be REALLY difficult, since you have your own copy of the FE and the BE.
If you HAVE TO make changes to the BE?
I open up a work document. And if I add a new column to say tblCusotmers?
Then I enter this:
Add new column TaxRate, Currency, to table tblCustomers
So, you build up a "log" of changes. Now, when you travel to the customer site, and want to roll out and deploy the new FE? Well, you have to FIRST open up their BE, and make the above changes to their production BE database.
Now in some cases, where is was not possible for me to be on site? (in fact, I had a automatic update system to automatic roll out a new version of my software - and it would automatic down load from the internet. In this case? I had to write code in the FE on startup that would use VBA code to MAKE the changes to the data tables. This can be REALLY hard to do, but is possible. I just recommend the plane jane word document, and you keep track of your changes.
So, the above is quite much how this development process works.
Since you will have to re-link the tables? Then near everyone has googled for a VBA table re-link routines. You want that, since having such code is MUCH easier then say using the linked table manager each time. And we even often have a table in the FE that saves the BE server location, and on startup will check, and if the location of the files don't match, we launch the re-link code.
that way, you can deploy the applcation to different sites, and have it automatic re-link. Another way is to have a simple text file in the same location as the FE on each computer, and on startup read the text file with the BE location - and re-link if required.
So, the typical process to role out a new FE (which is placed on each work station - do NOT break this rule!!!).
So, I point/relink my front end to the production BE. I then compile down to a accDE, and then deploy that new compiled FE to all the work stations. In fact, I have some code in VBA at start up that compares a version number, and if the version number is lower, then the VBA code will copy down the next FE sitting in a folder.
This might not be a big deal if you have 2-4 users. But, if you have two sites, and each has 35 users, then you want to figure out a automated approach.
However, do not prolong jumping over to the split development cycle, as you really for all practical purposes MUST develop in a split environment. So, for the first part, you can develop un-split. But, once you split - that's it, and from that point on-wards, you are to develop as split. There are boatloads of benefits, but it also really quite much standard approach from a developer point of view.
So, you have to master the linked table manager rite quick, and then VERY much consider adding some re-link code, since you want with great ease to point to a different back end - including at deployment time.
so, as a general rule, you should think of your FE like a .exe program, for a new version roll out, yes, you copy (over write) their existing FE's on each work station. And as noted, in most cases, it should be a compiled accDE, and not a un-compiled accDB for the FE.
For reducing risk:
you should have your development version, a test version for live and the live version.
You are developing on develop
Customer tests the changes on test (with test data)
after that you move to live
For the move from develop to test I create an update/migration script.
In this script are all alterations included that needs to be done on the back-end.
I use the script to create the test version and with this I can check if it is working properly.
In case there are database changes that I can't reflect in my script (either insufficient skill or restrictions from db) I add them to my checklist.
I am using version control to see changes during development and to import modules, queries etc. to the new version.
Updating the front end is done via Import of the latest version (without not needed forms / reports).
I have an Access 2003 database using MS-JET linked tables (that is, there are two *.mdb files). The overall task is to move to SQL server. However the existing databases have multiple fields with spaces and other characters that upset both of the Access migration wizards. So the intermediate task is to make a version of the existing mdb's with updated field names.
"Simply" changing field names in the UI mdb file doesn't work, because of the linked tables being read-only in that mdb. Updating the Data mdb file field names first doesn't work because the UI mdb then throws a wobbly on startup and Access gives up.
I'm looking for suggestions, but options I can see are:
re-integrate the Data mdb back into the UI one, do the updates, then re-export the data. Seems very risky to me (system is live, don't want to play with data any more than strictly necessary).
I've looked at several answers here relating to changing ODBC details with VBA code, which is interesting, and I can see how I might be able to programmatically edit the linked tables' names. Is this going to work? Is there a better way?
Unlink all linked tables, then edit in UI mdb the remaining forms, reports, queries (but would that work??) and edit the Data mdb fields, and finally relink everything. Will it fall apart?? ... seems likely.
Any suggestions?
Ruth
I go through this process every time I take over an existing Access application -- I have to bring it up to my own standards for naming conventions before I do anything significant with it. Recently I built a quick-and-dirty utility to rename fields. It was made for me and has very little error recovery, and a UI that is ugly, but it might be faster than doing it yourself.
See what track name autocorrect offers. How Name AutoCorrect Works in Microsoft Access
Make a copy of your UI MDB, delete the linked tables, then import them (as tables rather than links) from the Data MDB. Once you enable autocorrect, Access will attempt to propagate your field name changes wherever they are used (in forms for example). Unlikely it will catch 100% of the necessary changes, but it should resolve a big chunk of them. You would then need to manually track down and fix the rest.
Once you have everything fixed, you could use that MDB version to "upsize" your tables to SQL Server. However, one huge complication is if the original database is "live" while you're making changes to the new version. If you absolutely can't take the database out of service in the interim, you could discard the data from the new tables, then use "append" queries which alias the old to new field names to pull in the latest data.
INSERT INTO NewTable (emp_id, another_field)
SELECT
[emp ID],
[another field]
FROM OldTable IN 'C:\somefolder\Data.mdb';
Finally, a warning about autocorrect: do not leave it enabled in the production version of your database because strange things can happen. The safest approach IMO is to turn it on, complete your object name changes, then turn it back off again.
You might find Rick Fisher's Find and Replace tool helpful if you need to propagate name changes through the project. I've relied on it for years (though I don't do much Access development lately): http://www.rickworld.com/products.html
I think this tool will change the names of tables and fields for you but I am not sure. I mostly used it for finding references to tables and queries in other queries, form and report properties, and VBA code.
How can I track changes to a development database and apply those changes to a production database (SQL Server 2008)?
I keep a local copy of a database on my development server, and as I'm adding new features, I may add new tables or change field and table names in the database. What's a good way to track such changes and then apply them to the main database?
Is there some way to do a "diff"-like operation between two databases and merge definition changes?
I considered merge-replication, but I'm not sure how well than handles schema changes. For example, here: http://technet.microsoft.com/en-us/library/ms151870.aspx it mentions that I basically cannot use SSMS to make definition changes, because it drops and recreates tables, which is not allowed for published objects.
A smart piece of software could compare column counts, types, positions, and apply other fuzzy matching/logical deduction methods to figure out that a table was renamed or a new table was added or a column name changed, after which it could present the differences to the user for confirmation and automatic application.
Does anything like what I've described above exist, or am I stuck remembering to save DDL statements in SSMS and running them manually in the production database?
Maybe you need a migration tool like (for example) FluentMigrator, which helps you track database changes in source code.
Here is a tutorial from the original author of Fluent Migrator, explaining what Fluent Migrator is, why you might need it and how it works.
Another alternative would be what you already mentioned:
A smart piece of software could compare column counts, types,
positions, and apply other fuzzy matching/logical deduction methods to
figure out that a table was renamed or a new table was added or a
column name changed, after which it could present the differences to
the user for confirmation and automatic application.
I never tried it myself, but I've seen lots of recommendations for Redgate SQL Compare (which apparently does exactly what you asked for) here at Stack Overflow.
I have lots of stuff in an app.config, and when changes are necessary, an app restart is required. Bad for my 24x7 web server system (it really is 24x7, not even 23x7). I would like to use a good strategy for keeping the config information in a DB table and query/use it as needed. I googled around a bit and am coming up dry. Does anyone have any suggestions before I re-invent the wheel?
Thanks.
I needed exactly this for my recent application, and couldn't use any application server specific techniques as I needed some console apps run on cronjobs to access them too.
I basically made a couple of small tables to create a registry-style configuration database. I have a table of keys (which all have parent-keys so they can be arranged in a tree structure) and a table of values which are attached to keys. All keys and values are named, so my access functions look like this:
openKey("/my_app");
createKey("basic_settings");
openKey("basic_settings");
createValue("log_directory","c:\logs");
getValue("/my_app/basic_settings","log_directory");
The tree structure allows you to logically separate similar data (e.g. you can have a "log_directory" value under several different keys) and avoids having the overly verbose names you find in properties files.
All the values are just strings (varchar2 in the db), so there's some overhead in converting booleans and numbers: but it's only config data, so who cares?
I also create a "settings_changed" value that has a datetime string in it: so any app can quickly tell if it needs to refresh it's configuration (you obviously need to remember to set it when you change anything though).
There may be tools out there to do this kind of thing already: but this was only a days worth of coding and works a treat. I added command line tools to edit and upload/download parts or all of the tree, then made a quick graphical editor in Java Swing.
I have a small amount of experience using SVN on my development projects, and I have just as little experience with relational databases. I know the basic concepts like tables, and SQL statements, but I'm far from being an expert.
What I'd like to know is if there are any generic version control type systems like SVN, but that work with a database rather than files. I would like the same kind of features you get with SVN like the ability to create branches, create tags, and merge branches together. Rather than a revision number being associated to a version of a file repository it would be associated with a version of the database.
Are their any generic solutions available that can add this kind of functionality independent of the actual database schema? I'd be interested in solutions that work with MySQL or MS SQL Server.
I should also clarify that I'm trying to version control the data not the schema. I would expect the schema to remain constant. So really it seems like I want a way to create a log of all the INSERT, UPDATE, and DELETE requests sent the the database between each version of the data. That way any version could be recreated by resending all the SQL statements that have been saved up to the desired version.
You can script all your DDL, stored procedures and such to regular text files.
Then you can simply use SVN for database versioning.
I've never found a solution that works as well as Subversion, but here's a few things I've done that have helped:
Make scripts that will create the schema and populate any initial data. Then make an update script for each change after that. It's a fairly manual process, but it works. There's extra things that help like storing the current version number in a table in the db and making sure that the scripts are idempotent.
Store the full development db in Subversion. This doesn't usually work out too well for me if there is a lot of data or it is frequently changed. But in some projects is could work.
I keep and maintain create scripts in my version control system.
There are two things I can think of:
http://www.liquibase.org/ - provides a way of generally managing database changes. Creates files that get committed into source control, and it helps manage changes across different development databases, etc.
http://www.viget.com/extend/backup-your-database-in-git/ - this describes a strategy for backing up a database into source control, but the same strategy can be used just on the schema. In this scheme, the database would be in a separate area from your main code. (This can be used with other source control systems too.)