Using one package with multiple configurations in SSIS - ssis

I know this question has been asked before and I've been doing some research on using a single package with multiple configurations.
An example of this might be: A package that has an FTP control in it.
We might have two or three different FTP sites to connect to, but we only need one package. What I'm looking for is to run that package with three different configurations.
Example configurations:
- Username
- Password
After doing some research on the use of a configuration table, I'm still confused as to whether or not the standard configuration table would work with this scenario or not in SSIS (2008 R2). The article I found that I'm confused by is:
SSIS TABLE DRIVEN PACKAGE CONFIGURATIONS WITH ROW LEVEL FILTERING
If I read the article, some of it makes sense but I'm still not entirely certain that I would need a custom solution to do this or not.
What I'm after is this: I am designing a three tier architecture (three layers of packages).
The first layer will always do file acquisition, file preparation, and translations to a common XML format based on a provider's proprietary format, whatever it may be.
The second layer would take the common format and transform that into a "destination format" and then the third layer would take the destination XML format and insert it into the database.
My thoughts are to possibly use a table to configure the second and third layers and have the top layer using an XML configuration file since it will change based on the provider. Any thoughts on this would be appreciated.

I highly recomend this book and the package configuration framework contained therein:
Microsoft SQL Server 2008 Integration Services: Problem, Design, Solution
ISBN: 978-0-470-52576-0
For me, this book took all the mystery out of package configuration with easy to understand step by step set up and examples. It presents a robust beginning to end solution that I was able to implement in less than a day. Code download link here.
The authors do an excellent job of walking you through package configuration theory and practical setup as part of their larger package management framework.
I would like to second their recommendation to make a "Template Package" from which you build all other packages. The template has all the SSIS PDS goodies already in it so you only need build those once.
One of the most useful features is their approach to package configuration which stores all your configurations in one table in sql server and shows you have to access these at the system, application, and package levels for a layered configuration approach that I find most useful for situations like the one I think you are describing.
Setting up a general configuration as part of the template package once and then tweaking it from there as needed has saved me hundreds of hours in development time.
Not to gush on about this solution, but I found it particularly useful for Username and Password configurations as these are two of the core examples in the book. It can be a bit tricky to simultaneously configure the package with a Username and Password while simultaneously giving these two key pieces of data an adequate level of security.
For example: If you are interested in keeping Username and password secure, you would almost never want to just store these in package scoped variables. This book shows you how to add Username and password as configurations, keep them secure, and how to easily keep them up to date.
Although hopefully when your packages run in production (at least) you are using a Service account where the password never expires (all the more reason to keep those credentials secret and safe) and not your own user credentials.

Related

How To share configurations between Two or more TRAC environments

hope this is a good spot for my question,
for it i SW Related, but not code related.
We, in our company are using TRAC for Issue tracking and management of the Code links,
I am very satisfied by it, and like how it is working.
i have about several environments (1 per project) and every time we change a setting in the Configurations (e.g. Users & Permissions, Severity, Ticket types, etc...) we need to change all of them.
I Use
[inherit]
file=../../../sharedTrac.ini
and delete the shared parts from the file.
for the preferences, but i didn't find a way to share the Configurations.
this is bad for several reason and the head reason is that is "Bugs me !!!" :p
Can TRAC read its configurations from a central definition, and the data from a local DB?
EDIT:
I noticed all these configurations are in the .db file (sqlite file)...
Is there a Ready made tool to copy the configurations from DB to DB ?
or should i go ahead and analyse what should be copied and how ?
You're almost there. Note though, that local settings will always over-rule inherited ones, so you must delete them in your <env>/conf/trac.ini files to make central configuration effective.
Specifically to the part of configuration inside Trac db: No, there is no sync tool yet. Given that there was one for user accounts that is still a beta after years, there's not much interest. You should use the trac-admin command-line tool (as already advised here) or start to directly sync parts the db by means of own (Python) scripts or custom db syncronisation. For a start have a look at the Trac db schema.
You can try to do this through command line. Just call appropriate "trac-admin" command for each instance. Example one-liner to add user profile:
for D in */; do trac-admin $D session add username "Full Name" user#email.com ; done

SSIS - 2008 - Use a single config table for multiple copies of the same package

I am somewhat new to SSIS.
I have to deliver a 'generic' SSIS package, that the client will make multiple copies of, deploy and schedule each copy for different source databases. I have a single SSIS Configuration table in a separate common database. I would like to use this single configuration table for all connections. However the challenge is with the configuration filter. When client makes a copy of my package, it will have the same configuration filter just like others. I would like to give an option to the client to change the configuration filter before deploying, because for this new copy, the source database can be different. I do not find an option to control this.
Is there a way to change the configuration filter from outside the package (without editing the executable .dtsx file)? Or is there a better approach that I can follow? I do not prefer XML configuration files, the primary reason being my packages are deployed onto SQL server.
Any help would be greatly appreciated.
-Shahul
Your preferred solution does not align well with the way that SSIS package configurations are typically used. See Jamie Thomson's answer to a similar question on the MSDN forums.
I have created a package with the same requirements for my company. It loads data from different sources and loads them into different destinations based on individual configurations for the instances. It is used as an internal ETL.
We have adapters that connect to different sources and pass data to a common staging table in XML format and the IETL Package loads this data into different tables depending on a number of different settings etc.
i.e. Multiple SSIS package instances can be executed with different configurations. You are on the right track. It can be achieved using SQL Server to hold configurations and XML Config file to hold the database info that has this configurations. When an instance of the package executes it will load the default values configured with the package, but needs to update all variables to reflect the purpose of the new instance.
I have created a Windows app to configure these instances and they settings in the database to make it really easy for the client or consultant to configure them without actually opening the package.

Configuration Information in DB?

I have lots of stuff in an app.config, and when changes are necessary, an app restart is required. Bad for my 24x7 web server system (it really is 24x7, not even 23x7). I would like to use a good strategy for keeping the config information in a DB table and query/use it as needed. I googled around a bit and am coming up dry. Does anyone have any suggestions before I re-invent the wheel?
Thanks.
I needed exactly this for my recent application, and couldn't use any application server specific techniques as I needed some console apps run on cronjobs to access them too.
I basically made a couple of small tables to create a registry-style configuration database. I have a table of keys (which all have parent-keys so they can be arranged in a tree structure) and a table of values which are attached to keys. All keys and values are named, so my access functions look like this:
openKey("/my_app");
createKey("basic_settings");
openKey("basic_settings");
createValue("log_directory","c:\logs");
getValue("/my_app/basic_settings","log_directory");
The tree structure allows you to logically separate similar data (e.g. you can have a "log_directory" value under several different keys) and avoids having the overly verbose names you find in properties files.
All the values are just strings (varchar2 in the db), so there's some overhead in converting booleans and numbers: but it's only config data, so who cares?
I also create a "settings_changed" value that has a datetime string in it: so any app can quickly tell if it needs to refresh it's configuration (you obviously need to remember to set it when you change anything though).
There may be tools out there to do this kind of thing already: but this was only a days worth of coding and works a treat. I added command line tools to edit and upload/download parts or all of the tree, then made a quick graphical editor in Java Swing.

EDA based SOA and NServiceBus: Why not just use SSIS packages?

I have been investigating NServiceBus. I liked the idea of the pub-sub model, and that the only real coupling of the publisher and subscriber is the semantic of the message. Right now we use SQL replication to sync our data between the databases of different functional areas of our software. I hate this because our private schema is directly coupled to by the subscriber, and it makes it difficult to change on our side. I was thinking it would be great to replace this with NServiceBus publications, but the change seems a little drastic. What about just using something like SSIS? Can I accomplish the same decoupling using SSIS instead of NServiceBus?
SSIS is based on meta-data and therefore will still need to understand the inner schema of all your data sources and sinks. If the underlying meta-data for each source/sink changes your packages will have to change. You also are connecting via MS technologies and are thusly platform coupled. Since you are moving whole sets of data around it sounds like you may not be temporally coupled(system A has to wait for something to respond in system B). It's hard to tell without knowing more about the systems. Lastly SSIS must be aware of the physical locations of all the players in the exchange so you are spatially coupled as well.
In my opinion I don't think you can get to the same place as NSB without developing a lot of the NSB concepts into the packages. This would require using XML messages over the Sql Broker or something to that effect which has already been solved in NSB(see the NSB Contrib project on Github for the Sql Broker transport).

Working with multiple programmers on MS Access

Would you recommend working with multiple programmers on an MS Access application?
One of our MS Access application has grown to the point where the number of changes (bug fixes) and new features can no longer be handled by one programmer in the requested time frame.
We are trying to introduce version control using the undocumented SaveAsText and LoadFromText procedures in VBA to make collaboration on this application possible. Unfortunately we have already run into problems loading modified forms and reports back into Access as a checksum is stored in every form text file.
Before putting time into building an import/export application to compile text files into an Access database, we would like to hear your recommendations.
I think you should avoid this path at all cost, and try and persuade management into redevelopment.
It's a bitter pill to swallow, but this is going to need to be redeveloped sooner or later, and you are just saving them time and money.
We were using Microsoft's own version control add-in for MS Access 2000/2002/2003 for about 5 years now, and I can't remember a single serious problem. Usability of this add-in barely deserves a "B", but it must be much, much more convenient than fiddling with any ad-hoc method involving manual or semi-manual exporting/importing of Access forms, modules, etc.
We were using VSS as a version control system all the time. No problems whatsoever. However, if you have some good reasons to avoid VSS, you may have some options:
The version control add-in that we were using does not require VSS. Theoretically it can be used with any version control system that implements Microsoft Source Code Control Interface (MSCCI). For example, when we had to let somebody work on this project remotely, we used SourceOffsite by SourceGear. Access version control add-in worked with this third-party product fairly well (not without some quirks, but well enough). So, if your favorite version control system complies with MSCCI, you could try to use it.
Now that Microsoft has this Team Foundation thingy, apparently there are other options to be used to integrate MS Access with version control. We did not explore this path, though. This article may be a good start for exploring it.
Hope this would be of some help. :-)
P.S. I am not a big fan of MS Access. In fact, I rather hate it as a platform for a user front-end. If I had a choice, I would run away from it yesterday. :-) However, I must admit that existence of this version control add-in is one of the few things that makes maintenance of our old Access+SQLServer project more or less tolerable. :-))
In addition to what I already said here, I should add that the whole system works very well. The comparison process takes less than 30 minutes a week, for a team of 3 programmers. So let's describe it a little bit.
We have basically 2 versions of our Access program:
The "Developer's version", with all the stuff in it.
We each begin to work with an identical version of our developer's edition. As each one modifies or add parts of the code, we have to run some comparison routine on a regular basis. To do so, we have an object-export routine to a common "comparison" folder. An object (module for example) is exported as a text file (saveAsText command, do not work with tables, see infra), it will be compared to the existing equivalent text files in the folder. If files are identical, there is no file exported. If files are different, the new module is exported with the developer's name as an addition to the file name (if modQueries.txt exists, then modQueries_philippe.txt is created...). Of course if there is no equivalent .txt file in the folder, it will be created at first export.
At the end of the period, we would get in our folder the following files
modQueries.txt, being the first "original", last common version of the module
modQueries_Philippe.txt, with Philippe's modifications
modQueries_Denise.txt, with Denise's modifications
As the module was not modified by other developers, their export did not lead to the creation of a specific modQueries_developersName.txt file
If for any reasons Denise exported many times her module, only the last version is in the comparison folder.
We can then compare (with a "text file" comparer) the different versions and create the "updated" version of the module. We have a screen giving us the number of objects in the comparison folder, number of version for each object, and it is even possible to open the file comparer directly from the developer's interface (We use "File Compare Tool" which has a command-line mode and can then be started directly from Access).
The forms compare issue is quite special, as one of our rules is to have no specific code in our forms (please see here for more details). Forms are then only for display, so usually we do not even compare them. We just make sure that each one of them is updated by only one person (which is quite logical).
The table compare issue (we have local tables) can be only made between mdb files. As we export one text file per module, we also export one mdb file per table. We have a small routine allowing us to identify table differences at the structure level or at the record level.
After each comparison procedure, a subroutine will use all the objects available ini the comparison folder and create a whole new clean mdb file from scratch. This is the new developer's version. Every developer can then copy it on his computer and continue his work.
Developer's versions do not have numbers, but contains last client version number.
The client version, with limited stuff, automatically distributed to users
Each developer has the possibility to build a "client" mdb for final users. This mdb is created from scratch, in a way quite similar to our developer's version, but not all objects are exported. Some specific switches are turned off (special keys, access to code, etc). This mdb holds a version number as a property. The version number is used to build the name of the mdb file.
At production time, this mdb file is zipped and placed in a specific "distribution" folder. Each time a user starts the app, it will automatically check this folder to see if a new version is available. If yes, the client mdb file is updated from the distribution folder, and the app is restarted.
This distribution folder is replicated at night time with our overseas agencies. Users abroad will then be able to install the new version on the following day.
Following the direction provided by Yarik we settled on continuing developing in Access using the Access Add-in Source Code Control, the SVN SCC API Plugin by PushOk Software and Subversion. This stack provides us with seamless Access integration, full-backup and restore and an open version control system.
We had to install a hotfix to Access 2003 and make sure the default database file type matched our database file type to make it work.
We will continue to update this answer with our findings.
Have look at this thread:
How do you use version control with Access development?
Sounds like a terribly painful way to do team development. If you have any options for porting to another environment like VS2008 that would be my recommendation.
There is no easy way to work on Access as a team and even version control might be a bit tricky.