Background: Server1 has source db with 4 tables. Server2 is the destination db with those 4 to 5 tables (and some extra fields). We've a field mapping and some extra processing (i.e. truncate, concate, ..) before moving data from Server1 to Server2. We've 4 independent dtsx packages - one for each and there's an SQL job which calls each dtsx package in a sequence one by one.
Platform: SQL 2005 & SSIS 2005
Now, as you might guess each dtsx package will replicate a "source"
and "destination" db connection in connection manager. When we change
any setting in the remote destination db we've to change connection in
each dtsx - what are the options to persist the remote db connection
info at a single place?
Here're some I know -
Create a parent dtsx that will call each of the 4 dtsx in sequence (same as done in job) and pass connection variable defined in this(parent) package(Example). But this would mean a single dtsx package step in the job which doesn't make it easy to debug in future (at present we can easily identify which step failed - so which table/dtsx has issues)
A bit traditional but works - using database synonyms. We're already using synonyms for some remote tables in a data comparison SP. If I create 4 new synonyms - one for each remote table. I can refer those tables directly in dtsx by their synonym. Later in case of any change - I need to alter and re-run the script that will drop and re-create synonyms (I can do that remotely as well!)
I do simple dtsx designing and programming so pardon my knowledge and let me know the best approach.
What you are looking for is Package Configurations.
Basically, a configuration value is just a stored key/value pair of some property in SSIS (for example, your connection manager's connection string.)
You can store it in a number of formats (see the link) - I personally use XML files since I have a support team who handles deployment and they "get" files.
Then each package just references that XML file in its own configurations. When the package is run it will use the value of the connection string for that connection, so each package will use whatever's in the XML file (or whatever configuration format you choose.)
Basic steps:
Create connection in package.
Under SSIS -> Package Configurations, choose "Enable package configuration", choose "Add ..", choose the format you want, select the connection's connection string property, save the configuration.
In the other 3 packages, go to the same area, choose "Add.." and this time browse to the configuration you created, it'll ask you to Overwrite the values in that configuration with what's in the package or Reuse Existing (meaning keep the value you already created) - choose Reuse Existing. Name it, save it.
Check the link out above for more info, it's what you want.
Related
I have a production SSIS project in SQL Server 2016 that creates and exports a flat file to another server. The destination server has reached end-of-life and I need to change the destination path to the new server so we can decommission the old server. Can I just edit the package or project in Visual Studio or do I need to recompile (redeploy? republish?)? I have never edited before, only created new projects however that was a few years ago and I am a little rusty.
Alternatively, I could copy the existing job, edit the copy, then run them in parallel first. Then I can disable the old project/package once I am confident the new one works. I'm not having much luck figuring out how to do this either.
Any help would be appreciated. Thanks!
If I'm understanding your question correctly, it sounds like your goal is to simply change the destination location for your report. You would need to update the connection manager for your flat file connection inside of your SSIS package.
You would need to edit your flat file connection manager's connection string (by loading up the connection manager properties and changing the value yourself or by changing the expressions, if it is parameterized). Once you have verified it is now pointing at the right server's location to save the report you are generating, you would rebuild the entire package and redeploy it, which would essentially overwrite the .ispac and .dtsx files on the deployment server with your updates.
Standard caveat of "it depends" but the most common case is that you can solve this through Configuration.
Right click on the Package (or Project) depending on how things are set up. In the "Connection Managers" tab, find the connection manager that corresponds to the flat file output (a strong naming standard helps). I have selected SO_61794511.dtsx and the Name is Flat File Conn... which then allows the right side menu to be populated.
Of interest here is ConnectionString. I am going to directly edit this to change from C:\ssisdata\input\so_61794511.txt to my new path D:\path\here\something\so_newthing.txt
Click OK 2x and the next time the package runs, it will use the configured value.
That's the easiest approach. You could accomplish a similar thing if you edit the job that runs the package to set the value at every execution but this just does it at a global scale.
Where this can go off the rails is if there's a expression applied to the ConnectionString property, e.g. the output file has a dynamic date in the file name. This is why I advocate for exposing Package or Project level parameters of a "base file path" concept. This allows me to change the path from C: (local development) to D: (server deployment) or even to a UNC path \server\share by setting a configuration instead of hard coding a path into the packages themselves.
I've inherited some SSIS Packages that need to be modified for a SQL 2008 R2 to SQL 2012 migration. Unfortunately, the "configuration" was done rather poorly.
We have a global XML Config file for all SSIS Packages, and just a few variables in the XML Config file--Server Name, ODS Server Name, and Environment (Development, Integration, PreProd, and Production).
To "configure", they write code in the SSIS Package: If 'Development' Then ... Else If 'Integration' Then ...
In order to change the "configuration", one has to change the code in the Package.
I have unsuccessfully tried to negotiate a change, but no one is budging so the one XML Config file remains.
If I can add a second XML Config file, with my Package specific variables that need to be configured in each Environment, that's what I will do. However, I have not found a way to do this. Is is possible?
My second choice is put variables in a SQL Server table.
I think your best bet is to replace all of your script tasks with OLE DB Commands to call a stored procedure which would take the following parameters: Environment, ConnectionName, PackageName, and ServerType(standard or ODS). The output of this stored procedure would be the ServerName, which would be assigned to a variable. This variable could be then be used to set the server name for the connection. The stored procedure could depend on table(s) or global XML file(s). I would suggest tables. Either way, the logic in the package would be minimal and would allow you to implement in whatever way you see fit.
I hesitate to post because I feel I'm missing something simple, but after reading a ton of articles, I need a second set of eyes because mine are glazed over.
SSIS 2012 Solution.
Goal: Import data from 47 MS Access Databases into corresponding SQL Server databases. This means 47 packages. I want to avoid 47 different sets of Connection Managers and parameterize out what makes sense on the project level so common data is easily configurable.
I added project-level Parameters to hold the SQL Server, SQL UserName, SQL Password, and the Path to the Access Databases. These will be mapped out to the environment to allow changing the destination SQL Server easily as well as the source directory for the Access DBs.
I added two project-level OLEDB Connection Managers - one for SQL 2012 (NativeClient 11.0), one for Access (MSJET 4.0) (they're access 2000 db's). They are set to use an initial database configuration that I hope will be overridden by the individual packages.
For each package, I want to specify the databases being used, so I create a Package Variable (SQLDatabase and AccessDatabase)
In the Connection Managers, I add Expressions to include the Package Variable - e.g. for InitialCatalog on the SQL Connection Manager = #[User::SQLDatabase] and the ServerName in the Access CM combines the project path + the #[User::AccessDatabase]
WHAT WORKS:
I can update the Package Variable values and see them reflected in the Properties of the Connection Managers.
I can go into the Data Flow and add an OLE DB Source, select the SQL Connection Manager, and successfully receive a list of tables from that database.
WHAT FAILS:
When I click Preview, or Columns, after selecting a table I get a big error that reads:
"Error at Package1: The variable 'User::SQLDatabase' was not found in the Variables collection. The variable might not exist in the correct scope."
After clicking OK to that error, then the Preview or Column window shows up with no further problems.
However, after clicking OK on the OLE DB Source Editor, there is a big red X sitting on the box indicating the same error as above.
MY CONFUSION:
If I open the Variables for the package, the variable is sitting right there, scoped to Package1. If I update the value it and save, the Connection Manager updates itself accordingly, so the Expression mapping seems to be okay.
It's as if this is only partially working, and I'm unsure where the flag is to make the rest of it work, or where I failed in my configuration. Can you not set an expression in a project-level CM to use a package-level variable? If that's the case, I'm at a loss on how to correctly set this up to avoid package-level CMs.
Any help appreciated. Please be gentle if I've been an idiot somewhere.
If the connection managers are at project level, then you cannot use package level variables to set their value. If you go to the expression builder for Initial Catalog then you can see that only your project level variables are available. Hence you can use your project level variable to set it instead. If you really want to use the package level variable then you would need to create a connection manager per package which would be a pain.
I ran into a problem with some of our packages. The basic layout is the connection strings are stored in a SSIS Configuration table in the database.
I've noticed lately that the server name for my oledb connections seem to remain static. I have seen where I move the package from one environment to another and the validation fails. If I change the password to what is used in the first environment it validates.
Is there some other property besides the connection string that I need to store as well? I'm not using variables or expressions.
Has anyone seen this before? The server version in question is 2008 R2.
This is likely because it's trying to validate using the connection string stored in the package itself and not the one provided by the configuration file. I know of two ways to get around this problem:
Each connection manager and data flow task has a property called DelayValidation. When set to true, that property will prevent SSIS from trying to validate the connections and data flow tasks until after the configuration has changed the connection string. Trouble is, the default value is false, and you have to go through and set it on every single connection manager and data flow task. You'll also need to remember to to flip the setting every time you create a new one.
You can either manually or programmatically change the actual value of the connection string in the SSIS package to match the configuration file before you deploy it. That admittedly does seem to defeat the purpose of having a configuration file in the first place, but it does ensure that it will work. This is the option I often end up taking. Before I move from my test environment to production, I pop open the package in a text editor and do a find/replace of the connection string. I've determined that to be safe for my packages, but, as with any hacky solution like that, your mileage may vary.
I have a SSIS package built in Business Intellegience Development Studio which have both source and destination database specified. Now I want to use some variables to parameterize the database connections and run the package in a command line.
I try to replace the database name and sql server instance with my variable #[User::SourceDb]. #[User::SourceHost], but it failed to connect to the database.
Is that possible to paramterize the database and is there anything wrong with my variable useage? Thanks in advance!
There is a nice post here that details one way of doing this.
You will need to use a ConnectionManager and set the ConnectionString property of that from a Configuration Package.
The ConnectionString property is a fully qualified database connection string, like
Data Source=localhost;Initial Catalog=SSISConfig;Provider=SQLNCLI.1;Integrated Security=SSPI;Auto Translate=False;
There are a few gotchas and annoyances with using configuration packages so you may have to fiddle around a bit. I'd be more detailed but it has been about a year since I last worked on an SSIS project.
If you have any specific problems, please come back and comment about what you've hit. I'll try and refire the old memories.
You need to define "expressions" in the connection manager which then substitute into server/database.
You can't use variables directly quite often in SSIS
See "Using Variables in Packages" whcih links to Using Property Expressions in Packages.
And after a quick Bingle: http://sqlrs.blogspot.com/2006/03/using-expression-variables-in-ssis.html
You can right-click on any connection on the project or package and "Parameterize" all the fields from connection, in project or in package parameters. Connection string differs for OLE DB connection to .NET Providers(SqlClient Data Provider) type, so be careful. We have manage to have one centralized place for connection in Project.params