I ran into a problem with some of our packages. The basic layout is the connection strings are stored in a SSIS Configuration table in the database.
I've noticed lately that the server name for my oledb connections seem to remain static. I have seen where I move the package from one environment to another and the validation fails. If I change the password to what is used in the first environment it validates.
Is there some other property besides the connection string that I need to store as well? I'm not using variables or expressions.
Has anyone seen this before? The server version in question is 2008 R2.
This is likely because it's trying to validate using the connection string stored in the package itself and not the one provided by the configuration file. I know of two ways to get around this problem:
Each connection manager and data flow task has a property called DelayValidation. When set to true, that property will prevent SSIS from trying to validate the connections and data flow tasks until after the configuration has changed the connection string. Trouble is, the default value is false, and you have to go through and set it on every single connection manager and data flow task. You'll also need to remember to to flip the setting every time you create a new one.
You can either manually or programmatically change the actual value of the connection string in the SSIS package to match the configuration file before you deploy it. That admittedly does seem to defeat the purpose of having a configuration file in the first place, but it does ensure that it will work. This is the option I often end up taking. Before I move from my test environment to production, I pop open the package in a text editor and do a find/replace of the connection string. I've determined that to be safe for my packages, but, as with any hacky solution like that, your mileage may vary.
Related
I have a production SSIS project in SQL Server 2016 that creates and exports a flat file to another server. The destination server has reached end-of-life and I need to change the destination path to the new server so we can decommission the old server. Can I just edit the package or project in Visual Studio or do I need to recompile (redeploy? republish?)? I have never edited before, only created new projects however that was a few years ago and I am a little rusty.
Alternatively, I could copy the existing job, edit the copy, then run them in parallel first. Then I can disable the old project/package once I am confident the new one works. I'm not having much luck figuring out how to do this either.
Any help would be appreciated. Thanks!
If I'm understanding your question correctly, it sounds like your goal is to simply change the destination location for your report. You would need to update the connection manager for your flat file connection inside of your SSIS package.
You would need to edit your flat file connection manager's connection string (by loading up the connection manager properties and changing the value yourself or by changing the expressions, if it is parameterized). Once you have verified it is now pointing at the right server's location to save the report you are generating, you would rebuild the entire package and redeploy it, which would essentially overwrite the .ispac and .dtsx files on the deployment server with your updates.
Standard caveat of "it depends" but the most common case is that you can solve this through Configuration.
Right click on the Package (or Project) depending on how things are set up. In the "Connection Managers" tab, find the connection manager that corresponds to the flat file output (a strong naming standard helps). I have selected SO_61794511.dtsx and the Name is Flat File Conn... which then allows the right side menu to be populated.
Of interest here is ConnectionString. I am going to directly edit this to change from C:\ssisdata\input\so_61794511.txt to my new path D:\path\here\something\so_newthing.txt
Click OK 2x and the next time the package runs, it will use the configured value.
That's the easiest approach. You could accomplish a similar thing if you edit the job that runs the package to set the value at every execution but this just does it at a global scale.
Where this can go off the rails is if there's a expression applied to the ConnectionString property, e.g. the output file has a dynamic date in the file name. This is why I advocate for exposing Package or Project level parameters of a "base file path" concept. This allows me to change the path from C: (local development) to D: (server deployment) or even to a UNC path \server\share by setting a configuration instead of hard coding a path into the packages themselves.
I hesitate to post because I feel I'm missing something simple, but after reading a ton of articles, I need a second set of eyes because mine are glazed over.
SSIS 2012 Solution.
Goal: Import data from 47 MS Access Databases into corresponding SQL Server databases. This means 47 packages. I want to avoid 47 different sets of Connection Managers and parameterize out what makes sense on the project level so common data is easily configurable.
I added project-level Parameters to hold the SQL Server, SQL UserName, SQL Password, and the Path to the Access Databases. These will be mapped out to the environment to allow changing the destination SQL Server easily as well as the source directory for the Access DBs.
I added two project-level OLEDB Connection Managers - one for SQL 2012 (NativeClient 11.0), one for Access (MSJET 4.0) (they're access 2000 db's). They are set to use an initial database configuration that I hope will be overridden by the individual packages.
For each package, I want to specify the databases being used, so I create a Package Variable (SQLDatabase and AccessDatabase)
In the Connection Managers, I add Expressions to include the Package Variable - e.g. for InitialCatalog on the SQL Connection Manager = #[User::SQLDatabase] and the ServerName in the Access CM combines the project path + the #[User::AccessDatabase]
WHAT WORKS:
I can update the Package Variable values and see them reflected in the Properties of the Connection Managers.
I can go into the Data Flow and add an OLE DB Source, select the SQL Connection Manager, and successfully receive a list of tables from that database.
WHAT FAILS:
When I click Preview, or Columns, after selecting a table I get a big error that reads:
"Error at Package1: The variable 'User::SQLDatabase' was not found in the Variables collection. The variable might not exist in the correct scope."
After clicking OK to that error, then the Preview or Column window shows up with no further problems.
However, after clicking OK on the OLE DB Source Editor, there is a big red X sitting on the box indicating the same error as above.
MY CONFUSION:
If I open the Variables for the package, the variable is sitting right there, scoped to Package1. If I update the value it and save, the Connection Manager updates itself accordingly, so the Expression mapping seems to be okay.
It's as if this is only partially working, and I'm unsure where the flag is to make the rest of it work, or where I failed in my configuration. Can you not set an expression in a project-level CM to use a package-level variable? If that's the case, I'm at a loss on how to correctly set this up to avoid package-level CMs.
Any help appreciated. Please be gentle if I've been an idiot somewhere.
If the connection managers are at project level, then you cannot use package level variables to set their value. If you go to the expression builder for Initial Catalog then you can see that only your project level variables are available. Hence you can use your project level variable to set it instead. If you really want to use the package level variable then you would need to create a connection manager per package which would be a pain.
I'm in the process of creating a suite of packages to import data from our ERP system running on Informix IDS 11.7 into SQL server (2012).
Using SSIS to import this data I've come across an issue. I can access the data in 2 ways, using an ODBC connection and an ADO.NET data source, or using the OLEDB connection and provider.
Using ODBC is about 3 times slower (conservatively!), so naturally I'm keen to move away from that.
However the problem is, when I use OLEDB the data source is reporting the wrong data types.
NVARCHAR data types on the server are being reported as VARCHAR (DT_STR) to SSIS. This causes problem when importing data as any unicode data that come in causes the package to fail. There's no opportunity to do data conversions here, the package is failing when the data hits the data source component. I can set the component to ignore these errors and it will run fine, but with missing data, which isn't acceptable at all.
I've tried setting the DB_LOCALE and CLIENT_LOCAL in setnet32, doesn't have any effect.
When you create the OLEDB data source it complains about the default code page not being able to be found, so you have to set the property to "AlwyasUseDefaultCodePage" to true for that warning to go away, however setting the default code page doesn't seem to actually change it's behavior, it's still trying to bring this data through as VARCHAR even if I set the code page to something like 65001.
So, my question is, how can I either get the OLEDB driver from Informix working correctly, or set/force SSIS to think this data is actually DT_WSTR, not DT_STR?
Thanks!
So, to summarise the steps required to get this working (for posterities sake at least).
When setting up your OLEDB connection you need to add the 2 parameters -
RSASWS=TRUE;
UNICODE=TRUE;
These 2 parameters are NOT shown in the GUI for the connection, at least with the Informix 4.1 driver.
To add these in you need to modify the "ConnectionString" property of the connection, adding these 2 properties to the end of the connection string. Be aware that this property gets overwritten each time you open the connection GUI. You will need to make sure you modify the connection string manually after each time you enter this interface.
The other option for setting the connection string is using variables (or parameters in SSIS on SQL 2012), so any changes that get automatically made to the connection string will be corrected at run time (they actually get corrected at design time when using parameters).
One final caveat I've found with this data source, is appears to be padding out nvarchar's with empty spaces, as if they were nchar's. This could be an issue with my source data though, but something to check if you setting this up, you may need to add a trim step in.
Been struggling with this for quite some time so any help is appreciated. Spent a lot of time looking over MSDN, Google, etc, and still no luck.
I have an SSIS project I intent to always execute from within BID. The project has multiple packages, so I created a Data Source to share amongst all the packages.
In each package, I create a Connection Manager based of that Data Source. Now when I go and edit this Data Source's connection string to point to a different server, the change is NOT reflected in the packages.
Now, MSDN says the opposite. In fact, one of the main purposes of Data Sources is to update connection strings easily from within BID; at least according to MSDN.
And for me its not working. This has got to be the most annoying thing I have seen thus far in BID.
Any one have any suggestions?
UPDATE
It looks like BID does not synchronize a Data Source change with Connection Managers if the data source's authentication is SQL server. If it is Windows authentication, it detects the change fine.
Very interesting indeed.
Just a guess. This kind of behavior happens when you have a "package configuration" configured.
Right click on any blank area of your control flow and select "package configuration". If the "Enable package configurations" check box is selected and you have a configuration on your connection string, it will override any value you had specified manually.
I figured it out for those that are interested. I am not sure if this is a bug in BID or not but I hope this helps someone else struggling with this problem.
All you have to do is simply open up all your packages at one time. This is as simple as highlighting all the packages in the solution, right clicking, and choose "open".
Once all the packages have been opened, go ahead and edit your Data Source's connection properties. BID will automatically apply the new changes to all the open packages that have a connection manager based on that Data source.
Hope this helps someone.
I found I could not change any of the data connection configuration values because there was a variable defined for the connection string. Enabling/disabling package configurations had no effect on the behavior.
To access the variables menu, right click anywhere on the DTSX designer background and select Variables. Make sure you do not have any values in the variables list that are being referenced by your data connection. If you can not find the references anywhere, try fixing the values for the variables that might be affecting it.
You need to change in respective dtsx package. Open it in notepad++ and search for the connection string, modify it and save it. You can see the connection change in solution now.
I have a SSIS package built in Business Intellegience Development Studio which have both source and destination database specified. Now I want to use some variables to parameterize the database connections and run the package in a command line.
I try to replace the database name and sql server instance with my variable #[User::SourceDb]. #[User::SourceHost], but it failed to connect to the database.
Is that possible to paramterize the database and is there anything wrong with my variable useage? Thanks in advance!
There is a nice post here that details one way of doing this.
You will need to use a ConnectionManager and set the ConnectionString property of that from a Configuration Package.
The ConnectionString property is a fully qualified database connection string, like
Data Source=localhost;Initial Catalog=SSISConfig;Provider=SQLNCLI.1;Integrated Security=SSPI;Auto Translate=False;
There are a few gotchas and annoyances with using configuration packages so you may have to fiddle around a bit. I'd be more detailed but it has been about a year since I last worked on an SSIS project.
If you have any specific problems, please come back and comment about what you've hit. I'll try and refire the old memories.
You need to define "expressions" in the connection manager which then substitute into server/database.
You can't use variables directly quite often in SSIS
See "Using Variables in Packages" whcih links to Using Property Expressions in Packages.
And after a quick Bingle: http://sqlrs.blogspot.com/2006/03/using-expression-variables-in-ssis.html
You can right-click on any connection on the project or package and "Parameterize" all the fields from connection, in project or in package parameters. Connection string differs for OLE DB connection to .NET Providers(SqlClient Data Provider) type, so be careful. We have manage to have one centralized place for connection in Project.params