I hesitate to post because I feel I'm missing something simple, but after reading a ton of articles, I need a second set of eyes because mine are glazed over.
SSIS 2012 Solution.
Goal: Import data from 47 MS Access Databases into corresponding SQL Server databases. This means 47 packages. I want to avoid 47 different sets of Connection Managers and parameterize out what makes sense on the project level so common data is easily configurable.
I added project-level Parameters to hold the SQL Server, SQL UserName, SQL Password, and the Path to the Access Databases. These will be mapped out to the environment to allow changing the destination SQL Server easily as well as the source directory for the Access DBs.
I added two project-level OLEDB Connection Managers - one for SQL 2012 (NativeClient 11.0), one for Access (MSJET 4.0) (they're access 2000 db's). They are set to use an initial database configuration that I hope will be overridden by the individual packages.
For each package, I want to specify the databases being used, so I create a Package Variable (SQLDatabase and AccessDatabase)
In the Connection Managers, I add Expressions to include the Package Variable - e.g. for InitialCatalog on the SQL Connection Manager = #[User::SQLDatabase] and the ServerName in the Access CM combines the project path + the #[User::AccessDatabase]
WHAT WORKS:
I can update the Package Variable values and see them reflected in the Properties of the Connection Managers.
I can go into the Data Flow and add an OLE DB Source, select the SQL Connection Manager, and successfully receive a list of tables from that database.
WHAT FAILS:
When I click Preview, or Columns, after selecting a table I get a big error that reads:
"Error at Package1: The variable 'User::SQLDatabase' was not found in the Variables collection. The variable might not exist in the correct scope."
After clicking OK to that error, then the Preview or Column window shows up with no further problems.
However, after clicking OK on the OLE DB Source Editor, there is a big red X sitting on the box indicating the same error as above.
MY CONFUSION:
If I open the Variables for the package, the variable is sitting right there, scoped to Package1. If I update the value it and save, the Connection Manager updates itself accordingly, so the Expression mapping seems to be okay.
It's as if this is only partially working, and I'm unsure where the flag is to make the rest of it work, or where I failed in my configuration. Can you not set an expression in a project-level CM to use a package-level variable? If that's the case, I'm at a loss on how to correctly set this up to avoid package-level CMs.
Any help appreciated. Please be gentle if I've been an idiot somewhere.
If the connection managers are at project level, then you cannot use package level variables to set their value. If you go to the expression builder for Initial Catalog then you can see that only your project level variables are available. Hence you can use your project level variable to set it instead. If you really want to use the package level variable then you would need to create a connection manager per package which would be a pain.
Related
I'm in the process of creating a suite of packages to import data from our ERP system running on Informix IDS 11.7 into SQL server (2012).
Using SSIS to import this data I've come across an issue. I can access the data in 2 ways, using an ODBC connection and an ADO.NET data source, or using the OLEDB connection and provider.
Using ODBC is about 3 times slower (conservatively!), so naturally I'm keen to move away from that.
However the problem is, when I use OLEDB the data source is reporting the wrong data types.
NVARCHAR data types on the server are being reported as VARCHAR (DT_STR) to SSIS. This causes problem when importing data as any unicode data that come in causes the package to fail. There's no opportunity to do data conversions here, the package is failing when the data hits the data source component. I can set the component to ignore these errors and it will run fine, but with missing data, which isn't acceptable at all.
I've tried setting the DB_LOCALE and CLIENT_LOCAL in setnet32, doesn't have any effect.
When you create the OLEDB data source it complains about the default code page not being able to be found, so you have to set the property to "AlwyasUseDefaultCodePage" to true for that warning to go away, however setting the default code page doesn't seem to actually change it's behavior, it's still trying to bring this data through as VARCHAR even if I set the code page to something like 65001.
So, my question is, how can I either get the OLEDB driver from Informix working correctly, or set/force SSIS to think this data is actually DT_WSTR, not DT_STR?
Thanks!
So, to summarise the steps required to get this working (for posterities sake at least).
When setting up your OLEDB connection you need to add the 2 parameters -
RSASWS=TRUE;
UNICODE=TRUE;
These 2 parameters are NOT shown in the GUI for the connection, at least with the Informix 4.1 driver.
To add these in you need to modify the "ConnectionString" property of the connection, adding these 2 properties to the end of the connection string. Be aware that this property gets overwritten each time you open the connection GUI. You will need to make sure you modify the connection string manually after each time you enter this interface.
The other option for setting the connection string is using variables (or parameters in SSIS on SQL 2012), so any changes that get automatically made to the connection string will be corrected at run time (they actually get corrected at design time when using parameters).
One final caveat I've found with this data source, is appears to be padding out nvarchar's with empty spaces, as if they were nchar's. This could be an issue with my source data though, but something to check if you setting this up, you may need to add a trim step in.
I'm having this exact same problem I've been looking aroung but this is the only place I've seen the same issue and it is not resolved.
Does anyone knows what the problem might be?
I checked in Visual Studio and my SSIS version is 11.0.2100.60 (not a trial, not a beta).
EDIT: These are the steps I'm taking and the issue
First I choose a Connection Manager, right-click, select properties and click in Expressions option
Then in Property Expression Builder choose Connection String property and click in Expression option
Finally, in the Expression Builder dialog there is no option for variables, in every page I've read says that there should be a Variables node in there
Am I missing something?
SSIS 2012 has introduced the concept of Project level connection managers. What I see on the referenced post on the MSDN forums it the user has created a project level flat file connection manager and is unable to configure it with a local variable. Assuming that is the problem, my answer follows.
An SSIS project is generally more than one package. To simplify lives, the SSIS team now allows for the sharing of common resources across projects, connection managers being one of those resources.
Logically, if a thing is shared across a project, how can something that only exists in one file configure that resource? That configuration change would only work when Package1 is executing. When Package2 fires, unless the same variable and same expression was applied to the shared resource, you would experience different outcomes. That'd be a maintenance nightmare, which you might already experience if you don't have strong configuration practices.
If I create a Flat File Connection Manager at the project level, I can only reference variables that are also at the project level. Except there are not variables at the project level. Instead, they are called Parameters.
To that end, I created a Parameter called SomeProjectParameter
I then created a package, Package1.dtsx, and added 2 Flat File Connection Managers: FlatFileConnectionManagerLocal and FlatFileConnectionManagerProject
Instead that package, I also created a variable called SomeLocalVariable.
This screenshot shows me applying an expression to the ConnectionString property of FlatFileConnectionManagerLocal. There you can see that both the package variable, SomeLocalVariable is available as well as SomeProjectParameter
Now, if I try to apply an expression to the project's connection manager, you will only have project parameters available to you.
It's interesting to note that you can't apply an expression to a project level Connection Manager outside of the context of an SSIS package. There's simply no editor available to you until you have an open SSIS package. But, once applied, all the packages in the project will be similarly configured.
Quirk of the IDE I suppose. Also, don't be alarmed by the lack of color in these screenshots, I'm running with the 2012 version of SSDT.
i had the same problem and it was because the flat file source was set as a project source so i had to convert it into a package connection.
Right-click on the source and choose convert to package connection
Maybe this will help.
The Microsoft SSIS tutorial have these confusing statements:
In the Connection Managers pane, right-click Sample Flat File Source Data, and select Properties.
In the Properties window make sure the PackagePath starts with \Package.Connections. If not, in the
Connection Managers pane, right-click Sample Flat File Source Data, and select Convert to Package Connection.
\Package.Connections was initially displayed in the properties window, although I had not yet converted to Package Connection.
After reading this thread, I right clicked on Sample Flat File Source Data and then Convert to Package Connection and was able to complete the lesson. Thanks for the answers!
Background: Server1 has source db with 4 tables. Server2 is the destination db with those 4 to 5 tables (and some extra fields). We've a field mapping and some extra processing (i.e. truncate, concate, ..) before moving data from Server1 to Server2. We've 4 independent dtsx packages - one for each and there's an SQL job which calls each dtsx package in a sequence one by one.
Platform: SQL 2005 & SSIS 2005
Now, as you might guess each dtsx package will replicate a "source"
and "destination" db connection in connection manager. When we change
any setting in the remote destination db we've to change connection in
each dtsx - what are the options to persist the remote db connection
info at a single place?
Here're some I know -
Create a parent dtsx that will call each of the 4 dtsx in sequence (same as done in job) and pass connection variable defined in this(parent) package(Example). But this would mean a single dtsx package step in the job which doesn't make it easy to debug in future (at present we can easily identify which step failed - so which table/dtsx has issues)
A bit traditional but works - using database synonyms. We're already using synonyms for some remote tables in a data comparison SP. If I create 4 new synonyms - one for each remote table. I can refer those tables directly in dtsx by their synonym. Later in case of any change - I need to alter and re-run the script that will drop and re-create synonyms (I can do that remotely as well!)
I do simple dtsx designing and programming so pardon my knowledge and let me know the best approach.
What you are looking for is Package Configurations.
Basically, a configuration value is just a stored key/value pair of some property in SSIS (for example, your connection manager's connection string.)
You can store it in a number of formats (see the link) - I personally use XML files since I have a support team who handles deployment and they "get" files.
Then each package just references that XML file in its own configurations. When the package is run it will use the value of the connection string for that connection, so each package will use whatever's in the XML file (or whatever configuration format you choose.)
Basic steps:
Create connection in package.
Under SSIS -> Package Configurations, choose "Enable package configuration", choose "Add ..", choose the format you want, select the connection's connection string property, save the configuration.
In the other 3 packages, go to the same area, choose "Add.." and this time browse to the configuration you created, it'll ask you to Overwrite the values in that configuration with what's in the package or Reuse Existing (meaning keep the value you already created) - choose Reuse Existing. Name it, save it.
Check the link out above for more info, it's what you want.
I ran into a problem with some of our packages. The basic layout is the connection strings are stored in a SSIS Configuration table in the database.
I've noticed lately that the server name for my oledb connections seem to remain static. I have seen where I move the package from one environment to another and the validation fails. If I change the password to what is used in the first environment it validates.
Is there some other property besides the connection string that I need to store as well? I'm not using variables or expressions.
Has anyone seen this before? The server version in question is 2008 R2.
This is likely because it's trying to validate using the connection string stored in the package itself and not the one provided by the configuration file. I know of two ways to get around this problem:
Each connection manager and data flow task has a property called DelayValidation. When set to true, that property will prevent SSIS from trying to validate the connections and data flow tasks until after the configuration has changed the connection string. Trouble is, the default value is false, and you have to go through and set it on every single connection manager and data flow task. You'll also need to remember to to flip the setting every time you create a new one.
You can either manually or programmatically change the actual value of the connection string in the SSIS package to match the configuration file before you deploy it. That admittedly does seem to defeat the purpose of having a configuration file in the first place, but it does ensure that it will work. This is the option I often end up taking. Before I move from my test environment to production, I pop open the package in a text editor and do a find/replace of the connection string. I've determined that to be safe for my packages, but, as with any hacky solution like that, your mileage may vary.
When I open up the solution that contains SSIS packages created by a colleague, I get this awkward error that tells me nothing about what I'm supposed to do to fix it.
He left instructions to take all the "variables" out of the connection string in the dtsx file manually before opening up the solution. I have done that, now when try to view the package in the designer I just get an image of a red x and this message.
EDIT: You cannot see any design elements, no tabs across the top to switch to errors or data flows. Just a gray center area on the screen with a red x, and the message, its like VisualStudio dies in the process of reading the dtsx file.
The question is rather unspecific so it’s of course difficult to get on the right track here. All of the given answers focus different issues. I would say that PeterX had the best guess. The reason for the error could be as simple as a modified data source.
I came across with a bug "error output has no corresponding output" quite often when adding a new column to a table that needs to be processed by an existing SSIS package. This bug came along with an error message saying that a "Value does not fall within the expected range".
A newly added column needed to be processed by an existing SSIS Package. The expected behavior is that SSIS will recognize that there is a new column and select this column on the columns page of the OLEDB Source Task SSIS to be processed. However, when opening the OLEDB Source Task for the first time after having modified the table I got twice the following error message: "Value does not fall within the expected range." The error message showed up when opening the editor and when opening the Columns page of the editor. Within the Advanced Editor of the OLEDB Source Task the new column showed up in the OLEDB Source Output Columns Tree, but not in the OLEDB Source Error Output Columns Tree. This is the actual underlying problem of the error message. Unfortunately, there seems to be no way to add the missing column manually.
To solve the problem, remove and re-add the newly added column on the Columns Page of the normal Editor as mentioned by Jeff.
It is worth to be mentioned that the data source of the OLEDB Source task was a modified MDS View. Microsoft CRM Dynamics – as mentioned in the related thread – is using views, too. That leads me to the conclusion, that using views as a data source may produce either of the above mentioned errors, when modifying datatypes or adding/removing columns.
Related Thread: Error" ...The OLE DB Source.Outputs[OLE DB Source Output].Columns[XXXXXXXX] on the non-error output has no corresponding output
The described workaround refers to Visual Studio 2008 Version 9.0.30729.4462 QFE with Mircorsoft.NET Framework 3.5 SP1. The database is SQL Server 2008 R2 (SP2).
I had to delete and recreate the OLE DB Data source in my Data Flow - this is where I got the error. I also noted I had to "re-select" the "OLE DB connection manager" in the drop-down-list to force it to recognise the new connection.
This was probably a combination of getting the solution from TFS (where I noticed the data-sources didn't come-across properly and it complaining about a missing connection GUID) and/or copying and pasting the elements from another package.
(For BIDS 2008).
I had this issue for my OLE DB Source component with an SQL command after adding new columns to the database, and it wouldn't let me select columns or anything else to add the new columns.
I'm working with an Oracle database, and the only way I could get it to update was to change the SQL query to select 1 from dual, and preview it. Then revert it back to my old query.
You get a similar message if someone uses EncryptAllWithUserKey as the ProtectionLevel. However, I believe the message is slightly different (even though you get a grey design surface with a red X).
Have you tried viewing the file in Notepad? Is it just a series of GUIDs or is there anything in it that is humanly readable? If it doesn't have any readable code, then it was probably encyrpted with the user key.
If the employee deployed the packages to a server and used SQL Server as the deployment destination (not File System or SSIS Pacakge Store) then you can download the packages to your machine. Just connect to the SQL Server Integration Services engine, expand Stored Packages, expand MSDB, expand the relevant folder, right-click on the package, and click Export Package. Save the file on your local machine and open it. The package will probably lose annotations and pretty formatting, but otherwise it should be identical to what the employee deployed.
I just struck the same issue. After flailing about for a bit, I found the solution was to edit the Solution Configuration.
The Solution Configuration appeared to have a matching Project configuration, as shown:
However clicking the drop-down arrow for that Project (SSIS-Advance in this example) revealed that there was no Project Configuration for that project called Production - Sub Reports. I'm not sure how that came about - this Solution has a 7-year history and many developers.
Anyway once I created a New Project configuration (using that same drop-down menu), it is all happy now.
If it has Oracle data sources, you may need to install the Microsoft Connectors v4.0 for Oracle by Attunity:
https://www.microsoft.com/en-us/download/details.aspx?id=52950
I also had to use VS 2015 - the version originally used to create the project and package.
I had this exact problem and installing these connectors and using VS 2015 fixed the issue.
I had this occur as well when I tried to call a stored procedure with OUTPUT parameters with OLE DB.
I found this: http://sqlsolutions.blogspot.com/2013/04/ssis-value-does-not-fall-within.html, which resolved my issue. The relevant action was to rename the SSIS parameter mappings to '0', '1', etc.
So for example, when calling dbo.StoredProc #variable0 = ?, #variable1 = ? OUTPUT, #variable2 = ?;, in the parameter mapping dialog, you would name the parameters '0', '1', 2' to correspond to those. Ah, SSIS <3
I get this when I do not follow the convention for parameter naming, e.g. not name parameters 0,1,2,... in the right order for OLE DB connections.
The details are documented here.
In your connection manager, convert your connections to package level instead of project level
Delete connection manager and re-create and setup ssis package solve the problem.
I got this issue after I Add Existing Connection Manager in a SSIS project. I was just importing a Project Connection Manager from a different project (.conmgr) to my project. My solution to fix the issue was:
Deleting the imported .conmgr
Recreating it from scratch