1)
I have a SSIS package and am using package parameter to configure the connections to point to dev. db . Now if i use the environment variables to pass a different connection value pointing to test, after i deploy to the SSIS catalog and further schedule it using the SQL Agent Job.
What connection info. will be taken at runtime , when the job is scheduled to run. Will it be dev or a test conn??
2)
I have a SSIS package and am using project parameter to configure the connections to point to dev. db . Now if i use the environment variables to pass a different connection value pointing to test, after i deploy to the SSIS catalog and further schedule it using the SQL Agent Job.
What connection info. will be taken at runtime , when the job is scheduled to run. Will it be dev or a test conn??
Answer on the first question depends on whether you are using SSIS Package Configurations or not. Possibly not, since you are talking about environment variables, but if you still do and use it together with Environment Variables - see Microsoft article on that, and note this behavior changes with SSIS 2008 version.
If you are using SSIS Catalog and Package/Project variables, then mind this simple rule - more specific wins. In your case, the following precedence will take place, going to the next value if former is missing:
Value specified at package start, either with execution or dtexec parameters.
Value mapped with Environment Variable.
Vales specified at Package design time.
Here is another Microsoft article on package parameter value mapping.
Related
I hesitate to post because I feel I'm missing something simple, but after reading a ton of articles, I need a second set of eyes because mine are glazed over.
SSIS 2012 Solution.
Goal: Import data from 47 MS Access Databases into corresponding SQL Server databases. This means 47 packages. I want to avoid 47 different sets of Connection Managers and parameterize out what makes sense on the project level so common data is easily configurable.
I added project-level Parameters to hold the SQL Server, SQL UserName, SQL Password, and the Path to the Access Databases. These will be mapped out to the environment to allow changing the destination SQL Server easily as well as the source directory for the Access DBs.
I added two project-level OLEDB Connection Managers - one for SQL 2012 (NativeClient 11.0), one for Access (MSJET 4.0) (they're access 2000 db's). They are set to use an initial database configuration that I hope will be overridden by the individual packages.
For each package, I want to specify the databases being used, so I create a Package Variable (SQLDatabase and AccessDatabase)
In the Connection Managers, I add Expressions to include the Package Variable - e.g. for InitialCatalog on the SQL Connection Manager = #[User::SQLDatabase] and the ServerName in the Access CM combines the project path + the #[User::AccessDatabase]
WHAT WORKS:
I can update the Package Variable values and see them reflected in the Properties of the Connection Managers.
I can go into the Data Flow and add an OLE DB Source, select the SQL Connection Manager, and successfully receive a list of tables from that database.
WHAT FAILS:
When I click Preview, or Columns, after selecting a table I get a big error that reads:
"Error at Package1: The variable 'User::SQLDatabase' was not found in the Variables collection. The variable might not exist in the correct scope."
After clicking OK to that error, then the Preview or Column window shows up with no further problems.
However, after clicking OK on the OLE DB Source Editor, there is a big red X sitting on the box indicating the same error as above.
MY CONFUSION:
If I open the Variables for the package, the variable is sitting right there, scoped to Package1. If I update the value it and save, the Connection Manager updates itself accordingly, so the Expression mapping seems to be okay.
It's as if this is only partially working, and I'm unsure where the flag is to make the rest of it work, or where I failed in my configuration. Can you not set an expression in a project-level CM to use a package-level variable? If that's the case, I'm at a loss on how to correctly set this up to avoid package-level CMs.
Any help appreciated. Please be gentle if I've been an idiot somewhere.
If the connection managers are at project level, then you cannot use package level variables to set their value. If you go to the expression builder for Initial Catalog then you can see that only your project level variables are available. Hence you can use your project level variable to set it instead. If you really want to use the package level variable then you would need to create a connection manager per package which would be a pain.
What is the difference between Variables and Parameters in SSIS Denali?
If there is any difference then What is that which Variables cannot do that Parameters can do ? or vice versa.
When should one go with SSIS Parameters and Variables?
I tried searching on Google, but I failed to get some information.
Thanks In Anticipation!
I think a little bit background will be beneficial to understand the Parameter concept. Here I will explain it in the context of comparing with Variables. To fully grasp the Parameter concept, you might need to look up for the new Project Deployment Model, Environment, Build Configuration as well..
Usage Of Variable
With SSIS prior 2012, if we need to pass any external values to the package before the execution (as we all do all the time), I normally use configuration file (or a couple of other ways). Say we have a file server, which will be used to access a shared file, I will use variable to store the server name, and expose this variable to the configuration file. If the actual file server is changed (dev env to test env etc.), we just need the change the value of that variable in the configuration file and SSIS package remains intact.
Everything looked good, but there are a couple of things that I always ask myself why and could not figure out why:
100% of the time when I am exposing variables to configuration file, I just expose the "Value" properties. Why does SSIS allow to expose all the other variable properties?
Why does SSIS not have "private" variable? By "private" I mean when I chose the variables to configure, the "private" ones just did not get shown on the pick list. The SSIS package could have dozens of variables, for the internal value-holders, what's the point to expose them? Why I have to scroll all the way to find the only one I need to expose?
New Project Deployment Model
SSIS 2012 introduces a new deployment model, Project Deployment Model. For short, this model deploys SSIS project as a single unit to SQL Server SSIS catalog, and package configuration is NOT available in this model (it is available in the old model referenced as Package Deployment Model, with SSIS 2012 you can choose which one to use, 2012 default to the new model).
If we want the pass some values into the SSIS packages, we have to pass them in via Parameters, and use SSIS catalog in SSMS to configure the value for the parameters(only the value, nothing else we can configure). Parameters and connection managers are exposed automatically in SSIS catalog which can be configured, nothing else previously available via configuration files can be configured in Project Deployment Model (The world is much cleaner). Inside SSIS package, parameters can be used in the same way as variables in terms of building up expressions. However, parameters can NOT be modified within the SSIS package, which makes perfect sense. (Why do we need to change a value which is passed in from external? If we have to, pass the value to an variable, and do the changes there..)
Sum Up
Parameter is only available in the Project Deployment Model, and it provides the only mechanism for passing values from external to SSIS packages in this model. If we think SSIS pacakge as an OO class, Parameters could be thought as public properties, which externals can access and assign value to it (the class itself can/will use it, but cannot modify it). Where Variables could be thought as private variables, which are used internally, external world does not need to know anything about it.
For the old Package Deployment model, there is no Parameter, and the world remains the same.
FYI, in short, variable's value can be changed during the runtime, but parameter cannot. Parameter can help you do the project deployment and you can set it up in SSISDB catalog, while variable cannot.
There are many differences between Variables and parameters, few of them mentioned below:
Variables values can change at run time but parameters value can't change .
Variables can be used only with in the package we can't use it for other package with in the solution but we can use Parameters for multiple package (package exist with in the Solution Explorer).
The variables & parameters are similar to that are in java,
we pass/through some values to certain method/task in the form of parameters and we use them in that particular task we cant change those values since they are external things for that method similarly in SSIS the Project Parameters are used to set certain variables or connections dynamically in the package. where as variables are limited internal to the package level.
It works like this:
say you have a project parameter called ServerName:
Lets say you deploy an SSIS package into two integration catalog environments, one which is configured for prod server and another which is configured for test server:
Then your ServerName 'parameter' will be set in prod with prod server address and in test environment to contain test server address.
If any variable in your ssis package needs a runtime value(say the variable is used to set a connection at run time for prod or test servers respectively) then the variable will use the parameter from above to find the right server to connect to.
So parameters are usually needed in environment specific scenarios.
There are two types of parameters based on how you've configured your solution in Visual Studio: Project parameters or Package parameters. Project parameters are accessible to all packages in the project.
Parameters are using send data from outside of the package like usernames, passwords or connectionstrings etc. Variables are using inside of the package. It means you can define a variable in one of your SSIS package and use it in package level.
Parameters in SSIS are like Global Constants in Programming, so if they be applied to the Project can be used anywhere. Their
highest access would be the whole project.
Variables as they are named, can be assigned, from a Query or even a Parameter, etc. Also their highest level access would the
package.
I'm having this exact same problem I've been looking aroung but this is the only place I've seen the same issue and it is not resolved.
Does anyone knows what the problem might be?
I checked in Visual Studio and my SSIS version is 11.0.2100.60 (not a trial, not a beta).
EDIT: These are the steps I'm taking and the issue
First I choose a Connection Manager, right-click, select properties and click in Expressions option
Then in Property Expression Builder choose Connection String property and click in Expression option
Finally, in the Expression Builder dialog there is no option for variables, in every page I've read says that there should be a Variables node in there
Am I missing something?
SSIS 2012 has introduced the concept of Project level connection managers. What I see on the referenced post on the MSDN forums it the user has created a project level flat file connection manager and is unable to configure it with a local variable. Assuming that is the problem, my answer follows.
An SSIS project is generally more than one package. To simplify lives, the SSIS team now allows for the sharing of common resources across projects, connection managers being one of those resources.
Logically, if a thing is shared across a project, how can something that only exists in one file configure that resource? That configuration change would only work when Package1 is executing. When Package2 fires, unless the same variable and same expression was applied to the shared resource, you would experience different outcomes. That'd be a maintenance nightmare, which you might already experience if you don't have strong configuration practices.
If I create a Flat File Connection Manager at the project level, I can only reference variables that are also at the project level. Except there are not variables at the project level. Instead, they are called Parameters.
To that end, I created a Parameter called SomeProjectParameter
I then created a package, Package1.dtsx, and added 2 Flat File Connection Managers: FlatFileConnectionManagerLocal and FlatFileConnectionManagerProject
Instead that package, I also created a variable called SomeLocalVariable.
This screenshot shows me applying an expression to the ConnectionString property of FlatFileConnectionManagerLocal. There you can see that both the package variable, SomeLocalVariable is available as well as SomeProjectParameter
Now, if I try to apply an expression to the project's connection manager, you will only have project parameters available to you.
It's interesting to note that you can't apply an expression to a project level Connection Manager outside of the context of an SSIS package. There's simply no editor available to you until you have an open SSIS package. But, once applied, all the packages in the project will be similarly configured.
Quirk of the IDE I suppose. Also, don't be alarmed by the lack of color in these screenshots, I'm running with the 2012 version of SSDT.
i had the same problem and it was because the flat file source was set as a project source so i had to convert it into a package connection.
Right-click on the source and choose convert to package connection
Maybe this will help.
The Microsoft SSIS tutorial have these confusing statements:
In the Connection Managers pane, right-click Sample Flat File Source Data, and select Properties.
In the Properties window make sure the PackagePath starts with \Package.Connections. If not, in the
Connection Managers pane, right-click Sample Flat File Source Data, and select Convert to Package Connection.
\Package.Connections was initially displayed in the properties window, although I had not yet converted to Package Connection.
After reading this thread, I right clicked on Sample Flat File Source Data and then Convert to Package Connection and was able to complete the lesson. Thanks for the answers!
Morning
I've been reading "SQL Server 2008 Integration Services Problem - Design - Solution". It outlines a way of logging variable changes which I'm trying to replicate in SQL 2005.
Create variables e.g. PackageId, RecordsAffected. - Set Raise ChangeEvent to true.
Create a string variable g.g. strVariableValue. - Set Raise ChangeEvent to false.
On the package event handler: OnVariableValueChanged add a script task "SCR Convert value to string".
Add ReadOnlyVariables: System::VariableValue
Add ReadWriteVariables: User::strVariableValue
In the script, set a local variable to System::VariableValue.value.tostring
Set the variable User::strVariableValue to the local variable
Add an "Execute SQL Task" component "SQL Log Variable Value Changed" calling a SP with no resultsets.
Set parameter mapping to User::PackageId, System::VariableName, User::strVariableValue
When this is run, I get a deadlock on User::PackageID
Error: 0xC001405B at SQL Log Variable Value Changed: A deadlock was detected while trying to lock variable "User::_PackageID" for read access. A lock could not be acquired after 16 attempts and timed out.
The script step succeeds but the Execute SQL task fails. I'm using Visual Studio 2005 Version 8.0.50727.42, Microsoft SQL Server Integration Services Designer Version 9.00.4035.00 and BIDSHelper Version 1.4.3.0.
Any ideas?
Eureka!
I had the same problem and led to a few deadend posts, then I discovered the root.
I had the framework working just fine and wanted to force some info to be logged.
So I changed the value of the framework variable "strVariableValue" and this caused the deadlock with the change event task.
I fixed by creating my own variable "strLogMe" and putting whatever I wanted to log.
Moral: don't touch the framework variables
Did you use the code sample from the book? All the files are available on the Wiley website for free. The code sample includes a SSIS package, sql scripts, and VB code for the script. If this doesn't work for you, then let me know since one of my team members found a way to log variable changes that was different from this methodology.
I was getting this error ("a deadlock was detected" etc), suddenly, which seemed to coincide with I.T. having done a Microsoft Windows patch on the server. There were packages which were using script tasks, with read-only and/or read-write variables in the SSIS UI. Even though it seemed to have been an environmental issue (because the packages had worked for months, then suddenly stopped working, even though I hadn't changed any code), I thought, well (as I had seen from various blog posts from years gone by), there were instances of companies doing server patches, then having their SSIS packages break; and the blogs seemed to say, change the way you're locking the variables, don't reference them in the UI; instead, lock them explicitly in code. So I tried the same thing. It didn't fix it.
It turns out some individual had removed the permissions of the user under whose identity the packages run, from the AD group; those permissions were required because it was trying to copy a file from a directory which required read permissions on the directory. These packages are typically called by a SQL agent job using a proxy identity. When the package was executed manually from SSMS, it worked. But when it was run by calling the SQL agent job, it failed.
The bottom line is, it was just coincidence that the packages started failing around the time of the Windows update. But the other (main) point is, if your package is trying to access a file on the network, and the identity (or proxy identity) under which that package runs does not have permissions to the source or target directory, then your package could fail and the problem could manifest itself in this cryptic way, where it looks like a variable deadlock issue, but it's actually a file share permissions issue. I only wasted a day on this, but... maybe this will be useful to somebody in the future.
I have a SSIS package built in Business Intellegience Development Studio which have both source and destination database specified. Now I want to use some variables to parameterize the database connections and run the package in a command line.
I try to replace the database name and sql server instance with my variable #[User::SourceDb]. #[User::SourceHost], but it failed to connect to the database.
Is that possible to paramterize the database and is there anything wrong with my variable useage? Thanks in advance!
There is a nice post here that details one way of doing this.
You will need to use a ConnectionManager and set the ConnectionString property of that from a Configuration Package.
The ConnectionString property is a fully qualified database connection string, like
Data Source=localhost;Initial Catalog=SSISConfig;Provider=SQLNCLI.1;Integrated Security=SSPI;Auto Translate=False;
There are a few gotchas and annoyances with using configuration packages so you may have to fiddle around a bit. I'd be more detailed but it has been about a year since I last worked on an SSIS project.
If you have any specific problems, please come back and comment about what you've hit. I'll try and refire the old memories.
You need to define "expressions" in the connection manager which then substitute into server/database.
You can't use variables directly quite often in SSIS
See "Using Variables in Packages" whcih links to Using Property Expressions in Packages.
And after a quick Bingle: http://sqlrs.blogspot.com/2006/03/using-expression-variables-in-ssis.html
You can right-click on any connection on the project or package and "Parameterize" all the fields from connection, in project or in package parameters. Connection string differs for OLE DB connection to .NET Providers(SqlClient Data Provider) type, so be careful. We have manage to have one centralized place for connection in Project.params