What is the 'best practices' way to configure connections in SSIS 2012 project (that will be deployed to the server)? Documentation and Google shows multiple ways to accomplish this, but which way takes full advantage of the 2012 project/deployment model and is easily customizable, maintainable etc?
Consider a project (NorthwindETL) with three packages with each package referencing localhost.Northwind. On SSIS server, there is a project 'NorthwindETL', and an Environment 'Dev'.
To configure the connection, I have the following options
IN BIDS: Hard code the connection (via connection manager) manually in each package. IN SSIS: Configure the SSISDB 'NorthwindETL' project, under the connection manager tab, modify the Northwind connection string, once for each package.
IN BIDS: Using the Connection Manager 'Parameterize…' option, create a project (or package) parameter to specify the connection string (Northwind_Conn). IN SSIS: Configure the SSISDB 'NorthwindETL' project, specify the Northwind_Conn parameter value.
IN BIDS: Create a project level Connection Manager (Project_Northwind_Conn). IN SSIS: Configure the Configure the SSISDB 'NorthwindETL' project, under the connection manager tab, modify the 'Project_Northwind_Conn' connection string.
IN SSIS: Create an Environment on SSISDB called 'DEV'. In the 'Dev' environment properties, under variables, create a variable 'Env_Northwind_Conn'. Configure the NorthwindETL project, set 'Northwind_Conn' to the environmental variable 'Env_Nothwind_Conn'
(Also, I would prefer a solution that allows us to specify items separately such as InitialCatalog and Server, but this is not necessary. Although the connection manager allows you to modify the InitialCatalog and Server properties, this does not seem to actually modify the ConnectionString.)
For connection managers that are to be used across all the packages, typically database connections, the 2012 release gives us Project Connection Manager (option 3). I find that to be the most enjoyable for connection managers as when I go to apply configuration, I'm applying it once at the project level instead of once per package.
Unless you run your dev/test/prod ETL from the same dedicated server, I am not a fan of naming my SSIS Environment Variables as such. The primary reason is that my deployment script then has to have the intelligence built into it to not only switch server names per tier, but also the environment name. Just makes more opportunity for the dumb to enter.
I'm also a fan of just creating an empty Folder within the SSISDB, call it Configs and then establish my SSIS Environment variables there. All projects then reference that folder's variables. At my clients, it's generally been the case that they're all referencing the same Sales database so it seems like more work for me to have to maintain N configurations to satisfy N projects instead of having a single shared configuration repository.
Script everything. Click the scroll icon as you create and assign your configurations. It's going to make it a far easier row to hoe when you need to migrate from one environment to the next.
Related
Hello is it possible to define the SMTP server address from a query in a SSIS package
I understand it might be through the expressions section see image
Any help is appreciated
SSIS provides a way of configuring packages with parameters so that they can run with different values in different environments.
In your case, right click the connection manager for the smtp server and select parameterize.... In the dialogue, select the property, name the parameter and select whether you want it scoped to the project or package. If you select project, other packages will be able to reuse it.
When this is finally deployed to the SSISDB catalog, you can configure the parameters in the sql agent job step or you can use Environments to preconfigure the project and packages. Here are more details about parameters:
https://learn.microsoft.com/en-us/sql/integration-services/integration-services-ssis-package-and-project-parameters?view=sql-server-ver16#parameterize-dialog-box
First let me give some background
We have our own VPS, so we do not wish to use Azure to host our web applications.
We have already successfully created a CI/CD pipeline to our VPS by installing an agent on it for a .NET Core project.
We use Azure DevOps (formerly known as VSTS) to host our code in GIT and handle our backlogs and CI/CD pipelines.
We have several .NET Framework projects where we use XTD transforms to transform our web.config files on delivery/deployment to have the correct connection strings and other configuration properties.
This makes it possible to pull the master branch from our remote repo and have it working in seconds on a previously unused (for this application) development environment without the need for any configuration.
Now to get to my question
The master branch of the .NET Core project for which we already have the CI/CD pipeline in place holds the configuration in the json files for the staging environment it is continuously delivered to. When a developer pull the master branch, he/she first needs to configure these to suite the local debug environment.
This is an undesirable situation for us.
How can we make it so that if we use .NET Core we can use a mechanism that will allow us to have the project work on a local debug environment without any configuration and in the CI/CD pipeline?
What have we already tried?
we have found that we can have multiple versions of the appsettings.json file for the different environments like appsettings.debug.json and than in the static method CreateWebhost of the Program class we can call on or the other. But how we can automate this is something that we haven't been able to figure out or find documentation about.
Okay, so here are some options you can take advantage of TODAY. (there are im sure more options/approaches)
Option A
Configure the master branch to have appsetting.development.json with connection string to DEV database( or lowest environment)
remove any connection string from appsettings.json
Merge master accordingly.
Create environment variables on each of the backend servers for the connection string; ex, system environment variable named ConnectionStrings:cartDB with connection string to the database for the environment for which the backend server used.
The result of this will be that when running using DEVELOPMENT as the environment variable, then it will be able to connect to database everyone can access.
However, since all OTHER web servers have environment variables with connection string, they will take highest level of precedence, and will therefore be the values set when calling something such as
string connectionString = Configuration.GetConnectionString("cartDB");
This will satisfy the requirements you mentioned above.
Option B:
Configure the master branch to have appsetting.development.json with connection string to DEV database( or lowest environment)
remove any connection string from appsettings.json
Place appsetting.staging.json, appsettings.prod.json in source control, and set environment name variable in web servers. :/ not the best of options/advised against.
(its worth mentioning since I have seen this happen, we all have)
Option C
Add appsetting.staging.json, appsettings.prod.json to source control and use a token in place of the connection string value. Then, leverage some type of Tokenization Task to replace those tokens with the appropriate values.
I'm creating SQL Server Integration Services 2014 project in project deployment model.
For example, I'm connecting to a database and I want this connection to be centralized within the project scope.
My question, why do I need project connection manager? I like project parameter, as far as I know they do similar job as project connection managers + additional things e.g. they bind with SSDT Configuration Manager so I can have dev/QA configuration etc…
Under what circumstances will I need project connection manager instead of project parameter?
A project connection manager ensures that all the consumers of that resource use a consistent value. Using project level parameters to specify user name, server, etc, allows all the packages to have access to the correct value but there is no guarantee that all packages will consistently apply the correct pattern of expressions to get the configuration managers properly configured for an environment. Simply stated, the project level connection managers remove the opportunity for mistake.
Once deployed to the SSIDB, you can use Configurations there to ensure your projects are using the correct resource. It can be an all/some/none approach to configure your project level entities: connection managers and parameters.
I am working on my first SSIS package that connects to a SQL server. While I am developing it, I am connecting using Windows authentication which works fine since my Windows user name was added to the security of the database I am working on. Now, my IT department created a service account to deploy the package with. My question is, how can I change the user name/password of the connection before I deploy it? Is there a configuration file that the connection can read from? How can this be handled?
You actually have two security contexts here to be concerned with. The first is the account required to deploy the package or project you've created. The second is the account required to be able to execute the package you've created.
End-to-end Windows Authentication (deployment, execution and data sources)
The deployment account would need to have correct permissions to the server or filesystem on which it will reside. The execution account may be configured with a very different set of permissions, primarily related to the permissions required to execute whatever tasks you've built into the package.
In order to deploy the package under a different user that your own, it may be as simple as opening an application (like Command Prompt, Windows Explorer or SSIS Deployment Utility) as that user account and moving the package to the correct location. This can be handled on your workstation or the server.
For the execution account, you have options depending on how you're going to operationalize the execution of the package. Here's a few scenarios:
If you will have the package be executed by the SQL Server Agent and the account you need to execute the package with is the SQL Server Agent service account, you only need to create the job to run the package. Unless otherwise programmed, packages called from a SQL Agent job will run as the SQL Agent account
If you will have the package be executed through a SQL Server Agent job and the account you need to use for executing the package is NOT the SQL Agent service account, you can create an SSIS Proxy Account and specify that in the SSIS Package execution job step. After creating the credential inside SQL Server, it is as simple as changing a dropdown.
If you will be using command line execution from a SQL Agent job, the above two scenarios are still applicable.
If you will be using another mechanism (like Windows Scheduler or another Enterprise Scheduling tool) that uses a command line execution-like method, you should still be able to have that process "run as" the execution account.
Windows Authentication for deployment and execution only (SQL authentication for data)
The above details still apply for executing SSIS packages via jobs and/or command line, but you will need to pass the username and password to the connection manager at the time the package runs. You have several options to do this and should follow any established patterns or standard your organization has in place. SSIS has long supported using an XML-based .dtsConfig file which can be read into the package at run-time. There is a GUI within SSDT/BIDS that will lead you through the process of creating the file and telling it which package properties you want it to be able to configure.
A word of caution
Be careful when you're trying to SAVE sensitive information inside SSIS packages. There is a property named PackageProtectionLevel which can be set at the project and package level. By default, it is set to EncryptSensitiveWithUserKey. Now, don't let this property trick you into thinking the entire package is encrypted. It is not. This setting specifically refers to how SSIS will handle properties that are typed as sensitive. For example, the passwords in the connection managers are typed as sensitive information. SSIS will encrypt that field so that it doesn't store passwords in plain text. But it ONLY pertains to saving/storing the package. You can pass in plain text through a variable or configuration file that will be read into a sensitive field (like a password) at run-time.
If you need to be able to save a password with the package you've developed, I would recommend changing the PackageProtectionLevel to EncryptSensitiveWithPassword and setting it to something your team is able to remember or uses to protect other assets. Once that setting is in place, you will be able to check the "Save Password" box within the connection manager and have that go where ever the package goes. If you don't need to save that password with the package, change the property to DontSaveSentitive. Like I mentioned, you can still pass in credentials through configurations or other means, but it won't be stored INSIDE the package itself.
I am trying to run my SSIS Package via SQL Server Agent Jobs. In the SSIS Project I have Connections to My SQL Database as PROJECT Connections. I have set them up like this as all Packages call this connection. However when running it in SQL Server Agent I get error saying:
The connection "{}" is not found. This error is thrown by Connections collection when the specific connection element is not found.
It obviously can't locate the connection so how do I do this ? - The package executes successfully when executed via Visual Studio.
Many Thanks In Advance !
Bal
First, you need to create an Integration Services Catalog on your server instance. You will deploy your packages to the catalog.
A best practice for specifying the server in your connection managers is to use a single period, which references the local machine. That way, when you deploy your packages, the local machine will always be used:
After you've created your package(s) with project-level connections, you need to deploy the project. Right-click the project folder in SQL Server Data Tools/BIDS, then click Deploy. In the deployment wizard, specify the destination server (or just use "." again to deploy to the local instance) and the Integration Services (IS) Catalog folder:
Once deployed to your SQL Server instance's IS Catalog, you can set the Package Source in the Job Step Properties to "SSIS Catalog" and select the package that you deployed.
If you're exporting/importing to/from files, you'll want to ensure that the SQL Server Agent Service Account has appropriate rights to the folder where files are imported/exported. The easiest way to do that is to create a credential (usually a Windows user account), then create a SQL Server Agent Proxy that uses the credential, and then specify that proxy in the job step's Run as field.
Here's what your Job Step Properties window would look like after following all of the steps above:
The job should then run successfully.
FYI, you can also execute the package directly from the SSIS Catalog. Simply drill down into the Integration Services Catalog node of your server | right-click the package | click Execute....
Helpful Links
Schedule a Package by using SQL Server Agent
SSIS Tutorial: Deploying Packages
If you setup your configurations on the configurations tab in the Step properties of the SQL Server Job that should allow you to accomplish what you asked.