I am implementing a custom auditing framework, logging ETL events such as start, end, error, insertrows etc.
As well as logging at a package level, I'm implementing "session logging" where a sequence of package executions, i.e. a controller package that executes several packages, is a session. In order to keep track of the "session", the stored procedures always return a SessionLogID.
I was hoping I could map this result set to a project parameter as otherwise, I will have to save it to a user var and then pass it around between packages via parameters. This will mean every single package will have a Package Parameter and User Variable called SessionLogID. I don't want to do this if I don't need to.
Open to other suggestions.
Thanks,
Adam
Parameters cannot change at runtime. They are a set once kind of deal whereas variables can change at any time. You can set the variable once in the parent package and map the variable to the child package's using a parameter.
Related
Using SSIs 2016 trying to set an input parameter to a sql task to System::ErrorDescription, but that variable isnt listed. Is this variable not used any longer?
I have sql task that, if it fails, I want to log the error desciption to a table. Yes, I know I can enable logging, but I want also use this message in another task.
[Edit]
I think this variable is only visible within the context of an event handler such as OnError, correct?
The System::ErrorDescription variable is not available for all components, specifically it is only in the OnError, OnInformation, and OnWarning event handlers.
https://learn.microsoft.com/en-us/sql/integration-services/system-variables?view=sql-server-2016
It seems like a very common issue with SSIS packages is releasing a package to Production that ends up with running the wrong connectionstring parameters. This could happen by making any one of many mistakes or ommisions. As a result, I find it helpful to dump all ConnectionString values to a log file. This helps me understand what connectionstrings were actually applied to the package at run time.
Now, I am considering having my packages check to see if every connnection object in my package had its connectionstring overriden by an entry in the config file and if not, return a warning or even fail the package. This is to allow easier configuration by extracting all environment variables to a config file. If a connectionstring is never overridden, this risks that a package, when run in production, may use development settings or a package, when run in a non production setting when testing, may accidentily be run against production.
I'd like to borrow from anyone who may have tried to do this. I'd also be interested in suggestions on how to accomplish this with minimal work.
Thx
Technical question 1 - what are my connection string
This is an easy question to answer. In your package, add a Script Task and enumerate through the Connections collection. I fire the OnInformation event and if I had this scheduled, I'd be sure to have the /rep iew options in my dtexec to ensure I record Information, Errors and Warnings.
namespace TurnDownForWhat
{
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
/// <summary>
/// ScriptMain is the entry point class of the script. Do not change the name, attributes,
/// or parent of this class.
/// </summary>
[Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
{
public void Main()
{
bool fireAgain = false;
foreach (var item in Dts.Connections)
{
Dts.Events.FireInformation(0, "SCR Enumerate Connections", string.Format("{0}->{1}", item.Name, item.ConnectionString), string.Empty, 0, ref fireAgain);
}
Dts.TaskResult = (int)ScriptResults.Success;
}
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
}
}
Running that on my package, I can see I had two Connection managers, CM_FF and CM_OLE along with their connection strings.
Information: 0x0 at SCR Enum, SCR Enumerate Connections: CM_FF->C:\ssisdata\dba_72929.csv
Information: 0x0 at SCR Enum, SCR Enumerate Connections: CM_OLE->Data Source=localhost\dev2012;Initial Catalog=tempdb;Provider=SQLNCLI11;Integrated Security=SSPI;
Add that to ... your OnPreExecute event for all the packages and no one sees it but every reports back.
Technical question 2 - Missed configurations
I'm not aware of anything that will allow a package to know it's under configuration. I'm sure there's an event as you will see in your Information/Warning messages that a package attempted to apply a configuration, didn't find one and is going to retain it's design time value. Information - I'm configuring X via Y. Warning - tried to configure X but didn't find Y. But how to have a package inspect itself to find that out, I have no idea.
That said, I've seen reference to a property that fails package on missed configuration. I'm not seeing it now, but I'm certain it exists in some crevice. You can supply the /w parameter to dtexec which treats warnings as errors and really, warnings are just errors that haven't grown up yet.
Unspoken issue 1 - Permissions
I had a friend who botched an XML config file as part of their production deploy. Their production server started consuming data from a dev server. Bad things happened. It sounds like you have had a similar situation. The resolution is easy, insulate your environments. Are you using the same service account for your production class SQL Server boxes and dev/test/uat/qa/load/etc? STOP. Make a new one. Don't allow prod to talk to any boxes that aren't in their tier of service. Someone bones a package and doesn't set a configuration? First of all, you'll catch it when it goes from dev to something-before-production because that tier wouldn't be able to talk to anything else that's not that level. But if you're in the ultra cheap shop and you've only got dev and prod, so be it. Non-configured package goes to prod. Prod SQL Agent fires off the package. Package uses default connection manager and fails validation because it can't talk to the dev sales database.
Unspoken issue 2 - template
What's your process when you have a new package to build? Does your team really start from scratch? There are so many ways to solve this problem but the core concept is to define your best practices for Configuration, Logging, Package Protection Level, Transaction levels, etc into some easily consumable form. Maybe that's 3 starter packages: one for raw acquisition, maybe one stages and conforms the data and the last one moves data from conformed into the final destination. Teammates then simply have to pick one to start from and fill in the spots that need it. If they choose to do their own thing, that's the stick you beat them with when their package fails to run in production because they didn't follow the standard path.
There are other approaches here. If you're a strong .NET crew, you can gen your template packages that way. At this point, I create my templates with Biml and use that to drive basic package creation.
If I am understanding you correctly the below solution should work.
My suggestion to you is to turn on the Do not save sensitive option for the ProtectionLevel property at the top level of the package.
This will require you to use package configurations for every connection, otherwise it will not have the credentials to make a connection.
I have a very simple SSIS package and an absurdly simple Script task
var x = Dts.Variables["User::OrderPath"];
That fails with:
The element cannot be found in a collection. This error happens when
you try to retrieve an element from a collection on a container during
execution of the package and the element is not there.
I have a variable, OrderPath, that is scoped to the package. Any time I try to add the variable to the script's ReadOnlyVariables, it disappears whenever I execute the task.
This really shouldn't be this difficult so I assume I'm missing something monumentally basic. Any idea what that might be?
When accessing variables through the Dts.Variables collection, you do not prefix the variable name with the namespace; thus
var x = Dts.Variables["User::OrderPath"];
fails, while
var x = Dts.Variables["OrderPath"];
succeeds. (Assuming, of course, that OrderPath is added to either the ReadWriteVariables or ReadOnlyVariables task properties.)
See Using Variables in the Script Task in MSDN for another example.
PROBLEM
Let's say I have a jenkins/hudson job (for example free-style) that takes two parameters PARAM_ONE and PARAM_TWO. Now, I do not know the values of those parameters, but I can run some script (perl/shell) to find values of those parameters and then I want the user to select from a dropdown list after which I can start the build.
Is there any way of doing that?
Sounds like you've found a plug-in that does what you need, that is pretty similar to the built-in Parameterized Builds functionality.
To answer your second question: when you define parameterized builds, the parameters are typically passed to your job as environment variables. So you'd access them however you access environment variables in your language, for instance, if you defined a parameter PARAM_ONE, you'd access it as:
In bash:
$PARAM_ONE
In Windows batch:
%PARAM_ONE%
In Python:
import os
os.getenv('PARAM_ONE')
etc.
I imagine this would be the same for the Extended Choice Parameter plugin you are using.
Just install this, and give the parameter in the build script like:
Windows
"your build script" %PARAMONE% %PARAMTWO%
In Java, you can access these parameters off the run object
EnvVars envVars = new EnvVars();
envVars = run.getEnvironment(listener);
for (String envName2 : envVars.keySet()) {
listener.getLogger().println(envName2 + " = " + envVars.get(envName2));
}
I have a Access 2003 database that will dynamically load MDB databases as a library reference. The reason for this is this database is a menu front-end for 60+ application databases. Rather than deal with permanently referencing all these databases, the menu front-end will dynamically reference what is needed when the user makes a selection. I was working on moving this database to Access 2010 and creating a custom ribbon. I started using the technique from here to capture the ribbon object in a global variable when the ribbon loads. I then ran into the problem where I could verify the code was running and the global variable was correctly being assigned the ribbon reference but after the database would run through it's startup routine, that global variable would get reset to Nothing.
To verify what was going on, I created a simple database for testing. In this database, I had a module with a global variable:
Public obj as Object
I then had a function like this:
Public Function SetObj()
Set obj = Application
Debug.Print "IsNothing=" & (obj Is Nothing)
References.AddFromFile "Test.mdb"
Debug.Print "IsNothing=" & (obj Is Nothing)
End Function
Obviously, in my code, "Test.mdb" refers to an actual file. If I run this code, Debug.Print gives me "IsNothing=False" for both instances, but after the function finishes and if I wait a couple seconds, Debug.Print will give me "IsNothing=True". If I comment out References.AddFromFile, Debug.Print gives me "IsNothing=False" no matter how long I wait.
It makes sense to me that since Access has to re-compile the VBA code after loading the library that all global variables are reset. I've experimented with moving the global variable into a class, but since I then need a global variable for the class, the class variable then gets reset instead. I tried using a local variable in the function to save the value of the global variable, but it looks like Access waits a couple seconds after the code is finished running to do the re-compile, so that doesn't work either. Does anyone have any other ideas to accomplish this?
I don't really know if this will solve the problem for this kind of reference, but in general, I don't use public variables for this kind of thing, but instead use a STATIC variable inside your function. It would be something like this:
Public Function SetObj() As Object
Static obj As Object
If (obj Is Nothing) Then
Set obj = Application
End If
Set SetObj = obj
End Function
Then you'd just use SetObj as an object for using your application. In a production app, you'd need tear-down code, too, but I've omitted that here.
I doubt this helps, but your code struck me as rather inefficient and incomplete.
I figured out a solution to my problem, and thanks #David-W-Fenton, as your answer gave me the idea. I use your approach in a library database for caching frequently-accessed values that are stored in a table but don't change after the initial startup. Those values aren't lost every time the references change, and that's when the light bulb lit up.
The solution is to put the global variable in a library database. Access looks to be only resetting global variables in the database that the reference is being loaded into - which makes sense after thinking about it. So since the library database isn't the one being re-compiled, it doesn't get it's global (or private or static) variables reset.
What I ended up doing was creating a new module in an existing library database. It has a private variable and two methods - one to set the variable, one to retrieve the variable value. In my menu front-end database, when the ribbon loads and calls my callback function, rather than saving the ribbon object in the front-end database, I pass it to this module for saving. I now no longer lose that ribbon reference whenever new databases are added to the library references on the fly.