I just added error handling functionality to an SSIS package that I am upgrading, and I need to add this same error handling to about 30 more packages. Is there a way to extract the error handling control flow, parameters, variables, etc. so that I can easily add them to the rest of the packages?
I am using Visual Studio Enterprise 2019 and SSIS 15.0.
I found a bunch of articles on BIML, but it looks like that is only for creating new packages. I am aware that copy and paste exists, but I would like to try to find a solution that is easy to apply across future packages as well as the current packages being updated. Apologies if this question has already been asked, I searched, but I'm not sure that I even really know what search terms would be applicable.
Yes, Biml is an excellent choice for creating consistent packages going forward. Even if you're only generating empty packages with error handling logic, that's a pattern and that's the power of Biml.
With the change to BimlExpress and the now free ability to reverse engineer packages, an approach could be to reverse engineer the packages to Biml. That would all be static tier but you'll need to select all and then in a new BimlScript file, add the error handling like so
<#
foreach(AstPackageNode apn in this.RootNode.Packages)
{
if (!apn.Events.Where(x => x.EventType==EventType.OnError).Any())
{
AstTaskEventHandlerNode onError = new AstTaskEventHandlerNode(null);
onError.EventType = EventType.OnError;
onError.Name = "OnError";
// TODO: add tasks and such
apn.Events.Add(onError);
}
//WriteLine(apn.GetBiml());
}
#>
Once that's looking good, you right click on everything at once and generate packages.
A non-Biml approach is going to test your C# (or VB.NET) skills. I've not touched this type of SSIS dev in more than a decade but the concept will remain the same. https://billfellows.blogspot.com/2016/10/what-packages-still-use-configuration.html
You'll need to find all the SSIS packages. For each one of those, use a reference to the DTS Runtime application to load it. Then look at the package's Events collection and if there isn't an OnError, you're going to have to add one to the collection and then add all the associated tasks, configure them and then save.
Related
I was trying to merge two git branches and encountered Merge conflict error.
I tried to resolve them and saved it. But now the whole package is unable to load.
The error shown is " An item with the same key is already added"
The error msg is as below. I am unable to find out where exactly I should make the change.
Could anyone help me to resolve it?
Please let me know if I need to add more info for the troubleshooting to be easier.
at Microsoft.SqlServer.Dts.Runtime.Project.OpenProject(IProjectStorage storage, String projectPassword, IDTSEvents events)
at Microsoft.DataTransformationServices.Project.DataTransformationsProjectLoader.<>c__DisplayClass21_0.<LoadProject>b__0(String password, IDTSEvents events)
at Microsoft.DataTransformationServices.Controls.ProjectProtectionUtils.LoadProjectWithPassword(Boolean askedPasswordOnce, ProjectLoader loader, IWin32Window dialogParent, String& password, ProjectProtectionEvents errorListener)
at Microsoft.DataTransformationServices.Project.DataTransformationsProjectLoader.LoadProject(XmlNode manifestNode, String& projectPassword, ProjectProtectionEvents errorListener)
at Microsoft.DataTransformationServices.Project.DataTransformationsProjectLoader.DeserializeManifestInProjectMode(XmlNode manifestNode)
at Microsoft.DataTransformationServices.Project.DataTransformationsProjectLoader.ConstructProjectHierarchyFrom(ProjectSerialization projectSerialization)
at Microsoft.DataTransformationServices.Project.DataTransformationsProjectLoader.Deserialize(TextReader reader)
at Microsoft.DataWarehouse.VsIntegration.Shell.Project.Serialization.BaseProjectLoader.Load(IFileProjectHierarchy projectHierarchy)
at Microsoft.DataWarehouse.VsIntegration.Shell.Project.FileProjectHierarchy.Load(String pszFilename, UInt32 grfMode, Int32 iReadOnly)
An SSIS package is an XML based file format.
Yes, you should absolutely use source control to version your packages. But you would be best off treating them as binaries because no source tool I am aware of knows how to merge XML documents.
The error you're experiencing is that you have an invalid package declaration. Without seeing the two files and the merge record, it's super hard to guess what's been done, much less rectify it.
SSIS Source Control guidance
After doing SSIS for nearly 20 years, I have a few thoughts on the matter.
Design your packages to be as small and tightly focused on solving a single business problem (Populate Sales table from Excel)
Use package orchestration to solve the dependent package problem (Run the Employee Package, then Customer, then Sales)
Only one developer works on a package at a time. Decompose the package into smaller packages if the business problem supports it to get more developers working on a task
If adding new packages to a project/solution, have a captain/leader create empty/shell packages and commit the project to source control - because the SSDT project artifacts are also XML and subject to the same botched merge logic.
I have spent the days looking for an Opensource JavaDoc editor or a way to edit JavaDocs easily or creating package-info files and come back empty.
I am basically after a way to change the way the packages are listed. The automation framework I work on is undergoing some potential future changes. These changes in a nutshell will organise our test scripts in a way where they are split up into business areas and then into smaller packages of business functions within that area.
We use IntelliJ and the default JavaDoc generator creates the JavaDocs in a way where it makes a list of all the packages, what I want to do is have it create a list of the main packages (business areas) and then the sub packages within that contain our test scripts (business functions).
For example:
BusinessAreaOne contain BusinessFunctionOne, BusinessFunctionTwo and BusinessFunctionThree.
BusinessAreaTwo Contains BusinessFunctionFour and BusinessFunctionFive.
The BusinessFunction packages in turn contain scripts that cover just that business function.
I haven't been able to find an editor that does what I would like and I don't think editing the files manually is practical each time there is an update.
If there is no opensource java editor can do this are there any suggestions of a work around?
As I mentioned I have tried creating package-info files but as far as I can tell you can only add a description to that package. Is there any tutorials out there or does anybody know a way I can get it to display the list of packages within the selected package?
Thanks in advance.
Javadoc doesn't treat package hierarchies as nested hierarchies, it just flattens them all into a list, as you probably noticed, so your business functions will just appear as entries in the list between the business areas.
Looking for a Javadoc editor is not going to help you because the Javadocs you end up with is the output from the Javadoc compilation process, and there's no specific tag you can put in your code that will influence the compilation process in the way you're looking for.
You would have to write something to override Javadoc's implementation of its HTML formatter specifically for the package list using:
Javadoc Doclets
You can customize the content and format of the Javadoc tool's output by using doclets. The Javadoc tool has a default "built-in" doclet, called the standard doclet, that generates HTML-formatted API documentation. You can modify or subclass the standard doclet, or write your own doclet to generate HTML, XML, MIF, RTF or whatever output format you'd like. Information about doclets and their use is at the following locations:
Javadoc Doclets
You would use your own implementation of the Standard Doclet, finding the right class to override - try looking first at the PackageDoc class. I'm going through this learning process myself at the moment, but here's the info on Doclet and there's a link at the bottom of that page to the source code which will illustrate how to do it.
Of course there could be an easier way of doing it and if I discover it I'll come back to update this answer.
I was wondering if doing operations such as changing var in a SSIS editing the xml code, rather than the cumbersome var rename/replacing in bids, if was possible to force to rebuild all the SCR task scripts that are actually using the vars. I noticed that the built version of the Tasks stored in the package are not updated so I need to open and close each of them to rebuild the tasks using the changes. Is there any option to force to rebuild all of them ?
Mario
Unfortunately, there's nothing built into SSIS to automatically rebuild Script Components or Transformations.
However, there are third party tools that offer that functionality. My shop is currently using Pragmatic Works' BIxPress. I've found it very helpful, although not cheap; see their website for details.
We have noticed an issue recently that redeployed SSIS packages sometime don't seem to include the latest changes... When I search the dtsx using notepad I see the amended script in the code so the changes are definitely there.
My assumption was that script components of SSIS packages are eventually compiled into an assembly somewhere in the process - this is quite likely since I would imagine C# code cannot run without something compiling it first. So in theory if these assemblies would then end up being cached and not immediately overwritten (for some reason) that would explain this issue.
The only "evidence" that makes me think that my theory is correct is if I keep running the package at some point it suddenly shifts to the new code.
However, so far I haven't found why and how this is happening, if is... Can anybody help?
UPDATE:
MSDN says: "Unlike earlier versions where you could indicate whether the scripts were precompiled, all scripts are precompiled in SQL Server 2008 Integration Services (SSIS) and later versions." - If by pre-compiled they mean that instead of the actual package a pre-compiled version runs (I think this because the package itself does not seem to be compiled since the code is visible in Notepad) there must be a way to force the engine to overwrite the pre-compiled assembly... but how?
UPDATE:
One of the four core components of SSIS is the SQL ServerIntegration Services service, which is a windows service. Apparently this service will cache component/task metadata so that the SSIS runtime engine can poll the cache to see what is installed, which may help speed up package load times. However, if the packages are stored in the file system (not in SQL Integration Services) and executed by Agent Jobs, the agent job will use the 64 bit version of DTEXEC to execute the packages. I haven't yet found evidence that any caching would be involved there, but there are certainly options to check a number of parameters in the validation phase of the execution, such as version numbers - may be for a reason.
Have you looked at sysssispackages to compare the version build number of the package in msdb to your build number in Visual Studio / SSIS?
SELECT name, verbuild
FROM msdb.dbo.sysssispackages
WHERE name LIKE '%bla%'
(Adjust WHERE-clause as necessary to find your package. Do NOT ever "SELECT * FROM msdb.dbo.sysssispackages" as it contains the package XML in one of the columns.)
And in Visual Studio, open the package, then right-click at the background of the package and select "Properties" from the context menu. Look at the field VersionBuild. It should match the number from the SELECT above!
I know this is not an actual solution to your problem but it may help locate where the cause of the problem is. If the number is older, it means that your package deployment did not work.
This sounds somewhat familiar to something I ran into a while back. Unfortunately, I don't remember exactly when I ran into this (so I can't check for sure), but I believe the fix I found was to make sure that I explicitly invoked the Build | Build st_5bd541c294054c25b9e7eb55b92bd0e2 command from the script editor (VSTA) menu before closing the window. (The specific project name will be different for each script, obviously, since it's based on a GUID; however, there will only be one possible submenu under Build.)
Explicitly invoking the Build command ensures that the binary code for the script gets ASCII-encoded and saved in the XML of the resulting .dtsx file. I'd gotten used to SSIS 2005 always building for me whenever I closed the script editor. Apparently, there are bizarre edge cases where SSIS 2008 doesn't always build the script project when the editor closes.
BTW, the precompiled binaries appear to be stored in a tag of the source XML called BinaryItem:
<DTS:Executable DTS:ExecutableType="Microsoft.SqlServer.Dts.Tasks.ScriptTask.ScriptTask, Microsoft.SqlServer.ScriptTask, Version=10.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91" DTS:ThreadHint="0">
<DTS:Property DTS:Name="ObjectName">SCR_StepOne</DTS:Property>
<DTS:ObjectData>
<ScriptProject Name="ST_5bd541c294054c25b9e7eb55b92bd0e2" VSTAMajorVersion="2" VSTAMinorVersion="1" Language="CSharp" EntryPoint="Main" ReadOnlyVariables="User::FileOneName,User::OutputFolder" ReadWriteVariables="">
<BinaryItem Name="\bin\release\st_5bd541c294054c25b9e7eb55b92bd0e2.csproj.dll">
TVqQAAMAAAAEAAAA//8AALgAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAgAAAAA4fug4AtAnNIbgBTM0hVGhpcyBwcm9ncmFtIGNhbm5vdCBiZSBydW4gaW4gRE9TIG1v
ZGUuDQ0KJAAAAAAAAABQRQAATAEDADuOb04AAAAAAAAAAOAAAiELAQgAABAAAAAIAAAAAAAAPi8A
AAAgAAAAQAAAAABAAAAgAAAAAgAABAAAAAAAAAAEAAAAAAAAAACAAAAAAgAAAAAAAAMAQIUAABAA
It might be worth checking your source code control system history to see if that was getting updated for some of those screwy errors.
Caveat: I haven't found official Microsoft documentation on this.
This doesn't specifically solve the mystery you have, but if you are running file system-based packages and want to verify that the package that is running is the package you deployed, there is a way to do that.
Build your package.
Open the properties on your package and note down the "Version Build" property (alternatively, open the .dtsx in notepad and find the DTS:VersionBuild attribute.)
Deploy your package.
In your SQL Agent job step, go to the Verification tab.
Enter the Version Build in the "Verify package build" input box.
Execute the job step.
I don't know if this will force SSIS to throw out its cache and get the newly deployed package, but I do know if you modify the .dtsx package's build number by hand and then try to re-run the job step it fails because the package build doesn't match what it's looking for so it is definitely doing a run-time check of that value.
We have trouble with keeping apart our different configurations. Allow me to explain with a little
Scenario:
Let's say you have two AX projects, e.g. P and M, that both alter the ProdRouteJob table, calling methods in one of their own project specific classes.
You have all these classes on your developer machine, of course, and ProdRouteJob compiles fine, but for an installation on a new server, you don't want to add stub classes for every non-installed project, do you? So you wrap these calls to project classes in something like
if( Global::isConfigurationkeyEnabled( <projectPConfKey> ) )
// call project P stuff
to encapsulate them cleanly. You declare configuration keys for all your projects, and you're ready to go, right?
However, this throws an error if you haven't installed project P on this machine, because its projectPConfKey is unknown. Now you could at every installation install configuration keys for all your projects, just to tell the server that there is such a thing as a projectPConfKey, but then all these ifs would evaluate to true...
In short: how do you include configuration keys in your project XPOs so that your code compiles, but at the same time so that some configuration keys are disabled from the start?
Or is there something totally basic that I'm missing here?
EDIT:
Consensus among the answers (thank you, demas; thank you, Mr. Kjeldsen) is that it's impractical to attempt a more or less automatic client-side configuration via macros or configuration keys. Thus, when we are installing at the client, we will have to import the standard tables with our changes and manually take out all changes not pertinent to the current installation.
I am a bit at a loss which answer to accept, because demas answered the question I asked, while Mr. Kjeldsen answered the questions that arose in the comment conversation under demas' answer. I shall, with apologies to Mr. Kjeldsen, mark demas' answer as accepted.
if( Global::isConfigurationkeyEnabled( <projectPConfKey> ) )
This check works only execution time and when you compile code you get the error.
So you need create stub classes or compare objects when you import them on production server and import only changes of project M.
There is another one approach but I don't have Axapta to check it. Try:
#if.never
someMissingCall();
#endif
But i'm not sure that it will works now.
Big solution centers (VAR) typically have separate applications for each module.
If you have 2 modules, you have 2 applications.
Good modules have very small collision areas to the standard application, these are kept in a separate project.
Collisions with the standard and other modules are then handled manually at install time. As the collision area is small and rarely change (it should not contain buisness logic) the problem is minimal.