I have an application I am writing that accepts files of various formats. Then I write code for each format (csv, excel, xml) to convert it and enter it into a SQL Server database. This is fine but I was just looking into SSIS and wondering if this would help.
The main question though is how do I run these packages within my own code? Is it easy to pass parameters? If I move this web app to another server does it depend on other components being on that server (maybe SQL Server 2012 installed)? Or is it just some DLLs I can reference from my web app?
All the demos I see are about using the SSIS tool but I am more interested in how difficult it is to call packages with parameters from my code.
The BIDS/SSDT installation includes a complete client SDK that makes it pretty simple to run SSIS packages via code.
This MSDN article explains how to load and run a package via C# in detail but the actual code boils down to:
using Microsoft.SqlServer.Dts.Runtime;
.
.
.
Application app = new Application();
Package pkg = app.LoadPackage(PKG_FILE_NAME, null);
DTSExecResult pkgResults = pkg.Execute();
The Package object has a lot of properties and methods you can look into, in particular there is a Parameters collection that allows you to pass parameters into your SSIS package before execution.
There is a very cool library called EzAPI where you can generate your own packages and call them from within C#. It gives you a lot of flexibility to generate SSIS on the fly and execute.
http://sqlsrvintegrationsrv.codeplex.com/releases/view/21238
I built a couple console apps inside this project to test some of these methods and you might find the examples useful:
https://github.com/thevinnie/SyncDatabases
Look at "BuildingAPackage" and "BuildALookupPackage"
You can programmatically build SSIS packages using C# or VB.NET and then run the packages. You can also load an existing package programmatically to execute it. I have little experience with building packages this way since most of the packages can be built easily using the following tools.
Read the article Building Packages Programmatically on MSDN for more details.
I feel that it is easier to build the packages through these IDEs depending on which version of SSIS you are targeting. You can also create initial packages through SQL Server Import and Export Wizard and save the SSIS packages to the local disk, which you can later modify according to your needs.
SSIS Version Development IDE Visual Studio Shell
------------------- ----------------------------------------------- -------------------
SSIS 2005 Business Intelligence Development Studio (BIDS) Visual Studio 2005
SSIS 2008 - 2008 R2 Business Intelligence Development Studio (BIDS) Visual Studio 2008
SSIS 2012 SQL Server Data Tools (SSDT) Visual Studio 2010
You need to reference the appropriate SSIS specific DLLs in your code to create/load SSIS packages.
However, you will need a SQL Server Integration Services license to run the package. You cannot simply reference the DLLs alone. The license is usually part of your SQL Server license, if you already have one.
Response to your comment:
We will have SQL 2012 or 2008R2 on a separate server.. So I just need to paste the needed DLLs on my web server and reference them. correct ?
The packages will execute on that server. You are just remotely invoking to execute them and you should reference the appropriate DLLs in the code within your web/other form of external application. I usually schedule the packages to run on the database servers under SQL Server Agent Job. You can try that if that is an option for you.
Package parameters are read-only from an external application. You'll need to pass them in as variables (which I have done). If you are dealing with an existing package with parameters, you may consider writing a script task that populates the parameters from the variables (which I have not done).
Related
I have created several packages in Visual Studio 2017 (SSDT) SSIS solution, they are showing target server 2017. This was on my development PC. Now, upon completion of it, I am into a challenge. The Server on which, I am moving my packages, is having Visual Studio 2008.
I can understand that downgrading may not be right away possible, but I will still be happy if I can find a way to minimize my work (i can accept package by package or so). Please note I am accessing the other server on the Remote and not able to use Ctrl+C and Ctrl+V to copy and paste package objects, so if I don't get some easy ways then have to almost write the whole package again.
Thanks for your support.
Install BimlExpress in your Visual Studio 2017 environment. Once installed, you should be able to right click on the package and with your new BimlExpress options, select Convert SSIS Packages to Biml
In the Miscellaneous section of you SSIS project, you should have a BimlScript.biml file. That is the Biml representation of an SSIS package but in a much more terse, yet user friendly syntax. This XML describes in a version independent way what the SSIS package did. Here's where it's cool as a cucumber: The Biml for a 2005 SSIS package is exact same as a 2019 package and vice versa. The Biml Engine has to "know" how to translate the package definition into version specific .dtsx but the smart folks at Varigence have already solved that problem.
Now, you need to take that generic Biml and match it up to a Visual Studio 2008/2010 installation that has the Business Intelligence Design Studio (BIDS) bits installed on it. For SQL Server 2005/2008, BIDS only came on the SQL Server installation media as that is how the product is licensed. A developer license cost 25-50 USD but you can no longer purchase it. Hopefully, the target server has BIDS installed on it.
If so, then you need to find a copy of BidsHelper or maybe the renamed product BI Developer Extensions and then you can use the Biml engine embedded in that product to emit a 2008 correct package.
If none of that works, you can try getting a free 7/14 day license for BimlStudio and the 2018 release still allows for the targeting of SSIS 2008.
Now, all of that supposes you haven't used any features in SSIS 2012+ that didn't exist in 2008 (Expression tasks, Parameters, project connection managers, etc) and I doubt a Script Task/Component is going to downgrade but the good news there is you should be able to mostly copy/paste the code into the 2008 version of the object.
I am trying to deploy a Script component to SSIS toolbox in the Data Flow for reuse. I'm using visual studio 2010, and SQL server 2014.
I have created the script, Signed the assembly, used Gacutil.exe /i on the DLL file to add it. It now recedes in the GAC_MSIL folder under it's strong name folder.
The above procedure is the current way I'm trying to get it to work, but I have additionally tried to use SN utility to generate a strong name and add this key back into the build. Didn't work any better than the above procedure. Is it supposed to be next to impossible to add custom components to SSIS?
In my simple mind it should be just sign, build, GAC and done.
The editor for SSIS packages is tightly bound to the version of the SQL Server you're working with.
As a general reference, we have the following
VS 2005 = SQL Server 2005 {90}
VS 2008 = SQL Server 2008 & SQL Server 2008 R2 {100}
VS 2010, VS 2012 = SQL Server 2012 {110}
VS 2013 = SQL Server 2014 {120}
VS 2015 = SQL Server 2016 (once it's RTMed) {130}
Now, you could be using VS 2010 to build your custom DLL, as long as you've referenced the correct version of the SSIS DLLs, and then used SSDT-BI edition for VS 2013 to author your SSIS packages --- that's fine.
Making it work
Your current build and deploy cycle is what you need to do on your servers. That will allow SSIS to find the components when it runs.
However, you need the development experience so you need to also add that same DLL into the correct folder in your SQL Server installation. For SQL Server 2014, a custom "Script Component", or anything that lives in the data flow, would also need to be copied to
C:\Program Files\Microsoft SQL Server\120\DTS\PipelineComponents
If you've done it properly, then items automatically show up for VS 2010+. VS 2005/2008 required an additional step of explicitly adding tasks and components into the Toolbox.
If they aren't showing then you've either
built your DLL against the wrong version of the SSIS framework
using the wrong version of SSIS tools
not deployed the DLL to both the GAC and the correct DTS subfolder based upon your SQL Server version.
Clear as mud?
I have several custom SSIS Tasks and Components implemented for SQL Server 2008 R2. Now we want to switch to SQL Server 2012. First I tried just to open my DTSX packages in VS 2010 and no package could be loaded. Does anybody knows what I have to change in my projects that I can use my tasks and components with SQL Server 2012.
Thanks
Assuming you've installed SQL Server 2012 and SQL Server Data Tools, you should be prompted to upgrade your solution when you open it. I'm also assuming your 2008R2 assemblies are setup for signing.
After going thru the upgrade wizard you need to do several things:
Change your custom tasks and components to target the .NET 4
Framework in each of the project properties, in the Application Tab,
under Target framework.
Update the assembly references in each of your projects to point to
the appropriate SqlServer dlls.
Build and copy your project dlls to the appropriate directory,
depending on what kind of components you are updating, i.e. Program
Files (x86)\Microsoft SQL Server\110\DTS\PiplelineComponents
Program Files (x86)\Microsoft SQL Server\110\DTS\Tasks
Register your components in the GAC.
Close Visual Studio and then reopen; open your SSIS packages and
refresh the SSIS Toolbox, by right clicking in the tool box and
selected Refresh Toolbox.
You can get more detailed info here http://msdn.microsoft.com/en-us/library/ms345161.aspx
On my development machine I have Visual Studio 2010 and SQL Server 2008 Express. On our production server we have SQL Server 2008 Standard. I am going to create a WCF service that will reside on the production SQL server that will fire my SSIS package when called.
Because I have SQL Express on my development machine, I do not have access to create SSIS packages from this machine at all. This is needed so I can write the WCF service in the first place.
So, I could simply download SQL Server 2008 Standard from our msdn subscription to my development machine, but for reasons I cannot get into, this is not immediately possible. It will be in the near future but not soon enough.
How can I get this done? Are there tools to download? Where? I have researched this at length but there appears to be 5 different ways that lead to no where.
Since you have a licensed Edition of SQL Server on your network you can install the full suite of tools on your development server or your own desktop if you wish. This will give you access to BIDS software that SliverNinja mentioned.
I would say since you are limited to what can be done you only option would be to deploy a simple/basic package to your production server. If your development server can communicate with that server, I would say create your WCF service on the development box and have it simply call the package on your production server. The package does not have to do anything fancy to simply verify that you can call it correctly. Just have the package import a simple file and output it to a different file; or just export some catalog data from a database.
I do not see, since you are limited by not being able to duplicate your production environment on your dev server, that this would be unreasonable to do in this situation. If it has to get done, it has to get done.
So you want to create a package without BIDS?
Feel free to create one using plain text, there you go :)
<DTS:Executable xmlns:DTS="www.microsoft.com/SqlServer/Dts" DTS:ExecutableType="MSDTS.Package.1">
<DTS:Property DTS:Name="PackageFormatVersion">2</DTS:Property>
<DTS:Property DTS:Name="CreationDate" DTS:DataType="7">5/18/2012 1:21:47 PM</DTS:Property>
<DTS:Property DTS:Name="ProtectionLevel">1</DTS:Property>
<DTS:Property DTS:Name="DisableEventHandlers">0</DTS:Property>
.....a lot more properties.....
</DTS:Executable>
I'm joking, of course. My points are:
dtsx packages are merely XML files created by BIDS
you can create your package anywere, if you cant use your local box,
use what you have, even if that means to use your server (if that's
your only option, do it)
you dont need BIDS or even the SSIS service to run pacakges. DTEXEC,
DTEXECUI and SQl Server Agent are capable of running packages by
themselves (unless the packages are deployed to the package store)
I am not aware of any tool that builds packages other than BIDS. It
wouldn't make much sense, is like asking if there is a tool where you
can build C# application other than visual studio
You need BIDS (Business Intelligence Development Studio) to create the DTSX packages and SSIS which isn't available with SQL Express.
You also need Visual Studio to create WCF services, which it sounds like you already have.
I've got a number of SSIS packages spread across multiple projects in SQL Server Data Tools (formerly BIDS), which I want to deploy to the SSIS Catalog DB on my server.
When I want to deploy an entire project, I can just right-click the project and hit Deploy, to start the SSIS deployment wizard. However, there doesn't seem to be an option to do this for a single package only, without redeploying the rest of the packages in the same project.
Does anyone know if it's possible to deploy just a single package using the deployment wizard, independently from the others in the project?
This does not appear to be possible via the deployment wizard or SSDT. Similar discussion in this thread...where the original poster proposes a (terribly manual) workaround.
You might also want to get BIDSHelper from Codeplex (free tool). It contains a utility to allow you to deploy SSIS packages from BIDS. I haven't tested BIDS Helper with SSDT yet. Also you might want to grab SSIS Package Manager from Codeplex as well.
You can use SQL Server Management Studio to import a single package:
In SSMS, connect to your Integration Services instance.
Right-click the appropriate node (File System or MSDB) and select Import Package.
Set Package Location to File System.
Click the three dots next to Package Path. Browse to your SSDT project and select the .dtsx file you want to import.
Click OK. If prompted to overwrite the existing package, click Yes.