I am using SSIS for ETL and I need to monitor a source folder for the source file to be arrived. When ever a file arrives I need to move that file into another location and rename the file and start executing another SSIS Package. Here we don't have an option to use use any other tool to automate the execution. We have only choice to use SQL Server, SSIS.
I need the mechanism and the logic to implement this logic.
I'm assuming that by "File Watcher" you don't mean FileSystemWatcher class in .NET, as there wouldn't be any point in using this class if you're limited to SQL Server and SSIS (you'd need a job with forever-running SSIS package containing ScriptTask with FileSystemWatcher).
The only solution is to create two-step job. First step would contain SSIS package for reading directory content and comparing it to files history log. Second step would contain your main package and would execute only if first steps succeeds or returns value indicating that there are new files to process.
Your answer is here and here. My personal favorite way of doing it is having an infinite loop package. Yet another way of doing it would be to encapsulate the entire logic in an SSIS package and fire it every X minutes. Vary the value of X depending on the urgency.
Related
I have hundreds of BIML scripts and I have to convert each into SSIS package. The only process I figured out is to manually right click the biml file and convert it click the Generate SSIS Package. (Please follow the link to visualize it). How do I automate this process. In other words, how can I programmatically convert all the biml scripts into their corresponding SSIS packages..
https://www.google.com/url?sa=i&url=http%3A%2F%2Fwww.erikhudzik.com%2Ftag%2Fssis%2F&psig=AOvVaw3vHH8scEdHu5w-JUDrHyLi&ust=1657797254349000&source=images&cd=vfe&ved=0CAkQjRxqFwoTCLDFmZfe9fgCFQAAAAAdAAAAABAR
You should be able to select multiple files, right click and generate them all at the same time.
You can also reference one biml script from another. So you can have your main entry point which contains a <packages> element and then reference other scripts within that which define each package.
Finally, if you have biml studio, this comes with a command line utility which would allow you to do it programmatically.
There are 3 tools for transforming Biml into DTSX packages: BimlExpress, BimlStudio (formerly known as Mist) and the precursor to BimlExpress - BidsHelper. That product no longer exists and the Biml bits in it and has been rebranded.
Under the covers, BimlStudio is going to invoke Bimlc.exe which is the Biml Compiler and that is how scripts become packages. Buy it outright or rent it monthly, depending on your needs. This is your only choice for unattended/automatic/automated builds.
BimlExpress is the free tool that can also transform scripts into packages. It requires mouse clicks to build packages.
The big difference between the two, for the beginner at least, is convenience. I have ScriptedPackageA and ScriptedTablesB which makes Package1. In BimlStudio, I can set the properties so that one is live (tables) and just evaluate/expand the Package script. In BimlExpress, I need to shift/control click the scripts I want to be compiled/referenced.
Also if you have hundreds of Biml scripts... you might not have understood the idea behind Biml. For reference, I have about about 7000 .biml files on my machine but I bet I have less than 30 describing package patterns. The only way I'm at my large number of Biml files is that I have scripted a number of databases with an file per table.
Generally speaking, you want to distill your approach down to distinct patterns and then throw your metadata against it. How many ways can you have a package that loads from file to database?
I have a production SSIS project in SQL Server 2016 that creates and exports a flat file to another server. The destination server has reached end-of-life and I need to change the destination path to the new server so we can decommission the old server. Can I just edit the package or project in Visual Studio or do I need to recompile (redeploy? republish?)? I have never edited before, only created new projects however that was a few years ago and I am a little rusty.
Alternatively, I could copy the existing job, edit the copy, then run them in parallel first. Then I can disable the old project/package once I am confident the new one works. I'm not having much luck figuring out how to do this either.
Any help would be appreciated. Thanks!
If I'm understanding your question correctly, it sounds like your goal is to simply change the destination location for your report. You would need to update the connection manager for your flat file connection inside of your SSIS package.
You would need to edit your flat file connection manager's connection string (by loading up the connection manager properties and changing the value yourself or by changing the expressions, if it is parameterized). Once you have verified it is now pointing at the right server's location to save the report you are generating, you would rebuild the entire package and redeploy it, which would essentially overwrite the .ispac and .dtsx files on the deployment server with your updates.
Standard caveat of "it depends" but the most common case is that you can solve this through Configuration.
Right click on the Package (or Project) depending on how things are set up. In the "Connection Managers" tab, find the connection manager that corresponds to the flat file output (a strong naming standard helps). I have selected SO_61794511.dtsx and the Name is Flat File Conn... which then allows the right side menu to be populated.
Of interest here is ConnectionString. I am going to directly edit this to change from C:\ssisdata\input\so_61794511.txt to my new path D:\path\here\something\so_newthing.txt
Click OK 2x and the next time the package runs, it will use the configured value.
That's the easiest approach. You could accomplish a similar thing if you edit the job that runs the package to set the value at every execution but this just does it at a global scale.
Where this can go off the rails is if there's a expression applied to the ConnectionString property, e.g. the output file has a dynamic date in the file name. This is why I advocate for exposing Package or Project level parameters of a "base file path" concept. This allows me to change the path from C: (local development) to D: (server deployment) or even to a UNC path \server\share by setting a configuration instead of hard coding a path into the packages themselves.
I am trying to zip excel files using SSIS Process Task. This task asks for Executable. Unfortunately on the machine where I am executing SSIS, does not have any 3rd party provided compression utility. All I have is Windows built in compressor. I want to know how this can be used as executable and zip my files.
Thank you for your time.
You could use 7zip's command line version without installing it.
http://www.7-zip.org/download.html
Also, if your excel files are outputed as '.xlsx' they are already compressed. This is also true for any other office type that ends in an 'x'. Try changing the extension to '.zip' and open them.
Edit: Depending on how much you're doing you might want to consider buying the Cozyroc package which includes a zip task as well as many other useful ones. Not too costly either.
http://cozyroc.com/
I just want to add this note about cozyroc
You can test and develop with the SSIS+ components for as long as needed to determine if the components are suitable for your needs.
When you run the SSIS package from the command line with DTEXEC or SQL Server Agent, the license verification code is activated.
If a license is not found, the package execution will fail, the first time it tries to use SSIS+ library component.
I'm trying to programmatically add an Execute SQL task from within a script task of my SSIS package. I know that the Microsoft.SqlServer.Dts.Runtime.Package class has an Executables collection which I can add my new task to but how to I get a reference to the package Im inside of?
It is not possible, SSIS does not support self-modifying packages.
To prevent any attempts of doing what you are trying to do, the task code does not have access to the package API, so you can't obtain reference to Package object from a task. But even if you find a way to curcumvent this - the results are not predictable, as package is not allowed to modify itself at runtime.
If you could describe what you really want to achieve (rather than asking for particular way to do it) - someone might find a way to do it. Maybe you can use child package - it is OK to modify child package before its execution, or maybe it is enough to just change some variables that are used by Execute SQL task later in this package?
As Michael says above, it's not possible to do what you are asking. However, you may be able to find a solution by setting variables at runtime, or by enabling or disabling certain packages at runtime using dtexec. For example:
dtexec /f e:\ssis\master.dtsx /set \Package\YourPackageName.Disable;True
For my requirement I need to write Script task from One SSIS package to another. As my server is 64bit machine, when I execute the overriden SSIS package, it throws the error "Binary Script not found". Hence I need to compile the script programmatically in the Parent SSIS package and put the binary code into the another one. I dont know how to compile the Script programmatically.
Let me know how to do that
I don't think you can do this with SSIS. If there's a way to accomplish this using a Parent Package / Child Package architecture, I'm unaware of it.
You can get more flexibility creating packages dynamically in .Net. Although I haven't tried to dynamically compile script task code.
I'm curious: Why not copy the code out of the parent package and paste it into a child package?
:{> Andy
So you need to somehow programatically execute a (Debug>Build). This typically gets around the problem of binary not being found, when you do it manually in VS. Doesn't strike me as possible at runtime though.