Suppose I have a SSIS package which is having almost 20 sequence containers and while running the job, a few sequence container got completed successfully but a few fail. So, here I want an automated SQL query to disable the completed sequence container and enable the failed sequence container so that if I run the package again, then it will run only failed sequence container.
Is there any chance to disable the completed sequence container and run the failed one?
Personally, I detest checkpoints. They don't work within the context of a loop and maybe it's just my bad luck, but I have far too often experienced a corruption of the checkpoint file
I always advise clients to design their ETL solutions with restartability in mind. At some point, the process will fail and you know it's going to be in the middle of something crucial so remove the "thinking" aspect of getting it restarted. Don't make it depend on someone running a script to clean up half committed loads, etc. But that's a different question...
Every object in your control flow will support a property called Disable. When you run a package, you can deactivate tasks via the /SET command. Given the following example, each sequence container has a Script Task that does nothing but fires an information event
When I run that from the command line,
C:\Users\bfellows\source\repos\SO_Trash\SO_Trash>dtexec /file SO_61477591.dtsx /rep i
Microsoft (R) SQL Server Execute Package Utility
Version 14.0.3238.1 for 32-bit
Copyright (C) 2017 Microsoft. All rights reserved.
Started: 10:45:43 AM
Info: 2020-04-28 10:45:43.71
Code: 0x00000000
Source: SCR0 SCR Echo Back
Description: System::ParentContainerGUID : {3910E325-F3D0-4EC6-96FD-5E125FEC0ED2}
End Info
Info: 2020-04-28 10:45:43.74
Code: 0x00000000
Source: SCR1 SCR Echo Back
Description: System::ParentContainerGUID : {08bf88d5-71c3-4105-824e-bb8b1e942b24}
End Info
Info: 2020-04-28 10:45:43.77
Code: 0x00000000
Source: SCR2 SCR Echo Back
Description: System::ParentContainerGUID : {59823604-1caf-4a90-9303-45d55126d8b3}
End Info
Info: 2020-04-28 10:45:43.80
Code: 0x00000000
Source: SCR3 SCR Echo Back
Description: System::ParentContainerGUID : {977567f1-6bcf-4c84-909e-eb18ba17165d}
End Info
DTExec: The package execution returned DTSER_SUCCESS (0).
Started: 10:45:43 AM
Finished: 10:45:43 AM
Elapsed: 0.609 seconds
By specifying the path to the executable, I can set the property disable property like /set \Package\SEQC1.Properties[Disable];"True" where SEQC1 is the case sensitive name of the object I want to disable
C:\Users\bfellows\source\repos\SO_Trash\SO_Trash>dtexec /file .\SO_61477591.dtsx /rep i /set \Package\SEQC1.Properties[Disable];"True"
Microsoft (R) SQL Server Execute Package Utility
Version 14.0.3238.1 for 32-bit
Copyright (C) 2017 Microsoft. All rights reserved.
Started: 10:45:50 AM
Info: 2020-04-28 10:45:50.62
Code: 0x00000000
Source: SCR0 SCR Echo Back
Description: System::ParentContainerGUID : {3910E325-F3D0-4EC6-96FD-5E125FEC0ED2}
End Info
Info: 2020-04-28 10:45:50.66
Code: 0x00000000
Source: SCR2 SCR Echo Back
Description: System::ParentContainerGUID : {59823604-1caf-4a90-9303-45d55126d8b3}
End Info
Info: 2020-04-28 10:45:50.69
Code: 0x00000000
Source: SCR3 SCR Echo Back
Description: System::ParentContainerGUID : {977567f1-6bcf-4c84-909e-eb18ba17165d}
End Info
DTExec: The package execution returned DTSER_SUCCESS (0).
Started: 10:45:50 AM
Finished: 10:45:50 AM
Elapsed: 0.609 seconds
Notice that SCR1 is no longer in the output due to the SET command disabling it.
Knowing how to disable tasks, you could leave the package as-is and modify your start to be text generated from SQL Queries (give me all the tasks that have completed and build the correct /SET statements)
A better approach that would require modification to the existing package would be have an Execute SQL Task(s) that runs at the start of the package and it sets Boolean SSIS variables with an indicator of whether the Task should be disabled. And then modify each object to have an expression on the Disable property that is the new SSIS variable.
Personally, my approach is to make many SSIS packages. Each package is a unit of work (load sales table). When a package fires, it records that execution has begun. When it completes successfully, it records that it has completed (as well has how many rows inserted/updated/deleted)
I then have a orchestrator package that queries the list of known packages and identifies any package that has not run successfully. It then uses a foreach loop with an Execute package task to then start running the child processes.
An advantage to my approach is that, assuming no dependencies between task 1 and 3, I could have two copies of this running - one "stack" handles the package 0 and 2, while a different stack handles 1 and 3. One of the advantages of SSIS is that it makes it trivial to parallelize your workload.
You can use CHECKPOINT file feature in SSIS to automatically run only the failed containers in the control flow task. It is available out of the box in SSIS. You can configure it in the package.
Read more on Checkpoint usage & Restart packages using checkpoint on MSDN
Related
Note: I am not an SSIS developer and have a very limited knowledge on it, so may be missing on very small details, please read carefully.
I have an existing SSIS package which has script task in one of the steps.
I had to update one single value in script task, so I opened it in notepad++ and updated the value and saved it. Now when the scheduled SSIS job runs, it still picks up the old value and not the updated one.
I realized script task is a VB file, so probably would need to re-compile it for the binaries. I read about it and found out, if you open it through VS and save it, VS automatically compiles it and you don't have to do anything further. I can be wrong here.
I can't do project deployment on the server, so I would just need the updated package file to replace it on the server.(I am taking this updated package file from the Visual studio project that I created to edit the SSIS package. I am not sure, if there is another way to get the package or even if this is a right package.
Now the problem is,
when I updated the old package using notepad++, it didn't throw any error when it was running through scheduled job, but it took the old value
Now when I am putting the new package that I edited using Visual studio, it throws below error
Executed as user: sampleUser(nameupdated). Microsoft (R) SQL Server Execute Package Utility Version 13.0.5026.0 for 32-bit Copyright (C) 2016 Microsoft. All rights reserved. Started: 5:08:09 PM Error: 2021-11-02 17:08:10.83 Code: 0xC0016016 Source: OraclePackage Description: Failed to decrypt protected XML node "DTS:Password" with error 0x8009000B "Key not valid for use in specified state.". You may not be authorized to access this information. This error occurs when there is a cryptographic error. Verify that the correct key is available. End Error Error: 2021-11-02 17:08:11.26 Code: 0x00000005 Source: Rename files in archive folder to _yyyymmddhhmm Rename files in archive folder to _yyyymmddhhmm Description: Failed to migrate scripts contained in the package to the VSTA 14.0 format. Move the scripts to a new Script task. End Error Error: 2021-11-02 17:08:14.19 Code: 0x00000005 Source: Script Task Script Task Description: Failed to migrate scripts contained in the package to the VSTA 14.0 format. Move the scripts to a new Script task. End Error Error: 2021-11-02 17:08:17.17 Code: 0x00000001 Source: Script Task Description: Exception has been thrown by the target of an invocation. End Error DTExec: The package execution returned DTSER_FAILURE (1). Started: 5:08:09 PM Finished: 5:08:17 PM Elapsed: 7.25 seconds. The package execution failed. The step failed.
I read about this error and found that package's security level can cause this issue, so update the protection level of the package from "EncryptSensitiveWithUserKey" to "DontSaveSensitive"
Now when I am running the project I get below error
Executed as user: sampleuser(nameUpdated). Microsoft (R) SQL Server Execute Package Utility Version 13.0.5026.0 for 32-bit Copyright (C) 2016 Microsoft. All rights reserved. Started: 5:31:44 PM Error: 2021-11-02 17:31:53.93 Code: 0x00000001 Source: Script Task Description: Exception has been thrown by the target of an invocation. End Error DTExec: The package execution returned DTSER_FAILURE (1). Started: 5:31:44 PM Finished: 5:31:53 PM Elapsed: 9.047 seconds. The package execution failed. The step failed.
Please let me know if there is another way to edit the script task directly and compile it in place so that it takes the updated values else, if there is something else I can do.
Thanks in advance.
I found a solution(more of an error or less knowledge from my side actually)
Updating the security from "EncryptSensitiveWithUserKey" to "DontSaveSensitive"
When you open an existing SSIS package in Visual studio, the read and write variables are wiped out from script task for some reason. Reassigning them resolved the problem.
How did I find out if that was the problem, I put some exception handling in the script task to notify me of further issues.
Hope this helps to someone in the similar problem.
I am trying to run a Windows scheduler task using schtasks.exe in an SSIS job. The task will be called using the same ID as is used to execute the task. The task itself works; It is currently started manually after the SSIS job is done.
I've tried running a script on my own PC with a space in the name and this worked. All other settings are the same. The name of this task is "DWH Upload DEV" and it's location is set to "\".
I'm calling the script from an Execute Process Task block, using schtasks.exe as the executable (with the full file path of course) and /RUN /TN "DWH Upload DEV" as the arguments.
I have RequireFullFileName set to False, FailTaskIfReturnCodeIsNotSuccessValue set to True and SuccessValue and TimeOut to 0
I am receiving the following error: [Execute Process Task] Error: In Executing "[location]\schtasks.exe" "/RUN /TN "DWH Upload DEV"" at "", The process exit code was "1" while the expected was "0".
The part here that confuses me is the error being at seemingly nothing.
Thanks in advance for any help!
EDIT:
I can't run the program that is performed by the scheduler from SSIS itself, due to the program needing to be run by a specific user. The use of task scheduler is a workaround for this.
So, as it turned out, the task DID trigger but it didn't actually properly work (I later heard the task stopped working about an hour before I started working with it). The task can now properly be run by using the following settings:
Executable: C:\Windows\System32\schtasks.exe
Arguments: /RUN /TN "DWHUploadDEV" (the new version has no spaces in the name)
The rest of the settings is as normal.
I've got this question within a project in ssis. I had to divide the original project into different packages because this was to big and sometimes it could provoke some problems with the memory.
So, in order to link the different packages I'm using the "Execute Package Task" to refer one to each other.
If I execute the package directly from SSIS it works perfect, there's not problem.
But if I use a scheduler to program the time of execution I'm getting this error message:
Error: 2015-09-22 14:54:37.98 Code: 0xC00220E6 Source: Execute Package
Task Description: There is no project to reference. End Error Error:
2015-09-22 14:54:37.99 Code: 0xC0024107 Source: Execute Package Task
Description: There were errors during task validation. End Error
DTExec: The package execution returned DTSER_FAILURE (1).
I wonder what can be happening with the project and its execution.
Regards
I'm not sure what scheduler you are using, or how it's configured, but you can debug your way through this by simulating the scheduler with DTExec. It sounds like you've crossed your Setup and Execution Method (see below for those definitions).
Here's the summary.
Option 1: For the Setup, use Project References and for the Execution Method, use Project/Package.
Option 2: For the Setup, for each child package, use External References and for Execution Method, use File.
(It sounds like you're using a combination of Project References and File, which in turn sends the There is no project to reference error off of the child packages.)
Option 1
Setup
Open your parent package in SSDT, and then double click a child package. It should look like this:
Execution Method
This setting means you need to execute the package via the Project/Package method via DTExec. So build your project - this generates an ispac file. And to execute via dtexec, it would look like:
dtexec /Proj Path\To\MyProject.ispac /Pack Path\To\The\ParentPackage.dtsx
Note: If you specify the dtsproj file instead of the ispac file in the /Proj parameter, you will receive a File contains corrupted data error!
Option 2
Setup
Open your parent package in SSDT, and then double click a child package. Change it to look like this:
This is done by
1. Changing the Reference Type to External Reference
2. Changing the Location to File system (SQL Server is another option)
3. Select <New connection...> to create a new file connection for the child package to run (or a SQL server connection)
Execution Method
This setting means you can now use the File method, which is likely the way you're attempting to execute the package.
dtexec /f Path\To\My\ParentPackage.dtsx
I two SSIS packages that both run fine on my local machine. I can import them to SSIS on the server and execute them both with no problems. However, I need to run them in series every night at 3AM. Thus, I created a SQL Server Agent Job with two steps. The first step runs the first package successfully. The second step should run the second package but it fails with this error message:
Message
Executed as user: IT-DEV\itdev. Microsoft (R) SQL Server Execute Package Utility Version 11.0.2100.60 for 64-bit Copyright (C)
Microsoft Corporation. All rights reserved. Started: 1:27:21 PM
Error: 2013-05-30 13:27:21.53 Code: 0x00000001 Source: Create
text files on IT DEV Description: Exception has been thrown by
the target of an invocation. End Error DTExec: The package execution
returned DTSER_FAILURE (1). Started: 1:27:21 PM Finished: 1:27:21
PM Elapsed: 0.375 seconds. The package execution failed. The step
failed.
Both packages move some files from one directory to another and do some work. They both run in the same security context using the same proxy. I can see the work completed successfully from step 1, but again step 2 fails. I have looked through all possible step settings for each and see no differences. Thus, how can I get a more descriptive message so I know what the real issue is. I'm guessing its permissions but again both packages do similar work in similar locations requiring the same permissions. One runs, the other doesn't. Please help. Thanks in advance.
FYI I've also turned SSIS logging on and write everything to a file but the message below is the only error I see.
UPDATE
I ended up writing to a text file all over inside the script task to try to identify the problem. I seem to be getting stuck here:
string connStr = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + myPath + ";Extended Properties=\"text;HDR=NO;FMT=TabDelimited\"";
OleDbDataAdapter adapter = new OleDbDataAdapter("SELECT * FROM MYFILE.txt", connStr);
outfile.WriteLine("Adapter Created");
DataTable dtFiles = new DataTable();
adapter.Fill(dtFiles);
outfile.WriteLine("Filled DataTable");
I never see "Filled DataTable" in the file I'm writing to... the last entry says "Adapter Created".
Turns out I needed to replace this: Microsoft.Jet.OLEDB.4.0 with this: Microsoft.ACE.OLEDB.12.0 because I'm running the agent job on Windows Server 2008 64 bit.
Is there someway to reduce the detail level that is output when running a DTEXEC job ?
e.g. in the cmd window I've got loads of ...
Progress: 2012-05-10 12:24:20.70
Source: Import Email_to
Post Execute: 9% complete
End Progress
Progress: 2012-05-10 12:24:20.70
Source: Import Email_to
Post Execute: 18% complete
End Progress
Progress: 2012-05-10 12:24:20.70
Source: Import Email_to
Post Execute: 27% complete
End Progress
Progress: 2012-05-10 12:24:20.71
Source: Import Email_to
Post Execute: 36% complete
End Progress
Progress: 2012-05-10 12:24:20.71
Source: Import Email_to
Post Execute: 45% complete
End Progress
.... etc ...
You can use the flag /Reporting with level N (N stands for No reporting.) to prevent any status being reported to the command line console.
There are other levels like E for only Errors, W for only Warning. You can pick the choice of output that you would like to see on the console.
Refer MSDN to know more about dtexec Utility.
Syntax usage:
dtexec /f Package.dtsx /Reporting N
Alternative usage:
dtexec /f Package.dtsx /Rep N
Example:
Created a simple package with one empty data flow task.
Following screenshot shows when the package was executed using dtexec utility in the command line without the flag /Reporting. You can notice the status of data flow task is being displayed.
Following screenshot shows when the package was executed using dtexec utility in the command line with the flag /Reporting. You can notice the status of data flow task is not being displayed.
What I manage to do so far, to eliminate some lines produced at the stdout was:
DTExec.exe | findstr /v /c:"Microsoft (R)" | findstr /v /c:"Copyright (C)" | findstr /v /c:"Version 10"
and the output looks like that:
Started: 1:40:22 PM
Error: 2018-09-05 13:40:22.78
...
End Error
Warning: 2018-09-05 13:40:22.79
...
End Warning
DTExec: The package execution returned DTSER_FAILURE (1).
Started: 1:40:22 PM
Finished: 1:40:22 PM
Elapsed: 0.265 seconds
I wish to have the awk tool on every Windows. I will try to do something more with what we have on our Windows nodes: cscript.exe and some JavaScript, as I'm usually do with awk and sed.
I see no way to suppress not wanted part of the output from DTExec.exe with /ConsoleLog and /Reporting options - the only wanted parts from the stdout below are "Error:" and "Warning" blocks. Why?!?
I will have to create a tool which will be put in the pipeline after DTExec.exe and will behave like "grep -v", to filter out all the noises which should be added to the log files. The only problem is that this is a poor Windows Server, which doesn't have powerful, standard and easy to use GNU tools like "grep", and I will have to use something ugly, what every Windows Server has by default already installed (scrtipt.exe maybe).
Microsoft (R) SQL Server Execute Package Utility
Version 10.50.2500.0 for 32-bit
Copyright (C) Microsoft Corporation 2010. All rights reserved.
Started: 1:40:22 PM
Error: 2018-09-05 13:40:22.78
...
End Error
Warning: 2018-09-05 13:40:22.79
...
End Warning
DTExec: The package execution returned DTSER_FAILURE (1).
Started: 1:40:22 PM
Finished: 1:40:22 PM
Elapsed: 0.265 seconds