Is there someway to reduce the detail level that is output when running a DTEXEC job ?
e.g. in the cmd window I've got loads of ...
Progress: 2012-05-10 12:24:20.70
Source: Import Email_to
Post Execute: 9% complete
End Progress
Progress: 2012-05-10 12:24:20.70
Source: Import Email_to
Post Execute: 18% complete
End Progress
Progress: 2012-05-10 12:24:20.70
Source: Import Email_to
Post Execute: 27% complete
End Progress
Progress: 2012-05-10 12:24:20.71
Source: Import Email_to
Post Execute: 36% complete
End Progress
Progress: 2012-05-10 12:24:20.71
Source: Import Email_to
Post Execute: 45% complete
End Progress
.... etc ...
You can use the flag /Reporting with level N (N stands for No reporting.) to prevent any status being reported to the command line console.
There are other levels like E for only Errors, W for only Warning. You can pick the choice of output that you would like to see on the console.
Refer MSDN to know more about dtexec Utility.
Syntax usage:
dtexec /f Package.dtsx /Reporting N
Alternative usage:
dtexec /f Package.dtsx /Rep N
Example:
Created a simple package with one empty data flow task.
Following screenshot shows when the package was executed using dtexec utility in the command line without the flag /Reporting. You can notice the status of data flow task is being displayed.
Following screenshot shows when the package was executed using dtexec utility in the command line with the flag /Reporting. You can notice the status of data flow task is not being displayed.
What I manage to do so far, to eliminate some lines produced at the stdout was:
DTExec.exe | findstr /v /c:"Microsoft (R)" | findstr /v /c:"Copyright (C)" | findstr /v /c:"Version 10"
and the output looks like that:
Started: 1:40:22 PM
Error: 2018-09-05 13:40:22.78
...
End Error
Warning: 2018-09-05 13:40:22.79
...
End Warning
DTExec: The package execution returned DTSER_FAILURE (1).
Started: 1:40:22 PM
Finished: 1:40:22 PM
Elapsed: 0.265 seconds
I wish to have the awk tool on every Windows. I will try to do something more with what we have on our Windows nodes: cscript.exe and some JavaScript, as I'm usually do with awk and sed.
I see no way to suppress not wanted part of the output from DTExec.exe with /ConsoleLog and /Reporting options - the only wanted parts from the stdout below are "Error:" and "Warning" blocks. Why?!?
I will have to create a tool which will be put in the pipeline after DTExec.exe and will behave like "grep -v", to filter out all the noises which should be added to the log files. The only problem is that this is a poor Windows Server, which doesn't have powerful, standard and easy to use GNU tools like "grep", and I will have to use something ugly, what every Windows Server has by default already installed (scrtipt.exe maybe).
Microsoft (R) SQL Server Execute Package Utility
Version 10.50.2500.0 for 32-bit
Copyright (C) Microsoft Corporation 2010. All rights reserved.
Started: 1:40:22 PM
Error: 2018-09-05 13:40:22.78
...
End Error
Warning: 2018-09-05 13:40:22.79
...
End Warning
DTExec: The package execution returned DTSER_FAILURE (1).
Started: 1:40:22 PM
Finished: 1:40:22 PM
Elapsed: 0.265 seconds
Related
Note: I am not an SSIS developer and have a very limited knowledge on it, so may be missing on very small details, please read carefully.
I have an existing SSIS package which has script task in one of the steps.
I had to update one single value in script task, so I opened it in notepad++ and updated the value and saved it. Now when the scheduled SSIS job runs, it still picks up the old value and not the updated one.
I realized script task is a VB file, so probably would need to re-compile it for the binaries. I read about it and found out, if you open it through VS and save it, VS automatically compiles it and you don't have to do anything further. I can be wrong here.
I can't do project deployment on the server, so I would just need the updated package file to replace it on the server.(I am taking this updated package file from the Visual studio project that I created to edit the SSIS package. I am not sure, if there is another way to get the package or even if this is a right package.
Now the problem is,
when I updated the old package using notepad++, it didn't throw any error when it was running through scheduled job, but it took the old value
Now when I am putting the new package that I edited using Visual studio, it throws below error
Executed as user: sampleUser(nameupdated). Microsoft (R) SQL Server Execute Package Utility Version 13.0.5026.0 for 32-bit Copyright (C) 2016 Microsoft. All rights reserved. Started: 5:08:09 PM Error: 2021-11-02 17:08:10.83 Code: 0xC0016016 Source: OraclePackage Description: Failed to decrypt protected XML node "DTS:Password" with error 0x8009000B "Key not valid for use in specified state.". You may not be authorized to access this information. This error occurs when there is a cryptographic error. Verify that the correct key is available. End Error Error: 2021-11-02 17:08:11.26 Code: 0x00000005 Source: Rename files in archive folder to _yyyymmddhhmm Rename files in archive folder to _yyyymmddhhmm Description: Failed to migrate scripts contained in the package to the VSTA 14.0 format. Move the scripts to a new Script task. End Error Error: 2021-11-02 17:08:14.19 Code: 0x00000005 Source: Script Task Script Task Description: Failed to migrate scripts contained in the package to the VSTA 14.0 format. Move the scripts to a new Script task. End Error Error: 2021-11-02 17:08:17.17 Code: 0x00000001 Source: Script Task Description: Exception has been thrown by the target of an invocation. End Error DTExec: The package execution returned DTSER_FAILURE (1). Started: 5:08:09 PM Finished: 5:08:17 PM Elapsed: 7.25 seconds. The package execution failed. The step failed.
I read about this error and found that package's security level can cause this issue, so update the protection level of the package from "EncryptSensitiveWithUserKey" to "DontSaveSensitive"
Now when I am running the project I get below error
Executed as user: sampleuser(nameUpdated). Microsoft (R) SQL Server Execute Package Utility Version 13.0.5026.0 for 32-bit Copyright (C) 2016 Microsoft. All rights reserved. Started: 5:31:44 PM Error: 2021-11-02 17:31:53.93 Code: 0x00000001 Source: Script Task Description: Exception has been thrown by the target of an invocation. End Error DTExec: The package execution returned DTSER_FAILURE (1). Started: 5:31:44 PM Finished: 5:31:53 PM Elapsed: 9.047 seconds. The package execution failed. The step failed.
Please let me know if there is another way to edit the script task directly and compile it in place so that it takes the updated values else, if there is something else I can do.
Thanks in advance.
I found a solution(more of an error or less knowledge from my side actually)
Updating the security from "EncryptSensitiveWithUserKey" to "DontSaveSensitive"
When you open an existing SSIS package in Visual studio, the read and write variables are wiped out from script task for some reason. Reassigning them resolved the problem.
How did I find out if that was the problem, I put some exception handling in the script task to notify me of further issues.
Hope this helps to someone in the similar problem.
Suppose I have a SSIS package which is having almost 20 sequence containers and while running the job, a few sequence container got completed successfully but a few fail. So, here I want an automated SQL query to disable the completed sequence container and enable the failed sequence container so that if I run the package again, then it will run only failed sequence container.
Is there any chance to disable the completed sequence container and run the failed one?
Personally, I detest checkpoints. They don't work within the context of a loop and maybe it's just my bad luck, but I have far too often experienced a corruption of the checkpoint file
I always advise clients to design their ETL solutions with restartability in mind. At some point, the process will fail and you know it's going to be in the middle of something crucial so remove the "thinking" aspect of getting it restarted. Don't make it depend on someone running a script to clean up half committed loads, etc. But that's a different question...
Every object in your control flow will support a property called Disable. When you run a package, you can deactivate tasks via the /SET command. Given the following example, each sequence container has a Script Task that does nothing but fires an information event
When I run that from the command line,
C:\Users\bfellows\source\repos\SO_Trash\SO_Trash>dtexec /file SO_61477591.dtsx /rep i
Microsoft (R) SQL Server Execute Package Utility
Version 14.0.3238.1 for 32-bit
Copyright (C) 2017 Microsoft. All rights reserved.
Started: 10:45:43 AM
Info: 2020-04-28 10:45:43.71
Code: 0x00000000
Source: SCR0 SCR Echo Back
Description: System::ParentContainerGUID : {3910E325-F3D0-4EC6-96FD-5E125FEC0ED2}
End Info
Info: 2020-04-28 10:45:43.74
Code: 0x00000000
Source: SCR1 SCR Echo Back
Description: System::ParentContainerGUID : {08bf88d5-71c3-4105-824e-bb8b1e942b24}
End Info
Info: 2020-04-28 10:45:43.77
Code: 0x00000000
Source: SCR2 SCR Echo Back
Description: System::ParentContainerGUID : {59823604-1caf-4a90-9303-45d55126d8b3}
End Info
Info: 2020-04-28 10:45:43.80
Code: 0x00000000
Source: SCR3 SCR Echo Back
Description: System::ParentContainerGUID : {977567f1-6bcf-4c84-909e-eb18ba17165d}
End Info
DTExec: The package execution returned DTSER_SUCCESS (0).
Started: 10:45:43 AM
Finished: 10:45:43 AM
Elapsed: 0.609 seconds
By specifying the path to the executable, I can set the property disable property like /set \Package\SEQC1.Properties[Disable];"True" where SEQC1 is the case sensitive name of the object I want to disable
C:\Users\bfellows\source\repos\SO_Trash\SO_Trash>dtexec /file .\SO_61477591.dtsx /rep i /set \Package\SEQC1.Properties[Disable];"True"
Microsoft (R) SQL Server Execute Package Utility
Version 14.0.3238.1 for 32-bit
Copyright (C) 2017 Microsoft. All rights reserved.
Started: 10:45:50 AM
Info: 2020-04-28 10:45:50.62
Code: 0x00000000
Source: SCR0 SCR Echo Back
Description: System::ParentContainerGUID : {3910E325-F3D0-4EC6-96FD-5E125FEC0ED2}
End Info
Info: 2020-04-28 10:45:50.66
Code: 0x00000000
Source: SCR2 SCR Echo Back
Description: System::ParentContainerGUID : {59823604-1caf-4a90-9303-45d55126d8b3}
End Info
Info: 2020-04-28 10:45:50.69
Code: 0x00000000
Source: SCR3 SCR Echo Back
Description: System::ParentContainerGUID : {977567f1-6bcf-4c84-909e-eb18ba17165d}
End Info
DTExec: The package execution returned DTSER_SUCCESS (0).
Started: 10:45:50 AM
Finished: 10:45:50 AM
Elapsed: 0.609 seconds
Notice that SCR1 is no longer in the output due to the SET command disabling it.
Knowing how to disable tasks, you could leave the package as-is and modify your start to be text generated from SQL Queries (give me all the tasks that have completed and build the correct /SET statements)
A better approach that would require modification to the existing package would be have an Execute SQL Task(s) that runs at the start of the package and it sets Boolean SSIS variables with an indicator of whether the Task should be disabled. And then modify each object to have an expression on the Disable property that is the new SSIS variable.
Personally, my approach is to make many SSIS packages. Each package is a unit of work (load sales table). When a package fires, it records that execution has begun. When it completes successfully, it records that it has completed (as well has how many rows inserted/updated/deleted)
I then have a orchestrator package that queries the list of known packages and identifies any package that has not run successfully. It then uses a foreach loop with an Execute package task to then start running the child processes.
An advantage to my approach is that, assuming no dependencies between task 1 and 3, I could have two copies of this running - one "stack" handles the package 0 and 2, while a different stack handles 1 and 3. One of the advantages of SSIS is that it makes it trivial to parallelize your workload.
You can use CHECKPOINT file feature in SSIS to automatically run only the failed containers in the control flow task. It is available out of the box in SSIS. You can configure it in the package.
Read more on Checkpoint usage & Restart packages using checkpoint on MSDN
I have a ssis job scheduled to run.
It executes a procedure and then selects from a table to output to a file.
The ssis package runs fine in ssis and manually running the dtsx file.
When it runs as a job, all of it completes fine except the part where it tries to save to a file.
Here's the error ( i removed hostname\username)
Executed as user: {hostnamexxx\usernamexxx}. Microsoft (R) SQL Server
Execute Package Utility
Version 10.0.1600.22 for 32-bit
Copyright (C) Microsoft Corp 1984-2005. All rights reserved.
Started: 2:33:29 PM
Error: 2014-09-30 14:36:23.57
Code: 0xC020200E
Source: Data Flow Task Flat File Destination [22]
Description: Cannot open the datafile "P:\\Ordbsjia_ORDER_0003637.DAT".
End Error
Error: 2014-09-30 14:36:23.57
Code: 0xC004701A
Source: Data Flow Task SSIS.Pipeline
Description: component "Flat File Destination" (22) failed the pre-execute phase and returned error code 0xC020200E.
End Error
DTExec: The package execution returned DTSER_FAILURE (1).
Started: 2:33:29 PM
Finished: 2:36:23 PM
Elapsed: 174.422 seconds. The package execution failed. The step failed.
I two SSIS packages that both run fine on my local machine. I can import them to SSIS on the server and execute them both with no problems. However, I need to run them in series every night at 3AM. Thus, I created a SQL Server Agent Job with two steps. The first step runs the first package successfully. The second step should run the second package but it fails with this error message:
Message
Executed as user: IT-DEV\itdev. Microsoft (R) SQL Server Execute Package Utility Version 11.0.2100.60 for 64-bit Copyright (C)
Microsoft Corporation. All rights reserved. Started: 1:27:21 PM
Error: 2013-05-30 13:27:21.53 Code: 0x00000001 Source: Create
text files on IT DEV Description: Exception has been thrown by
the target of an invocation. End Error DTExec: The package execution
returned DTSER_FAILURE (1). Started: 1:27:21 PM Finished: 1:27:21
PM Elapsed: 0.375 seconds. The package execution failed. The step
failed.
Both packages move some files from one directory to another and do some work. They both run in the same security context using the same proxy. I can see the work completed successfully from step 1, but again step 2 fails. I have looked through all possible step settings for each and see no differences. Thus, how can I get a more descriptive message so I know what the real issue is. I'm guessing its permissions but again both packages do similar work in similar locations requiring the same permissions. One runs, the other doesn't. Please help. Thanks in advance.
FYI I've also turned SSIS logging on and write everything to a file but the message below is the only error I see.
UPDATE
I ended up writing to a text file all over inside the script task to try to identify the problem. I seem to be getting stuck here:
string connStr = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + myPath + ";Extended Properties=\"text;HDR=NO;FMT=TabDelimited\"";
OleDbDataAdapter adapter = new OleDbDataAdapter("SELECT * FROM MYFILE.txt", connStr);
outfile.WriteLine("Adapter Created");
DataTable dtFiles = new DataTable();
adapter.Fill(dtFiles);
outfile.WriteLine("Filled DataTable");
I never see "Filled DataTable" in the file I'm writing to... the last entry says "Adapter Created".
Turns out I needed to replace this: Microsoft.Jet.OLEDB.4.0 with this: Microsoft.ACE.OLEDB.12.0 because I'm running the agent job on Windows Server 2008 64 bit.
I'm having an issue scheduling a .dtsx package to execute as a SQL Server Agent job. I've Googled the problem, but can't find the required solution for my particular issue.
The SQL Server version is 2012 Business Intelligence Edition.
The package runs fine within Visual Studio.
The job is set to run under the "SQL Server Agent Service Account".
The file is on the same server as the DB.
The error log in part is:
...complete End Progress Progress: 2013-01-09 20:51:48.58
Source: Inheader File Import Validating: 33% complete End
Progress Progress: 2013-01-09 20:51:48.58 Source: Inheader File
Import Validating: 66% complete End Progress Progress:
2013-01-09 20:51:48.59 Source: Inheader File Import
Validating: 100% complete End Progress Progress: 2013-01-09
20:51:48.59 Source: Product File Import ... The command line
parameters are invalid... The step failed.
This looks similar to the log generated when running the package within VS, so it doesn't appear to be a permissions issue - it looks to be processing the script.
I haven't changed the command line parameters from those generated when configuring the step, they are(I've altered sensitive data):
/FILE "\"C:\...filepath...\package.dtsx\"" /CHECKPOINTING OFF /REPORTING E
A lot of the links I have found don't really mention the file type packages, and I'm a bit stuck so any help would be much appreciated.
I'm also confused as to why the error wouldn't occur prior to the execution of the package.
Assuming this is accurate:
/FILE "\"C:\...filepath...\package.dtsx\"" /CHECKPOINTING OFF /REPORTING E
then the backslash and quote before the start of the path, and the backslash and quote after the .dtsx extention, are the likely culprits.
Things should look more like this:
/FILE "C:\...filepath...\package.dtsx" /CHECKPOINTING OFF /REPORTING E