I am building out a new SQL Server 2019 database and using SSIS 2019 to import data from various sources. Most of my SSIS packages are randomly failing on the server (but not when I run them from Visual Studio), but then will run successfully, and then fail again. The failure error can change from failure to failure on the same package.
To simplify things, I built a brand new Test package that executes a stored procedure to see if I could isolate the problem. The Test package has only the one SQL task and the Connection Manager.
The Test package also randomly fails but is always for the same reason: cannot find the Connection, which it lists by the GUID. It will run several times, then fail once, sometimes twice, then run 5 or 6 times in a row, then fail. Off and on.
The packages are deployed to Integration Services Catalog and I'm calling them using Autosys to run the DTEXEC command.
Any ideas on what would cause a random failure like this?
Related
We have a package that will be triggered through MVC application by invoking store procedure which creates a execution request for the Package. MVC app is hosted on different server and DB is on different server. This error started occurring after 3 months of Windows Server 2019 migration.
Package executes successfully most of the time, but fails once or twice per day with the following errors. After re-executing the package immediately it succeeds. Error occurs in the below order.
Illegal operation attempted on a registry key that has been marked for deletion
Failed to acquire connection. Connection may not be configured correctly or you may not have the right permissions on this connection.
Upon capturing the "ISServexec.exe" process dump and analyzed through DebugDiag, we came to know the below details.
System.Runtime.InteropServices.COMException: Retrieving the COM class factory for component with CLSID {2206CDB2-19C1-11D1-89E0-00C04FD7A829} failed due to the following error: 800703fa Illegal operation attempted on a registry key that has been marked for deletion. (Exception from HRESULT: 0x800703FA).
System.InvalidOperationException: The .Net Framework Data Providers require Microsoft Data Access Components(MDAC). Please install Microsoft Data Access Components(MDAC) version 2.6 or later.
System.Runtime.InteropServices.COMException: Illegal operation attempted on a registry key that has been marked for deletion.
Microsoft.SqlServer.Dts.Runtime.DtsGenericException: Illegal operation attempted on a registry key that has been marked for deletion.
Note:
Package doesn't have any excel reading operations. Just source table to target table load (both same database).
We have validated the MDAC version in the DB server as 6.3.
App Server: Windows 2019 with .Net framework 4.8 installed.
DB Server: Windows 2019, SQL Server 2019, SSIS v15, .Net framework 4.8.
Package executed using domain service account.
We have triggered the Package through SQL job and Package did not fail at all. Almost 200 executions tested. This failure is random and only fails when store procedure invoked through MVC application.
We have used ProcMon to collect the "ISServexec.exe" logs and only thing we noticed is hive unloaded and no errors.
SQL Profiler didnt even capture the error.
Package is not failing at the same step when this error occurs.
Package logging is configured to write errors in event log, but in event log only "package is failed" information is captured.
Package configured to use Oledb connection type. SQL Native client and MSOLEDB providers are used and error occurs in both the providers.
Package has been upgraded to SQL 2019. Hence we created a new Package in Visual Studio 2019 from scratch and this also fails with same error.
No errors captured in SQL Server extended events.
My Questions:
What are possible ways to validate the service account holds necessary permissions when the above error happens. Is this required ?
Is there any tools that I can use to debug and find the cause of the error or additional information ?
Is this oledb driver issue ?
How to capture any logs at windows server level when package execution happens.
Sorry for the detailed explanation. Any suggestions would be greatly appreciated.
I am currently migrating a database from an old SQL Server 2012 to a new system running SQL Server 2019. In addition to the database I am also migrating several SQL Server Agent Jobs.
One of these jobs consists of multiple different steps, one that just runs a PowerShell script and two other steps that are implemented as an SSIS package. When I execute one of the steps that are implemented as SSIS package, I get the following error:
The process could not be created for step 2 of job <long hexadecimal number> (reason: The system cannot find the file specified). The step failed.
How am I supposed to diagnose this error? I have no idea which file cannot be found and why. Is there no way to get any additional information about this error? I already checked all files involved in the SSIS package such as the package file itself, the configuration file and all files that are accessed during the execution of the step but I cannot find the reason for this error.
Could there be a compatibility issue since the packages are executed in a new SQL Server 2019 instead of the older SQL Server 2012? If so, how can I check if this is actually the case?
Thanks a lot in advance!
Having the exact same issue.
Found a workaround, don't like it, but it did get the package run.
Change "Type" from SQL Server Integration Services Package" to "Transact-SQL script (T-SQL)"
Make your SQL Statement something like:
EXEC xp_cmdshell '""C:\Program Files\Microsoft SQL Server\150\DTS\Binn\DTExec.exe" /FILE "\"<Path To Package>\<PackageName>.dtsx\"" /WARNASERROR /X86 /CHECKPOINTING OFF /REPORTING E'
Note that the start of the command '"" is NOT a type-o, neither is the \"" right after the .dtsx package name.
Don't like this answer, but until the correct solution comes along... it is working.
For the past week, multiple SSIS packages running on SQL Server Agent that load data into Snowflake have started returning the follow message randomly.
"Failed to acquire connection "snowflake". Connection may not be configured correctly or you may not have the right permissions on this connection."
We are seeing this message across multiple jobs and each of the jobs is loading multiple tables and its not happening on each call to Snowflake within the projects, but just on one or two tasks in jobs that have 100s.
We are using the 2.20.2 drivers from Snowflake
We have ran the jobs while WireShark was capturing network traffic and were received by the network team. They didn't have much luck because the ACK messages were not being shown.
We also ran Process Monitor while the jobs ran and we did not find anything that alluded to any issues
We also dug though the logs from the Snowflake driver and found the calls right before and right after, but no messages for the task that failed. Since those logs bounce around on which file they are sending to, its a bit hard to track sequential actions when multiple task on a job are running together.
We also installed SnowCD and ran it and it returned a full success message.
The user that runs the jobs on SQL Server Agent is an Admin on the server and has SysAdmin rights on the Sql Sever instance.
The warehouse the drivers are connected to are a size Large with a max of 3 clusters (was at 1 when the issue started, but upped it to 3 to see if that helped)
Jobs are running on Windows Server 2016 DataCenter in Azure
SQL Server instance is Sql Sever 2016 13.0.4604.0
We cannot figure out why we are suddenly and randomly using connection to Snowflake.
Some ideas to help get these packages working:
Add a retry to the tasks that are failing. The task would move onto the next step only upon success:
https://www.mssqltips.com/sqlservertip/5625/how-to-retry-sql-server-integration-services-ssis-control-flow-tasks/
You can also combine the truncate and insert into one step using the insert overwrite into command which will allow your package to run quicker and have one less task for failure:
https://docs.snowflake.net/manuals/sql-reference/sql/insert.html#insert-using-overwrite
Once the SSIS packages are consistently completing, you can analyze the logs at the point of failure to see if there is any pattern to help you identify the root cause.
I am running an SSIS package created with SSDT for Visual Studio 2015. The package has a project level connection to an AS400 DB, and my admin put a specific username and password into the environment variables on SQL Server 2016. The password is sensitive. I receive intermittent errors when running the package:
I have the package set to restart up to 3 times on failure, and it always runs successfully the next time. Also, it does not always fail on the same container. I am the only user who uses this username/password combination, and it does not expire. The package runs every 10 minutes for 8 hours, but the errors only occur once or twice per day.
I have read numerous posts about running packages in 32 bit mode. I'm not sure if this is what I need to do. I would appreciate any help. Let me know if more detail is required.
I have a SSIS 2005 package that is up and running in our production environment. The package uses an SMTP Connection Manager to send an e-mail message out to a designated user. We have a scheduled job that executes this package, and also overrides the SMTP connection string so that the package can target the test or production mail server, which makes it possible to keep a single service on both our test and production servers, just configured differently.
We recently changed the server name of our production mail server and went into the scheduled job and changed the command prompt values that run to point to the new server. However, the next morning the job failed and the error log indicated that the job tried to connect to the old mail server.
Is there something I'm missing to updating the SSIS package parameters? Do I have to delete the existing package, and then reimport and reschedule the job again in order for the new server change to hold?
The dba's from where I work had a similar issue. They had to change a job's running parameters, but it seems like running the job with modified parameters only worked the first time they ran it. After that run, it kept using the old values in subsequent runs. They had to repackage the damned thing.
If you are using the package configuration feature? It can be picky on the order of execution. There's some more info here: http://msdn.microsoft.com/en-us/library/ms141132.aspx