In our software, we have a few JUnit tests that occasionally hang during execution due to race conditions in the tests. When one of these hangs occurred last night, I just killed the forked Java process that was running the test. The tests continued running as expected until the build completed.
After the build, I got an error message (I was running with --continue) about having killed Gradle Test Executor xxx. When I went to look at the test results, I discovered that the HTML test report wasn't there, and neither were the XML test result files.
From this I have two questions: How can I generate the test reports without rerunning all of the tests (they take a night to run)? How can I gracefully handle a hung test without effecting the build output?
Related
I am building out a new SQL Server 2019 database and using SSIS 2019 to import data from various sources. Most of my SSIS packages are randomly failing on the server (but not when I run them from Visual Studio), but then will run successfully, and then fail again. The failure error can change from failure to failure on the same package.
To simplify things, I built a brand new Test package that executes a stored procedure to see if I could isolate the problem. The Test package has only the one SQL task and the Connection Manager.
The Test package also randomly fails but is always for the same reason: cannot find the Connection, which it lists by the GUID. It will run several times, then fail once, sometimes twice, then run 5 or 6 times in a row, then fail. Off and on.
The packages are deployed to Integration Services Catalog and I'm calling them using Autosys to run the DTEXEC command.
Any ideas on what would cause a random failure like this?
I am receiving the following error in SSIS "An OLE DB error has occurred. Error code: 0x80004005" for each of my data flow tasks.
When I set 'delay validation' to 'True' for all data flow tasks and execute my packages the integration works okay.
However the SQL agent job doesn't run.
As far as I can tell, the reason for this is due to the 'to_update' temporary tables I have set up to act as a middle man. The Microsoft article below seems to back this up.
https://support.microsoft.com/en-us/topic/error-message-when-an-ssis-package-runs-that-is-scheduled-to-run-as-a-sql-server-agent-job-an-ole-db-error-has-occurred-error-code-0x80004005-6a687a1f-917a-d3ae-4d3a-44e7dae82988
As the article says my next step would be to 'change the permissions for the Temp directory of the SQL Server Agent Service startup account. Grant the Read permission and the Write permission to the SQL Server 2005 Agent proxy account for this directory.' however I honestly have no idea where I would do this (I'm new to the world of SSIS!)
If someone could point me in the right direction that would be much appreciated.
There are a few things to take into account, the delay validation might be a red herring and nothing to do with the error or it might be that the Agent is calling an older version of the package that doesn't delay validation. So first of all make sure the package with the delay validation is deployed correctly and it is the one being run by SQL Agent.
Then, if the problem continues it could be permission or that the Temp directory is running out of space, worth checking it.
Finally, if it comes to permission problems then whoever is in charge of maintaining the DB, I'm assuming there is a DBA, should check what account is running the package from SQL Agent and make sure it has the right permissions.
another thing to check:
make sure the Run as is correct and if not running as 32 bit try it or viceversa.
After spending some more time it looked like the issue was actually related to disk space, the memory on the server I was using was running at 100% and the SQL agent jobs were running but very slowly and never actually ran successfully. I ended several processes and staggered when each of the SQL jobs run so they do not all trigger at once, and these are running successfully again.
For the past week, multiple SSIS packages running on SQL Server Agent that load data into Snowflake have started returning the follow message randomly.
"Failed to acquire connection "snowflake". Connection may not be configured correctly or you may not have the right permissions on this connection."
We are seeing this message across multiple jobs and each of the jobs is loading multiple tables and its not happening on each call to Snowflake within the projects, but just on one or two tasks in jobs that have 100s.
We are using the 2.20.2 drivers from Snowflake
We have ran the jobs while WireShark was capturing network traffic and were received by the network team. They didn't have much luck because the ACK messages were not being shown.
We also ran Process Monitor while the jobs ran and we did not find anything that alluded to any issues
We also dug though the logs from the Snowflake driver and found the calls right before and right after, but no messages for the task that failed. Since those logs bounce around on which file they are sending to, its a bit hard to track sequential actions when multiple task on a job are running together.
We also installed SnowCD and ran it and it returned a full success message.
The user that runs the jobs on SQL Server Agent is an Admin on the server and has SysAdmin rights on the Sql Sever instance.
The warehouse the drivers are connected to are a size Large with a max of 3 clusters (was at 1 when the issue started, but upped it to 3 to see if that helped)
Jobs are running on Windows Server 2016 DataCenter in Azure
SQL Server instance is Sql Sever 2016 13.0.4604.0
We cannot figure out why we are suddenly and randomly using connection to Snowflake.
Some ideas to help get these packages working:
Add a retry to the tasks that are failing. The task would move onto the next step only upon success:
https://www.mssqltips.com/sqlservertip/5625/how-to-retry-sql-server-integration-services-ssis-control-flow-tasks/
You can also combine the truncate and insert into one step using the insert overwrite into command which will allow your package to run quicker and have one less task for failure:
https://docs.snowflake.net/manuals/sql-reference/sql/insert.html#insert-using-overwrite
Once the SSIS packages are consistently completing, you can analyze the logs at the point of failure to see if there is any pattern to help you identify the root cause.
Hi we have a SSIS package that calls an execute process task.
The task accesses a saved sql file in the directory, the agent that runs our task does have permission to access all these files. Yet when the package is executed it hangs at the execute process task, no processes are running but the package still is, all the previous tasks complete successfully until it reaches the execute process task
Hi if anyone is looking for the same problem we came across the answer, the cmd window required user input and therefore could not continue with the process until the cmd process task has finished and it never does.
So in short make sure your cmd process doesnt require user input if you are deploying to SSISDB or it will hang and you cant do anything about it
I have a problem with a hang using xp_cmdshell.
The executable is called, performs its work, and exits. It is not hanging because of a ui prompt in the exe. The exe is not hanging at all. The exe disappears from the process list in task manager, and internal logging from the exe confirms that it executed the very last line in the main function
the call to xp_cmdshell does NOT return control in SQL. It hangs on that line (it is the last line of the batch). Killing the process is ineffective. It actually requires a restart of sql server to get rid of the hung process (ugh)
The hang only happens the first time it is run. Subsequent calls to the procedure with identical parameters work and exit correctly so long as the first one is hung. Once SQL is restarted, the first subsequent call will hang again.
If it makes any difference, I am trying to receive the return value from the exe -- my sql procedure ends with:
exec #i = xp_cmdshell #cmd;
return #i;
Activity Monitor is reporting the process to be stuck on a wait type of PREEMPTIVE_OS_PROCESSOPS (what the other developer saw) or PREEMPTIVE_OS_PIPEOPS (what I'm seeing on my current testing)
Any ideas?
Just came across this situation myself where I've run an invalid comment via xp_cmdshell.
I managed to kill it without restarting SQL, what I've done was to identify the process that run the command and kill it from Task Manager.
Assume your SQL was running in Windows 2008 upward:
Under Task Manager, Processes tab. I enabled the column to show Command Line of each process (e.g.: View -> Select Columns..).
If you unsure what command you've run via xp_cmdshell, dbcc inputbuffer(SPID) should give you a clue.
We had the same issue, with SQL Server 2008, also with calls involving xp_cmdshell and BCP. Killing the sql process ID didn't help, it would just stay stuck in "KILLED/ROLLBACK" status.
Only way to kill it was to kill the bcp.exe process in Windows task manager.
In the end we traced the issue down to wrong SQL in sproc that was calling the xp_cmdshell. It was by mistake opening multiple transactions in a loop and not closing them. After BEGIN/COMMIT trans issues were fixed, PREEMPTIVE_OS_PROCESSOPS never came back again.
We actually did eventually figure out the problem here. The app being called was used to automatically dump some documents to a printer when certain conditions happened.
It turns out that a particular print driver popped up a weird little window in the notification tray on a print job. So it was hanging because of a ui window popping up -- but our app was exiting properly because it wasn't our window, it was a window triggered by the print driver.
That driver included an option to turn off that display window. Our problem went away when that option was set.