For the past week, multiple SSIS packages running on SQL Server Agent that load data into Snowflake have started returning the follow message randomly.
"Failed to acquire connection "snowflake". Connection may not be configured correctly or you may not have the right permissions on this connection."
We are seeing this message across multiple jobs and each of the jobs is loading multiple tables and its not happening on each call to Snowflake within the projects, but just on one or two tasks in jobs that have 100s.
We are using the 2.20.2 drivers from Snowflake
We have ran the jobs while WireShark was capturing network traffic and were received by the network team. They didn't have much luck because the ACK messages were not being shown.
We also ran Process Monitor while the jobs ran and we did not find anything that alluded to any issues
We also dug though the logs from the Snowflake driver and found the calls right before and right after, but no messages for the task that failed. Since those logs bounce around on which file they are sending to, its a bit hard to track sequential actions when multiple task on a job are running together.
We also installed SnowCD and ran it and it returned a full success message.
The user that runs the jobs on SQL Server Agent is an Admin on the server and has SysAdmin rights on the Sql Sever instance.
The warehouse the drivers are connected to are a size Large with a max of 3 clusters (was at 1 when the issue started, but upped it to 3 to see if that helped)
Jobs are running on Windows Server 2016 DataCenter in Azure
SQL Server instance is Sql Sever 2016 13.0.4604.0
We cannot figure out why we are suddenly and randomly using connection to Snowflake.
Some ideas to help get these packages working:
Add a retry to the tasks that are failing. The task would move onto the next step only upon success:
https://www.mssqltips.com/sqlservertip/5625/how-to-retry-sql-server-integration-services-ssis-control-flow-tasks/
You can also combine the truncate and insert into one step using the insert overwrite into command which will allow your package to run quicker and have one less task for failure:
https://docs.snowflake.net/manuals/sql-reference/sql/insert.html#insert-using-overwrite
Once the SSIS packages are consistently completing, you can analyze the logs at the point of failure to see if there is any pattern to help you identify the root cause.
Related
I am building out a new SQL Server 2019 database and using SSIS 2019 to import data from various sources. Most of my SSIS packages are randomly failing on the server (but not when I run them from Visual Studio), but then will run successfully, and then fail again. The failure error can change from failure to failure on the same package.
To simplify things, I built a brand new Test package that executes a stored procedure to see if I could isolate the problem. The Test package has only the one SQL task and the Connection Manager.
The Test package also randomly fails but is always for the same reason: cannot find the Connection, which it lists by the GUID. It will run several times, then fail once, sometimes twice, then run 5 or 6 times in a row, then fail. Off and on.
The packages are deployed to Integration Services Catalog and I'm calling them using Autosys to run the DTEXEC command.
Any ideas on what would cause a random failure like this?
I am receiving the following error in SSIS "An OLE DB error has occurred. Error code: 0x80004005" for each of my data flow tasks.
When I set 'delay validation' to 'True' for all data flow tasks and execute my packages the integration works okay.
However the SQL agent job doesn't run.
As far as I can tell, the reason for this is due to the 'to_update' temporary tables I have set up to act as a middle man. The Microsoft article below seems to back this up.
https://support.microsoft.com/en-us/topic/error-message-when-an-ssis-package-runs-that-is-scheduled-to-run-as-a-sql-server-agent-job-an-ole-db-error-has-occurred-error-code-0x80004005-6a687a1f-917a-d3ae-4d3a-44e7dae82988
As the article says my next step would be to 'change the permissions for the Temp directory of the SQL Server Agent Service startup account. Grant the Read permission and the Write permission to the SQL Server 2005 Agent proxy account for this directory.' however I honestly have no idea where I would do this (I'm new to the world of SSIS!)
If someone could point me in the right direction that would be much appreciated.
There are a few things to take into account, the delay validation might be a red herring and nothing to do with the error or it might be that the Agent is calling an older version of the package that doesn't delay validation. So first of all make sure the package with the delay validation is deployed correctly and it is the one being run by SQL Agent.
Then, if the problem continues it could be permission or that the Temp directory is running out of space, worth checking it.
Finally, if it comes to permission problems then whoever is in charge of maintaining the DB, I'm assuming there is a DBA, should check what account is running the package from SQL Agent and make sure it has the right permissions.
another thing to check:
make sure the Run as is correct and if not running as 32 bit try it or viceversa.
After spending some more time it looked like the issue was actually related to disk space, the memory on the server I was using was running at 100% and the SQL agent jobs were running but very slowly and never actually ran successfully. I ended several processes and staggered when each of the SQL jobs run so they do not all trigger at once, and these are running successfully again.
I am running an SSIS package created with SSDT for Visual Studio 2015. The package has a project level connection to an AS400 DB, and my admin put a specific username and password into the environment variables on SQL Server 2016. The password is sensitive. I receive intermittent errors when running the package:
I have the package set to restart up to 3 times on failure, and it always runs successfully the next time. Also, it does not always fail on the same container. I am the only user who uses this username/password combination, and it does not expire. The package runs every 10 minutes for 8 hours, but the errors only occur once or twice per day.
I have read numerous posts about running packages in 32 bit mode. I'm not sure if this is what I need to do. I would appreciate any help. Let me know if more detail is required.
I have an SSIS package which contains an Analysis Services Processing Task.
This package is kicked off by a SQL Server Job in SQL Server 2008 R2.
If I run this job myself or process the cube manually everything is fine.
However if I schedule the job and let the SQL server agent run it then the Analysis Services Processing task fails stating Errors in the OLAP storage engine and that an error occurred while processing the one of the measure groups.
Has anyone else every seen anything like this?
The SQL Server Agent service account may not have sufficient permissions. You can validate this by doing any of the following:
Add the service account to the Administrators group on the analysis services server to validate this issue. Let the job run as scheduled.
Create a proxy that runs under your credentials and set the job to execute under the proxy. Let the job run as scheduled.
Change the SQL Server Agent to use your credentials. Let the job run as scheduled.
If the job completes successfully after making any of the above changes, then you have a permission issue that you need to resolve.
So after months of looking at this I finally realised the answer; of course it's a simple one.
The SSIS job I had created was processing the Cube only, while every time I processed manually in management studio I was processing the whole SSAS database.
I've now changed the SSIS package to process the whole database and everything seems to be working correctly.
when This Error Generate :
reporting services catalog database file existence failed sql server 2008 r2
================================================================================
This issue occurs because the databases for the SQL Server 2008 Reporting Services instance that you want to install already exist on the computer with following path.
C:\Program Files\Microsoft SQL Server\MSSQL10.MSSQLSERVER\MSSQL\DATA
Manually Remove .MDF and .LDF files of Report from above path and Re-run Setup.
In my case I found the solution in other way. First of all I login into analysus services using SQL management studio.Second of all I searched into databases folder the name of my project, after I open that and double clic in Roles folder, clic in new role, in this window I selected Membership in the left pane, clic in add button and search the NT AUTHORITY\Service, clic in Ok button and finally restart Sql analysis services. I tried to run my job and It works.
At the start of the year I pushed to setting up TFS for a more structured approach to things (before, everyone would change things as they went, obviously A Bad Thing). I set up a very basic single server TFS 2010 installation. The TFS databases resided on one of our Dev servers (SQL 2008).
Everything went well until:
We uninstalled SQL 2008, installed SQL 2008 R2 and reattached the databases. Since then TFS has been impossible:
The clients (SQL Mgt Studio and VS2008/2010) could no longer connect (error 404 not found)
http://localhost:8080/tfs/ gave:
"Team Foundation services are not available from the server.
Technical information (for administrator):
The request could not be processed because the application is not configured correctly. No service host is available for the request."
Team Foundation Admin Console finds the collections, everything SEEMS ok.
In an effort to jumpstart things:
I restarted the website and it's application pool
I rebooted the server
No effect.
Then I stopped the collection (that worked) to re-enter the database information, save it and start the collection again. However, it kept hanging on the save. I tried to detach the collection, but that didn't do anything. So now I have a stopped collection with the following activities:
Prepare Collection (Success)
Create Collection (Success)
Servicing Collection (Queued)
Detach Collection (Queued) (3 times, since I tried this a couple of time)
and nothing is budging.
I have all source in my local folder, so in extremis I can delete and uninstall the whole thing and start over, but... I rather not.
Any way to unblock this?
ok, This was solved by re-adding the TFS machine account to the new SQL Server installation using
EXEC master.dbo.sp_grantlogin #loginame = N'DOMAIN\MACHINE$'
as detailed here. From then on all tasks proceeded as they should..
What tipped me off was the following error in the Application Log:
TF53010: The following error has
occurred in a Team Foundation
component or extension: Date (UTC):
22/06/2011 18:07:22 Machine: AZT-TS-02
Application Domain: TfsJobAgent.exe
Assembly:
Microsoft.TeamFoundation.Framework.Server,
Version=10.0.0.0, Culture=neutral,
PublicKeyToken=b03f5f7f11d50a3a;
v2.0.50727 Service Host: Process
Details: Process Name: TFSJobAgent
Process Id: 2980 Thread Id: 3804
Account name: NT AUTHORITY\NETWORK
SERVICE
Detailed Message: There was an error
during job agent execution. The
operation will be retried. Similar
errors in the next five minutes may
not be logged. Exception Message:
TF246017: Team Foundation Server could
not connect to the database. Verify
that the server that is hosting the
database is operational, and that
network problems are not blocking
communication with the server. (type
DatabaseConnectionException)
Good times,
Try running the following command:
TFSConfig registerDB /DatabaseName:Tfs_Configuration /SQLInstance:SERVERNAME /Continue
RegisterDB updates the name of the server that hosts the configuration database and in this case should resolve your DB issues. Another command you could try is RemapDBs.
Make sure you "Run As Admin" for these commands or they of course will not work.
I am guessing what is going on is attaching isn't going to be enough because TFS internal mappings no longer understands where your SQL Server db is.
Hope that helps.