I am running an SSIS package created with SSDT for Visual Studio 2015. The package has a project level connection to an AS400 DB, and my admin put a specific username and password into the environment variables on SQL Server 2016. The password is sensitive. I receive intermittent errors when running the package:
I have the package set to restart up to 3 times on failure, and it always runs successfully the next time. Also, it does not always fail on the same container. I am the only user who uses this username/password combination, and it does not expire. The package runs every 10 minutes for 8 hours, but the errors only occur once or twice per day.
I have read numerous posts about running packages in 32 bit mode. I'm not sure if this is what I need to do. I would appreciate any help. Let me know if more detail is required.
Related
I am building out a new SQL Server 2019 database and using SSIS 2019 to import data from various sources. Most of my SSIS packages are randomly failing on the server (but not when I run them from Visual Studio), but then will run successfully, and then fail again. The failure error can change from failure to failure on the same package.
To simplify things, I built a brand new Test package that executes a stored procedure to see if I could isolate the problem. The Test package has only the one SQL task and the Connection Manager.
The Test package also randomly fails but is always for the same reason: cannot find the Connection, which it lists by the GUID. It will run several times, then fail once, sometimes twice, then run 5 or 6 times in a row, then fail. Off and on.
The packages are deployed to Integration Services Catalog and I'm calling them using Autosys to run the DTEXEC command.
Any ideas on what would cause a random failure like this?
For the past week, multiple SSIS packages running on SQL Server Agent that load data into Snowflake have started returning the follow message randomly.
"Failed to acquire connection "snowflake". Connection may not be configured correctly or you may not have the right permissions on this connection."
We are seeing this message across multiple jobs and each of the jobs is loading multiple tables and its not happening on each call to Snowflake within the projects, but just on one or two tasks in jobs that have 100s.
We are using the 2.20.2 drivers from Snowflake
We have ran the jobs while WireShark was capturing network traffic and were received by the network team. They didn't have much luck because the ACK messages were not being shown.
We also ran Process Monitor while the jobs ran and we did not find anything that alluded to any issues
We also dug though the logs from the Snowflake driver and found the calls right before and right after, but no messages for the task that failed. Since those logs bounce around on which file they are sending to, its a bit hard to track sequential actions when multiple task on a job are running together.
We also installed SnowCD and ran it and it returned a full success message.
The user that runs the jobs on SQL Server Agent is an Admin on the server and has SysAdmin rights on the Sql Sever instance.
The warehouse the drivers are connected to are a size Large with a max of 3 clusters (was at 1 when the issue started, but upped it to 3 to see if that helped)
Jobs are running on Windows Server 2016 DataCenter in Azure
SQL Server instance is Sql Sever 2016 13.0.4604.0
We cannot figure out why we are suddenly and randomly using connection to Snowflake.
Some ideas to help get these packages working:
Add a retry to the tasks that are failing. The task would move onto the next step only upon success:
https://www.mssqltips.com/sqlservertip/5625/how-to-retry-sql-server-integration-services-ssis-control-flow-tasks/
You can also combine the truncate and insert into one step using the insert overwrite into command which will allow your package to run quicker and have one less task for failure:
https://docs.snowflake.net/manuals/sql-reference/sql/insert.html#insert-using-overwrite
Once the SSIS packages are consistently completing, you can analyze the logs at the point of failure to see if there is any pattern to help you identify the root cause.
I run a series of DTS packages via SQL Server (2000) Agent every night. They have been running ok for years until I added a couple of transform data tasks that pick up info from another local SQL Server (2008).
The odd thing is that the tasks run ok for about a week and then fail with the error 'Login failed for user domain\$duplicate-1889'. I don't run these packages using this user account, in fact I don't think it's a real account. I have a Windows account called dbadmin which has been set up on both servers to have access. This dbadmin account is used to start up the SQL Server Agent service and run the scheduled jobs.
When the jobs fail, all have to do to get them to run again is to stop and restart SQL Server Agent. But after a week they fail again with the same issue. Also, it's not the same day of the week each time.
I checked the Windows event log and although I can see the error, I can't figure out its cause. Any help or pointers would be appreciated.
The domain\$duplicate-1889 name looks like machine account. Every machine joined to a domain has such an account in the domain that the machine uses to authenticate to the domain.
The passwords for those accounts are automatically handled and the password is periodically changed automatically. By default that interval is 30 days but if is changed to once a week in your environment maybe there's a problem when the computer's password has been updated?
It is possible to disable the automatic changing maybe that's worth trying to see if the problem dissapears.
There is a Master package as below to call and run all child packages in my SSIS solution by Project Reference. Also, all connection managers are set at project level since all packages are sharing the same ones.
When manually running, everything works well from master to last child. However, when the master package is executed from a SQL Agent, it shows error as follows:
Code: 0xC001000E Source: Master
Description: The connection "{5A827D76-916C-4F22-ADE9-266ABBEB1E37}" is not found.
This error is thrown by Connections collection when the specific connection element is not found.
Then I checked the connection ID and found it is an OLE DB connection manager in the first child package as follows:
I'm not sure if I should replace project connections with package level to fix this issue. Hopefully someone could help me out. Thanks in advance!
Some more info that might be helpful:
All connection managers are connect to SQL-Server 2012 and my visual studio version is 2010.
An SSIS proxy is created with the same windows account of packages for SQL Agent to execute. It also has full control to all DBs used in solution.
SQL Agent is set as below:
At the start of the year I pushed to setting up TFS for a more structured approach to things (before, everyone would change things as they went, obviously A Bad Thing). I set up a very basic single server TFS 2010 installation. The TFS databases resided on one of our Dev servers (SQL 2008).
Everything went well until:
We uninstalled SQL 2008, installed SQL 2008 R2 and reattached the databases. Since then TFS has been impossible:
The clients (SQL Mgt Studio and VS2008/2010) could no longer connect (error 404 not found)
http://localhost:8080/tfs/ gave:
"Team Foundation services are not available from the server.
Technical information (for administrator):
The request could not be processed because the application is not configured correctly. No service host is available for the request."
Team Foundation Admin Console finds the collections, everything SEEMS ok.
In an effort to jumpstart things:
I restarted the website and it's application pool
I rebooted the server
No effect.
Then I stopped the collection (that worked) to re-enter the database information, save it and start the collection again. However, it kept hanging on the save. I tried to detach the collection, but that didn't do anything. So now I have a stopped collection with the following activities:
Prepare Collection (Success)
Create Collection (Success)
Servicing Collection (Queued)
Detach Collection (Queued) (3 times, since I tried this a couple of time)
and nothing is budging.
I have all source in my local folder, so in extremis I can delete and uninstall the whole thing and start over, but... I rather not.
Any way to unblock this?
ok, This was solved by re-adding the TFS machine account to the new SQL Server installation using
EXEC master.dbo.sp_grantlogin #loginame = N'DOMAIN\MACHINE$'
as detailed here. From then on all tasks proceeded as they should..
What tipped me off was the following error in the Application Log:
TF53010: The following error has
occurred in a Team Foundation
component or extension: Date (UTC):
22/06/2011 18:07:22 Machine: AZT-TS-02
Application Domain: TfsJobAgent.exe
Assembly:
Microsoft.TeamFoundation.Framework.Server,
Version=10.0.0.0, Culture=neutral,
PublicKeyToken=b03f5f7f11d50a3a;
v2.0.50727 Service Host: Process
Details: Process Name: TFSJobAgent
Process Id: 2980 Thread Id: 3804
Account name: NT AUTHORITY\NETWORK
SERVICE
Detailed Message: There was an error
during job agent execution. The
operation will be retried. Similar
errors in the next five minutes may
not be logged. Exception Message:
TF246017: Team Foundation Server could
not connect to the database. Verify
that the server that is hosting the
database is operational, and that
network problems are not blocking
communication with the server. (type
DatabaseConnectionException)
Good times,
Try running the following command:
TFSConfig registerDB /DatabaseName:Tfs_Configuration /SQLInstance:SERVERNAME /Continue
RegisterDB updates the name of the server that hosts the configuration database and in this case should resolve your DB issues. Another command you could try is RemapDBs.
Make sure you "Run As Admin" for these commands or they of course will not work.
I am guessing what is going on is attaching isn't going to be enough because TFS internal mappings no longer understands where your SQL Server db is.
Hope that helps.