We are trying to copy the data from on-premise system to Azure Data Warehouse. We want to use SSIS package for copy data from on premise to cloud. Azure Data Factory v1 does not support SSIS package and Azure Data factory v2 supports SSIS packages but it does not supported by Data Warehouse.
So My Question is, How to import data from On-premise database to Azure Data Warehouse using Azure Data Factory SSIS package?
Thanks in Advance,
Vinir Shah
I'm afraid we can't copy the data from on-premise system to Azure Data Warehouse with SSIS package running in Data Factory.
To copy the data from on-premise system to Azure Data Warehouse, we could do with two ways:
Run the SSIS package local: Load data into Azure SQL Data Warehouse with SQL Server Integration Services (SSIS).
Using Copy activity to copy the data from on-premise system to Azure
Data Warehouse with self-host integration runtime.
Data Factory SSIS-integration runtime can't get data from the on-premise SQL server. I tested run the package in Data Factory and always get the error bellow:
Error: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80004005. An OLE DB record is available. Source: "Microsoft SQL Server Native Client 11.0" Hresult: 0x80004005 Description: "Login timeout expired". An OLE DB record is available. Source: "Microsoft SQL Server Native Client 11.0" Hresult: 0x80004005 Description: "A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online.". An OLE DB record is available. Source: "Microsoft SQL Server Native Client 11.0" Hresult: 0x80004005 Description: "Named Pipes Provider: Could not open a connection to SQL Server [53]. ".
As we know about Data Factory, only the self-host integration runtime can help us connect to on-premise data source.
Hope this helps.
We have 2 servers, Server01 and Server02.
Scheduler for executing the SSIS package has been created in Server01. We are retrieving some tables from Server02 to Server01 via linked server. And those tables were created as views in Server01.
One table from server02 to server01 has approximately 43 million records. We are using this view in our script in Execute SQL task, it gets automatically failed with below error message, while manual execution or via scheduler.
Error Message:
OLEDB provided "linked server name" for linked server "server name" returned message "Protocal error in TDS stream.
OLEDB provided "linked server name" for linked server "server name" returned message "Communication link failure"
Physical connection is not usable (0xffffff)
Semaphore timeout period has expired
I am trying to connect my web api to an existing database using this command by going in tools-> nuget package manager -> nuget package manager console
Scaffold-DbContext "Server=(localdb)\mssqllocaldb;Database=Blogging;Trusted_Connection=True;" Microsoft.EntityFrameworkCore.SqlServer -OutputDir Models
But it says
A network-related or instance-specific error occurred while
establishing a connection to SQL Server. The server was not found or
was not accessible. Verify that the instance name is correct and that
SQL Server is configured to allow remote connections. (provider: Named
Pipes Provider, error: 40 - Could not open a connection to SQL Server)
It's supposed to get all tables from this database and show them in models, I've done this before and forgotten the procedure, I'm just trying to connect it to database first so I an copy the rest of the code and get the API working
Here is my latest effort at revising this question. But this time, I am trying to follow the good counsel given by Oded in his article Getting good answers on StackOverflow.
I need to find out how I can determine the root cause for the following error:
Communication link failure
TCP Provider: The specified network name is no longer available
From time to time, I am seeing this error when running a set of SSIS packages. This error can occur when one to many packages are run from:
A SQL Server Agent Job
A batch file
In debug mode from BIDS
The full error message I see is as follows:
SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80004005.
An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80004005 Description: "Communication link failure".
An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80004005 Description: "TCP Provider: The specified network name is no longer available.
".
SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80004005.
An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80004005 Description: "Protocol error in TDS stream".
An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80004005 Description: "Communication link failure".
An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80004005 Description: "TCP Provider: An existing connection was forcibly closed by the remote host."
This is an overview of how I have designed the ETL process:
Two servers
Both are virtual machines
The SSIS packages run on an application server
The SQL Server database lives on a database server
I use an OLE DB connection manager to connect from the SSIS package on the application server to the SQL Server database on the database server.
The packages run as a file system deployment on the application server and not as a database deployment on the database server.
The main reason for this is that the ETL is integrated with a set of tools no found on and drives not accessible to the database server. These tools include Apex Data Loader for Salesforce and pgAdmin III.
So far I cannot consistently reproduce this error. However, this is what I have observed:
Failure occurs more frequently during regular business hours
Failure occurs less frequently during off hours
For about a two hour period on a Friday morning I was able to successfully reproduce the error on a specific package.
The error occurred during a large data flow if a child package call that precedes the large data flow was enabled.
The error did not occur during the same large data flow if the child package call that precedes the large data flow was disabled.
The child package in question calls back to the database to retrieve a tiny amount of information for use in an email body and then sends the email.
It feels like maybe a resource limit is being exceeded?
Maybe a connection limit?
I am wondering what tools I should be using to try and determine the root cause of the error.
Technical details about the two servers involved are listed below:
SQL Server and Database Server info:
Microsoft SQL Server 2008 R2 (SP1) - 10.50.2500.0 (X64) Jun 17 2011 00:54:03 Copyright (c) Microsoft Corporation Enterprise Edition (64-bit) on Windows NT 6.1 (Build 7601: Service Pack 1) (Hypervisor)
SSIS info:
Microsoft Visual Studio 2008 Version 9.0.30729.1 SP
Microsoft .NET Framework Version 3.5 SP1
Application Server info:
OS name: Microsoft Windows Server 2008 R2 Standard
Version: 6.1.7601 Service Pack 1 Build 7601
I have researched the error message online and found these, but would really like to get an expert's insight before proceeding:
How to Disable TCP Chimney, TCPIP Offload Engine (TOE) or TCP Segmentation Offload (TSO).
Using Netsh Commands to Enable or Disable TCP Chimney Offload
Any help is appreciated.
Thanks
UPDATE:
Further testing shows that this is not "an SSIS thing" as the same error is seen at the same rate when using SQL Server Management Studio. The complexity of the query does not make the error more or less likely. In an attempt to resolve, we have tried one fix (below):
#1 How to Disable TCP Chimney, TCPIP Offload Engine (TOE) or TCP Segmentation Offload (TSO).
This was our first attempt. TCP Chimney is now disabled on the Application Server and the Database Server. Testing shows that the same error occurs at the same rate.
So where to go from here? Honestly I am not sure. One seemingly good option remains:
Application Server and Database Server SQL Server installations do not exactly match
Application Server = SQL Server 2008 (SP1) - 10.0.2531.0 (X64)
Database Server = SQL Server 2008 R2 (SP1) - 10.50.2500.0 (X64)
The plan is to upgrade the SQL Server installation on the Application server. Its kind of a hit and hope, but at this point this seems like the best option. Something in my brain tells me that this might be solved by fixing a hardware issue (by that I mean a repair or a replace) and that there might not be anything that hardware and software configuration can do about it.
However, I am still not sure how to go about determining a root cause. I am still left wondering what tools I should be using to diagnose the root cause.
Do you have AV software on Application Server side? If yes, try to disable AV - sometime AV blocks TCP/IP traffic. Issue with "The specified network name is no longer available" was solved by disabling AV here:
https://community.spiceworks.com/topic/239423-the-specified-network-name-is-no-longer-available-while-writing-to-shared-dir
The error message indicates that the connection was forcibly closed. You have also mentioned this occur when you run a lot of jobs. Chances are, the network firewall is to blame. You should reach out to the firewall administrator to research the logs in order to see if the firewall closed the connection. If this is the case then two potential solutions exist:
Add an exception to whatever firewall rule is being triggered and causing the connection to be closed.
Stop running so many jobs concurrently. You should consider running them in sequence. This also adheres to the idea of being a good network citizen.
First off did you tried to remove the large send offload setting on the nic ?
Second point, can you run a wireshark to capture the packets if you can reproduce the error ?
Third point, did you tried to change the vnic from the VM ? some model can cause issue. (If you use vmxnet3, try e1000, etc..)
Last point, does you have a vswitch between them, they are on same host, a physical switch between, etc... A badly configured switch can drop traffic, if inside the host the same host and same vswitch it's the best test, as the traffic never leave the server.
It seems there are multiple causes for this issue. For me it was the SQL file being too big at about 150MiB. Splitting it into smaller files worked for me.
One possible root cause could be the active directory membership count as we experiencer recently. That will among other influence the size of Kerberos token that can influence the SQL server login steps as well.
Details:
The login packet is structurally invalid
Kerberos Token Size
Note: We observed error log entries on the SQL Server like:
The login packet used to open the connection is structurally invalid; the connection has been closed. Please contact the vendor of the client library.'
On the client we observed the following error message:
A connection was successfully established with the server, but then an error occurred during the login process. (provider: TCP Provider, error: 0 - The specified network name is no longer available.) (.Net SqlClient Data Provider)
This issue was only affecting some AD users. There was no general service interruption.
Using the following command we determined the active directory membership:
PS> $groups = (az ad user get-member-groups --id {user identifier} |ConvertFrom-Json -AsHashtable)
PS> $groups.Count
294
Or we can also use the following command:
PS> $token = [System.Security.Principal.WindowsIdentity]::GetCurrent()
PS> $token.Groups.count
771
Connecting to the server with different providers yield different result:
Microsoft ODBC Driver for SQL Server Version 17.07.0002:
Data Source Name: SOME_NAME
Data Source Description:
Server: SQLSERVER.DOMAIN.COM
Database: CORE_DATABASE
Language: (Default)
Data Encryption: Yes
Trust Server Certificate: Yes
Multiple Active Result Sets(MARS): No
Mirror Server:
Translate Character Data: Yes
Log Long Running Queries: No
Log Driver Statistics: No
Use Regional Settings: No
Use ANSI Quoted Identifiers: Yes
Use ANSI Null, Paddings and Warnings: Yes
Authentication: ActiveDirectoryIntegrated
-----------
Running connectivity tests...
Attempting connection
Connection established
Verifying option settings
INFO: Connection was encrypted without server certificate validation.
Disconnecting from server
TESTS COMPLETED SUCCESSFULLY!
Microsoft SQL Server Native Client Version 11.00.7462:
Data Source Name: SOME_NAME
Data Source Description:
Server: SQLSERVER.DOMAIN.COM
Use Integrated Security: Yes
Database: (Default)
Language: (Default)
Data Encryption: No
Trust Server Certificate: No
Multiple Active Result Sets(MARS): No
Translate Character Data: Yes
Log Long Running Queries: No
Log Driver Statistics: No
Use Regional Settings: No
Use ANSI Quoted Identifiers: Yes
Use ANSI Null, Paddings and Warnings: Yes
-----------
Running connectivity tests...
Attempting connection
[Microsoft][SQL Server Native Client 11.0]TCP Provider: The specified network name is no longer available.
[Microsoft][SQL Server Native Client 11.0]Communication link failure
TESTS FAILED!
SQL Server - was not able to configure:
Try using ODBC instead of OLE DB for your connection to the database.
I am trying to configure a SSIS 2012 project on a couple of mirrored databases.
I was hoping to be able to use the OLEDB provider, but that doesn't seem to support the Failover Partner attribute in the connectionstring.
I have tried to add the keys "Failover Partner", "Failover_Partner" and "FailoverPartner", but none of them works - the connection never tries to connect to the failover parther even if that holds the principal database. All my tests have been performed with all three variants of the attribute key.
If I use the ADO.NET connection, there is a build-in setting called "Failover Partner" in the Connection Manager, and that works like a charm, and it can connect to the database with the principal being on either one of the servers... just like it should.
My question is: Is it correct that OLEDB does not work out of the box with mirrored databases, when using SSIS 2012 agains a SQL Server 2012 mirrored database?
How I test
My test is really simple: i have a "Execute SQL Task" with an OLEDB connection manager with the following connectionstring:
Data Source=srv1;Initial Catalog=MyDatabase;Provider=SQLOLEDB.1;Integrated Security=SSPI;Auto Translate=False;Application Name=My Application;
I have tried adding the different versions of Failover Partner as mentioned above:
Data Source=srv1.domain.local;Initial Catalog=MyDatabase;Provider=SQLOLEDB.1;Integrated Security=SSPI;Auto Translate=False;Application Name=My Application;Failover Partner=srv2.domain.local;
If MyDatabase is principal on srv1, then it works. If the database is principal on srv2, then SSIS fails with the following error:
Error: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80004005.
An OLE DB record is available. Source: "Microsoft OLE DB Provider for SQL Server" Hresult: 0x80004005 Description: "Cannot open database "MyDatabase" requested by the login. The login failed.".
and the following is logged on srv1 in the error log:
Login failed for user 'DOMAIN\MyUser'. Reason: Failed to open the explicitly specified database 'MyDatabase'. [CLIENT: 10.0.0.3]
What am I doing wrong? Or do I need to change to ADO.NET which works 100% as expected?
Thanx,
Sjang
There is a failover partner parameter if you use OleDB/SQLNCLI11.1 connection string:
Data Source=srv1;Initial Catalog=myDB;Provider=SQLNCLI11.1;Integrated Security=SSPI;Failover Partner=failover_partner_name;