I'm attempting to implement automatic failover via a MySQL Fabric Server Group using a JNDI Resource, Hibernate, and Connection Pooling. Our DB farm is properly setup with MySQL Fabric and I'm attempting to implement the client/connector.
I have a JNDI resource declared in my server.xml file which has a MySQL Fabric-based URL and Driver, and uses connection pooling with hibernate.
<Resource
name="jdbc/myApp"
type="javax.sql.DataSource"
driverClassName="com.mysql.fabric.jdbc.FabricMySQLDriver"
maxWait="1000"
url="proper.fabric.url/fabricServerGroup=myFabricGroup"
maxActive="45"
validationQuery="select 1 from dual"
/>
The problem I'm having is that the connection pooling and Fabric do not play well together. After taking down the master DB, Fabric automatically promotes a slave as expected. However, when I attempt to make a new connection to the DB I'm getting old now-defunct connections back, most notably I'm getting a read-only connection back when I'm requesting readwrite. As additional info if I restart my Tomcat server and ask for a fresh connection I get the correct results. The problem I'm having specifically occurs because of connection pooling.
What I need to know is if there's a key/value pair which I'm missing that would make my FabricDriver smart enough to know to wipe the connection pool when automatic failover occurs.
The best I've found is to use testOnBorrow, which unfortunately can only use a SELECT query (via validationQuery), which still succeeds when I'm getting a read-only connection back despite asking for readwrite.
Has anybody solved this issue before?
Note: If I attempt to use com.mysql.fabric.jdbc.FabricMySQLDataSource I can't even successfully make any connections. The regular javax.sql.DataSource seems to be the correct value here.
Second Note: I'm using mysql-connector-java version 5.1.35
Just heard back from Oracle, this is apparently a known bug and is going to be fixed in the upcoming release: 5.1.36
I have the same issue with Oracle MySQL Connector/J 5.1.39.
I am able to follow the failover requesting a new connection to Fabric only catching the Exception in my code:
...
} catch(java.sql.SQLException e) {
int errorCode = e.getErrorCode();
/*
* java.sql.SQLException: The MySQL server is running with the --read-only option so it cannot execute this statement
* java.sql.SQLException: Cannot execute statement in a READ ONLY transaction.
*
*/
if(errorCode == 1290 || errorCode == 1792) {
System.out.println("Failover");
System.out.println(e.toString());
connection.close();
rawConnection = DriverManager.getConnection(baseUrl, mysql_user, mysql_password);
connection = (FabricMySQLConnection) rawConnection;
}
Asking Oracle support about the automatic failover from the connector side they answer "Connector/J doesn't have desired feature".
There is someone able to integrate that catching inside the connection pool?
Related
I have an application in .net core 3.1 and I am using MySql with that. To connect to MySql I am using Pomelo.EntityFrameworkCore.MySql nuget.
My .net core application is deployed as a docker container on AWS ECS and I am using Aurora MySql RDS to store data. I have granted "Publicly accessible" access to the Aurora MySql and I can connect to the db using MySql workbench and also with my .net core application using localhost. But when I deploy the application and try to perform any db action then it starts giving throwing exception:
An exception has been raised that is likely due to a transient
failure. Consider enabling transient error resiliency by adding
'EnableRetryOnFailure()' to the 'UseMySql' call.
Then I have added retry pattern like this:
services.AddDbContextPool<DataContext>(options =>
options.UseMySql(Configuration.GetConnectionString("DefaultConnection"), builder =>
{
builder.EnableRetryOnFailure(5, TimeSpan.FromSeconds(5), null);
}
);
and my connection string is something like this:
"DefaultConnection": "Server=db-cluster-1-instance-1.cqb2fsjwx78p.us-east-2.rds.amazonaws.com;Database=dbName;User ID=admin;Password=password;port=3306"
After adding retry pattern. I am getting this error:
"Maximum number of retries (5) exceeded while executing database
operations with 'MySqlRetryingExecutionStrategy'. See inner exception
for the most recent failure."
So I suspect, it's something else. What am I doing wrong here? Or it might be something wrong at aws side
So the issue was related to docker image I was pulling.
I was using mcr.microsoft.com/dotnet/core/sdk:3.1. Changing it to mcr.microsoft.com/dotnet/core/sdk:3.1-bionic worked.
https://github.com/dotnet/SqlClient/issues/222
Add SslMode=None to your connection string and see if the connection works (see MySqlConnector's Connection String Options). The transient exception just means, that Pomelo (i.e. MySqlConnector) was unable to connect to the database server.
So this is related to connection problems (either fixable by altering the connection string or by changing your Aurora/firewall configuration).
I am creating .net core 2.1 MVC application and using Azure database for MySQL DB 5.7.
I have read below links but seems they are applicable for MS SQL DB.
https://learn.microsoft.com/en-us/azure/mysql/concepts-high-availability
https://learn.microsoft.com/en-us/azure/architecture/best-practices/retry-service-specific
Transient handling for MySQL not possible? Help me link to MYSQL related similar pages.
A transient error, also known as a transient fault, is an error that will resolve itself. Most typically these errors manifest as a connection to the database server being dropped. Also new connections to a server can't be opened. Transient errors can occur for example when hardware or network failure happens.
Transient errors should be handled using retry logic. Situations that must be considered:
An error occurs when you try to open a connection
An idle connection is dropped on the server side. When you try to issue a command it can't be executed
An active connection that currently is executing a command is dropped.
The first and second case are fairly straight forward to handle. Try to open the connection again. When you succeed, the transient error has been mitigated by the system. You can use your Azure Database for MySQL again. We recommend having waits before retrying the connection. Back off if the initial retries fail. This way the system can use all resources available to overcome the error situation. A good pattern to follow is:
Wait for 5 seconds before your first retry.
For each following retry, the increase the wait exponentially, up to 60 seconds.
Set a max number of retries at which point your application considers the operation failed.
Read more here.
And you can read more on how to troubleshoot connection issues to Troubleshoot connection issues to Azure Database for MySQL here.
I have installed MySQL Community server on local machine and am trying to access it from .NET Core API. Something must be wrong with either the connection string or the server configurations. I have checked the server configs to make sure TCP/IP connections are allowed, checked the connection string according to tutorials but still fail to connect, getting this exception in code:
A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections.
Tried connecting to MSSQL database and there are no issues, so nothing else could be wrong from code-side apart from the connection string itself.
The connection string:
"SqlConnectionString": "Server=127.0.0.1;Uid=admin;Pwd=admin;Database=sys;"
SQL query in code (using Dapper):
private const string SQL_INSERT_ENTITY = #"
INSERT INTO [sys].[adminportal_restapi] (Id, FeatureName, IsEnabled, LastInsertUpdate)
VALUES (#Id, #FeatureName, #IsEnabled, #LastInsertUpdate)
";
Database layout:
Found the solution - you have to use a different connector.
Basically these changes are needed:
in project.json add this dependency: "MySql.Data.Core": "7.0.4-IR-191"
and in code change new SqlConnection to new MySqlConnection
EDIT: that package got unlisted by MySQL devs without releasing a newer version (which is a complete douchebag move, IMO), so just use this one instead: "MySqlConnector": "0.11.5" syntax for connections is identical, just it doesn't support integrated security, which might suck in some cases. Good luck
I've always used Mongo with Node, but now due to an existing datasource I need to connect a node app with Mysql.
Sequelize seems a good solution, but I don't get how to handle connection error, reconnection and re-tries.
To check for connection error on first run .authenticate().then().catch(function(error){...});
But what if I loose connection and want to reconnect?
There is an open issue for this in Sequelize:
https://github.com/sequelize/sequelize/issues/2113
Based on that, this error is handled in sequelize.
I verified the version 4.11.1 of sequelize has this issue fixed.
The queries will fail when the database server is down, but will recover to reconnect and succeed when the database server is up.
(You don't need to restart the application as faced with previous versions.)
I have the following setup: Tomcat 7 and MySQL 5.6 are running on separate AWS EC2 instances, JDBC driver is used. Everything works fine under nominal conditions.
But when connection between the instances gets broken (due to instance shutdown or security group blocking), my JSP is stuck in trying to get the data source, but it never throws any exception (like timeout), which I could catch in JSP:
ds = (DataSource)envContext.lookup("jdbc/mydatabase");
After Tomcat restart, there is a lookup exception in catalina.out, but the JSPs are still stuck.
It would be great to have an exception, e.g. to detect that something is wrong with the database ec2 instance and switch to a backup on the software level. How to do it if there is no exception?
Note that this behavior is different from MySQL shutdown (ec2 instance still running), then there is a clear exception I can catch at the JSP level.