I want my Drupal8 installation to be more reliable in case of database failure, so I want to enable Master / Slave replication in MySQL, and make it work with Drupal 8.
I setted up two MySQL servers (one master, one slave), and replication is working.
Now, I want to setup Drupal to balance SELECT query to the slave, and INSERT/UPDATE/DELETE queries to the master. It will balance the load between the two servers, and I will have a (litle) HA concept with it.
To test this architecture, I installed a brand new Drupal 8 instance, no custom module at all. Only pure vanilla Drupal 8 if possible.
I change the settings.php file as follow :
$databases['default']['default'] = array (
'database'=>'drupaldb',
'username'=>'masteruser',
'host'=>'database-master.com'
...
);
$databases['default']['slave'] = array (
'database'=>'drupaldb',
'username'=>'slaveuser',
'host'=>'database-slave.com'
...
);
"slaveuser" is a MySQL user with only READ access ; "masteruser" have RW access on database.
With this settings, the application is working fine, but when I look at the Slave metrics, there is no connection. Even when I'm reloading page of the front-end.
If I switch off the master, or if I change the master settings (bad password, wrong database) in config php file, I expected the application, for the front-end pages, to use the Slave settings. But front pages are in error : "The website encountered an unexpected error. Please try again later". Error log indicate a SQL connection failure.
What I see here, is that there is no way to balance queries onto master (for writes) and slave (for reads).
I expected Drupal8 to manage this without any extra plugin.
Do I have to use the mysql_nd_ms PHP extension to do M/S balancing ?
The syntax for the replica database configuration can be found in settings.php
https://api.drupal.org/api/drupal/sites%21default%21default.settings.php/8.7.x
The syntax for the default db replica is:
// $info_array is the db connection details
$databases['default']['replica'][] = $info_array;
Note: D8 uses replica not slave for the terminology.
Note: so far as I can tell, adding 'replica' servers does nothing unless you are using the db_* functional calls (which are deprecated), or if you manually instantiate the database.replica connection in any of your custom queries
e.g.
/** #var \Drupal\Core\Database\Connection $database_replica */
$database_replica = \Drupal::service('database.replica');
$query = $database_replica->select('node', 'n');....
$database_replica = \Drupal::service('database.replica');
or
$database_replica = Database::getConnection('replica', 'default');
Related
According to other answers (1, 2), as well as some documentation in the mariadb-connector-j repo, the consensus seems to be:
On a master/slave cluster, driver will use underlying 2 connections:
one to a master instance, one to a slave instance.
A "connection" to aurora mean 2 underlying connection to instances:
one to master, one to slave. Driver will use the underlying connection
to master or slave according to Connection.setReadonly().
When using the "aurora" keyword, driver , under the hood, create 2
connections: a connection to the primary server, a connection to one
of the replicas if any. The goal is always to save resources on the
main server. Generally, only one pool is configured. The driver then
uses the connection to the primary / replica according to
[Connection.setReadOnly]
However, I am trying this for myself and do not see this behavior with MariaDB Connector/J 2.6.2 (in a Play 2.8.x application, which uses HikariCP as the connection pool). I created a new MySQL Aurora cluster (with a master and read-replica) and modified my test application to connect to this cluster with a HikariCP pool that's configured at a fixed size of 8 connections, using the cluster endpoint url as follows:
"jdbc:mysql:aurora://test.cluster-abcdefg.us-east-1.rds.amazonaws.com:3306/test"
From what I can see in the logs, as well as the attached image from the RDS web dashboard's monitoring tab, once the application starts up, the master goes to 8 connections (blue line) and the read-replica has 0 connections (orange line). This seems to be contrary to all the above points that, under the hood, the driver makes 2 underlying connections and chooses between them based on the status of a given Connection's read-only property.
The Play configuration for this connection is as follows:
test {
driver = org.mariadb.jdbc.Driver
url = "jdbc:mysql:aurora://test.cluster-abcdefg.us-east-1.rds.amazonaws.com:3306/test"
hikaricp {
autoCommit = false
readOnly = false
}
}
If I instead configure the connection like:
test {
driver = org.mariadb.jdbc.Driver
url = "jdbc:mysql:aurora://test.cluster-abcdefg.us-east-1.rds.amazonaws.com:3306/test"
hikaricp {
autoCommit = true
readOnly = true
}
}
then I do see connections made to both the master and read-replica:
Please help me understand this behavior.
What I'm trying to achieve is to have separate write and read pools. The write pool would always point to the master, and the read pool would always point to a read replica (assuming one exists). I want to avoid dual underlying connections, as the readOnly setting is configured at the pool level and will never change on a given Connection during its lifetime. How can I achieve this?
Additionally, I would like to know whether I am expected to use the autoReconnect parameter when specifying the aurora failover mode. Does the value of that parameter matter with the aurora failover mode?
I could see Stop Server and Bring Offline in mysql workbench.
I hope both are used to stop the services but still confused in difference between both in terms of when to use Stop Server and Bring Offline?
Stop server just stops the mysql process
Offline mode is a new feature introduced in MySQL 5.7.5, that basically throws out all users except DBAs :
MySQL Server now supports an “offline mode” with these
characteristics:
Connected client users who do not have the SUPER privilege are disconnected on the next request, with an appropriate error.
Disconnection includes terminating running statements and releasing
locks. Such clients also cannot initiate new connections, and receive
an appropriate error.
Connected client users who have the SUPER privilege are not disconnected, and can initiate new connections to manage the server.
Replication slave threads are permitted to keep applying data to the server.
Only users who have the SUPER privilege can control offline mode. To
put a server in offline mode, change the value of the new offline_mode
system variable from OFF to ON. To resume normal operations, change
offline_mode from ON to OFF. In offline mode, clients that are refused
access receive an ER_SERVER_OFFLINE_MODE error.
Source : Changes in MySQL 5.7.5 (2014-09-25, Milestone 15)
I've read a ton about persistent database connections between PHP and MySQL (mysql_connect vs. mysql_pconnect). Same with PDO and MySQLi. It's definitely just my lack of understanding on this one, but how can a database connection be persistent between webpages? In this code:
$conn = mysql_pconnect( $server , $user, $pass );
mysql_select_db( $dbname );
If two users load this page at the same time, with two different $dbname variables, will PHP only make one connection to the database or two? I am fairly certain that
$conn = mysql_connect( $server , $user, $pass );
would make two connections.
If pconnect reuses the connection opened by the first user, will the mysql_select_db call work for the second user?
Ideally, what I am looking for is a way to have fewer database connections but still be able to set the default database in each PHP script. I have clients who all use the same PHP scripts, but the data is stored in their own client database (hence, $dbname is always different, but the MySQL connection parameters are the same - same mysql ip address, user and password).
Hope that makes sense. We can use MySQL, MySQLi or PDO, just need to know how to accomplish this the best way without having the possibility for clients to accidently write data to someone else's database! Thanks in advance.
The persistence is done by the copy of the PHP that's embedded in the webserver. Ordinarily you'd be right- if PHP was running in CGI mode, it would be impossible to have a persistent connection, because there'd be nothing left to persist when the request is done and PHP shuts down.
However, since there's a copy of PHP embedded in the webserver, and the webserver itself keeps running between requests, it is possible to maintain a pool of persistent connections within that "permanent" PHP.
However, note that on Apache multi-worker type server models, the connection pools are maintained PER-CHILD. If you set your pool limit to 10, you'll have 10 connections per Apache child. 20 children = 200 connections.
Persistent connections will also lead to long-term problems with deadlocks and other hard-to-debug problems. Remember - there's no guarantee that a user's HTTP requests will be serviced by the SAME apache child/mysql connection. If a script dies part-way through a database transaction, that transaction will NOT be rolled back, because MySQL does not see the HTTP side of things - all it sees is that the mysql<->apache connection is still open and assumes all's well.
The next user to hit that particular apache/mysql child/connection combination will now magically end up in the middle of that transaction, with no clue that the transaction is open. Basically, it's the Web equivalent of an unflushed toilet - all the "garbage" from the previous user is still there.
With non-persistent connections, you're guaranteed to have a 'clean' environment each time you connect.
From my reading of documentation and comments, I see:
Docs on mysql_pconnect (deprecated method)
Second, the connection to the SQL server will not be closed when the execution of the script ends. Instead, the link will remain open for future use ( mysql_close() will not close links established by mysql_pconnect()).
and a comment on that page
Persistent connections work well for CGI PHP managed by fastCGI, contrary to the suggestion above that they only work for the module version. That's because fastCGI keeps PHP processes running between requests. Persistent connections in this mode are easily made immune to connection limits too, because you can set PHP_FCGI_CHILDREN << mysql's max_connections <<< Apache's MaxClients. This also saves resources.
Docs on mysqli_connect (new method)
Prepending host by p: opens a persistent connection. mysqli_change_user() is automatically called on connections opened from the connection pool.
Docs for mysqli_change_user:
Changes the user of the specified database connection and sets the current database.
So my understanding is as follows: pconnect keeps the connection open after a script ends but while a process (or maybe group of processes) is still alive (like in a server with FCGI set up). Only one script at a time uses a connection, and when a new script grabs that connection the user and database are updated.
Thus if you use FCGI and persistent connections you can reduce the number of db connections open, but scripts running simultaneously will not be sharing the same connection. There is no problem with the connection being confused as to which database is selected.
I've started to check mysql connector j's replication paradigm and see that we can seperate read and write operations on master and slave databases.
I've checked below page and get some clues on the operation but still need to know how does mysql-jdbc understands which server is master and which servers are slaves ? ( might be a silly one, sorry for this )
http://www.dragishak.com/?p=307
The ReplicationDriver or NonRegisteringReplicationDriver decides the first url as master and the rest considered as slaves
The point you should of take into consideration is : If you are using ReplicationDriver or NonRegisteringReplicationDriver you need to give at least two hosts contains the same db instance. Otherwise you will get an SQLException telling : "Must specify at least one slave host to connect to for master/slave replication load-balancing functionality".
One more point : You don't actually need to create an instance of NonRegisteringReplicationDriver. Because ReplicationDriver is also using it. You can check it by let your application throw and Exception. What you will see is; the DB connection was tried by NonRegisteringReplicationDriver.connect(..) method.
Edit(!) : You actually don't need to create non of spesific driver for your system. What you need to know is what are you doing and the correct connection url. Because the Driver class itself checks the url against replication pattern and loadbalance pattern. Then it triggers the required driver instance.
I've got a trio of Windows servers (data1, data2 and datawitness) that aren't part of any domain and don't use AD. I'm trying to set up mirroring based on the instructions at http://alan328.com/SQL2005_Database_Mirroring_Tutorial.aspx. I've had success right up until the final set of instructions where I tell data1 to use datawitness as the witness server. That step fails with the following message:
alter database MyDatabase set witness = 'TCP://datawitness.somedomain.com:7024'
The ALTER DATABASE command could not be sent to the remote server instance 'TCP://datawitness.somedomain.com:7024'. The database mirroring configuration was not changed. Verify that the server is connected, and try again.
I've tested both port 7024 as well as 1433 using telnet and both servers can indeed connect with each other. I'm also able to add a connection to the witness server from SQL Server Manager on the primary server. I've used the Configuration Manager on both servers to enabled Named Pipes and verify that IP traffic is enabled and using port 1433 by default.
What else could it be? Do I need any additional ports open for this to work? (The firewall rules are very restrictive, but I know traffic on the previously mentioned ports is explicitly allowed)
Caveats that are worth mentioning here:
Each server is in a different network segment
The servers don't use AD and aren't part of a domain
There is no DNS server configured for these servers, so I'm using the HOSTS file to map domain names to IP addresses (verified using telnet, ping, etc).
The firewall rules are very restrictive and I don't have direct access to tweak them, though I can call in a change if needed
Data1 and Data2 are using SQL Server 2008, Datawitness is using SQL Express 2005. All of them use the default instance (i.e. none of them are named instances)
After combing through blogs and KB articles and forum posts and reinstalling and reconfiguring and rebooting and profiling, etc, etc, etc, I finally found the key to the puzzle - an entry in the event log on the witness server reported this error:
Database mirroring connection error 2 'DNS lookup failed with error: '11001(No such host is known.)'.' for 'TCP://ABC-WEB01:7024'.
I had used a hosts file to map mock domain names for all three servers in the form of datax.mydomain.com. However, it is now apparent that the witness was trying to comunicate back using the name of the primary server, which I did not have a hosts entry for. Simply adding another entry for ABC-WEB01 pointing to the primary web server did the trick. No errors and the mirroring is finally complete.
Hope this saves someone else a billion hours.
I'd like to add one more sub answer to this specific question, as my comment on Chris' answer shows, my mirror was showing up as disconnected (to the witness) Apperently you need to reboot (or in my case i just restarded the service) the witness server.
As soon as i did this the mirror showed the Witness connection as Connected!
See: http://www.bigresource.com/Tracker/Track-ms_sql-cBsxsUSH/