Load balancing MySql with HAProxy on EC2 Servers - mysql

For our application, setup on the Amazon EC2 instances, we are using MySQL databases, installed on 2 EC2 instances, in master-master replication mode. To load balance these DB Servers I am using an HAProxy deployed on another EC2 instance with configuration
listen mysql <public dns of haproxy>:<port>
mode tcp
balance roundrobin
server mysql-db-s1 <elastic ip of db server1>:<port> maxconn 2000
server mysql-db-s2 <elastic ip of db server2>:<port> maxconn 2000
All three instances are assigned elastic ip's and I use the public dns of haproxy to connect from my application. But at times I get the error
`com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: C
ommunications link failure The last packet successfully received from the server was 58,085 milliseconds ago.
The last packet sent successfully to the server was 0 milliseconds ago.`
Has this anything to do with the configuration I have given here? Is it better to use the elastic ip's instead of Public DNS ?

This was not essentially an HAProxy issue, as I got the same error after I connected directly to one of the databases. Tuned the database a little and fixed some code in my application, and now I am not getting this atleast for the past 12 hours. Thanks for the suggestions. I would be moving one of the DB to another availability zone.

Web servers and databases load balancing concepts because of the request for the websites in large number. So we need to know how to handle this large number of requests through load balancing.
In this article, we are going to discuss this MySQL load balancing concepts using HAProxy. Let's consider the MySQL node 1 ip address is 192.168.1.172 and MySQL node 2 ip address is 192.168.2.82.
Step 1: Create MySQL Users And Install MySQL Client The first thing we need to make sure MySQL servers have to be configured to perform Master-Master replication as load balancing involves both reading and writing to all the backends. The initial thing we need to create a MySQL users for HAPorxy. The first user will be used by HAProxy to check the status of a server
$ mysql -u root -p -e "INSERT INTO mysql.user (Host,User) values ('192.168.1.172','haproxy_check'); FLUSH PRIVILEGES;"
$ mysql -u root -p -e "GRANT ALL PRIVILEGES ON *.* TO 'haproxy_root'#'192.168.1.172' IDENTIFIED BY 'password' WITH GRANT OPTION; FLUSH PRIVILEGES"
$ sudo apt-get install mysql-client
$ sudo apt-get install haproxy
$ sed -i "s/ENABLED=0/ENABLED=1/" /etc/default/haproxy
$ sudo nano /etc/haproxy/haproxy.cfg
and add this content to that new configuration file
global
log 127.0.0.1 local0 notice
user haproxy
group haproxy
defaults
log global
retries 2
timeout connect 3000
timeout server 5000
timeout client 5000
listen mysql-cluster
bind 127.0.0.1:3306
mode tcp
option mysql-check user haproxy_check
balance roundrobin
server mysql-1 192.168.1.172:3306 check
server mysql-2 192.168.2.82:3306 check
Step 3: Test Load Balancing
$ mysql -u haproxy_root -p -e "show variables like 'server_id'"

Related

Error ERROR 2013 (HY000): Lost connection to MySQL server during query setting up group replication on Lightsail

I am trying to configure sql group replication between 2 vps, one being a non AWS Lightsail instance and acting as primary.
However I cannot seem to be able to get the Lightsail instance working.
It launched without problems on the primary, but when I try to start the group replication on the Lightsail server
START GROUP_REPLICATION;
ERROR 2013 (HY000): Lost connection to MySQL server during query
(timemout is set to 20)
Ports 3306 and 33061 are open both servers in ufw and also open on the AWS Lightsail's firewall.
my.cnf on Lightsail looks like this
# Shared replication group configuration
loose-group_replication_group_name = "<uuid>"
loose-group_replication_ip_whitelist = "<primary_static_public_ip>,<lightsail_static_public_ip>"
loose-group_replication_group_seeds = "<primary_static_public_ip>:33061,<lightsail_static_public_ip>:33061"
# Single or Multi-primary mode? Uncomment these two lines
# for multi-primary mode, where any host can accept writes
loose-group_replication_single_primary_mode = OFF
loose-group_replication_enforce_update_everywhere_checks = ON
# Host specific replication configuration
server_id = 2
bind-address = "0.0.0.0"
report_host = "<lightsail_static_public_ip>"
loose-group_replication_local_address = "<lightsail_static_public_ip>:33061"
On the primary I only use the primary's address and whitelist Lightsail public address. I cannot figure out why it refuses to connect.
I could however connect from Lightsail to primary using mysql -h command and the replication username.
UPDATE
So, apparently there is no way of having this working remotely without servers being inside the same VPC or local network.
Therefore the only solution is to use the "deprecated" solution (according to mysql official website) of Master/Master replication.
To do so one has to first set-up a usual Master/Slave replication (I will not explain this here as there is plenty of tutorial on how to set it up, eg: https://www.digitalocean.com/community/tutorials/how-to-set-up-master-slave-replication-in-mysql).
Once the replication is in place and works:
on the SLAVE:
edit /etc/mysql/my.cnf to add the following lines
#use public IP address of the server or 0.0.0.0 if you wish to bind it to
#more than a single IP address (ie using localhost)
bind-address = "0.0.0.0"
binlog_do_db = db_to_replicate
binlog_ignore_db = db_to_ignore
Then if you are not syncing the mysql database you will need to create a slave user on the slave
CREATE USER 'slave_usr'#'%' IDENTIFIED BY 'password' REQUIRE SSL;
GRANT REPLICATION SLAVE ON *.* TO 'slave_usr'#'%';
then run service mysql restart
On the MASTER:
Get the binlog file and position using: SHOW MASTER STATUS;
Run the CHANGE MASTER command: CHANGE MASTER TO MASTER_SSL=1, MASTER_HOST='slave_host_public_ip',MASTER_USER='<slave_usr>', MASTER_PASSWORD='password', MASTER_LOG_FILE='mysql-bin.00000X', MASTER_LOG_POS= binlog_pos;
you might need run stop slave beforehand.
Then run START SLAVE;
You can check that all is working correctly using SHOW SLAVE STATUS\G on both servers.
If you bind-address to 0.0.0.0 it is recommended to reject all IP's that will not need to connect to the servers.
Example using ufw: sudo ufw allow from <ip> to any port 3306

mariadb remote connection fail in centos

I have visited many websites for remote connection of MariaDb.
I have executed the command as below to create user with password in sql.
GRANT ALL PRIVILEGES ON . TO 'root#(my server ip)' IDENTIFIED BY '(my password)'
And i've added one line below [mysqld] in the file of /etc/my.cnf.d
bind-address = 0.0.0.0
Then restart MariaDb service as below
sudo systemctl restart mariadb
Everything runs good.
However when i access by below command, it runs failed.
mysql -u root -p -h (my server ip)
I've turn off my firewall in my server, and turn on the port 3306 in GCP server, and it can be expected, i must fail to connect in my local machine.
Since you want to use a TCP connection, I assume that you want to connect to a remote server, not to a server running on the same machine.
Make sure that you are able to connect physically to the database server, e.g. with telnet server_ip:3306.
Determine the IP address of the computer from which you want to connect to the server (= client_ip).
Add a user on the server:
GRANT ... TO root#client_ip
If client and server are running on the same machine, the preferred way is to use a linux socket (user#localhost) which is way faster.

Connecting to MySQL via SSH ERROR 2013 (HY000)

The Setup
I am currently using the Premium Wordpress Hosting provided by MediaTemple. I have a very large data-set to import and I was hoping to get direct access to the database via an SSH tunnel.
--------------- ------------------- ------------
| My Machine | ---- SSH TUNNEL -----| Hosting Server | -- -- ? -- -- | Database |
--------------- ------------------- ------------
What Works
If I ssh into the Hosting Server and from the shell on the Hosting Provider, connect to mysql like this, I am able to get into MySQL.
mysql -uuser -ppassword -h123.456.789.1 -P3308
What Does Not Work
However, if I try to connect to MySQL using the -L flag with SSH to create a tunnel, I am unable to connect to the server.
ssh me#hostingserver.net -L 7002:123.456.789.1:3308
From a shell on My Machine:
mysql -uuser -ppassword -h127.0.0.1 -P7002
I get the following error:
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0
From reading other answers (StackOverflow , StackOverflow), I have reasoned that the issue stems from the IP address with which MySQL client tries to bind. I think that the ip address attach to the request to connect, when executed on my machine, is not on the white-list of the Database Server.
Is there anyway to get direct access to the MySQL Database from My Machine. From a system administration perspective, I obiviously have enough access to connect to the MySQL database from the shell but I cannot run the client on My Machine. I have a very large dataset that I would like to transfer from My Machine to Database. I would also like to be able access the database and exicute SQL whenever I need to. This and the large dataset kind of eliminates the possibility of just using a the source command from the MySQL Client on Hosting Server. What is the best workaround to give me something close to the ability to run SQL on the Database from My Machine?
I encountered roughly the same issue. That is, I simply could not connect to the MySQL server, even though I had successfully tunneled to the remote host.
TLDR: it was an iptables issue involving the loopback interface
In my situation, mysqld was running on the same VPS as sshd. However, the MySQL instance was bound only to 127.0.0.1 and listening on the default port. As you did, I confirmed that I could connect to the mysqld instance on the remote machine using the credentials used locally.
Here is the tunnel:
ssh -v -N -L 33306:127.0.0.1:3306 sshuser#sshanddbvps.org
Here is the connection string to the mysqld instance using the mysql client:
mysql -umysqluser -h127.0.0.1 -P 33306 -p
Even though ssh indicated that the connection was successful...
debug1: Connection to port 33306 forwarding to 127.0.0.1 port 3306 requested.
...the mysql client connection would error out after accepting the correct password with the message you mentioned:
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet'...
To check that data was flowing across the loopback interface, I logged into the remote server and ran three commands in three separate shells:
while true; do echo -e "HTTP/1.1 200 OK\n\n $(date)" | nc -l 127.0.0.1 1234; done
tcpdump -i lo src 127.0.0.1 -or dst 127.0.0.1
nc 127.0.0.1 1234
After running the third, output from the second command appeared:
13:59:14.474552 IP localhost.36146 > localhost.1234: Flags [S], seq 1149798272, win 43690, options [mss 65495,sackOK,TS val 48523264 ecr 0,nop,wscale 7], length 0
But nothing indicating that packets were flowing in the reverse direction.
Inserting a rule in the INPUT chain of the firewall that allowed traffic from the default loopback address solved the issue:
iptables -I INPUT 4 -i lo -s 127.0.0.1 -j ACCEPT

Mysql-cluster issue

I am trying to create a table in mysql Cluster.
When I type this I had this error:
/usr/local/mysql/mysqlc/bin/mysql -h 127.0.0.1 -P 1186
-u root
ERROR 1045 (28000): Access denied for user 'root'#'localhost' (using password: NO)
Someone has an idea?
Thank you very much.
Why are you trying to access MySQL on port 1186 ? MySQL should be available in port 3306 and NDB_MGMD in port 1186 in case you are using a cluster. Run this to check what port are you using :
$ sudo netstat -atnp | grep LISTEN
If you see 3306 for mysql and 1186 for ndb_mgmd there is no need to specify default port. Then to create a table in a cluster just make sure your cluster is up and running
$ ndb_mgm
ndb_mgm>show
Then connect normally to your mysql
$ mysql -u root -p
First stop the SQL service using command:
sudo service mysql stop
Then use the command:
/usr/bin/mysqld_safe --skip-grant-tables
Before you execute above command make sure that you kill all executing mysqld & mysql_safe processes.
And you can go ahead with the commands you mentioned and gain access.
You can grant the permissions to the another host using this command:-
GRANT ALL ON ​*.*​ to root#'192.168.1.4' IDENTIFIED BY 'your-root-password';
And then, you can check the grant access using this:-
show grants;
You can then start the mysql using this command:-
mysql -h 192.168.1.4 -P3306 -u root
I am on Linux platform with MySQL NDB 5.7. I am trying to monitor all traffic related to MySQL clustering - between data nodes, management node and sql nodes. To that end, I used netstat to list all open ports listening on my machine before starting MySQL cluster. Then, I started MySQL cluster and ran netstat again. I assumed that the ports that were listening the second time around, but not the first time, were related to MySQL clustering.
But there are two problems with this. First, there could be ports opened by other processes between the two netstat runs. Second, MySQL might open other ports after I ran the netstat command the second time.
What is the best way to go about finding all ports being used by MySQL for clustering purposes?
I believe ephemeral ports are picked dynamically, so perhaps if I knew all the MySQL clustering related processes that would be running, I can figure out every port that they are using. Pointers will be very welcome.

temporarily disable mysql remote access

I need to disble remote access during nightly mysql maintenance on Linux server so that no one can query the database during that time. I can't do SERVICE MYSQL STOP because then I couldn't do what I needed to do (truncate and rebuild a couple tables). Is there a way to turn off external access for a short time?
Thanks in advance.
Here is a great way without touching anything in the OS:
Step 1) Export all users to SQL file like this:
mysql -h localhost -u root
-p rootpassword --skip-column-names -A -e"SELECT CONCAT('SHOW GRANTS FOR ''',user,'''#''',host,''';') FROM
mysql.user WHERE user<>''" | mysql
-hlocalhost -uroot -prootpassword --skip-column-names -A | sed 's/$/;/g' > /root/MySQLGrants.sql
Step 2) Disable all users except root#localhost
DELETE FROM mysql.user WHERE CONCAT(user,host) <> 'rootlocalhost';
FLUSH PRIVILEGES;
Step 3) Perform your maintenance
Step 4) Reload the Grants
mysql -h localhost -u root -p rootpassword < /root/MySQLGrants.sql
Give this a Try !!!!
PS
service mysql restart --skip-networking
would still be the simplest and fastest way plus logging off all connections before maintenance
just run service mysql restart after your maintenance
Since 5.7.5 there's an "official way" to deal with that:
MySQL Server now supports an “offline mode” with these
characteristics:
Connected client users who do not have the SUPER privilege are
disconnected on the next request, with an appropriate error.
Disconnection includes terminating running statements and releasing
locks. Such clients also cannot initiate new connections, and receive
an appropriate error.
Connected client users who have the SUPER privilege are not
disconnected, and can initiate new connections to manage the server.
Replication slave threads are permitted to keep applying data to the
server.
Only users who have the SUPER privilege can control offline mode. To
put a server in offline mode, change the value of the new offline_mode
system variable from OFF to ON. To resume normal operations, change
offline_mode from ON to OFF. In offline mode, clients that are refused
access receive an ER_SERVER_OFFLINE_MODE error.
Disable the login credentials used by the remote apps?
Firewall the MySQL port so it can't be accessed by the remote apps?
There are some another method to temporary disable access to MySQL database during maintenance.
Edit /etc/mysql/my.conf'
Change bind-address to:
bind-address = 127.0.0.1
Restart MySQL:
service mysql restart --skip-networking
Nobody will access MySQL server from remote until you have change bind-address to original value and restart MySQL again.