Connection to mysql server in SYN_SENT - mysql

Question

We are facing a strange problem from last few days between our application server and database server(Mysql): connection to database server from application server hangs in SYN_SENT state and after that we are not able to make any connection to database server on mysql port(3306). When we checked the netstat output on database server its in SYN_RECV state.
What I can figure out is mysql server is receiving the SYN request and responding also and its not reaching to the client hence SYN_RECV at server side and SYN_SENT at client side. I think SYN_SENT state should go after some time and because of this other db connection attempts to same server should not hang.
Does anybody have any idea how can we resolve this issue?
Out setup details : Application server: RHEL 5.4, kernel-release = 2.6.18-164.el5, x86_64 Database server: Mysql Version : 5.1.49 RHEL 5.4, kernel-release = 2.6.18-164.el5, x86_64

Answer 1

Fix for server with only localhost access:
set 127.0.0.1 in the bind address in my.cnf
Fix for connection to remote ip's
(REMOTE_IP replace with remote ip)
iptables -A INPUT -p tcp -d 127.0.0.1 --dport 3306 -s REMOTE_IP -j ACCEPT
iptables -A INPUT -p udp -d 127.0.0.1 --dport 3306 -s REMOTE_IP -j ACCEPT
iptables -A INPUT -p tcp --dport 3306 -j DROP
iptables -A INPUT -p udp --dport 3306 -j DROP
Also you need to set bind ip in my.cnf to 0.0.0.0. Second rule you don't need, I just made it to be sure ;) (udp part)
Proof of concept:
first allow the connection from remoteip to the destination (-d 127.0.0.1 = localhost)
-p tcp / udp = protocoll tcp or udp
after this rules you need to make a rule to drop all requests to tcp / udp connections to port 3306.
Why is this working:
Because iptables is going is "numeric". Always 1 rule after another.
you can see your rules with the command:
iptables -L INPUT -n --line-numbers
the first rule which is displayed is the first rule so if you say accept all connections and afterward drop from ip x.x.x.x all connections then it doesn't work.
you need to pick as first rule to drop all connections from this ip and afterwards allow all connections. (it's a bad example..)
if you failed an entry you can display your rules and take the number in front of the rule and drop the rule with the command:
iptables -D INPUT <<number here>>