nginx connect() failed (110: Connection timed out) - mysql

Question

I know that, this question has been asked multiple times on different forums, but still I can not manage to find a solution, which solves my problem... The situation: We have a nginx, php-fpm and MySQL stack on a server. The server sits behind a nginx reverse proxy. The problem is that on the upstream server there are clean error logs and on the reverse proxy I am getting multiple messages
connect() failed (110: Connection timed out) while connecting to >upstream, client: ++++++++++, server: domain.com, request: "GET >/files/imagecache/FrontBullet/blog1/dknasda.jpg HTTP/1.1", upstream: >"http://192.168.158.142:80/files/imagecache/FrontBullet/blog1/dknasda.jpg>", host: "somedomain.com"
For some reason this error occurs every 1-5 minutes for different resources or files.
My nginx config on the reverse proxy is the following one :
user ++++;
worker_processes 3;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
events {
worker_connections 1024;
use epoll;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 60s; #keeps the connection open with the client; MS default is 60.
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
gzip on;
gzip_http_version 1.1;
gzip_vary on;
gzip_comp_level 6;
gzip_proxied any;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/javascript text/x-js;
gzip_buffers 16 8k;
gzip_disable "MSIE [1-6]\.(?!.*SV1)";
upstream main_upstream {
server 192.168.158.142:80 max_fails=3 fail_timeout=60s; # New Server. Sent to 192.168.90
# server 192.168.158.143:80; # HSB
keepalive 32;
}
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name domain.com;
location / {
proxy_buffers 32 32k;
proxy_buffer_size 64k;
proxy_pass http://main_upstream;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-By $server_addr:$server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_connect_timeout 600s;
proxy_send_timeout 600s;
proxy_read_timeout 600s;
send_timeout 600s;
proxy_http_version 1.1;
proxy_set_header Connection "";
client_max_body_size 32M;
client_body_buffer_size 512k;
}
error_page 404 /404.html;
location = /40x.html {
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
}
Any Idea why is this happening? I am using centos 7.1 abd nginx 1.6.3
Thanks in advance,
Todor

Answer 1

Finally found the reason and it has been clean for hours now. It turned out that there were 2 overlapping problems. The first one was the kernel dropping requests after the queue is full. Here is a nice manual about tuning the kernel linux kernel parameters - https://www.nginx.com/blog/tuning-nginx/
After the problems started we migrated the site to a new server and used DHCP to assign IP address - BIG MISTAKE. Every hour or so the dhcp restarted the network Interface. Examining the entire system log I noticed that the IP of the network interface is reassigned on regular intervals. Those intervals coincided with the bursts of errors in the log. So the solution was to go back to static IPs.

Answer 2

You have a upstream block like this:
upstream main_upstream {
server 192.168.158.142:80 max_fails=3 fail_timeout=60s; # New Server. Sent to 192.168.90
# server 192.168.158.143:80; # HSB
keepalive 32;
}
The first line fetches requests from 192.168.158.142:80 and stop trying after 3 fails for 60 seconds.
The second line is commented out, so it doesn't work.
Also there are other ip's listed on your other comments, which obviously won't work.
Assuming you want to fetch requests from 192.168.158.142:80 and failover to another ip 192.168.158.143:80 if it fails, this is how you do it:
upstream main_upstream {
keepalive 30;
server 192.168.158.142:80; # main server
server 192.168.158.143:80 backup; # failover
}
Or if you want to distribute the load evenly in a round robin style:
upstream main_upstream {
keepalive 30;
server 192.168.158.142:80; # server 1
server 192.168.158.143:80; # server 2
}
In both cases, make sure you can access both 192.168.158.142:80 and 192.168.158.143:80
Also bear in mind that you're using keepalive, meanning that if your backend servers are set to limit requests in any way, they might start refusing to serve requests to your frontend nginx (I will assume this is your load balancer).