I'm not sure if this is the correct place to post this. Please let me know if it is not.
Basically, exactly what the title of the question is. I have an haproxy configration file where I'm trying to set the timeout to 3600s. However, it seems that at random, the file will revert to a previous iteration, with much shorter timeout values.
What I set is as follows:
defaults
log global
mode http
retries 3
timeout client 3600s
timeout connect 3600s
timeout server 3600s
option tcplog
balance roundrobin
listen admin
bind 127.0.0.1:22002
mode http
stats enable
stats show-node
stats uri /admin
listen stats :1936
mode http
log global
maxconn 10
timeout client 3600s
timeout connect 3600s
timeout server 3600s
timeout queue 3600s
stats enable
stats hide-version
stats show-node
stats uri /haproxy?stats
However, it somehow changes to the following:
defaults
log global
mode http
retries 3
timeout client 50s
timeout connect 5s
timeout server 50s
option tcplog
balance roundrobin
listen admin
bind 127.0.0.1:22002
mode http
stats enable
stats show-node
stats uri /admin
listen stats :1936
mode http
log global
maxconn 10
clitimeout 100s
srvtimeout 100s
contimeout 100s
timeout queue 100s
stats enable
stats hide-version
stats show-node
stats uri /haproxy?stats
I have not found anything to indicate that haproxy 1.5 caches the configuration file, but I need to investigate all possibilities. Simply put: is there anything in haproxy that can cause this?
AFAIK, HAProxy doesn't implement anything that could explain that behavior.
How was installed HAProxy ? What's your OS ?
If you have auditd on your server, you can add a rule to watch which process is modifying the configuration file :
auditctl -w /etc/haproxy/haproxy.conf -p wa
Then watch for any activity in /var/log/audit/audit.log.
To delete the auditing :
auditctl -W /etc/haproxy/haproxy.conf
Related
Works fine under little load on our test bench but once we add to production the whole thing crashes and we are unable to get asterisk to function correctly. Almost as if there is a lag or delay in accessing the MariaDB cluster.
Our architecture and configs below;
Asterisk 13 Realtime with HAProxy(1.5.18) --> 6 x MariaDB(10.4.11) on independent Datacentres with Galera syncing them (1 only as backup)
Galera Sync is working fine and other services are able to read/write via the HAProxy 100%
Only seems to become and issue when we add load or we reload the dialplan or restart asterisk etc.
[haproxy.cfg]
global
user haproxy
group haproxy
defaults
mode http
log global
retries 2
timeout connect 3000ms
timeout server 10h
timeout client 10h
listen stats
bind *:8404
stats enable
stats hide-version
stats uri /stats
listen mysql-cluster
bind 127.0.0.1:3306
mode tcp
option mysql-check user haproxy_check
balance roundrobin
server mysql_server1 10.0.0.1:3306 check
server mysql_server2 10.0.0.2:3306 check
server mysql_server3 10.0.0.3:3306 check
server mysql_server4 10.0.0.4:3306 check
server mysql_server5 10.0.0.5:3306 check
server mysql_server6 10.0.0.6:3306 check backup
Really we would like to know if firstly Asterisk 13 Realtime will work via HAProxy and if so are there config changes we need to make to get it working.
Can provide more info if required
Try use Realtime->ODBC->haproxy.
If not help, use debugging, for example, gdb traces.
There is no way to determine what issue you have. Need more logs and configs.
I have implemented InnoDB cluster using MySql router(version - 2.1.4) for HA
This is my mysqlrouter.conf file
[DEFAULT]
user=mysqlrouter
logging_folder=
runtime_folder=/tmp/mysqlrouter/run
data_folder=/tmp/mysqlrouter/data
keyring_path=/tmp/mysqlrouter/data/keyring
master_key_path=/tmp/mysqlrouter/mysqlrouter.key
[logger]
level = DEBUG
[metadata_cache:magentoCluster]
router_id=49
bootstrap_server_addresses=mysql://ic-1:3306,mysql://ic-2:3306,mysql://ic-3:3306
user=mysql_router49_sqxivre03wzz
metadata_cluster=magentoCluster
ttl=1
[routing:magentoCluster_default_rw]
bind_address=0.0.0.0
bind_port=6446
destinations=metadata-cache://magentoCluster/default?role=PRIMARY
mode=read-write
protocol=classic
[routing:magentoCluster_default_ro]
bind_address=0.0.0.0
bind_port=6447
destinations=metadata-cache://magentoCluster/default?role=ALL
mode=read-only
protocol=classic
[routing:magentoCluster_default_x_rw]
bind_address=0.0.0.0
bind_port=64460
destinations=metadata-cache://magentoCluster/default?role=PRIMARY
mode=read-write
protocol=x
[routing:magentoCluster_default_x_ro]
bind_address=0.0.0.0
bind_port=64470
destinations=metadata-cache://magentoCluster/default?role=ALL
mode=read-only
protocol=x
MySql Router split the read requests to slave nodes, if I down slave 1 then router takes some seconds to know the slave 1 is down. So the requests are sent to the down slave node and the request fails. Any Suggestion how to handle this failure?
The client should always check for errors. This is a necessity for any system, because network errors, power outages, etc, can occur in any configuration.
When the client discovers a connection failure (failure to connect / dropped connection), it should start over by reconnecting and replaying the transaction it is in the middle of.
For transaction integrity, the client must be involved in the process; recovery cannot be provide by any proxy.
I'm trying to start a Haproxy loadbalancer with the following configuration:
global
log 127.0.0.1 local0
log 127.0.0.1 local0 notice
resolvers docker
nameserver dnsmasq 1.2.3.4:53
defaults
mode http
log global
option httplog
option dontlognull
frontend ft_radix_real
bind *:61616
maxconn 6000
acl is_websocket hdr(Upgrade) -i WebSocket
acl is_websocket hdr_beg(Host) -i ws
use_backend bk_radix_real if is_websocket
backend bk_radix_real
balance roundrobin
option forwardfor
server radix-real-1 1.2.3.4:1884 check resolvers docker resolve-prefer ipv4
server radix-real-2 1.2.3.4:1884 check resolvers docker resolve-prefer ipv4
server radix-real-3 1.2.3.4:1884 check resolvers docker resolve-prefer ipv4
server radix-real-4 1.2.3.4:1884 check resolvers docker resolve-prefer ipv4
listen stats
mode http
option httplog
option dontlognull
bind *:1936
stats enable
stats scope ft_radix_real
stats scope bk_radix_real
stats uri /
stats realm Haproxy\ Statistics
stats auth admin:admin
This configuration works when all backend servers are up. However, I would like to be able to start Haproxy even if some(NOT ALL) of the backend servers are not running. I checked the configuration document but couldn't find a solution. Is it possible?
Since 1.7 you can start HAproxy without resolving all the hosts on startup:
defaults
# never fail on address resolution
default-server init-addr none
https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#init-addr
I don't see any problem, the checks are there for this. Servers will be checked, the dead ones will be marked down and only the remaining valid ones will handle the traffic. You probably need to describe what type of issue you're facing exactly.
I have an app hosted on Openshift. We have a funtionality that let the user upload file onto $OPENSHIFT_DATA_DIR, then a nodeJS funtion is called to insert into our DB. In case of big tables this operation may take 5-7 minutes to be completed.
BUT, before the server complete the operation the client side got disconected and a Gateway Time-out error appears at 120000ms, the server side process continue the operation, and after sometime is completed, but the client side goes with this horrible error.
I need to know where I can edit those 120000ms. I edited the haproxy with different values but timeout is still 120s. Is there another file somewhere?
retries 6
timeout http-request 8m
timeout queue 8m
timeout connect 8m
timeout client 8m
timeout server 8m
timeout http-keep-alive 8m
found 2 haproxy files:
haproxy/conf/haproxy/haproxy.cfg
haproxy/versions/1.4/configuration/haproxy.cfg
both are edited
I guess there is multiple timeouts out there, but need to know where they are, or how to change client-side timeout
The app Gears: 3
haproxy-1.4 (Web Load Balancer)
Gears: Located with nodejs-0.10
nodejs-0.10 (Node.js 0.10)
postgresql-9.2 (PostgreSQL 9.2)
Gears: 1 small
smarterclayton-redis-2.6 (Redis)
5-7 minutes is an awfully long time for a web request. It sounds like this would be the perfect opportunity for you to explore using background tasks. Try uploading your data from the client and processing it in the background with something similar to delayed_job in rails.
I'm trying to add an endpoint to an existing application that sends Server Sent Events. There often may be no event for ~5 minutes. I'm hoping to configure that endpoint to not cut off my server even when the response has not been completed in ~1min, but all other endpoints to timeout if the server fails to respond.
Is there an easy way to support server sent events in HAProxy?
Here is my suggestion for HAProxy and SSE: you have plenty of custom timeout options in HAProxy, and there is 2 interesting options for you.
The timeout tunnel specifies timeout for tunnel connection - used for Websockets, SSE or CONNECT. Bypass both server and client timeout.
The timeout client handles the situation where a client looses their connection (network loss, disappear before the ACK of ending session, etc...)
In your haproxy.cfg, this is what you should do, first in your defaults section :
# Set the max time to wait for a connection attempt to a server to succeed
timeout connect 30s
# Set the max allowed time to wait for a complete HTTP request
timeout client 50s
# Set the maximum inactivity time on the server side
timeout server 50s
Nothing special until there.
Now, still in the defaults section :
# handle the situation where a client suddenly disappears from the net
timeout client-fin 30s
Next, jump to your backend definition and add this:
timeout tunnel 10h
I suggest a high value, 10 hours seems ok.
You should also avoid using the default http-keep-alive option, SSE does not use it. Instead, use http-server-close.