I am getting intermittent 'Connection Timed Out' errors when a php script on my web server connects to the MySQL database server over the private network. However, if I tell the script to use the public network to connect, these errors do not appear.
My connection script is setup so that whenever I try to connect to mysql, it checks for errors, if there is an error, it sends me an email then automatically switches to the public network to try that connection. If the public connection fails, it sends me another email and displays a custom web page to the user.
I get about 5 to 10 connection errors every hour. There are hundreds of successful connections every minute.
These machines are dedicated machines. I contacted our hosting company and they tested the routers and cables and said everything is fine. I tried pinging the servers both ways and there are no errors at all for test periods over an hour.
I am using the latest Nginx with the latest PHP and PHP-FPM. Mysql is 5.5.27. These are Centos 6 64bit systems with that latest updates.
I've tried many network configuration options, adjustments to php-fpm & mysql config file and no matter what I do or change, nothing fixes it.
The weird thing is, everything works great over the public network and pings and file transfer work great over the private network between both machines.
Any ideas?
** UPDATE **
I made some changes to the PHP-FPM config file and to the MySQL config file and the errors are now about 2 to 3 per hour but still unresolved.
I'm not sure this is your case but still worth mentioning as it helped me in a similar situation. Basically, there is a cap on max number of connections in linux kernel: https://serverfault.com/questions/10852/what-limits-the-maximum-number-of-connections-on-a-linux-server
Not sure if it is shared between all the networks, but if you think it's worth checking I'd just raise those variable values say twice and see if it had any effect on how frequently the error happens.
Related
I have a running instance of VerneMQ (cluster of 2 nodes) on Google kubernets and using MySQL (CloudSQL) for Auth. Server accepts connections over TLS
It works fine, but after a few days i start seeing this message on the log:
can't authenticate client {[],<<"Client-id">>} from X.X.X.X:16609 due to plugin_chain_exhausted
The client app (paho) complains that the server refused the connection for being "not authorized (code=5 in paho error)"
after a few retry it finally connects. but every time it get's harder and harder until it just won't connect anymore
If i restart VerneMQ everything get's back to normal
I have only 3 clients currently connected at most, at the same time.
clients already connected have no issues in pub/sub.
In my configuration i have (among other things):
log.console.level=debug
plugins.vmq_diversity=on
vmq_diversity.mysql.* = all of them set
allow_anonymous=off
vmq_diversity.auth_mysql.enabled=on
it's like the server degrades over time. the status webpage reports no problem
My verne server was build from the git repository about a month ago and runs on a docker container
what could be the cause?
what else could i check to find posibles causes? maybe a diversity missconfiguration?
Tks
To quickly explain the plugin_chain_exhausted log: with Verne you can run multiple authentication/authorization plugins, and they will be checked in a chain. If one plugin allows the client, it will be in. If no plugin allows the client, you'll see the log above.
This does not explain the behaviour you describe, though. I don't think I have seen that.
In any case, the first thing to check is whether you actually run multiple plugins. For instance: have you disabled the vmq.passwd and the vmq.acl plugins?
Last night AWS RDS had an "Internet Connectivity Issue" that was resolved a short time later. However, my app (which runs in .NET Core and connects to an RDS MySQL instance via Pomelo.EntityFrameworkCore.MySql) could never re-establish a connection to the database even though the MySQL server was back online. I tested connecting from my own local machine and it worked just fine. I then re-deployed the .NET Core app it everything started working again.
Is there something that I need to re-create (the db context perhaps), or is there something that is being cached that I need to flush to try to connect again? I connect via hostname, and my connection string looks something like this:
server=something.somewhere.us-east-2.rds.amazonaws.com;userid=XXXX;password=YYYYY;database=ZZZZ
Here is the Exception being thrown:
MySqlException: Unable to connect to any of the specified MySQL hosts.
at MySqlConnector.Core.ServerSession+<ConnectAsync>d__56.MoveNext (C:\projects\mysqlconnector\src\MySqlConnector\Core\ServerSession.cs:239)
and here is how I create my db context in Startup.cs:
services.AddDbContext<BlayFapContext>(opt => opt.UseMySql(Settings.Instance.SQLConnectionString));
Any help would be greatly appreciated.
Giawa
Okay, we worked out what happened. Pomelo's MySQL wrapper had an issue as outlined in their git repo here: https://github.com/PomeloFoundation/Pomelo.EntityFrameworkCore.MySql/issues/434
Basically, if a MySQL database is not available when the connection string is first used then it will be cached as invalid and will never work again. You can easily confirm this by launching a service with no MySQL connectivity, verify it doesn't work, then launch MySQL and confirm that the service still doesn't work. It can never establish a MySQL connection after the first connection string is found to be invalid.
They patched it shortly after the 2.0.1 release, but they haven't updated Nuget with a new version since then, despite the issue being found 6 months ago. So, the fix is to checkout their repository source code, and patch it ourselves. We found the fix here works just fine: https://github.com/PomeloFoundation/Pomelo.EntityFrameworkCore.MySql/pull/456
So, why was the connection string retried? We already had a successful connection! It turns out that the internet connectivity issue with the Ohio data center was not limited to RDS, but also affected EC2. Our EC2 instance was rebooted as part of the fix, and the MySQL connection wasn't valid when it reboot due to the continued connectivity issues. The state of that connection was cached, and even though the MySQL server came back online our service was toast.
Giawa
Not sure what is going on....
MS Access front end (on Win7 and 10 PCs)
MySQL on DiskStation (MariaDB) back end.
Connection is through ODBC 5.3.
All was perfect and great until one day I lost internet service for about 2 hours. Once back up and running, we had a new dynamic IP. No problem. I changed the ODBC connection to the new IP and we're back up and running.
Then, a short time later, I start getting ODBC connection errors. Either "connection failed" or "gone away". Sometimes once a day, sometimes all machines (5 connections) randomly, many times throughout the day.
Sometimes if I just close and restart Access, the connection will enable. Sometimes it won't. Other times, I will "test" the ODBC driver and the first time it will fail, and the second time it will say "success", then all is well again.
Checked global wait timeout and it's set to 28800.
Since users connect both inside and outside of our LAN, the driver is set to our local IP, and then my Linksys router port forward 3806 to the internal IP of the DiskStation. Again, until Rogers dropped service and I was forced to use a different IP, all was well in our little network world.
Can anyone shed some light? Keep in mind, I am limited to my tech knowledge.
Yesterday, my website crashed. Or, what appears to be a crash.
My host (JustHost) have disclosed that it was a SAN problem and have been rebuilding it. It's hard to get anymore information out of them because they are playing their cards very close to their chest for some reason.
I have suffered almost 24 hours downtime in 2 days and I'm getting fed up.
The error I am receiving on the frontend of the website is:
Database connection error (2): Could not connect to MySQL.Database connection error (2): Could not connect to MySQL.
Trying to access my phpMyadmin from cPanel gives me the following error:
Accessing SSH and trying to start the mysql service via putty I get:
Starting MySQL. ERROR! The server quit without updating PID file (/var/lib/mysql /mysqld.pid).
Can anyone please help me resolve this issue as my host is rather useless!
Thanks!
Yep, mysqld is down. Did you try running "mysqld_safe&"?
For anyone who wishes to know the conclusion of this issue.
I have been left with 2 choices:
1) Be patient and wait for Just Host to fix the issue and potentially lose my Google search rankings and a lot more future revenue
2) Set up a new hosting account, transfer my data over and wait for the domain to propagate.
I have chosen the latter of the two. My confidence in Just Host as a customer service hub and web hosting provider has thoroughly diminished to the point where all of my hosting accounts will now be switched over to another provider.
They have let me down by not fixing the issue in good time and have passed me from pillar to post in live chat, email support and phone support. The final nail in the coffin was my request for them to backup my databases and send them to me via email being met with the resoundingly passive "sorry, we do not provide that service" response.
I would do it myself if only I had access to phpMyadmin, which I do not have because of a server error that they have been incapable of fixing for the past 72 hours.
Goodbye Just Host.
I'm running a MsAccess with Mysql background database and today I am experiencing some problems.
Depending on how far I can get into the program I am able to get a list of all the names and entry. But once I go into the edit form of any entry I get a variety of results. Sometimes I am lucky and I can see the first entry. But any entry after that will always get a MsAccess error: Object invalid or no longer set and then any subsequent calls after will yield a Mysql ODBC error of Mysql server has gone away.
I've looked around at several websites and even all the stack overflow sites with the same question and I've tried a variety of solutions. (Keep in mind that this database has been running for years and this is the first time I am getting this message) It also takes about 2m14s to 2m59s before it displays any error messages.
Here is what I tried so far:
I've tried changing a few of the ODBC connection settings.
Checked Allow Big Results - FAILED
Checked Enable Auto Reconnect - FAILED
I've Checked and repaired tables - FAILED
A good number of tables resulted in, "table needs update please repair table"
I've Doubled the numbers in the my.cnf configuration and restarted the mysql service as well as the msaccess software. FAILED
These were the settings of my my.cnf after I made the changes.
Here:
set-variable = max_connections=500
safe-show-database
log-error=/var/log/mysqld.log
connect_timeout=1000
interactive_timeout=28800
wait_timeout=288000 <**This was changed**
join_buffer_size=6M
key_buffer_size=300M <**THIS WAS CHANGED **
max_allowed_packet=300M <**THIS WAS CHANGED **
myisam_sort_buffer_size=300M <**THIS WAS CHANGED **
read_buffer_size=6M
sort_buffer_size=6M
table_cache=12288
thread_cache_size=24
tmp_table_size=132M
query_cache_limit=3M
query_cache_size=64M
query_cache_type=1
Basically I've tried every suggestion I could find so far and I can't seem to figure out the problem.
I've also had a look at MYSQL: Has gone away
If I go through the bulleted list there
Nobody has killed the running thread
I don't think the query was ran after the connection was closed (since it ran for years fine)
Client application does have the privileges needed
I don't know how to figure out if I had a timeout from the TCP/IP connection on client side
I don't know if I've encounted a timeout on the server side, but I do know that automatic reconnection in client is disabled
???
The query could potentially be large since it is a form with many subforms
???
DNS should be ok, since I can connect to it with a real sql viewer (HeidiSQL)
???Child forks???
???
I think this is one of those bugs that is hard to figure out since I'm exhausted :/ I'm probably missing some info but I am not sure what else to include.
---EDIT---
Thank you all for your comments, I'm still debugging this issue. It seems that it's not all the forms that are causing some issues. So I'm starting to think that this is a MSAccess issue more than it is an MySql issue. The forms that do break all have the same line in their VBA code:
Private Sub Form_Error(DataErr As Integer, Response As Integer)
If Screen.ActiveControl.Name = "UnboundTextBox" Then
Response = acDataErrContinue
End If
End Sub
I don't recall putting this code in, so I assume it's an automatic thing, but I'm still going through all my forms to see which ones cause this error and which do not. So more in a few.
---Today I'm doing some debugging trying to find whatever I can from whatever log I can muster ----
MSACCESS 17ec-b10 EXIT SQLStatisticsW with return code -1 (SQL_ERROR)
HSTMT 0AF82920
WCHAR * 0x00000000 [ -3] <empty string>
SWORD -3
WCHAR * 0x00000000 [ -3] <empty string>
SWORD -3
WCHAR * 0x0013AAE8 [ -3] "location\ 0"
SWORD -3
UWORD 1 <SQL_INDEX_ALL>
UWORD 0 <SQL_QUICK>
DIAG [08S01] [MySQL][ODBC 3.51 Driver][mysqld-5.0.92-community]Lost
connection to MySQL server during query (2013)
----------------------------- FINAL EDIT --------------------------
The past week or so, my boss has been working hard to install a new internal server for our office. We also switched IPs several times and ISP providers. Turns out, as a result he created for me an administrative nightmare, as I attempted to debug issues with the server, and issues with the software ect.. ect.. when the issue was in the settings the ISP providers established with the router settings etc... etc... SO basically this whole headache was nothing wrong with server, nor software but hardware inbetween.
So now, I have to backwards engineer everything I did and try to get it to work on both ips. It works on the important one now. But it's not working on the old one so... I guess I just have to figure that out. [Waving imaginary fist in the air] Lol
The problem is that the connection is timing out. This is not a setting that you can change at the ODBC level. My MySQL provider had set the timeout at 30 seconds (you can find out what yours is set at my running the PassThrough query "SELECT ##wait_timeout"). I got round it by polling the connection every 10 seconds with a simple form. That keeps the connection alive. Make sure you have an AutoExec macro which opens the form.
Private Sub Form_Open(Cancel As Integer)
Me.TimerInterval = 1000 ' 10 seconds
End Sub
Private Sub Form_Timer()
Me.Requery
End Sub
I have not used MySQL with Access in a production environment, but the problem you are having sounds suspiciously similar to one I've encountered with SQL Server. Basically, if the client workstation has incorrect DNS settings, it can get confused about how to resolve the connection to the SQL Server and lose the connection. In the cases where I've encountered this, the workstations were incorrectly configured with the Internet provider's DNS as primary DSN, but the SQL Server was a local server, available only on the local LAN (and not mapped by the ISPs DNS).
The solution is to use the local domain controller as the primary DNS. It will look up the local SQL Server's IP and pass off any non-local lookups to the Internet provider's DNS.
If you don't have a Windows/Samba domain controller, then you need to use whatever the local LAN's authorative name server is. Alternatively, you could connect to the MySQL server by IP address and avoid the problem, or map it in the client workstation's HOSTS file (which is not recommended -- it's a great way to cause really hard to troubleshoot problems when the IP addresses change).
[mysqld-5.0.92-community]Lost
connection to MySQL server during query (2013)
This genary means you have found a bug in MySQL-Server
can you post a tail of your mysqld.err and mysqd.log
this makes the server restart and will effectefly shut down all open connections