I want to write a MySQL daemon plugin that monitors queries on other MySQL servers and compares them with queries running on where the daemon is running. Basically this is in a spider engine setting, where a query initiated on the head node get run on the shards. Whenever a query is killed on the head, I want to have a daemon on the shard nodes that will kill the associated query there.
The idea was to initiate a pthread that uses mysql_real_connect, mysql_real_query... to access "show processlist" on the head node and compare them with local thread list. If I issue mysql_real_connect in the thread of the daemon, I get a segmentation fault. I think this is due to a threading issue in mysql_real_connect. I've used #define MYSQL_SERVER 1 and follow the approach taken in ha_federated::real_connect().
Is it actually possible to run mysql_real_connect from within a daemon plugin?
Thanks for any hints.
After meditating over the handlersocker daemon plugin I realised that in order for mysql_real_connect and friends to work in the daemon thread, a mysql internal thread environment needs to be set up and registered with the server.
This basically involves calling the following in the daemon thread:
my_thread_init();
thd = new THD;
mysql_mutex_lock(&LOCK_thread_count);
thd->thread_id = thread_id++;
mysql_mutex_unlock(&LOCK_thread_count);
thd->store_globals();
... setting up various options in THD ...
mysql_mutex_lock(&LOCK_thread_count);
threads.append(thd);
++thread_count;
mysql_mutex_unlock(&LOCK_thread_count);
Related
I am very frequently getting this error in MySQL:
OS errno 24 - Too many open files
What's the cause and what are the solutions?
I was getting the errno: 24 - Too many open files too often when i was using many databases at the same time.
Solution
ensure that the connections to db server close propertly
edit /etc/systemd/system.conf. Uncomment and make
DefaultLimitNOFILE=infinity
DefaultLimitMEMLOCK=infinity
then run systemctl daemon-reload and service mysql restart.
You can check the results with the query: SHOW GLOBAL VARIABLES LIKE 'open_files_limit' and you may notice that the value has changed. You should not have any errno 24 now.
Please notice that the solution may differ from other OS/versions. You can try to locate the variables first.Tested with Ubuntu 16.04.3 and mysql 5.7.19.
In my case it was useless to setting up the open_files_limit variable in mysql configuration files as the variable is flagged as a readonly.
I hope it helped!
You probably have a connection leak in your application, that is why open connections are not closed once the function completes it's execution.
I would probably look into the application code and see where the connections/preparedstatement (if it's java) objects are not closed and fix it.
A quick workaround is to increase ulimit of the server (explained here) which would increase number of open file descriptors (i.e. connections). However, if you have a connection leak, you will encounter this error again, at later stages.
I faced the same problem and found a solution on another stackoverflow-question.
By running the following snippet with Bash:
ulimit -n 30000
OK something odd and maybe im missing something simple here. I am trying to diagnose a problem I think that max connections are being reached and need to up them. Upon further looking I ran # service --status-all and then I noticed that mysqld was not in the list so I tried service mysqld status which returned mysqld: unrecognized service . but if I run mysql - p MYPASS I am logged in to mysql and can see all my databases as normal. also I log into phpmyadmin just fine all databases are there. Also have several scripts running using 'localhost' and my mysqlusername and pass, and they are running just fine...
So I am not sure why I am not seeing the mysql service, anyone have some idea? or can point me in some direction?
I am using Centos, I did not set this machine up that person is no longer here, I have several other servers running mysql as well and I checked all those all is right.
It sounds like the service has not been registered.
On a working box do;
chkconfig --list mysqld it should return something like;
mysqld 0:off 1:off 2:off 3:on 4:off 5:off 6:off
Now do the same on the box in question - if nothing comes back then you need to chkconfig --add mysqld then re run the above commands to verify its registered, then you should be able to access it via the service manager.
When I start infinidb, I am getting the following error message, making it impossible for the service to start. In the log file, it says there is a problem with the rollback. Is there a way to abort the rollback so I can start the service again? I am ok with losing some data.
Starting Calpont InfiniDB Database Platform: ....... DONE
Warning! The DBRM is currently in Read-Only mode!
Updates will not propagate!
Starting Calpont InfiniDB MySQL:
Starting MySQL
. *
Validate InfiniDB System Catalog:
Validation Succesfully Completed
Perform Functionality test:
InfiniDB Logging check: DONE
Platform Process check: ERROR: Problem with InfiniDB process DMLProc, should be a single version running
***Stopping InfiniDB to allow the process problem to be resolved.
Shutting down Calpont InfiniDB MySQL:
Shutting down MySQL
... *
Shutting down Calpont InfiniDB Database Platform:
/var/log/Calpont/crit.log
DMLProc[4292]: 41.458706 |0|0|0| C 20 CAL0002: DMLProc failed to start due to : problem with rollback. Version buffer file does not exists.
I already tried removing the files in /usr/local/Calpont/data1/systemFiles/dataTransaction but it still won't start.
This is how I was able to solve it.
While the service is starting:
# service infinidb start
In another ssh session go to /usr/local/Calpont/bin and look for the transaction ID causing the problem:
# ./rollback -p
Once you find out, immediately run the rollback command with the transaction id that you see:
./rollback -r 64536
It will respond with OK and the service started successfully this time. I had to do this twice because my first attempt was probably not able to rollback the transaction on time.
We have 8 phusion passengers with 20 connections each. should be 160 max connection. Today our Mysql connection crossed 300 and our server stopped responding.
What would happen if the thread dies unnaturally ? how do db connections associated with it get cleaned-up?
How to debug this type of scenario ?
What would happen if the thread dies unnaturally ?
If you are using your statements under transaction then all statements under un-successful transactions will be rolled back but if you are executing your statements individually then how many have been executed will be saved in db but other un-successful will not and data can be in-consistant.
how do db connections associated with it get cleaned-up?
As per my assumption your application is opening connections more than required and also not closing properly. In this case if you check connections in mysql admininstrator then you will get so many connections in sleep mode. So you can kill all sleep connections to clean them.
How to debug this type of scenario ?
Step1: Enable general logs on your server:
Step2: execute below command in any gui tool like sqlyog etc.
show full processlist;
Step3: Take first sleep process from above command:
Step3: find/grep above process id (suppose it is 235798) in your general log. You can use below command.
$ cat /var/log/mysqldquery.log | grep 235798
Note: there can be different file name and path for your general log file.
Above command will show you few lines, check if you are closing connection at the end, should show "quite" statement at the end of line. In this way you need to check few processes those are in sleep mode and you can jugde which type statements/from which module opening extra connectiions and not closing and accordingly you can take action.
Can anyone tell me why after reboot a Google Compute Engine instance i get a ssh connection time out. I reboot the instance by sudo reboot and by Google Compute Engine Console and both do the same.
When the OS shuts down to reboot, all network connections are closed, including SSH connections. From the client side, this can look like a connection time out.
When you use gcutil resetinstance, it does the same thing as pushing the power button on a physical host. This is different from e.g. sudo reboot, because the former does not give the operating system a chance to perform any shutdown (like closing open sockets, flushing buffers, etc), while the latter does an orderly shutdown.
You should probably prefer logging in to the instance to do a reboot rather than using gcutil resetinstance if the host is still ssh-able; resetinstance (or the "Reboot Instance" button in the GUI) is a hard reset, which allows you to recover from a kernel crash or SSH failing.
In more detail:
During OS-initiated reboot (like sudo reboot), the operating system performs a number of cleanup steps and then moves to runlevel 6 (reboot). This causes all the scripts in /etc/init.d to be run and then a graceful shutdown. During a graceful shutdown, sshd will be killed; sshd could catch the kill signal to close all of its open sockets. Closing the socket will cause a FIN TCP packet to be sent, starting an orderly TCP teardown ("Connection closed" message in your ssh client). Alternatively, if sshd simply exits, the kernel sends a RST (reset) packet on all open TCP sockets, which will cause a "Connection reset" message on your ssh client. Once all the processes have been shut down, the kernel will make sure that all dirty pages in the page cache are flushed to disk, then execute one of two or three mechanisms to trigger a BIOS reboot. (ACPI, keyboard controller, or triple-fault.)
When triggering an external reset (e.g. via the resetinstance API call or GUI), the VM will go immediately to the last step, and the operating system won't have a chance to do any of the graceful shutdown steps above. This means your ssh client won't receive a FIN or RST packet like above, and will only notice the connection closed when the remote server stops responding. ("Connection timed out")
Thank you Brian Dorsey, E. Anderson and vgt for answering my question. The problem was other. Every time that i reseted the connection previously i up an ethernet bridge with the brigde-util utility between the "eth0" inferface and a new brigde interface called "br0". After reset the instance by sudo reboot or by GCE Console, ssh connection stopped working.
But if i don't up the ethernet bridge the instance restart ok by both methods.
If your instance image is CentOS, try to remove selinux.
sudo yum remove selinux*
Slightly orthogonal to Brian's answer. To gracefully reboot a GCE VM you can use:
gcutil resetinstance <instancename>