I am trying to create a valgrind (cachegrind) analysis of MySQL client connections.
I am running valgrind with --trace-children=yes.
What I want to find is one of the internal method calls, to see the call graph when it is being used...
After running valgrind --trace-children=yes ./bin/mysqld_safe
I get many dump files that were written that moment.
I am waiting 5 minutes (for letting the new files that I expect to be created to have a different "last modified" date.
After these 5 minutes I open 30 sessions, and floud the system with small transactions, and when I am done - shutdown the MySQL.
Now the questions:
1. After running 30 transactions and shutting down the system, only 3 files are modified. I expected to see 30 files, cause I though MySQL spans processes. So first - can someone confirm MySQL spans threads and not processes for each session?
I see three different database log calls: one to a DUMMY, one to binlog, and one to the innodb log. Can someone explain why the binlog and the DUMMY are there, and what's the difference between them? (I guess the DUMMY is because of the innodb, but I don't understand why the binlog is there if my first guess is true).
Is there a better way to do this analysis?
Is there a tool like kcachegrind that can open multiple files and show the summery from all of them? (or is it possible somehow within kcachegrind?)
Thanks!!
btw - for people who extend and develop MySQL - there are many interesting things there that can be improved....
I can only help you on some issues: Yes, MySQL does not create processes, but threads, see the manual on the command which lists what is currently done by the server:
When you are attempting to ascertain what your MySQL server is doing,
it can be helpful to examine the process list, which is the set of
threads currently executing within the server.
(Highlighting by me.)
Concerning the logs: Binary log is the log used for replication. This contains all executed statements (or changed rows) and will be propagated to slaves.
The InnoDB log is independent from the binary log and is used to assure that InnoDB performs ACID conform. Transactions are inserted there first and this file is used if the server crashed and InnoDB starts a recovery.
It is kind of normal that both logs are filled on a normal server.
I cannot help you with your other questions though. Maybe you want to ask your question on dba.stackexchange.com
Related
I have MySQL 5.7.24 running on a Windows VM. It has a few thousand databases (7000). I understand this is not the recommended set up for MySQL but some business requirements have necessitated this multi-tenant db structure and I cannot change that unfortunately.
The server works fine when it is running but the startup time can get pretty long, almost 20-30 mins after a clean shutdown of the MySQL service and 1+ hours after a restart of the Windows VM.
Is there any way to reduce the startup time?
In my configuration, I observed that innodb_file_per_table = ON (which is the default for MySQL 5.7 I believe) and so I think that at startup it is scanning every .ibd file.
Would changing innodb_file_per_table = OFF and then altering each table to get rid of the .ibd files be a viable option. One thing to note is that in general, every database size is pretty small and even with 7000 databases, the total size of the data is about 60gb only. So to my understanding, innodb_file_per_table = ON is more beneficial when there are single tables that can get pretty large which is not the case for my server.
Question: Is my logic reasonable and could this innodb_file_per_table be the reason for the slow startup? Or is there some other config variable that I can change so that each .ibd file is not scanned before the server starts accepting connections.
Any help to guide me in the right direction would be much appreciated. Thanks in advance!
You should upgrade to MySQL 8.0.
I was working on a system with the same problem as yours. In our case, we had about 1500 schemas per MySQL instance, and a little over 100 tables per schema. So it was about 160,000+ tables per instance. It caused lots of problems trying to use innodb_file_per_table, because the mysqld process couldn't work with that many open file descriptors efficiently. The only way to make the system work was to abandon file-per-table, and move all the tables into the central tablespace.
But that causes a different problem. Tablespaces never shrink, they only grow. The only way to shrink a tablespace is to move the tables to another tablespace, and drop the big one.
One day one of the developers added some code that used a table like a log, inserting a vast number of rows very rapidly. I got him to stop logging that data, but by then it was too late. MySQL's central tablespace had expanded to 95% of the size of the database storage, leaving too little space for binlogs and other files. And I could never shrink it without incurring downtime for our business.
I asked him, "Why were you writing to that table so much? What are you doing with the data you're storing?" He shrugged and said casually, "I dunno, I thought the data might be interesting sometime, but I had no specific use for them." I felt like strangling him.
The point of this story is that one naïve developer can cause a lot of inconvenience if you disable innodb_file_per_table.
When MySQL 8.0 was being planned, the MySQL Product Manager solicited ideas for scalability criteria. I told him about the need to support instances with a lot of tables, like 160k or more. MySQL 8.0 included an all-new implementation of internal code for handling metadata about tables, and he asked the engineers to test the scalability with up to 1 million tables (with file-per-table enabled).
So the best solution to your problem is not to turn off innodb_file_per_table. That will just lead to another kind of crisis. The best solution is to upgrade to 8.0.
Re your comment:
As far as I know, InnoDB does not open tables at startup time. It opens tables when they are first queried.
Make sure you have table_open_cache and innodb_open_files tuned for your scale. Here is some reading:
https://dev.mysql.com/doc/refman/5.7/en/table-cache.html
https://www.percona.com/blog/2009/11/18/how-innodb_open_files-affects-performance/
https://www.percona.com/blog/2018/11/28/what-happens-if-you-set-innodb_open_files-higher-than-open_files_limit/
https://www.percona.com/blog/2017/10/01/one-million-tables-mysql-8-0/
I hope you are using an SSD for storage, not a spinning disk. This makes a huge difference when doing a lot of small I/O operations. SSD storage devices have been a standard recommendation for database servers for about 10 years.
Also this probably doesn't help you but I gave up on using Windows around 2007. Not as a server nor a desktop.
We are creating a data pipeline from Mysql in RDS to elastic search for creating search indexes,
and for this using debezium cdc with its mysql source and elastic sink connector.
Now as the mysql is in rds we have to give the mysql user LOCK TABLE permission for two tables we wanted cdc, as mentioned in docs.
We also have various other mysql users performing transactions which may require any of the two tables.
As soon as we connected the mysql connector to our production database there was a lock created and our whole system went down, after realising this we soon stopped the kafka and also removed the connector, but the locks where still increasing and it only solved after we stop all the new queries by stopping our production code from running and manually killing the processes.
What could be the potential cause for this, and how could we prevent this ?
I'm only guessing because I don't know your query traffic. I would assume the locks you saw increasing were the backlog of queries that had been waiting for the table locks to be released.
I mean the following sequence is what I believe happened:
Debezium starts table locks on your two tables.
The application is still working, and it is trying to execute queries that access those locked tables. The queries begin waiting for the lock to be released. They will wait for up to 1 year (this is the default lock_wait_timeout value).
As you spend some minutes trying to figure out why your site is not responding, a large number of blocked queries accumulate. Potentially as many as max_connections. After all the allowed connections are full of blocked queries, then the application cannot connect to MySQL at all.
Finally you stop the Debezium process that is trying to read its initial snapshot of data. It releases its table locks.
Immediately when the table locks are released, the waiting queries can proceed.
But many of them do need to acquire locks too, if they are INSERT/UPDATE/DELETE/REPLACE or if they are SELECT ... FOR UPDATE or other locking statements.
Since there are so many of these queries queued up, it's more likely for them to be requesting locks that overlap, which means they have to wait for each other to finish and release their locks.
Also because there are hundreds of queries executing at the same time, they are overtaxing system resources like CPU, causing high system load, and this makes them all slow down too. So it will take longer for queries to complete, and therefore if they are blocked each other, they have to wait longer.
Meanwhile the application is still trying to accept requests, and therefore is adding more queries to execute. They are also subject to the queueing and resource exhaustion.
Eventually you stop the application, which at least allows the queue of waiting queries to gradually be finished. As the system load goes down, MySQL is able to process the queries more efficiently and finishes them all pretty soon.
The suggestion by the other answer to use a read replica for your Debezium snapshot is a good one. If your application can read from the master MySQL instance for a while, then no query will be blocked on the replica while Debezium has it locked. Eventually Debezium will finish reading all the data, and release the locks, and then go on to read only the binlog. Then the app can resume using the replica as a read instance.
If your binlog uses GTID, you should be able to make a CDC tool like Debezium read the snapshot from the replica, then when that's done, switch to the master to read the binlog. But if you don't use GTID, that's a little more tricky. The tool would have to know the binlog position on the master corresponding to the snapshot on the replica.
If the locking is problem and you cannot afford to tradeoff locking vs consistency then please take a look at snapshot.locking.mode config option.
Use the replica to prevent lock table statement getting executed, why debezium need lock table? all CDC tool fetch the events from bin logs.
The reason is that debezium is not as written in the document (version 1.5). Once FTWRL acquisition fails, it will execute the lock table. It will be released after the snapshot is read. If you see in the log that "Unable to refresh and obtain the global read lock, the table read lock will be used after reading the table name", congratulations, lucky one
After a recent DB crash with no RCA from our vendor. I'm left want to expand my knowledge on what sort of things can cause a database to crash.
In our specific case the logs show record index mis-matches just before the crash. We believe this was due to copying .frm and .idb files to another db rather than using mysqldump. There are logged warnings about that for about a week prior to the crash, starting when they were copied. But would it really take so long for the DB to crash?
In my experience, most crashes are due to hardware errors, i.e. your disk is failing. Second most common is user error, like moving InnoDB tablespaces around like they're normal files (you already know, don't do that). Third is bugs in MySQL, because all software has bugs.
It's certainly possible to take an indefinite amount of time before user activity accesses the code path that causes a crash. Can't make any conclusion from that.
Ultimately, you have to create redundancy to protect against crashes. This is especially important for databases. Examples of redundancy:
Use RAID 1 or RAID 10 to do disk mirroring
Use replication to copy data another MySQL instance continuously
Host the other MySQL instance on another physical computer, best if it's on a separate rack in your data center
You might even have another replica in another region of the country
Last week I noticed after a crash that my mysql log file had become so large that it consumed the disk - not a massive disk. I recently implemented a new helpdesk/ticketing system which was adopted by the entire company much quicker than was anticipated thus a log file with 99% selects.
So my question is this; Can I retain mysql logging but exclude select statements? Further more can I keep select statements but exclude certain databases(i.e. helpdesk)?
Thanks for any response
You can't restrict MySQL General log file to certain database or certain DML statements. It logs everything being executed on your MySQL server and ofcourse it's a overhead on a MySQL server in production environment.
I suggest you to turn-off General log on production server and enable slow query log with appropriate settings so that only problamatic queries will be logged which needs attention, later you can optimize those queries to achieve better MySQL performance.
If you still needs general log to be enabled then make sure that logrotate script is used for General log file which will keep it's size to a certain limit.
http://www.thegeekstuff.com/2010/07/logrotate-examples/
Assume a number of conventional LAMP-style applications which use MySQL as a back-end to record the 'current durable state' for the applications.
I am interested in establishing an 'audit' of transitions at the database level - and storing them as a log. The idea is that - assuming the MySQL database has been 'dumped' at the beginning of the day, it would be possible to 'replay' transactions against the back-up to recover any state during the working day.... A bit like time-machine for MySQL - I guess.
I have found some documentation about "Audit plugins" which look relevant but leaves me with more questions than answers.
http://dev.mysql.com/doc/refman/5.6/en/writing-audit-plugins.html
Essentially, I'd like to establish if it would be feasible to write a MySQL plugin to achieve my goal - such that it would work 'seamlessly' with existing MySQL applications?
The principal detail I'm finding it difficult to ascertain is this: When the audit-plugin is notified of an event, what is the mechanism by which the new data can be established in order to log it? How are data types encoded? How hard would it be to write a tool to 'replay' this audit against a 'full-system-backup' using mysqldump, for example?
Are there any existing examples of such plugins?
You just want MySQL's Point-in-Time (Incremental) Recovery Using the Binary Log:
Point-in-time recovery refers to recovery of data changes made since a given point in time. Typically, this type of recovery is performed after restoring a full backup that brings the server to its state as of the time the backup was made. (The full backup can be made in several ways, such as those listed in Section 7.2, “Database Backup Methods”.) Point-in-time recovery then brings the server up to date incrementally from the time of the full backup to a more recent time