I have a mysql (Percona) 5.7 instance with over 1Million tables.
When I start the database, it can take more than an hour to start.
Errorlog doesn't show anything, but when I trace mysqld_safe, I found out that MySQL is getting a stat on every file in the DB.
Any idea why this may happen?
Also, please no suggestion to fix my schema, this is a blackbox.
Thanks
This turned out to be 2 issues (other than millions of tables)!
When MySQL start, and a crash recovery is needed, as of 5.7.17, it needs to traverse your datadir to build it's dictionary. This will be lifted in future releases(8.0), as MySQL will have it's own catalog, and will not rely on datadir content anymore. Doc states that this isn't done anymore. It's true and false. It doesn't read the 1st page of ibd files, but It does a file stat. Filed Bug
Once it finished (1), it starts a new process, "Executing 'SELECT * FROM INFORMATION_SCHEMA.TABLES;' to get a list of tables using the deprecated partition engine.". That of course open all the files again. Use disable-partition-engine-check if you think you don't need it. Doc
All this can be observed using sysdig. very powerful handy dtrace like tool.
sysdig proc.name=mysqld | grep "open fd="
Ok, now, it's time to reduce the number of files.
Related
I'm using MardiaDB and i'm wondering if there is a way how to install a 'recycle bin' on my server where if someone deleted a table or anything it gets shifted to the recycle bin and restoring it is easy.
Not talking about mounting things to restore it and all that stuff but litterly a 'save place' where it gets stored (i have more then enough space) until i decide to delete it or just keep it there for 24 hours.
Any thoughts?
No such feature exists. http://bugs.mysql.com takes "feature requests".
Such a feature would necessarily involve MySQL; it cannot be done entirely in the OS's filesystem. This is because a running mysql caches information in RAM that the FS does not know about. And because the information about a table/db/proc/trigger/etc is not located entirely in a single file. Instead extra info exists in other, more general, files.
With MyISAM, your goal was partially possible in the fs. A MyISAM table was composed of 3 files: .frm, .MYD',.MYI`. Still MySQL would need to flush something to forget that it know about the table before the fs could move the 3 files somewhere else. MyISAM is going away; so don't even think about using that 'Engine'.
In InnoDB, a table is composed of a .ibd file (if using file_per_table) plus a .frm file, plus some info in the communal ibdata1 file. If the table is PARTITIONed, the layout is more complex.
In version 8.0, most of the previous paragraph will become incorrect -- a major change is occurring.
"Transactions" are a way of undoing writes to a table...
BEGIN;
INSERT/UPDATE/DELETE/etc...
if ( change-mind )
then ROLLBACK;
else COMMIT;
Effectively, the undo log acts as a recycle bin -- but only at the record level, and only until you execute COMMIT.
MySQL 8.0 will add the ability to have DDL statements (eg, DROP TABLE) in a transaction. But, again, it is only until COMMIT.
Think of COMMIT as flushing the recycle bin.
Work has a web site that uses large data sets, load balanced between two MySQL 5.6.16-64.2 servers using MyISAM, running on Linux (2.6.32-358.el6.x86_64 GNU/Linux.) This data is being updated hourly from a text based file set that is received from a MS-SQL database. To avoid disruption on reads from the web site and at the same time make sure the updates doesn't take too long following process was put in place:
Have the data one a third Linux box (only used for data update processing,) update the different data tables as needed, move a copy of the physical table files to the production servers under a temporary name, and then do a table swap by MySQL TABLE RENAME.
But every time the table (under the temporary name) is seen by the destination MySQL servers as being crashed and require repair. The repair takes too long, so it cannot be forced to do a repair before doing the table swap.
The processing is programmed in Ruby 1.8.7 by having a thread for each server (just as a FYI, this also happens if not doing it in a thread to a single server.)
The steps to perform file copy is as follows:
Use Net::SFTP to transfer the files to a destination folder that is not the database folder (done due to permissions.) Code example of the file transfer for the main table files (if table also has partition files then they are transferred separately and rspFile is assigned differently to match the temporary name.) For speed it is parallel uploaded:
Net::SFTP.start(host_server, host_user, :password => host_pwd) do |sftp|
uploads = fileList.map { |f|
rcpFile = File.basename(f, File.extname(f)) + prcExt + File.extname(f)
sftp.upload(f, "/home/#{host_user}/#{rcpFile}")
}
uploads.each { |u| u.wait }
end
Then assign the files the owner and group to the mysql user and to move the files to the MySQL database folder, by using Net::SSH to execute sudo shell commands:
Net::SSH.start(host_server, host_user, :port => host_port.to_i, :password => host_pwd) do |ssh|
doSSHCommand(ssh, "sudo sh -c 'chown mysql /home/#{host_user}/#{prcLocalFiles}'", host_pwd)
doSSHCommand(ssh, "sudo sh -c 'chgrp mysql /home/#{host_user}/#{prcLocalFiles}'", host_pwd)
doSSHCommand(ssh, "sudo sh -c 'mv /home/#{host_user}/#{prcLocalFiles} #{host_path}'", host_pwd)
end
The doSSHCommand method:
def doSSHCommand(ssh, cmd, pwd)
result = ""
ssh.open_channel do |channel|
channel.request_pty do |c, success|
raise "could not request pty" unless success
channel.exec "#{cmd}" do |c, success|
raise "could not execute command '#{cmd}'" unless success
channel.on_data do |c, data|
if (data[/\[sudo\]|Password/i]) then
channel.send_data "#{pwd}\n"
else
result += data unless data.nil?
end
end
end
end
end
ssh.loop
result
end
If done manually by using scp to move the files over, do the owner/group changes, and move the files, then it never crashes the table. By checking the file sizes compared between scp and Net::SFTP there are no difference.
Other process methods has been tried, but experience they take too long compared to using the method described above. Anyone have an idea of why the tables are being crashed and if there a solution to avoid table crash without having to do a table repair?
The tables are marked as crashed because you're probably getting race conditions as you copy the files. That is, there are writes pending to the tables during your execution of your Ruby script, so the resulting copy is incomplete.
The safer way to copy MyISAM tables is to run the SQL commands FLUSH TABLES followed by FLUSH TABLES WITH READ LOCK first, to ensure that all pending changes are written to the table on disk, and then block any further changes until you release the table lock. Then perform your copy script, and then finally unlock the tables.
This does mean that no one can update the tables while you're copying them. That's the only way you can ensure you get uncorrupt files.
But I have to comment that it seems like you're reinventing MySQL replication. Is there any reason you're not using that? It could probably work faster, better, and more efficiently, incrementally and continually updating only the parts of the tables that have changed.
The issue was found and solved:
The process database had the table files copied from one of the production databases, and did not show crashed on the process server and no issues when query and updating the data.
While searching the web following SO answer was found: MySQL table is marked as crashed
So by guessing that when the tables was copied from production to the process server, that the header info stayed the same and might interfere when copied back to the production servers during the processor. So it was tried by repairing the table on the process server and then run a few tests on our staging environment where the issue was also experienced. And surely enough that corrected the issue.
So the final solution was to repair the tables once on the process server before having the process script run hourly.
I see you've already found an answer, but two things struck me about this question.
One, you should look at rsync which gives you many more options, not the least of which is a speedier transfer, that may better suit this problem. File transfer between servers is basically why rsync exists as a thing.
Second, and I'm not trying to re engineer your system but you may have outgrown MySQL. It may not be the best fit for this problem. This problem may be better served by Riak where you have multiple nodes, or Mongo where you can deal with large files and have multiple nodes. Just two thoughts I had while reading your question.
We have a MySQL slow query killer that kills process IDs after a specified number of seconds passes and that works fine. However we'd like in our notifications to also see the full query that was being so slow.
The problem is that while mysqladmin -v processlist|grep process_id should work, it truncates queries that have newlines in them e.g.:
SELECT * FROM table WHERE x=y {
... stuff
};
Here stuff will be cut off and the query thusly truncated. (I realize that that may not be syntactically correct as I'm not a DBA, but I just wanted to give an example of the kind of query flow we sometimes have to deal with in our applications; please don't complain about the format, it wasn't my decision nor is it under my control.)
Doing a query in information_schema would solve this, I believe, but the team does not want to do this due to the performance impact queries against that database often involve. So is there a better way to approach this then grepping mysqladmin?
I would advise to activate the slow log. It displays the statement that is taking long time,, how long and with extra details. Unless its in a STORED PROCEDURE.
You first need to activate the slow log in case it isnt allready.
To enable the slow query log, start mysqld with the --log-slow-queries[=file_name] option or change the values in your config file and restart the service, see below.
If the slow query log file is enabled but no name is specified, the
default name is host_name-slow.log and the server creates the file in
the same directory where it creates the PID file. If a name is given,
the server creates the file in the data directory unless an absolute
path name is given to specify a different directory.
You can set the output (where the data can be read) to FILE or TABLE
Change your my.ini file and change/add them
log-output = TABLE
slow-query-log=1
long_query_time = 10
log-queries-not-using-indexes
This will log any query that takes longer than 10 seconds to complete. Plus any query that is not using a index.
If you go with log-output=table then you can simply execute
select * from mysql.slow_log
If you go with log-output=file then you have to open the physical file in the MySql folder.
here we have a x64 Debian Lenny with MySQL 5.1.47 and some InnoDB databases on it. ibdata files and other stuff is on the same file system (ext3). I've noticed, that on some situations there are many processes in MySQL process list which hang on "freeing items" state. This happens when I do the following on shell (file1 and file2 are about 2.5gb)
cat file1 file2 >new_file
or execute the following SQL statement
SELECT 'name' AS col UNION SELECT col FROM db_name.table_name INTO OUTFILE ('/var/xxx/yyy')
When one of these two things is running, then I can see many MySQL processes running endless with "freeing items" state (I'm using innotop). When killing this shell process (or SQL statement) then these blocked transactions disappear.
In internet I found some hints to disable InnoDB adaptive hash index and common query cache, but this doesn't help. Is there someone who has the same experience?
Regards
We've found turning on the deadline i/o scheduler to be of great help keeping our db from starving during hi external load on the file system. Try
echo deadline > /sys/block/sda/queue/scheduler
And test if the problem becones smaller. (Replace sda for the device your db is on)
Just stepping back a bit, have you done some basic InnoDB tuning? The MySQL defaults can be pretty limiting. A good overview:
http://www.mysqlperformanceblog.com/2007/11/01/innodb-performance-optimization-basics/
The interesting is:
all databases on this server are kept in one InnoDB file (one_file_per_table is disabled)
we've written a small shell script, inserting 10 rows per second in a new table in a new database on this server
then we've copied some big files via "cat" (like described above) parallel...
and: nothing happend, no locks, no endless "freeing items"
so we've changed the shell script to insert the lines in a new table in the existing database, which causes this problems and copied the files again
and: the locks are back! huh?
last try: we've copied the "buggy" database on the server itself into a new database (with full structure and data) and started shell script again
and: no "freeing items" processes!?
I don't understand this behavior: inserting in the old database let these "freeing items" processes appear, but inserting into a new database (full copy of the old database) is fine?
I want to run an import of a large file into MySQL. However, I don't want it written to the binary log, because the import will take a long time, and cause the slaves to fall far behind. I would rather run it seperately on the slaves, after the fact, because it will be much easier on the system. The table in question is a new one, and therefore I don't really have to worry about it getting out of sync, as the master and all the slaves will have the same data in the end, because they will all import the same file eventually. I also don't want to change any of the replicate-ignore-* or binlog_ignore-* options, because they require a restart of the servers in question. Is there a a way to run a single query without writing it to the binary log?
I already know the answer, but I couldn't find it very easily on the net, so I'm letting somebody post the answer to get some rep points. If there's no answers in a little white I'll post the answer I found.
You can use the sql-log-bin session variable to turn off logging for the current session. Just simply enter:
SET sql_log_bin = 0;
and all queries on your current session will not be sent to the binary log. If you want to turn binary logging back on, run:
SET sql_log_bin = 1;
This is only for the currently running session you are in. All other queries from all other connections will still continued to be logged. Also, you must have SUPER privileges for this to work.
If you want to do this from the cli, try this, it worked for me:
$ mysqldump old_DB | mysql --init-command="SET SQL_LOG_BIN = 0;" new_DB
Possible that the "init-command" param was added in a newer MySQL version.