I have a question for large mysql queries. Is it possible to skip the copying to tmp table on disk step that mysql takes for large queries or is it possible to make it go faster? because this step is taking way too long to get the results of my queries back. I read on the MySQL page that mysql performs this to save memory, but I don't care about saving memory I just want to get the results of my queries back FAST, I have enough memory on my machine. Also, my tables are properly indexed so that's not the reason why my queries are slow.
Any help?
Thank you
There are two things you can do to lessen the impact by this
OPTION #1 : Increase the variables tmp_table_size and/or max_heap_table_size
These options will govern how large an in-memory temp table can be before it is deemed too large and then pages to disk as a temporary MyISAM table. The larger these values are, the less likely you will get 'copying to tmp table on disk'. Please, make sure your server has enough RAM and max_connections is moderately configured should a single DB connection need a lot of RAM for its own temp tables.
OPTION #2 : Use a RAM disk for tmp tables
You should be able to configure a RAM disk in Linux and then set the tmpdir in mysql to be the folder that has the RAM disk mounted.
For starters, configure a RAM disk in the OS
Create a folder in the Linux called /var/tmpfs
mkdir /var/tmpfs
Next, add this line to /etc/fstab (for example, if you want a 16GB RAM disk)
none /var/tmpfs tmpfs defaults,size=16g 1 2
and reboot the server.
Note : It is possible to make a RAM disk without rebooting. Just remember to still add the aforementioned line to /etc/fstab to have the RAM disk after a server reboot.
Now for MySQL:
Add this line in /etc/my.cnf
[mysqld]
tmpdir=/var/tmpfs
and restart mysql.
OPTION #3 : Get tmp table into the RAM Disk ASAP (assuming you apply OPTION #2 first)
You may want to force tmp tables into the RAM disk as quickly as possible so that MySQL does not spin its wheels migrating large in-memory tmp tables into a RAM disk. Just add this to /etc/my.cnf:
[mysqld]
tmpdir=/var/tmpfs
tmp_table_size=2K
and restart mysql. This will cause even the tiniest temp table to be brought into existence right in the RAM disk. You could periodically run ls -l /var/tmpfs to watch temp tables come and go.
Give it a Try !!!
CAVEAT
If you see nothing but temp tables in /var/tmpfs 24/7, this could impact OS functionality/performance. To make sure /var/tmpfs does not get overpopulated, look into tuning your queries. Once you do, you should see less tmp tables materializing in /var/tmpfs.
You can also skip the copy to tmp table on disk part (not answered in the selected answer)
If you avoid some data types :
Support for variable-length data types (including BLOB and TEXT) not supported by MEMORY.
from https://dev.mysql.com/doc/refman/8.0/en/memory-storage-engine.html
(or https://mariadb.com/kb/en/library/memory-storage-engine/ if you are using mariadb).
If your temporary table is small enough : as said in selected answer, you can
Increase the variables tmp_table_size and/or max_heap_table_size
But if you split your query in smaller queries (not having the query does not help to analyze your problem), you can make it fit inside a memory temporary table.
Related
I am trying to index a varchar(255) column of a table with 2 billion rows. The indexation stops with the below error:
Error number 28 means ‘No space left on device
This post suggests I can just change the tmpdir to another partition with larger disk space.
Is this advised? Are there any downsides to doing so? (slower indexation ... etc)
If not what are the detailed steps I need to go through to change tmpdir of mysql effectively?
Advised: Yes. Easy: yes.
In the old days, disk drives were small. Often a machine would have multiple drives to make it possible to store lots of data. And file systems had limitations. (Long ago, FAT16 on DOS topped out at 2 billion bytes, no way for it to hold 2 billion rows.)
So, having a setting for tmpdir made it easy to chop up the disk usage. In rare cases, it can be beneficial to performance to do such separation, but even that is vanishing with the advent of SSDs. Meanwhile, for really huge datasets, RAID striping provides a better way to provide performance than manually splitting up disk allocation.
Find the my.cnf configuration file. In the [mysqld] section, add
tmpdir = /.../...
to point to some otherwise unused directory on a disk with lots of free space.
Meanwhile... "2 billion rows" is large enough to lead to numerous challenges. You may want to start a question in dba.stackexhange.com to discuss potential issues.
innodb_tmpdir
From the changelog for 5.6.29 and 5.7.11 (Feb 2016), innodb_tmpdir may be a better thing to change:
A new InnoDB configuration option, innodb_tmpdir, allows you to configure a separate directory for temporary files created during online ALTER TABLE operations that rebuild the table. This option was introduced to help avoid MySQL temporary directory overflows that could occur as a result of large temporary files created during online ALTER TABLE operations. innodb_tmpdir can be configured dynamically using a SET statement.
Online ALTER TABLE operations that rebuild a table also create an intermediate table file in the same directory as the original table. The innodb_tmpdir option is not applicable to intermediate table files. (Bug #19183565)
I use docker on local machine with Mysql container.
Steps:
Launch mysql docker. Uses memory about 2-2.5G.
Launch command "optimize table". Uses memory grow up to 4.5-5G;
Restart mysql docker and uses memory 2-2.5G again (see screenshot)
List item;
Database has 10 small tables (< 10 rows) and one huge table (10,000,000 rows with 10 columns, table has index for every column).
Why it happens?
P.S. Sorry for my bad english.
screenshot
Do not use OPTIMIZE TABLE on InnoDB tables. It almost never provides any benefit.
OPTIMIZE TABLE copies the table over. Apparently you have innodb_file_per_table = OFF, which implies that it will make a second copy of the table in ibdata1.
"An index for every column" -- This is usually a waste. Look at the SELECTs ou have to see what indexes you can use. (Ask for help here, but provide SHOW CREATE TABLE and the relevant queries.)
The buffer_pool is in RAM; its size is limited by innodb_buffer_pool_size. When MySQL starts (in your Docker, for example), it will start out small and grow, perhaps up to that size.
Doing the OPTIMIZE, especially with so many indexes, will consume some or all of the available buffer_pool.
I have a datawarehouse on mySQL size being almost 1.4 TB
when i checked the number of files the mysql server daemon is accessing I was surprised.
cmd user : lsof | grep mysql
cnd2 used : lsof | grep mysql | wc -l
2598
Please suggest how can this be reduced.What is the impact of the same on mysql performance.
Very likely, it's possible to reduce the number of files held open by MySQL. But note that this will likely decrease performance. In terms of performance, what's more important (than the number of open files) is the closing and opening of files. (That's more expensive than holding files open.)
MySQL caches open tables, keeping recently used files open. This avoids the performance overhead associated with closing files, and then reopening them again later. The limit on the number of tables held open is set with the table_open_cache variable.
The limit on the number of files MySQL has open is set in the max_open_files variable.
Reference: http://dev.mysql.com/doc/refman/5.5/en/table-cache.html
Note that a MyISAM table is stored on disk as three files: tablename.frm, tablename.MYD and tablename.MYI. If the table is partitioned, each partition requires separate files.
With InnoDB engine, if the innodb_file_per_table option is set when a table is created, that's two files, tablename.frm and tablename.idb.
If multiple sessions are accessing a table, there will be multiple "open files" for that table.
There's also a couple of innodb log files, as well as the mysql log file (in logging is enable), the mysql general log, slow query log, etc.
Any temporary table, derived table or an intermediate result set that exceeds the limit on the size of tables stored "in memory", that's additional files will be spun out to disk, and potentially more open files.
If you are hitting a "too many open files" OS error, you can have the OS limit increased. If that's not possible, then decrease the max_open_files variable; you may also need to reduce some other related variables as well.
If you are looking at limiting the number of open files as a solution to slow performance, it's possible you are barking up the wrong tree, because reducing the number of files that MySQL can hold open will likely decrease performance.
We have a series of tables that have grown organically to several million rows, in production doing an insert or update can take up to two seconds. However if I dump the table and recreate it from the dump queries are lightning fast.
We have rebuilt one of the tables by creating a copy rebuilding the indexes and then doing a rename switch and copying over any new rows, this worked because that table is only ever appended to. Doing this made the inserts and updates lightning quick.
My questions:
Why do inserts get slow over time?
Why does recreating the table and doing an import fix this?
Is there any way that I can rebuild indexes without locking a table for updates?
It sounds like it's either
Index unbalancing over time
Disk fragmentation
Internal innodb datafile(s) fragmentation
You could try analyze table foo which doesn't take locks, just a few index dives and takes a few seconds.
If this doesn't fix it, you can use
mysql> SET PROFILING=1;
mysql> INSERT INTO foo ($testdata);
mysql> show profile for QUERY 1;
and you should see where most of the time is spent.
Apparently innodb performs better when inserts are done in PK order, is this your case?
InnoDB performance is heavily dependent on RAM. If the indexes don't fit in RAM, performance can drop considerably and quickly. Rebuild the whole table improves performance because the data and indexes are now optimized.
If you are only ever inserting into the table, MyISAM is better suited for that. You won't have locking issues if only appending, since the record is added to the end of the file. MyISAM will also allow you to use MERGE tables, which are really nice for taking parts of the data offline or archiving without having to do exports and/or deletes.
Updating a table requires indices to be rebuilt. If you are doing bulk inserts, try to do them in one transaction (as the dump and restore does). If the table is write-biased I would think about dropping the indices anyway or let a background job do read-processing of the table (eg by copying it to an indexed one).
track down the in use my.ini and increase the key_buffer_size I had a 1.5GB table with a large key where the Queries per second (all writes) were down to 17. I found it strange that the in the administration panel (while the table was locked for writing to speed up the process) it was doing 200 InnoDB reads per second to 24 writes per second.
It was forced to read the index table off disk. I changed the key_buffer_size from 8M to 128M and the performance jumped to 150 queries per second completed and only had to perform 61 reads to get 240 writes. (after restart)
Could it be due to fragmentation of XFS?
Copy/pasted from http://stevesubuntutweaks.blogspot.com/2010/07/should-you-use-xfs-file-system.html :
To check the fragmentation level of a drive, for example located at /dev/sda6:
sudo xfs_db -c frag -r /dev/sda6
The result will look something like so:
actual 51270, ideal 174, fragmentation factor 99.66%
That is an actual result I got from the first time I installed these utilities, previously having no knowledge of XFS maintenance. Pretty nasty. Basically, the 174 files on the partition were spread over 51270 separate pieces. To defragment, run the following command:
sudo xfs_fsr -v /dev/sda6
Let it run for a while. the -v option lets it show the progress. After it finishes, try checking the fragmentation level again:
sudo xfs_db -c frag -r /dev/sda6
actual 176, ideal 174, fragmentation factor 1.14%
Much better!
I am running a mysql server. I would like to somehow make sure that the whole DB is loaded into the ram as I heard it would be alot faster.
Is this true? and how do I vertify it?
If you are using InnoDB, then you can set innodb_buffer_pool_size to be at least as large as your data and indexes. Then as you access data, it will be cached into memory and all subsequent accesses will occur via memory.
If you wanted to pre-cache the data, you could do some of your common queries, or full table/index scans to force it to load all of the data into memory.
There is an init-file option for mysqld which can have the server automatically execute some commands on startup, where you could include the pre-caching queries to execute.
Keep in mind you don't want to run out of memory, so be careful with setting innodb_buffer_pool_size too large.
Well, one option is to use a ramdrive.. though it's not permanent.
http://www.linuxforums.org/forum/misc/670-how-create-ram-drive-linux.html
(Just for completeness,) You could use HEAP as table engine. But keep in mind that whatever data you put in a HEAP table is gone when you reboot the server.
Tables in RAM are very fast.
You can change your existing tables by altering their storage engine. But make a backup if you want to use the data after a server restart.
ALTER TABLE test ENGINE = MEMORY;
Keep in mind, that MEMORY tables cannot contain BLOB or TEXT columns and data and index size is limited to available memory.