I have a MySQL database that contains millions of rows per table and there are 9 tables in total. The database is fully populated, and all I am doing is reads i.e., there are no INSERTs or UPDATEs. Data is stored in MyISAM tables.
Given this scenario, which linux file system would work best? Currently, I have xfs. But, I read somewhere that xfs has horrible read performance. Is that true? Should I shift the database to an ext3 file system?
Thanks
What about a RAM disk?
it's not about the FS but it can improve your SELECTs. did you evaluated the mysql table partitioning ?
Related
Environment: JSF2, persistence with Hibernate, MySQL
I have a database that is rappidly filling because of a table with image data. That data is never searched but only directly accessed by id. Problem is that it stays in the database and so enlarges the backups and the runtime memory usage of the database.
I'm thinking that there could possibly be multiple solutions:
Tell MySQL that the table should not be cached and/or kept in memory.
Don't use MySQL at all for that table. Just let persistence know that this should be stored on disk directly.
???
But I haven't found a way to do either. Please advice.
Thanks,
Milo van der Zee
Storage type depends on storage engine in MySQL. Only tables having MEMORY storage engine are stored in RAM others are stored on disk.
In select queries you can use SELECT SQL_NO_CACHE to tell MySQL not to cache query data in MySQL query cache.
You can partition the table by defining partitions on table. This will make inserts and selects faster.
You can also create day wise tables like table_name_2012_07_20 and archive tables with old dates and to store data in compress format you can either use Archive storage engine or if you are using MyIsam storage engine then do myisamchk or myisampack to save disk space on the hard drive.
Basic info:
My Mysql database is using TokuDB, InnoDB, MyIsam tables.
Server info:
16 core, 64GB RAM, CentOS 6.2, MySQL v 5.5
Process:
1. Import large amount data from one text file to one TokuDB table.
2. Select data by joining different table.
When process 1 and 2 running at the same time, the whole operation speed will be much slower.
Does anyone know specific reason?
Any suggestions to improve it?
Separate the IO to different disks/arrays. Having all IO on a single partition/array results in horrible performance. If possible, invest in a dedicated drive array such as IBM's DS3524 or HP Smart Array. Connecting the DB Server though Fibre Channel (or better yet SAS2) will give you an incredible performance gain. I stopped putting lots of disks into the server itself a few years ago. I get 5X the performance with MySQL on a drive array than disk in the server.
in tokudb, load data infile works much faster when importing on empty tables (especially when you have non-increment primarey key, or unique index)
I noticed that my database server supports the Memory database engine. I want to make a database I have already made running InnoDB run completely in memory for performance.
How do I do that? I explored PHPMyAdmin, and I can't find a "change engine" functionality.
Assuming you understand the consequences of using the MEMORY engine as mentioned in comments, and here, as well as some others you'll find by searching about (no transaction safety, locking issues, etc) - you can proceed as follows:
MEMORY tables are stored differently than InnoDB, so you'll need to use an export/import strategy. First dump each table separately to a file using SELECT * FROM tablename INTO OUTFILE 'table_filename'. Create the MEMORY database and recreate the tables you'll be using with this syntax: CREATE TABLE tablename (...) ENGINE = MEMORY;. You can then import your data using LOAD DATA INFILE 'table_filename' INTO TABLE tablename for each table.
It is also possible to place the MySQL data directory in a tmpfs in thus speeding up the database write and read calls. It might not be the most efficient way to do this but sometimes you can't just change the storage engine.
Here is my fstab entry for my MySQL data directory
none /opt/mysql/server-5.6/data tmpfs defaults,size=1000M,uid=999,gid=1000,mode=0700 0 0
You may also want to take a look at the innodb_flush_log_at_trx_commit=2 setting. Maybe this will speedup your MySQL sufficently.
innodb_flush_log_at_trx_commit changes the mysql disk flush behaviour. When set to 2 it will only flush the buffer every second. By default each insert will cause a flush and thus cause more IO load.
Memory Engine is not the solution you're looking for. You lose everything that you went to a database for in the first place (i.e. ACID).
Here are some better alternatives:
Don't use joins - very few large apps do this (i.e Google, Flickr, NetFlix), because it sucks for large sets of joins.
A LEFT [OUTER] JOIN can be faster than an equivalent subquery because
the server might be able to optimize it better—a fact that is not
specific to MySQL Server alone.
-The MySQL Manual
Make sure the columns you're querying against have indexes. Use EXPLAIN to confirm they are being used.
Use and increase your Query_Cache and memory space for your indexes to get them in memory and store frequent lookups.
Denormalize your schema, especially for simple joins (i.e. get fooId from barMap).
The last point is key. I used to love joins, but then had to run joins on a few tables with 100M+ rows. No good. Better off insert the data you're joining against into that target table (if it's not too much) and query against indexed columns and you'll get your query in a few ms.
I hope those help.
If your database is small enough (or if you add enough memory) your database will effectively run in memory since it your data will be cached after the first request.
Changing the database table definitions to use the memory engine is probably more complicated than you need.
If you have enough memory to load the tables into memory with the MEMORY engine, you have enough to tune the innodb settings to cache everything anyway.
"How do I do that? I explored PHPMyAdmin, and I can't find a "change engine" functionality."
In direct response to this part of your question, you can issue an ALTER TABLE tbl engine=InnoDB; and it'll recreate the table in the proper engine.
In place of the Memory storage engine, one can consider MySQL Cluster. It is said to give similar performance but to support disk-backed operation for durability. I've not tried it, but it looks promising (and been in development for a number of years).
You can find the official MySQL Cluster documentation here.
Additional thoughts :
Ramdisk - setting the temp drive MySQL uses as a RAM disk, very easy to set up.
memcache - memcache server is easy to set up, use it to store the results of your queries for X amount of time.
Is there any limit to maximum row of table in DBMS (specially MySQL)?
I want create table for saving logfile and it's row increase so fast I want know what shoud I do to prevent any problem.
I don't think there is an official limit, it will depend on maximum index sizes and filesystem restrictions.
From mySQL 5.0 Features:
Support for large databases. We use MySQL Server with databases that contain 50 million records. We also know of users who use MySQL Server with 200,000 tables and about 5,000,000,000 rows.
You should periodically move log rows out to a historical database for data mining and purge them from the transactional database. It's a common practice.
There's probably some sort of limitation, dependent on the engine used and the table structure. I've got a table with appx 45 million entries in a database I administrate, I've heard of (much) higher numbers.
I've a database with 3 of the tables having rows in excess of 20 million each. I've used GUIDs as primary keys (unfortunately). Now our database is about 20GB and growing 5GB per month.
It takes about 2 hrs to take full backup of the database, and 30hrs to restore on a box with 4GB RAM.
We once have all the tables from database disappeared. other mysql databases in same server were alright except one - for which only data was disappeared leaving empty tables.
A select query (among many slow queries) - which get max of a date column in one of 20m table takes about 5 mins to return result. This query used pretty frequently.
What I'm looking answers for
recommended db design changes
ways to improved select query performance - max date column on 20m records
other queries' performance
how to go about handling future db growth
Thanks all for your attention.
I've seen setups of larger size (with InnoDB as storage engine and a GUID as a primary key), and there were no such problems.
We once have all the tables from database disappeared. other mysql databases in same server were alright except one - for which only data was disappeared leaving empty tables.
The tables may seem empty if the system LSN has gone below the each page's LSN. This may happen if the InnoDB logfiles are corrupt. InnoDB, however, will issue a warning in this case.
A select query (among many slow queries) - which get max of a date column in one of 20m table takes about 5 mins to return result. This query used pretty frequently.
Create an index on this column, the query will be instant.
Please post the exact query and I'll tell you how to create the best index.
I see no problem in the DB design as such, most probably it's something with your server.
Is it possible to reproduce this behavior on another server with a clean vanilla MySQL installation?
You may also want to try to split data between the tables. Set innodb_file_per_table and restore from the backup.
A free alternaive to innodb hot backup is Percona XtraBackup Tool.
For backup, you could use the innodb hot backup tool. This not only lets you do consistent backups while your database is up, but the restore is much faster than the one you're doing (I'm assuming mysqldump?). It does cost money.
You might also try Mydumper: http://www.mydumper.org/
It is a great tool and is free and open source