I recently conducted a mysqlslap stress test to determine which type of insert is best from a performance standpoint. The files containing the queries were stored on a linux system and read from it. I hope someone could through some insight on this. Following are the benchmark obtained through mysqlslap test:
Time taken for Individual/Separate Insert: 5.24 seconds
Time taken for Batch Insert: 2.77 seconds
Time taken for Bulk Insert: 0.52 seconds
MySQL Version: 5.6.14 (InnoDB)
Total Records inserted on empty table: 5000
The records put together has size < 10kb
Innodb_buffer_pool_pages_flushed is disabled
Query cache turned off
Does it have to do anything with parsing, Created_tmp_tables or Innodb_buffer_pool_pages_flushed or any global variables?
Related
I have a very large MySql Insert Script that is 70GB and has 22,288,088,954 records for inserting.
What is the best (Fastest) way to import this file?
I already set up this:
innodb_buffer_pool_size = 4G
innodb_log_buffer_size = 256M
innodb_log_file_size = 1G
innodb_write_io_threads = 16
innodb_flush_log_at_trx_commit = 0
And I use MySql Command Line Client to execute this file, but it is executing almost 5 days and when I try to see how much total records are inserted after 5 days there are only 4,371,106 records inserted.
I use this sql to find total inserted records:
SELECT SUM(TABLE_ROWS)
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_SCHEMA LIKE 'mydatabase'
MySql is installed on Win 10 Pro PC, with I5 CPU and 32GB RAM, SSD disk
Can anybody help me and suggest a faster way to import this script because if only 4,000,000 records from 22,288,088,954 in 5 days, it will take a very long time to finish.
Thank you in advance.
First, I don't believe you. 22 billion rows in 70 GBytes is just over 3 characters per row. If you are using one insert, that is the (, ), and about one more character.
Second, you no doubt have a problem with the transaction log. Are you doing commits after every insert or few inserts? For optimal performance, you should probably be committing after every 1,000 - 100,000, so the log file doesn't get too big.
Third, the best way to insert the data is not via inserts at all. You should be using load data infile. In less than five days, you should be able to write a script to convert the insert statements into an input data file. Then you can just load that. Better yet, go back to the original data and extract it in a more suitable format.
I would like to optimize MySQL for a single user (me). Noone else is connected to the database. MySQL is installed locally on my PC. The data covers around 70 tables with 150 GB of data in total.
The data is static, so I won't do changes to it (no UPDATE or DELETE). I want to analyze the data, so I will run large and complex queries including SELECT and JOIN over large proportion of the data.
Machine:
Windows 7 64-bit
Intel Core i7-4800MQ # 2,70 GHz
32 GB RAM
2x 512 GB SSD
MySQL 5.7, INNODB
What I did so far:
Deactivated HyperThreading (MySQL uses only one virtual Core per
query - CPU usage 12.5% --> 25%)
Declare primary keys and
Indexed all foreign keys
innodb_buffer_pool_size = 25G
max_connections=10
Use of InnoDB
So what can I do to optimize the configuration (not the query itself) for single-user-queries?
database engine?
general configuration?
better cache all the information of the joins?
Note:
Under the current configuration, the CPU usage is the bottleneck because it is on 25% when I run a complex query. As test i tried some huge queries (fetching a lot of data). If I can believe the timing of MYSQL Workbench the duration was only 4 seconds, but after 10 hours of running it couldn't finish fetching the data...
Thanks a lot!
You could try MySiam. It is especially good for read-intensive (select) tables. And you should select a large key_buffer_size. But I am not sure about the JOIN performance... Give it a try
i've imported a mysql table stored in a .sql file using my localhost phpymadmin, the table has 14000 records (simple data, 5 fields only) and it took almost 10 minutes. is this normal? i'm running a laptop with win8, core i7 quad and my xampp seems to be configured properly.
thanks
Your hard drive is the limit in this case. Having a single insert per row means your inserting is limited on your hard drives IOPS (I/O operations per second).
Bulk inserting reduces the IOPS but increases the MB/s transfer which is what you want in this case.
so rewriting like
INSERT INTO table VALUES (1,2,3,4),(1,2,3,4)
with comma separated rows will give a huge boost
Putting in a hard drive with higher IOPS will speed it up also if the rewritten query is still slow
I have created a benchmark of multiple LOAD DATA operations inserting millions of records into different tables.
i have surprised when i discovered that on windows my Benchmark finished after 1.15 hours, vs Unix where it finished after 1.45 hours.
Then i have created different benchmark for read operations which involves full table scanning using single time index (my measurement has a time value which is non unique).
For the second benchmark i received 5 times better results on Unix vs Windows.
What may be the reason for this? (I have only MyISAM tables)
Short Version:
INSERT INTO SELECT into a table with fulltext index with the same data takes sometimes 25 seconds and sometimes 2500 seconds. We have no idea where this huge gap is coming from.
Long Version:
I have a problem with a cronjob which imports new data into an import table and copies this data via an INSERT INTO SELECT statement to the production tables. I have split the tables because of the time an update takes with a mysql fulltext index – it seems to be faster to insert the data into this table with one INSERT INTO SELECT than with many single insert statements
The cron to import new data is running every 5 minutes. There is a function that checks if an instance of the cron is running to disallow parallel running of the script. Usually there are about 500 new records with every cron call.
In the night at 1-2 am there are a lot of more new data (about 5.000 – 15.000 new records) and the cron is running much more longer than 5 minutes.
When the cron is running long in the night and while tracking the performance of these queries I detected that the performance of the INSERT INTO SELECT statement is very (!) slow. To copy about 15.000 new records (with a filesize of about 30 MB) the query takes more than 2.500 seconds!
The query is:
INSERT IGNORE INTO mentiondata
SELECT * FROM mentionimport
WHERE id <= 1203780;
I profile the query and with the following result:
2012-10-31 06:52:06 Queryprofile: {
"starting":"0.000036",
"checking permissions":"0.000003",
"Opening tables":"0.000132",
"System lock":"0.000003",
"Table lock":"0.000007",
"init":"0.000041",
"optimizing":"0.000007",
"statistics":"0.000023",
"preparing":"0.000005",
"executing":"0.000002",
"Sending data":"999.999999",
"end":"0.000017",
"query end":"0.000005",
"freeing items":"1.458159",
"logging slow query":"0.000050",
"cleaning up":"0.000007"}
In the process-list the sending-data was over 2.500 – in the profile it is just 999.999999. Maybe this is the profiler-limit – whatever…
The really strange thing is: When I try to reproduce the problem via deleting the records from the fulltext-table (DELETE FROM mentiondata WHERE id >= 1203780;) and starting the copy process manually it just takes about 25 seconds!!!
So I don’t get it and I really need help! I don’t understand why there is such a performance-difference between the same query! I checked the mysql-processlist while the cron-copy-statement was running – there are no other queries which lock tables or something. There’s just the single copy-query in the processlist – and a “sending data” with more than 2.500 seconds. There is no other cron or any other tasks which influence the performance of the server running. It seems that the mysql-server slows down every night or that the sql-query takes an extreme long time when the connection was opened a long time before the insert statement takes place (to insert the data into the copy import tables).
Are there any status-variables I can check why mysql is so slow? Is there a possibility to check why these queries are so slow? Here some server-variables for info:
bulk_insert_buffer_size: 268435456
key_buffer_size: 536870912
query_cache_size: 536870912
Thanks for any help!
Timo