I have schema A and B in my mysql,when I do a pressure test on schema A which include lots of concurrent batch insert and update sql operation and import many tablespace from files which was exported by xtrabackup on schema B.I found the import operation is very slow and cost lots of time (more than one hour). If I don't do pressure test on schema A and the import operation just cost 20 seconds nearly.
show processlist:
856 guest 10.142.90.17:51671 pinc_0002 Query 733 Table lock alter table bill import tablespace 0
857 guest 10.142.90.17:51700 pinc_0002 Query 733 Table lock alter table company import tablespace 0
858 guest 10.142.90.17:51731 pinc_0002 Query 733 Table lock alter table car_new import tablespace 0
859 guest 10.142.90.17:51758 pinc_0002 Query 733 Table lock alter table dialing_his import tablespace 0
860 guest 10.142.90.17:51799 pinc_0002 Query 733 Table lock alter table car import tablespace 0
861 guest 10.142.90.17:51846 pinc_0002 Query 732 Table lock alter table employee_his_new import tablespace 0
862 guest 10.142.90.17:51869 pinc_0002 Query 732 Table lock alter table book import tablespace 0
863 guest 10.142.90.17:51914 pinc_0002 Query 732 Table lock alter table goods import tablespace 0
864 guest 10.142.90.17:51975 pinc_0002 Query 732 Table lock alter table order_details import tablespace 0
result of SHOW ENGINE INNODB STATUS
select * from information_schema.INNODB_LOCKS:
115155367:9417:4:2 115155367 X RECORD `pinc_0003`.`testcolumn` GEN_CLUST_INDEX 9417 4 2 0x00001D331DDF
115153055:9417:4:2 115153055 X RECORD `pinc_0003`.`testcolumn` GEN_CLUST_INDEX 9417 4 2 0x00001D331DDF
115150974:9417:4:2 115150974 X RECORD `pinc_0003`.`testcolumn` GEN_CLUST_INDEX 9417 4 2 0x00001D331DDF
115148337:9417:4:2 115148337 X RECORD `pinc_0003`.`testcolumn` GEN_CLUST_INDEX 9417 4 2 0x00001D331DDF
result of select * from information_schema.INNODB_TRX
top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
54996 apps 20 0 111g 71g 8664 S 1794.7 14.2 2254:20 mysqld
iostat -dxm：
```
[apps#cs1n3 ~]$ iostat -dxm 2
Linux 2.6.32-358.el6.x86_64 (cs1n3) 02/08/2017 _x86_64_ (64 CPU)
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
sda 0.04 485.19 8.12 76.89 0.19 2.43 63.20 0.19 2.25 0.16 1.38
sdb 0.00 8.14 0.03 2.44 0.00 0.04 35.10 0.01 4.51 0.14 0.04
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 841.50 0.00 7003.50 0.00 63.91 18.69 3.13 0.45 0.13 94.20
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
```
```
iostat -dxm
Linux 2.6.32-358.el6.x86_64 (cs1n3) 02/08/2017 _x86_64_ (64 CPU)
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
sda 0.04 485.19 8.12 76.89 0.19 2.43 63.20 0.19 2.25 0.16 1.38
sdb 0.00 8.14 0.03 2.44 0.00 0.04 35.10 0.01 4.51 0.14 0.04
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 841.50 0.00 7003.50 0.00 63.91 18.69 3.13 0.45 0.13 94.20
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
```
Maybe answer is in your table architecture. If you have PRIMARY_KEY in yours tables, your database has B-Tree structure, but if you have tables without PRIMARY_KEY, your database has table structure. So, maybe you have table structure in your database, so operation add/update costs more than B-Tree structure.
Related
Edit:
The queries are executed from AWS Lambda and it is a AWS Aurora MySQL compatible database.
It is a M5.large database that has about 70 database connections and around 7% cpu load.
End Edit.
We have a dead lock that we are struggling to understand. It happens very rarely and of course when the requests are very close to each other in time.
How can we get better understanding when this happens?
What is best practice in cases like this? Is it ok to accept this and write your code to rerun the query?
Below is more information output from show engine innodb status
2019-07-31 10:44:08 0x2b0f9939d700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 14 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 457179 srv_active, 0 srv_shutdown, 158871 srv_idle
srv_master_thread log flush and writes: 0
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 353267
OS WAIT ARRAY INFO: signal count 363003
RW-shared spins 0, rounds 60860, OS waits 3736
RW-excl spins 0, rounds 278008, OS waits 4241
RW-sx spins 834, rounds 12583, OS waits 94
Spin rounds per wait: 60860.00 RW-shared, 278008.00 RW-excl, 15.09 RW-sx
------------------------
LATEST DETECTED DEADLOCK
------------------------
2019-07-29 00:01:57 0x2b122b42b700
*** (1) TRANSACTION:
TRANSACTION 25004785, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 7 lock struct(s), heap size 1136, 4 row lock(s), undo log entries 1
MySQL thread id 1067258, OS thread handle 47346011883264, query id 192217819 160.170.30.251 root update
INSERT INTO logs (date, time, duration, source, another_table_id, date_id) VALUES ('2019-07-29', '00:00:06', 20047, 0, 6793, 3629) ON DUPLICATE KEY UPDATE duration = VALUES(duration)
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 252 page no 15293 n bits 0 index logs_another_table__c5da8410_uniq of table `database`.`logs` trx id 25004785 lock_mode X locks gap before rec insert intention waiting
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len=8; bufptr=0x2b1111b00070; hex= 73757072656d756d; asc supremum;;
*** (2) TRANSACTION:
TRANSACTION 25004783, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 8 lock struct(s), heap size 1136, 5 row lock(s), undo log entries 1
MySQL thread id 1067257, OS thread handle 47345995474688, query id 192217812 160.170.10.38 root update
INSERT INTO logs (date, time, duration, source, another_table_id, date_id) VALUES ('2019-07-28', '23:59:05', 20047, 0, 6792, 3631) ON DUPLICATE KEY UPDATE duration = VALUES(duration)
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 252 page no 15293 n bits 0 index logs_another_table__c5da1234_uniq of table `database`.`logs` trx id 25004783 lock_mode X
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len=8; bufptr=0x2b1111b00070; hex= 73757072656d756d; asc supremum;;
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 252 page no 15293 n bits 0 index logs_another_table__c5da1234_uniq of table `database`.`logs` trx id 25004783 lock_mode X locks gap before rec insert intention waiting
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len=8; bufptr=0x2b1111b00070; hex= 73757072656d756d; asc supremum;;
------------
TRANSACTIONS
------------
Trx id counter 30159757
Purge done for trx's n:o < 30159757 undo n:o < 0 state: running but idle
History list length 2976
--------
FILE I/O
--------
I/O thread 0 state: waiting for i/o request (insert buffer thread)
I/O thread 1 state: waiting for i/o request (log thread)
I/O thread 2 state: waiting for i/o request (read thread)
I/O thread 3 state: waiting for i/o request (read thread)
I/O thread 4 state: waiting for i/o request (read thread)
I/O thread 5 state: waiting for i/o request (read thread)
I/O thread 6 state: waiting for i/o request (write thread)
I/O thread 7 state: waiting for i/o request (write thread)
I/O thread 8 state: waiting for i/o request (write thread)
I/O thread 9 state: waiting for i/o request (write thread)
Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] ,
ibuf aio reads:, log i/o's:, sync i/o's:
Pending flushes (fsync) log: 0; buffer pool: 0
0 OS file reads, 0 OS file writes, 0 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 0, seg size 2, 0 merges
merged operations:
insert 0, delete mark 0, delete 0
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 2303461, node heap has 0 buffer(s)
Hash table size 2303461, node heap has 0 buffer(s)
Hash table size 2303461, node heap has 0 buffer(s)
Hash table size 2303461, node heap has 0 buffer(s)
Hash table size 2303461, node heap has 0 buffer(s)
Hash table size 2303461, node heap has 0 buffer(s)
Hash table size 2303461, node heap has 0 buffer(s)
Hash table size 2303461, node heap has 0 buffer(s)
0.00 hash searches/s, 1.00 non-hash searches/s
---
LOG
---
Log sequence number 8204
Log flushed up to 0
Pages flushed up to 8204
Last checkpoint at 8192
0 pending log flushes, 0 pending chkp writes
0 log i/o's done, 0.00 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 0
Dictionary memory allocated 4272484
Buffer pool size 510568
Free buffers 271669
Database pages 238899
Old database pages 88146
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 2287, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 49550, created 189349, written 4828
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 238899, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
----------------------
INDIVIDUAL BUFFER POOL INFO
----------------------
---BUFFER POOL 0
Buffer pool size 255284
Free buffers 136125
Database pages 119159
Old database pages 43966
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 1238, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 24722, created 94437, written 68
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 119159, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 1
Buffer pool size 255284
Free buffers 135544
Database pages 119740
Old database pages 44180
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 1049, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 24828, created 94912, written 4760
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 119740, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
Process ID=8465, Main thread ID=47357039400704, state: sleeping
Number of rows inserted 3672324, updated 8861306, deleted 2146, read 1562714829
0.00 inserts/s, 0.50 updates/s, 0.00 deletes/s, 0.64 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
Slow query when IO wait is high.
Show from iotop command
-- TID -- PRIO -- USER -- DISK READ -- DISK WRITE -- SWAPIN -- IO> -- COMMAND
-- 2311 -- be/4 -- mysql -- 0.00 B/s -- 0.00 B/s -- 0.00% -- 96.25% -- mysql~l.sock
-- 2311 -- be/4 -- mysql -- 0.00 B/s -- 0.00 B/s -- 0.00% -- 96.25% -- mysql~l.sock
-- 2311 -- be/4 -- mysql -- 0.00 B/s -- 0.00 B/s -- 0.00% -- 96.24% -- mysql~l.sock
High IO wait start when 6:13:28 PM - 6:13:29 PM (sar command)
--------------------- CPU -- %usr -- %nice -- %sys -- %iowait -- %steal
-- 6:13:28 PM --- all ----- 2.53 --- 0.00 ---- 2.02 ----- 39.39 ------ 0.00
-- 6:13:29 PM --- all ----- 1.99 --- 0.00 ---- 1.00 ----- 49.25 ------ 0.00
Got slow query between that time
Time: 130329 18:13:29
User#Host: wdwdwd[wdwdwd] # localhost []
Query_time: 2.007902 Lock_time: 0.000025 Rows_sent: 0 Rows_examined: 1 SET timestamp=1364555609;
UPDATE log_product SET credit=credit+1 WHERE id_product='349721228' and id_user='2021841' LIMIT 1;
## Question are How to fix this process. What the real cause ? ##
Do you have an index on the "Log_Product" table by (id_user, id_product) as a single index, not TWO indexes, one by each column. Also, if the fields are numeric, you don't need quotes...
I'm having following table structure.
reservation: (InnoDB)
------------------------------------------
id INT,
date DATE,
item_id INT,
slot VARCHAR(50);
PRIMARY KEY(id), UNIQUE KEY(item_id,date).
Now i'm trying to use SELECT.....FOR UPDATE on reservation table inside a transaction to lock specific rows within date range(eg:2012-06-15 to 2012-06-16) of a particular item_id.
SELECT availability FROM reservation WHERE item_id={$item_id} AND (date>='{$to_date}' AND date<='{$from_date}') FOR UPDATE
NOW when i use the above statement, it is blocking all the rows of particular item_id even beyond the date range.I'm also using this UNIQUE KEY(item_id,date).
How to lock only particular date range of a specific item_id?
regards,
ravi.
ADDED:
|
=====================================
120615 20:14:13 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 24 seconds
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 12, signal count 12
Mutex spin waits 0, rounds 180, OS waits 4
RW-shared spins 10, OS waits 5; RW-excl spins 2, OS waits 2
------------
TRANSACTIONS
------------
Trx id counter 0 15147
Purge done for trx's n:o < 0 15136 undo n:o < 0 0
History list length 2
Total number of lock structs in row lock hash table 2
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 0 0, not started, OS thread id 2940
MySQL thread id 76, query id 1487 localhost 127.0.0.1 root
SHOW innodb status
---TRANSACTION 0 15143, not started, OS thread id 2588
MySQL thread id 10, query id 578 localhost 127.0.0.1 root
---TRANSACTION 0 15144, ACTIVE 11526 sec, OS thread id 2740
3 lock struct(s), heap size 320
MySQL thread id 11, query id 577 localhost 127.0.0.1 root
--------
FILE I/O
--------
I/O thread 0 state: wait Windows aio (insert buffer thread)
I/O thread 1 state: wait Windows aio (log thread)
I/O thread 2 state: wait Windows aio (read thread)
I/O thread 3 state: wait Windows aio (write thread)
Pending normal aio reads: 0, aio writes: 0,
ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 0; buffer pool: 0
187 OS file reads, 75 OS file writes, 29 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 0, seg size 2,
0 inserts, 0 merged recs, 0 merges
Hash table size 62383, used cells 0, node heap has 1 buffer(s)
0.00 hash searches/s, 0.00 non-hash searches/s
---
LOG
---
Log sequence number 0 8287163
Log flushed up to 0 8287163
Last checkpoint at 0 8287163
0 pending log writes, 0 pending chkp writes
24 log i/o's done, 0.00 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 23277934; in additional pool allocated 1430400
Buffer pool size 960
Free buffers 895
Database pages 64
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages read 64, created 0, written 51
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
1 read views open inside InnoDB
Main thread id 1544, state: waiting for server activity
Number of rows inserted 10, updated 0, deleted 0, read 66
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
|
I have the following lookup-table:
CREATE TABLE `widgetuser` (
`widgetuserid` char(40) NOT NULL,
`userid` int(10) unsigned NOT NULL,
`created_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
PRIMARY KEY (`widgetuserid`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 DELAY_KEY_WRITE=1;
I have a widgetuser_tmp Table with the same structure but no key and I fill the widgetuser table with this data (4mio rows):
mysql> insert into widgetuser select * from widgetuser_tmp limit 0,4000000;flush tables;
Query OK, 4000000 rows affected (33.14 sec)
Records: 4000000 Duplicates: 0 Warnings: 0
Query OK, 0 rows affected (0.91 sec)
While it is writing, it goes directly to RAID-1 with 15MB/s, disk util <50% and we see no reads, since I filled the disk cache with the source table:
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 3839.20 0.00 52.40 0.00 15.20 594.20 12.46 237.75 5.57 29.20
sdb 0.00 3839.00 0.00 52.60 0.00 15.20 591.94 14.50 275.59 7.19 37.80
I insert the next 1 Mio rows, it's all fine and the wMB/s goes back to 0 right after the flush:
mysql> insert into widgetuser select * from widgetuser_tmp limit 4000000,1000000;flush tables;
Query OK, 1000000 rows affected (10.18 sec)
Records: 1000000 Duplicates: 0 Warnings: 0
Query OK, 0 rows affected (0.87 sec)
mysql> insert into widgetuser select * from widgetuser_tmp limit 5000000,1000000;flush tables;
Query OK, 1000000 rows affected (10.21 sec)
Records: 1000000 Duplicates: 0 Warnings: 0
Query OK, 0 rows affected (1.02 sec)
mysql> insert into widgetuser select * from widgetuser_tmp limit 6000000,1000000;flush tables;
Query OK, 1000000 rows affected (10.67 sec)
Records: 1000000 Duplicates: 0 Warnings: 0
Query OK, 0 rows affected (1.17 sec)
But when I do the 7mio batch, the result still looks the same, but in the iostat -mdx sda sdb 5 suddenly we have 100% util for 30 seconds:
mysql> insert into widgetuser select * from widgetuser_tmp limit 7000000,1000000;flush tables;
Query OK, 1000000 rows affected (10.73 sec)
Records: 1000000 Duplicates: 0 Warnings: 0
Query OK, 0 rows affected (1.21 sec)
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 88.60 0.00 295.60 0.00 1.52 10.53 130.60 435.93 3.38 100.00
sdb 0.00 89.20 0.00 300.80 0.00 1.57 10.68 143.99 483.97 3.32 100.00
The data-files are not touched after the flush:
-rw-rw---- 1 mysql mysql 1032000000 2009-10-30 12:10 widgetuser.MYD
-rw-rw---- 1 mysql mysql 522777600 2009-10-30 12:11 widgetuser.MYI
And also the table status seams normal:
+----------------+--------+---------+------------+----------+----------------+-------------+-------------------+--------------+-----------+----------------+---------------------+---------------------+------------+-----------------+----------+-------------------+---------+
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment |
+----------------+--------+---------+------------+----------+----------------+-------------+-------------------+--------------+-----------+----------------+---------------------+---------------------+------------+-----------------+----------+-------------------+---------+
| widgetuser | MyISAM | 10 | Fixed | 8000000 | 129 | 1032000000 | 36310271995674623 | 522777600 | 0 | NULL | 2009-10-30 11:59:41 | 2009-10-30 12:10:59 | NULL | utf8_general_ci | NULL | delay_key_write=1 | |
+----------------+--------+---------+------------+----------+----------------+-------------+-------------------+--------------+-----------+----------------+---------------------+---------------------+------------+-----------------+----------+-------------------+---------+
And when I continue (since we have 100% drive utilization), it get's worse very fast:
mysql> insert into widgetuser select * from widgetuser_tmp limit 9000000,1000000;flush tables;
Query OK, 1000000 rows affected (31.93 sec)
Records: 1000000 Duplicates: 0 Warnings: 0
Query OK, 0 rows affected (2.34 sec)
mysql> insert into widgetuser select * from widgetuser_tmp limit 10000000,1000000;flush tables;
Query OK, 1000000 rows affected (2 min 39.72 sec)
Records: 1000000 Duplicates: 0 Warnings: 0
Query OK, 0 rows affected (7.82 sec)
The primary key is checked to see whether the new entry is unique or not. As soon as the key does not fit into memory (key_buffer_size=512MB = ca. 8Mio entries), it needs to fetch the missing key parts from the drive(-cache) for checking it. Therefore we should see more reads and slower insert times - we don't see the slower reads since the key is buffered in disk cache. But my question: who is writing suddenly so much and where and why and how can I fix this? Any ideas are appreciated!
Futher ideas and insights:
since the 1MB/s random writes follow the finished statement, the unique validation is already passed
it is a software raid-1 with ahci on, disks are 93% free and capable of about 80wMB/s
the machine has 8GB ram, 5GB cache, 600MB taken by MySQL, 1,7GB free
MySQL 5.1.31-1ubuntu2-log
the delay_key_write does not change this behavior
myisam_sort_buffer_size = 2 GB (not used here, though?)
key_buffer_size = 512 MB
bin_log is off
Linux 2.6.28-15-server #52-Ubuntu SMP Wed Sep 9 11:34:09 UTC 2009 x86_64 GNU/Linux
It's not entirely clear from your question what behaviour you're expecting, or getting. Here are some things you might not know
FLUSH TABLES blows away the MyISAM key cache - it doesn't just write dirty blocks, it also discards clean ones so every index block must be fetched again to be modified
MyISAM uses a block size of 1k by default which is probably smaller than your filesystem blocks; this can create performance problems
If you intend any kind of durability (which you presumably don't, because you're using MyISAM), then you should use hardware raid with a battery-backed cache in the controller.
My guess is that either the index no longer fits in the key buffer, or that it's having to do a lot more writes, which trigger reads because they're unbuffered writes off the block-size boundaries.
Try changing myisam_block_size to 4k or higher and rebuild the table (this is a my.cnf-only option which only takes effect on new tables after a restart).
You can examine the block size on a table with myisamchk -dv
i'm using mariadb5528,if usage of key_buffer_size >90% ,it seems that delay_key_write don't work
so enlarge the key_buffer_size to 2G.
I recently upgraded one of my applications to Rails 2.2.2. Having done that, I've encountered a strange performance bug that has caused renders that used to complete in a fraction of a second to take up to 10 seconds.
I've profiled the issue, and here are the results I've come up with. It looks like the issue is in the real_connect method of the Mysql class. My understanding is that the Ruby real_connect method is a wrapper around the C mysql_real_connect() function. This would lead me to believe that the issue must be with the database, since I've encountered the same problem when running the code on Windows and Linux (the database server is a separate system). I don't, however, believe this is the case, because when I roll back to a previous version (pre Rails 2.2.2) from my subversion repository, the performance issue goes away. This would seem the indicate that there is some kind of bug in ActiveRecord.
How do I go about identifying and fixing this bug? Does anyone have any insight? Is there something I'm missing?
Update: I just created a small profiler script to test the Mysql.real_connect method, and it appears that the problem isn't in Rails, but in the MySQL gem or the database server itself.
Upon running the following code:
result = RubyProf.profile do
5.times do
begin
# connect to the MySQL server
dbh = Mysql.real_connect(ip, user, pass, db)
# get server version string and display it
puts "Server version: " + dbh.get_server_info
rescue Mysql::Error => e
puts "Error code: #{e.errno}"
puts "Error message: #{e.error}"
puts "Error SQLSTATE: #{e.sqlstate}" if e.respond_to?("sqlstate")
ensure
# disconnect from server
dbh.close if dbh
end
end
end
printer = RubyProf::FlatPrinter.new(result)
printer.print(STDOUT, 0)
I came up with this performance result:
Server version: 5.0.32-Debian_7etch3-log
Server version: 5.0.32-Debian_7etch3-log
Server version: 5.0.32-Debian_7etch3-log
Server version: 5.0.32-Debian_7etch3-log
Server version: 5.0.32-Debian_7etch3-log
Thread ID: 18998180
Total: 50.402000
%self total self wait child calls name
99.99 50.40 50.40 0.00 0.00 5 <Class::Mysql>#real_connect (ruby_runtime:0}
0.00 0.00 0.00 0.00 0.00 10 IO#write (ruby_runtime:0}
0.00 0.00 0.00 0.00 0.00 5 Mysql#get_server_info (ruby_runtime:0}
0.00 0.00 0.00 0.00 0.00 5 Kernel#puts (ruby_runtime: 0}
0.00 0.00 0.00 0.00 0.00 5 String#+ (ruby_runtime:0}
0.00 0.00 0.00 0.00 0.00 5 Mysql#initialize (ruby_runtime:0}
0.00 50.40 0.00 0.00 50.40 1 Integer#times (ruby_runtime:0}
0.00 50.40 0.00 0.00 50.40 1 Global#[No method] (tmp/mysql_test/test.rb:12}
0.00 0.00 0.00 0.00 0.00 5 Mysql#close (ruby_runtime: 0}
It seems as though the problem isn't in ActiveRecord, it's either in the MySQL gem or in the database. Where do I go from here?
I was able to track down the problem. I started by connecting to the host using the MySQL command from my development machine using the command mysql --host=ip --user=user --password=password db. This was very slow, so I ssh'ed into the server, and connected from there using the same command. This was also slow.
I changed the command to mysql --host=localhost --user=user --password=password db and I was able to connect instantaneously. I added an entry for my development system in the /etc/hosts file, and was able to connect instantaneously from that as well. Apparently the MySQL server was attempting to perform a reverse dns lookup to resolve the host name associated with the IP address, as is listed in the MySQL Manual, and was timing out.
I added the --skip-name-resolve option to the start section of the /etc/init.d/mysql script, so that this check is skipped, and restarted the server. When I run the profile script I created earlier, I get the following result:
Server version: 5.0.32-Debian_7etch3-log
Server version: 5.0.32-Debian_7etch3-log
Server version: 5.0.32-Debian_7etch3-log
Server version: 5.0.32-Debian_7etch3-log
Server version: 5.0.32-Debian_7etch3-log
Thread ID: 52978590
Total: 0.016000
%self total self wait child calls name
87.50 0.01 0.01 0.00 0.00 5 <Class::Mysql>#real_connect (ruby_runtime:0}
6.25 0.00 0.00 0.00 0.00 10 IO#write (ruby_runtime:0}
6.25 0.00 0.00 0.00 0.00 5 Mysql#close (ruby_runtime:0}
0.00 0.00 0.00 0.00 0.00 5 Kernel#puts (ruby_runtime:0}
0.00 0.00 0.00 0.00 0.00 5 Mysql#initialize (ruby_runtime:0}
0.00 0.00 0.00 0.00 0.00 5 String#+ (ruby_runtime:0}
0.00 0.02 0.00 0.00 0.02 1 Global#[No method] (tmp/mysql_test/test.rb:12}
0.00 0.02 0.00 0.00 0.02 1 Integer#times (ruby_runtime:0}
0.00 0.00 0.00 0.00 0.00 5 Mysql#get_server_info (ruby_runtime:0}