I have the following lookup-table:
CREATE TABLE `widgetuser` (
`widgetuserid` char(40) NOT NULL,
`userid` int(10) unsigned NOT NULL,
`created_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
PRIMARY KEY (`widgetuserid`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 DELAY_KEY_WRITE=1;
I have a widgetuser_tmp Table with the same structure but no key and I fill the widgetuser table with this data (4mio rows):
mysql> insert into widgetuser select * from widgetuser_tmp limit 0,4000000;flush tables;
Query OK, 4000000 rows affected (33.14 sec)
Records: 4000000 Duplicates: 0 Warnings: 0
Query OK, 0 rows affected (0.91 sec)
While it is writing, it goes directly to RAID-1 with 15MB/s, disk util <50% and we see no reads, since I filled the disk cache with the source table:
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 3839.20 0.00 52.40 0.00 15.20 594.20 12.46 237.75 5.57 29.20
sdb 0.00 3839.00 0.00 52.60 0.00 15.20 591.94 14.50 275.59 7.19 37.80
I insert the next 1 Mio rows, it's all fine and the wMB/s goes back to 0 right after the flush:
mysql> insert into widgetuser select * from widgetuser_tmp limit 4000000,1000000;flush tables;
Query OK, 1000000 rows affected (10.18 sec)
Records: 1000000 Duplicates: 0 Warnings: 0
Query OK, 0 rows affected (0.87 sec)
mysql> insert into widgetuser select * from widgetuser_tmp limit 5000000,1000000;flush tables;
Query OK, 1000000 rows affected (10.21 sec)
Records: 1000000 Duplicates: 0 Warnings: 0
Query OK, 0 rows affected (1.02 sec)
mysql> insert into widgetuser select * from widgetuser_tmp limit 6000000,1000000;flush tables;
Query OK, 1000000 rows affected (10.67 sec)
Records: 1000000 Duplicates: 0 Warnings: 0
Query OK, 0 rows affected (1.17 sec)
But when I do the 7mio batch, the result still looks the same, but in the iostat -mdx sda sdb 5 suddenly we have 100% util for 30 seconds:
mysql> insert into widgetuser select * from widgetuser_tmp limit 7000000,1000000;flush tables;
Query OK, 1000000 rows affected (10.73 sec)
Records: 1000000 Duplicates: 0 Warnings: 0
Query OK, 0 rows affected (1.21 sec)
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 88.60 0.00 295.60 0.00 1.52 10.53 130.60 435.93 3.38 100.00
sdb 0.00 89.20 0.00 300.80 0.00 1.57 10.68 143.99 483.97 3.32 100.00
The data-files are not touched after the flush:
-rw-rw---- 1 mysql mysql 1032000000 2009-10-30 12:10 widgetuser.MYD
-rw-rw---- 1 mysql mysql 522777600 2009-10-30 12:11 widgetuser.MYI
And also the table status seams normal:
+----------------+--------+---------+------------+----------+----------------+-------------+-------------------+--------------+-----------+----------------+---------------------+---------------------+------------+-----------------+----------+-------------------+---------+
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment |
+----------------+--------+---------+------------+----------+----------------+-------------+-------------------+--------------+-----------+----------------+---------------------+---------------------+------------+-----------------+----------+-------------------+---------+
| widgetuser | MyISAM | 10 | Fixed | 8000000 | 129 | 1032000000 | 36310271995674623 | 522777600 | 0 | NULL | 2009-10-30 11:59:41 | 2009-10-30 12:10:59 | NULL | utf8_general_ci | NULL | delay_key_write=1 | |
+----------------+--------+---------+------------+----------+----------------+-------------+-------------------+--------------+-----------+----------------+---------------------+---------------------+------------+-----------------+----------+-------------------+---------+
And when I continue (since we have 100% drive utilization), it get's worse very fast:
mysql> insert into widgetuser select * from widgetuser_tmp limit 9000000,1000000;flush tables;
Query OK, 1000000 rows affected (31.93 sec)
Records: 1000000 Duplicates: 0 Warnings: 0
Query OK, 0 rows affected (2.34 sec)
mysql> insert into widgetuser select * from widgetuser_tmp limit 10000000,1000000;flush tables;
Query OK, 1000000 rows affected (2 min 39.72 sec)
Records: 1000000 Duplicates: 0 Warnings: 0
Query OK, 0 rows affected (7.82 sec)
The primary key is checked to see whether the new entry is unique or not. As soon as the key does not fit into memory (key_buffer_size=512MB = ca. 8Mio entries), it needs to fetch the missing key parts from the drive(-cache) for checking it. Therefore we should see more reads and slower insert times - we don't see the slower reads since the key is buffered in disk cache. But my question: who is writing suddenly so much and where and why and how can I fix this? Any ideas are appreciated!
Futher ideas and insights:
since the 1MB/s random writes follow the finished statement, the unique validation is already passed
it is a software raid-1 with ahci on, disks are 93% free and capable of about 80wMB/s
the machine has 8GB ram, 5GB cache, 600MB taken by MySQL, 1,7GB free
MySQL 5.1.31-1ubuntu2-log
the delay_key_write does not change this behavior
myisam_sort_buffer_size = 2 GB (not used here, though?)
key_buffer_size = 512 MB
bin_log is off
Linux 2.6.28-15-server #52-Ubuntu SMP Wed Sep 9 11:34:09 UTC 2009 x86_64 GNU/Linux
It's not entirely clear from your question what behaviour you're expecting, or getting. Here are some things you might not know
FLUSH TABLES blows away the MyISAM key cache - it doesn't just write dirty blocks, it also discards clean ones so every index block must be fetched again to be modified
MyISAM uses a block size of 1k by default which is probably smaller than your filesystem blocks; this can create performance problems
If you intend any kind of durability (which you presumably don't, because you're using MyISAM), then you should use hardware raid with a battery-backed cache in the controller.
My guess is that either the index no longer fits in the key buffer, or that it's having to do a lot more writes, which trigger reads because they're unbuffered writes off the block-size boundaries.
Try changing myisam_block_size to 4k or higher and rebuild the table (this is a my.cnf-only option which only takes effect on new tables after a restart).
You can examine the block size on a table with myisamchk -dv
i'm using mariadb5528,if usage of key_buffer_size >90% ,it seems that delay_key_write don't work
so enlarge the key_buffer_size to 2G.
Related
Source: MS Access on Windows network share
Target: MySQL/MariaDB on Ubuntu
Tools: mdb-export, mysqlimport
record count: 1,5 Mio +
I wonder if there is a fast and reliable way of comparing the imported data records.
Is there an SQL standard equivalent to e.g. md5 fingerprint hashes of files? Right now, I am building different import routines and I only want to fast check for similarity and (if failed) search for the detailed differences later on.
A somewhat of a quick-and-dirty approach for individual columns can be implemented using stored aggregate functions which should be SQL standard.
This is how you'd do it with MariaDB:
CREATE AGGREGATE FUNCTION IF NOT EXISTS my_checksum(x TEXT) RETURNS CHAR(40)
DETERMINISTIC
BEGIN
DECLARE cksum CHAR(40) DEFAULT SHA1('');
DECLARE CONTINUE HANDLER FOR NOT FOUND
RETURN cksum;
LOOP
FETCH GROUP NEXT ROW;
SET cksum = SHA1(CONCAT(cksum, x));
END LOOP;
END
You can then calculate a checksum from of a column as such:
MariaDB [test]> create or replace table t1(data varchar(20));
Query OK, 0 rows affected (0.063 sec)
MariaDB [test]> create or replace table t2(data varchar(20));
Query OK, 0 rows affected (0.064 sec)
MariaDB [test]> insert into t1 values ('hello'), ('world'), ('!');
Query OK, 3 rows affected (0.011 sec)
Records: 3 Duplicates: 0 Warnings: 0
MariaDB [test]> insert into t2 values ('Hello'), ('World'), ('!');
Query OK, 3 rows affected (0.015 sec)
Records: 3 Duplicates: 0 Warnings: 0
MariaDB [test]> select my_checksum(data) from t1;
+------------------------------------------+
| my_checksum(data) |
+------------------------------------------+
| 7f6fb9a61c2097f70a36254c332c47364c496e07 |
+------------------------------------------+
1 row in set (0.001 sec)
MariaDB [test]> select my_checksum(data) from t2;
+------------------------------------------+
| my_checksum(data) |
+------------------------------------------+
| 5f683ea3674e33ce24bff5f68f53509566ad4da2 |
+------------------------------------------+
1 row in set (0.001 sec)
MariaDB [test]> delete from t2;
Query OK, 3 rows affected (0.011 sec)
MariaDB [test]> insert into t2 values ('hello'), ('world'), ('!');
Query OK, 3 rows affected (0.012 sec)
Records: 3 Duplicates: 0 Warnings: 0
MariaDB [test]> select my_checksum(data) from t2;
+------------------------------------------+
| my_checksum(data) |
+------------------------------------------+
| 7f6fb9a61c2097f70a36254c332c47364c496e07 |
+------------------------------------------+
1 row in set (0.001 sec)
This of course relies on the SHA1 of the column being the same on all databases. Conversions into strings should make it mostly compatible but there might be differences in how these are implemented in different databases.
The percona toolkit has the tool you need.
https://docs.percona.com/percona-toolkit/
See pt-table-checksum and pt-table-sync
I found it.
It's very simple and very fast.
CHECKSUM TABLE tbl_name
Will give you a number value to compare.
And it's Transact-SQL so will hopefully work the same on MS Access, MySQL and MariaDB
I have a BBDD (MySQL 8) with several tables. In one of them I have a blob column (called sensitive_data) which has more or less legible content if we perform a SQL query:
SELECT * FROM sensitive_information WHERE sensitive_data LIKE '%name%';
This returns something like
????/namenKZ???Oj?B | ?!John Doe???????????????
I installed keyring plugin and it's active:
PLUGIN_NAME | PLUGIN_STATUS |
+--------------+-----------+ |
keyring_file | ACTIVE
If I execute the statement:
ALTER TABLE sensitive_information ENCRYPTION='Y';
The output console says:
Query OK, 36 rows affected (0,01 sec)
Records: 36 Duplicates: 0 Warnings: 0
But if I perform the same SELECT query I mentioned above I get the very same result, like no encryption was done.
What am I missing?
Thanks in advance.
I'm implementing a custom table-based sequence generator for MySQL database v5.7.16 with InnoDB engine.
The sequence_table looks as follows:
+-------------+-----------+
|sequence_name|next_value |
+-------------+-----------+
| first_seq | 1 |
+-------------+-----------+
| second_seq | 1 |
+-------------+-----------+
sequence_name column is a primary key.
This sequence table contains multiple sequences for different consumers.
I use the following strategy for the sequence updates:
Select current sequence value: select next_val from sequence_table where sequence_name=?.
Add the allocation size to current sequence value.
Update the sequence value if it's current value matches the value selected in the first step: update sequence_table set next_val=? where sequence_name=? and next_val=?.
If the update is successful return the increased sequence value, otherwise repeat the process from step 1.
The documentation contains the following information:
UPDATE ... WHERE ... sets an exclusive next-key lock on every record
the search encounters. However, only an index record lock is required
for statements that lock rows using a unique index to search for a
unique row. 14.5.3 Locks Set by Different SQL Statements in InnoDB
The part in bold is a bit confusing.
As you can see, I match the primary key in the WHERE clause of the UPDATE statement.
Is it possible that the search may encounter more than one record and therefore lock multiple rows in this sequence table?
In other words, will the update in the 3rd step of the algorithm block just one or multiple rows?
You didn't mention what transaction isolation level you're planning to use.
Lets assume you're using repeatable read (in read committed no such a problem should exist)
From here:
For locking reads (SELECT with FOR UPDATE or LOCK IN SHARE MODE),
UPDATE, and DELETE statements, locking depends on whether the
statement uses a unique index with a unique search condition, or a
range-type search condition
and
For a unique index with a unique search condition, InnoDB locks only
the index record found, not the gap before it
So at least in theory it should lock only a single record and no next-key lock will be used.
More quotes from other docs pages to back my thoughts:
innodb-next-key-locks
link
A next-key lock is a combination of a record lock on the index record
and a gap lock on the gap before the index record.
gap locks
link
Gap locking is not needed for statements that lock rows using a unique
index to search for a unique row
Don't grab the sequence numbers inside the main transaction; do it before the START TRANSCTION.
Do the task in a single statement with autocommit=ON.
Both of those lead to it being much faster, less likely to block.
(You code was missing BEGIN/COMMIT and FOR UPDATE. I got rid of those rather than explaining the issues.)
Set up test:
mysql> CREATE TABLE so49197964 (
-> name VARCHAR(22) NOT NULL,
-> next_value INT UNSIGNED NOT NULL,
-> PRIMARY KEY (name)
-> ) ENGINE=InnoDB;
Query OK, 0 rows affected (0.02 sec)
mysql> INSERT INTO so49197964 (name, next_value)
-> VALUES
-> ('first', 1), ('second', 1);
Query OK, 2 rows affected (0.00 sec)
Records: 2 Duplicates: 0 Warnings: 0
mysql> SELECT * FROM so49197964;
+--------+------------+
| name | next_value |
+--------+------------+
| first | 1 |
| second | 1 |
+--------+------------+
2 rows in set (0.00 sec)
Grab 20 nums from 'first' and fetch the starting number:
mysql> UPDATE so49197964
-> SET next_value = LAST_INSERT_ID(next_value) + 20
-> WHERE name = 'first';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0
mysql> SELECT LAST_INSERT_ID();
+------------------+
| LAST_INSERT_ID() |
+------------------+
| 1 |
+------------------+
1 row in set (0.00 sec)
mysql> SELECT * FROM so49197964;
+--------+------------+
| name | next_value |
+--------+------------+
| first | 21 |
| second | 1 |
+--------+------------+
2 rows in set (0.00 sec)
Grab another 20:
mysql> UPDATE so49197964
-> SET next_value = LAST_INSERT_ID(next_value) + 20
-> WHERE name = 'first';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0
mysql> SELECT LAST_INSERT_ID();
+------------------+
| LAST_INSERT_ID() |
+------------------+
| 21 |
+------------------+
1 row in set (0.00 sec)
mysql> SELECT * FROM so49197964;
+--------+------------+
| name | next_value |
+--------+------------+
| first | 41 |
| second | 1 |
+--------+------------+
2 rows in set (0.00 sec)
This question already has an answer here:
Mysql inconsistent number of rows count(*) vs table.table_rows in information_schema
(1 answer)
Closed 6 years ago.
Figures reported by MySQL count(*) and on information_schema.TABLES are totally different.
mysql> SELECT * FROM information_schema.TABLES WHERE TABLE_NAME = 'my_table'\G
*************************** 1. row ***************************
TABLE_CATALOG: def
TABLE_SCHEMA: my_db
TABLE_NAME: my_table
TABLE_TYPE: BASE TABLE
ENGINE: InnoDB
VERSION: 10
ROW_FORMAT: Compact
TABLE_ROWS: 31016698
AVG_ROW_LENGTH: 399
DATA_LENGTH: 12378439680
MAX_DATA_LENGTH: 0
INDEX_LENGTH: 4863262720
DATA_FREE: 5242880
AUTO_INCREMENT: NULL
CREATE_TIME: 2016-06-14 18:54:24
UPDATE_TIME: NULL
CHECK_TIME: NULL
TABLE_COLLATION: utf8_general_ci
CHECKSUM: NULL
CREATE_OPTIONS:
TABLE_COMMENT:
1 row in set (0.00 sec)
mysql> select count(*) from my_table;
+----------+
| count(*) |
+----------+
| 46406095 |
+----------+
1 row in set (27.45 sec)
Note that there are 31,016,698 rows according to information_schema, count() however report 46,406,095 rows...
Now which one can be trusted? Why these stats are different?
I'm using MySQL server v5.6.30.
The count in that metadata, similar to the output of SHOW TABLE STATUS, cannot be trusted. It's often off by a factor of 100 or more, either over or under.
The reason for this is the engine does not know how many rows are in the table until it calculates this. Under heavy load you might have a lot of contention on the primary key index which makes pinning down an exact value an expensive computation.
This approximation is computed based on the total data length divided by the average row length. It's rarely even close to what it should be unless your records are all about the same length and you haven't been deleting a lot of them.
The only value that can be truly trusted is COUNT(*) but that operation can take a long time to complete, so be warned.
There is quite a lot of confusion in the description of these variables, in official documentation of MySQL.
According to it, max_binlog_cache_size means,
If a transaction requires more than this many bytes of memory, the
server generates a Multi-statement transaction required more than
'max_binlog_cache_size' bytes of storage error.
max_binlog_cache_size sets the size for the transaction cache only
and binlog_cache_size means,
The size of the cache to hold changes to the binary log during a
transaction.
binlog_cache_size sets the size for the transaction cache only
On reading the documentation, I observed there is no difference among these two. There is also something very confusing in the documentation like,
In MySQL 5.7, the visibility to sessions of max_binlog_cache_size
matches that of the binlog_cache_size system variable; in other words,
changing its value effects only new sessions that are started after
the value is changed.
When I query the server variables it shows both. I have a MySQL 5.6 and a MySQL 5.7. All I need to know is, which variable I should consider and configure for which server.
binlog_cache_size for MySQL 5.6 and max_binlog_cache_size for MySQL 5.7??
There are additional confusing variables max_binlog_stmt_cache_size and binlog_stmt_cache_size, related to these.
Both variables can be configured in both versions, they have different meaning. Definitions in the manual and in the help are confusing; here is a much better explanation: http://dev.mysql.com/doc/refman/5.6/en/binary-log.html
binlog_cache_size defines the maximum amount of memory that the buffer can use. If transaction grows above this value, it uses a temporary disk file. Please note that the buffer is allocated per connection.
max_binlog_cache_size defines the maximum total size of a transaction. If the transaction grows above this value, it fails.
Below is a simple demonstration of the difference.
Setup:
MariaDB [test]> select ##binlog_cache_size, ##max_binlog_cache_size, ##binlog_format;
+---------------------+-------------------------+-----------------+
| ##binlog_cache_size | ##max_binlog_cache_size | ##binlog_format |
+---------------------+-------------------------+-----------------+
| 32768 | 65536 | ROW |
+---------------------+-------------------------+-----------------+
1 row in set (0.01 sec)
MariaDB [test]> show create table t1 \G
*************************** 1. row ***************************
Table: t1
Create Table: CREATE TABLE `t1` (
`a` text
) ENGINE=InnoDB DEFAULT CHARSET=latin1
1 row in set (0.00 sec)
1. Transaction size is below ##binlog_cache_size
(transaction succeeds, uses the cache, does not use the disk)
MariaDB [test]> flush status;
Query OK, 0 rows affected (0.00 sec)
MariaDB [test]> begin;
Query OK, 0 rows affected (0.00 sec)
MariaDB [test]> insert into t1 values (repeat('a',20000));
Query OK, 1 row affected (0.01 sec)
MariaDB [test]> insert into t1 values (repeat('a',10000));
Query OK, 1 row affected (0.04 sec)
MariaDB [test]> commit;
Query OK, 0 rows affected (0.05 sec)
MariaDB [test]> show status like 'Binlog_cache%';
+-----------------------+-------+
| Variable_name | Value |
+-----------------------+-------+
| Binlog_cache_disk_use | 0 |
| Binlog_cache_use | 1 |
+-----------------------+-------+
2 rows in set (0.01 sec)
2. Transaction size is above ##binlog_cache_size, but below ##max_binlog_cache_size
(transaction uses the cache, and the cache uses the disk)
MariaDB [test]> flush status;
Query OK, 0 rows affected (0.00 sec)
MariaDB [test]> begin;
Query OK, 0 rows affected (0.00 sec)
MariaDB [test]> insert into t1 values (repeat('a',20000));
Query OK, 1 row affected (0.10 sec)
MariaDB [test]> insert into t1 values (repeat('a',20000));
Query OK, 1 row affected (0.10 sec)
MariaDB [test]> commit;
Query OK, 0 rows affected (0.03 sec)
MariaDB [test]> show status like 'Binlog_cache%';
+-----------------------+-------+
| Variable_name | Value |
+-----------------------+-------+
| Binlog_cache_disk_use | 1 |
| Binlog_cache_use | 1 |
+-----------------------+-------+
2 rows in set (0.01 sec)
3. Transaction size exceeds ##max_binlog_cache_size
(transaction fails)
MariaDB [test]> flush status;
Query OK, 0 rows affected (0.00 sec)
MariaDB [test]> begin;
Query OK, 0 rows affected (0.00 sec)
MariaDB [test]> insert into t1 values (repeat('a',20000));
Query OK, 1 row affected (0.12 sec)
MariaDB [test]> insert into t1 values (repeat('a',20000));
Query OK, 1 row affected (0.15 sec)
MariaDB [test]> insert into t1 values (repeat('a',20000));
Query OK, 1 row affected (0.12 sec)
MariaDB [test]> insert into t1 values (repeat('a',20000));
ERROR 1197 (HY000): Multi-statement transaction required more than 'max_binlog_cache_size' bytes of storage; increase this mysqld variable and try again
So, if your transactions are big, but you don't have too many connections, you might want to increase ##binlog_cache_size to avoid excessive disk writes.
If you have many concurrent connections, you should be careful to avoid connections trying to allocate too much memory for the caches simultaneously.
If you want to make sure that transactions don't grow too big, you might want to limit ##max_binlog_cache_size.
##binlog_stmt_cache_size and ##max_binlog_stmt_cache_size should work in a similar way, the difference is that %binlog_cache% values are for transactional updates, and %binlog_stmt_cache% for non-transactional updates.
While experimenting, please note that the values are not 100% precise, there are some hidden subtleties with initially allocated sizes. It shouldn't matter for practical purposes, but can be confusing when you play with low values.