I have a table with a row count of 48769914. The problem is the bogus information when querying the database, i.e., the data_length. Any ideas on how to correct this misbehavior?
mysql> show table status like "events"\G
*************************** 1. row ***************************
Name: events
Engine: InnoDB
Version: 10
Row_format: Compact
Rows: 0
Avg_row_length: 0
Data_length: 16384
Max_data_length: 0
Index_length: 32768
Data_free: 7405043712
Auto_increment: 59816602
Create_time: 2012-06-05 05:12:37
Update_time: NULL
Check_time: NULL
Collation: utf8_general_ci
Checksum: NULL
Create_options:
Comment:
1 row in set (0.88 sec)
exact count:
mysql> select count(id) from events;
+-----------+
| count(id) |
+-----------+
| 48769914 |
+-----------+
1 row in set (5 min 37.67 sec)
Update The status information looks like the table was empty. Zero rows, zero row length and basically no data in the table. How can I get MySQL to show correct estimates for that data.
InnoDB row count is not precise, because InnoDB does not keep track of records count internally, it can only estimate this by amount of allocated space in the tablespace.
See InnoDB restrictions in the manual for more information
InnoDB doesn't store row count in the table status so that isn't bogus. You solve this by running your SELECT query where you want the row count.
Related
i recently upgraded from mySQL 5.6.34 -> 8.0.16 (on macOS 10.14.5) and i am noticing very strange behavior with the row counts returned from "SHOW TABLE STATUS" as well as the row counts in the "information_schema" table. consider this simple schema:
CREATE TABLE `test` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(255) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
INSERT INTO `test` (`id`, `name`) VALUES
(1, 'one'),
(2, 'two'),
(3, 'three'),
(4, 'four'),
(5, 'five');
when i then run the following query i see the expected output:
SELECT * FROM test;
+----+-------+
| id | name |
+----+-------+
| 1 | one |
| 2 | two |
| 3 | three |
| 4 | four |
| 5 | five |
+----+-------+
likewise when i then run the following query i see the expected output:
SELECT COUNT(*) FROM test;
+----------+
| COUNT(*) |
+----------+
| 5 |
+----------+
however when i then run the following query:
SHOW TABLE STATUS \G
*************************** 1. row ***************************
Name: test
Engine: MyISAM
Version: 10
Row_format: Dynamic
Rows: 0
Avg_row_length: 0
Data_length: 0
Max_data_length: 281474976710655
Index_length: 1024
Data_free: 0
Auto_increment: 1
Create_time: 2019-05-30 13:56:46
Update_time: 2019-05-30 16:02:24
Check_time: NULL
Collation: utf8_unicode_ci
Checksum: NULL
Create_options:
Comment:
it appears that there are no rows (even though there are 5). likewise i see the same results when i run:
SELECT table_name, table_rows FROM information_schema.tables WHERE table_schema = 'test';
+------------+------------+
| TABLE_NAME | TABLE_ROWS |
+------------+------------+
| test | 0 |
+------------+------------+
no rows? if i add/delete rows to the table the counts do not change. only after i run:
ANALYZE TABLE `test`
...do i see all of the row counts as correct. i am only seeing this on mySQL 8. everything worked as expected on mySQL 5. i am aware of problems with accurate row counts using InnoDB tables, but these are all MyISAM tables, which should always show the correct row counts. any help is appreciated. thanks.
The information schema tables underwent significant, incompatible changes in MySQL 8 with the introduction of the global data dictionary:
Previously, INFORMATION_SCHEMA queries for table statistics in the STATISTICS and TABLES tables retrieved statistics directly from storage engines. As of MySQL 8.0, cached table statistics are used by default.
The cache is controlled by the system variable information_schema_stats_expiry:
Some INFORMATION_SCHEMA tables contain columns that provide table statistics:
[...] TABLES.TABLE_ROWS [...]
Those columns represent dynamic table metadata; that is, information that changes as table contents change.
By default, MySQL retrieves cached values for those columns from the mysql.index_stats and mysql.table_stats dictionary tables when the columns are queried, which is more efficient than retrieving statistics directly from the storage engine. If cached statistics are not available or have expired, MySQL retrieves the latest statistics from the storage engine and caches them in the mysql.index_stats and mysql.table_stats dictionary tables. Subsequent queries retrieve the cached statistics until the cached statistics expire.
[...]
To update cached values at any time for a given table, use ANALYZE TABLE.
To always retrieve the latest statistics directly from the storage engine and bypass cached values, set information_schema_stats_expiry to 0.
This is consistent with your behaviour.
You can set information_schema_stats_expiry globally to 0, or per session whenever you need accurate statistics.
This question already has an answer here:
Mysql inconsistent number of rows count(*) vs table.table_rows in information_schema
(1 answer)
Closed 6 years ago.
Figures reported by MySQL count(*) and on information_schema.TABLES are totally different.
mysql> SELECT * FROM information_schema.TABLES WHERE TABLE_NAME = 'my_table'\G
*************************** 1. row ***************************
TABLE_CATALOG: def
TABLE_SCHEMA: my_db
TABLE_NAME: my_table
TABLE_TYPE: BASE TABLE
ENGINE: InnoDB
VERSION: 10
ROW_FORMAT: Compact
TABLE_ROWS: 31016698
AVG_ROW_LENGTH: 399
DATA_LENGTH: 12378439680
MAX_DATA_LENGTH: 0
INDEX_LENGTH: 4863262720
DATA_FREE: 5242880
AUTO_INCREMENT: NULL
CREATE_TIME: 2016-06-14 18:54:24
UPDATE_TIME: NULL
CHECK_TIME: NULL
TABLE_COLLATION: utf8_general_ci
CHECKSUM: NULL
CREATE_OPTIONS:
TABLE_COMMENT:
1 row in set (0.00 sec)
mysql> select count(*) from my_table;
+----------+
| count(*) |
+----------+
| 46406095 |
+----------+
1 row in set (27.45 sec)
Note that there are 31,016,698 rows according to information_schema, count() however report 46,406,095 rows...
Now which one can be trusted? Why these stats are different?
I'm using MySQL server v5.6.30.
The count in that metadata, similar to the output of SHOW TABLE STATUS, cannot be trusted. It's often off by a factor of 100 or more, either over or under.
The reason for this is the engine does not know how many rows are in the table until it calculates this. Under heavy load you might have a lot of contention on the primary key index which makes pinning down an exact value an expensive computation.
This approximation is computed based on the total data length divided by the average row length. It's rarely even close to what it should be unless your records are all about the same length and you haven't been deleting a lot of them.
The only value that can be truly trusted is COUNT(*) but that operation can take a long time to complete, so be warned.
I am trying to optimize my database by adjusting indices.
SHOW INDEXES FROM my_table
outputs
Table ... Key_name ... Column_name ... Cardinality ...
---------------------------------------------------------------------
my_table ... idx_field1 ... field1 ... 1 ...
while
SELECT field1 FROM my_table PROCEDURE ANALYSE()\G
outputs
*************************** 1. row ***************************
Field_name: my_db.my_table.field1
Min_value: ow
Max_value: rt
Min_length: 2
Max_length: 2
Empties_or_zeros: 0
Nulls: 0
Avg_value_or_avg_length: 2.0000
Std: NULL
Optimal_fieldtype: ENUM('ow','rt') NOT NULL
1 row in set (0.26 sec)
i.e., the reported cardinality (1) is not equal to the number of unique values (2). Why?
PS. I did perform
analyze table my_table
before running the queries.
The "cardinality" in SHOW INDEXES is an approximation. ANALYSE() gets the exact value because it is derived from an exhaustive scan of the table.
The former is used for deciding how to optimize a query. Generally, a low cardinality (whether 1 or 2) implies that an index on that field is not worth using.
Where are you headed with this question?
My primary key is an integer.
I get the error #1062 - Duplicate entry '4294967295' for key 'PRIMARY' on next insert.
Reason: Obviously maximum value is reached.
Is there any way i can find all tables whose integer columns have reached max values or close to reaching max values to avoid such errors
Query:
SELECT
id, IF(max(id)<4294967295, 'true', 'false') as insert_flag
FROM Table
Note: I have checked in above query, whether max id's value is less
then Int Range (4294967295). Query returns true and false in
insert_flag column. You can check this in your code before insertion.
If you were talking about id as some kind of AUTO_INCREMENT field, then you can get this information for all tables with show table status (please use help show table status for more details).
e.g.
mysql> show table status \G
*************************** 1. row ***************************
Name: test_table
Engine: InnoDB
Version: 10
Row_format: Compact
Rows: 0
Avg_row_length: 0
Data_length: 16384
Max_data_length: 0
Index_length: 65536
Data_free: 0
Auto_increment: 2
Create_time: 2014-11-07 13:17:16
Update_time: NULL
Check_time: NULL
Collation: latin1_swedish_ci
Checksum: NULL
Create_options:
Comment:
1 row in set (0.00 sec)
Here's a way to check all the auto-increment columns to see how close they are to their maximum:
SELECT c.TABLE_SCHEMA, c.TABLE_NAME, c.COLUMN_NAME,
(#max_auto_increment_size := CASE
WHEN c.COLUMN_TYPE like 'tinyint% unsigned' THEN 255
WHEN c.COLUMN_TYPE like 'tinyint%' THEN 127
WHEN c.COLUMN_TYPE like 'smallint% unsigned' THEN 65535
WHEN c.COLUMN_TYPE like 'smallint%' THEN 32767
WHEN c.COLUMN_TYPE like 'mediumint% unsigned' THEN 16777215
WHEN c.COLUMN_TYPE like 'mediumint%' THEN 8388607
WHEN c.COLUMN_TYPE like 'int% unsigned' THEN 4294967295
WHEN c.COLUMN_TYPE like 'int%' THEN 2147483647
WHEN c.COLUMN_TYPE like 'bigint% unsigned' THEN 18446744073709551615
WHEN c.COLUMN_TYPE like 'bigint%' THEN 9223372036854775807
ELSE 127
END) AS MAX_AUTO_INCREMENT_SIZE,
ROUND(#max_auto_increment_size - t.AUTO_INCREMENT) AS Headroom,
(CASE
WHEN ROUND(#max_auto_increment_size - t.AUTO_INCREMENT) < 1000 THEN 'CRITICAL'
WHEN ROUND(#max_auto_increment_size - t.AUTO_INCREMENT) < 10000 THEN 'LOW'
ELSE 'OK'
END) AS Remark
FROM information_schema.columns c
JOIN tables t ON (t.TABLE_SCHEMA = c.TABLE_SCHEMA AND t.TABLE_NAME = c.TABLE_NAME)
WHERE c.TABLE_SCHEMA NOT IN('information_schema','mysql','performance_schema')
AND c.EXTRA = 'auto_increment'
ORDER BY c.TABLE_SCHEMA, c.TABLE_NAME, c.COLUMN_NAME;
I have the following lookup-table:
CREATE TABLE `widgetuser` (
`widgetuserid` char(40) NOT NULL,
`userid` int(10) unsigned NOT NULL,
`created_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
PRIMARY KEY (`widgetuserid`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 DELAY_KEY_WRITE=1;
I have a widgetuser_tmp Table with the same structure but no key and I fill the widgetuser table with this data (4mio rows):
mysql> insert into widgetuser select * from widgetuser_tmp limit 0,4000000;flush tables;
Query OK, 4000000 rows affected (33.14 sec)
Records: 4000000 Duplicates: 0 Warnings: 0
Query OK, 0 rows affected (0.91 sec)
While it is writing, it goes directly to RAID-1 with 15MB/s, disk util <50% and we see no reads, since I filled the disk cache with the source table:
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 3839.20 0.00 52.40 0.00 15.20 594.20 12.46 237.75 5.57 29.20
sdb 0.00 3839.00 0.00 52.60 0.00 15.20 591.94 14.50 275.59 7.19 37.80
I insert the next 1 Mio rows, it's all fine and the wMB/s goes back to 0 right after the flush:
mysql> insert into widgetuser select * from widgetuser_tmp limit 4000000,1000000;flush tables;
Query OK, 1000000 rows affected (10.18 sec)
Records: 1000000 Duplicates: 0 Warnings: 0
Query OK, 0 rows affected (0.87 sec)
mysql> insert into widgetuser select * from widgetuser_tmp limit 5000000,1000000;flush tables;
Query OK, 1000000 rows affected (10.21 sec)
Records: 1000000 Duplicates: 0 Warnings: 0
Query OK, 0 rows affected (1.02 sec)
mysql> insert into widgetuser select * from widgetuser_tmp limit 6000000,1000000;flush tables;
Query OK, 1000000 rows affected (10.67 sec)
Records: 1000000 Duplicates: 0 Warnings: 0
Query OK, 0 rows affected (1.17 sec)
But when I do the 7mio batch, the result still looks the same, but in the iostat -mdx sda sdb 5 suddenly we have 100% util for 30 seconds:
mysql> insert into widgetuser select * from widgetuser_tmp limit 7000000,1000000;flush tables;
Query OK, 1000000 rows affected (10.73 sec)
Records: 1000000 Duplicates: 0 Warnings: 0
Query OK, 0 rows affected (1.21 sec)
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 88.60 0.00 295.60 0.00 1.52 10.53 130.60 435.93 3.38 100.00
sdb 0.00 89.20 0.00 300.80 0.00 1.57 10.68 143.99 483.97 3.32 100.00
The data-files are not touched after the flush:
-rw-rw---- 1 mysql mysql 1032000000 2009-10-30 12:10 widgetuser.MYD
-rw-rw---- 1 mysql mysql 522777600 2009-10-30 12:11 widgetuser.MYI
And also the table status seams normal:
+----------------+--------+---------+------------+----------+----------------+-------------+-------------------+--------------+-----------+----------------+---------------------+---------------------+------------+-----------------+----------+-------------------+---------+
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment |
+----------------+--------+---------+------------+----------+----------------+-------------+-------------------+--------------+-----------+----------------+---------------------+---------------------+------------+-----------------+----------+-------------------+---------+
| widgetuser | MyISAM | 10 | Fixed | 8000000 | 129 | 1032000000 | 36310271995674623 | 522777600 | 0 | NULL | 2009-10-30 11:59:41 | 2009-10-30 12:10:59 | NULL | utf8_general_ci | NULL | delay_key_write=1 | |
+----------------+--------+---------+------------+----------+----------------+-------------+-------------------+--------------+-----------+----------------+---------------------+---------------------+------------+-----------------+----------+-------------------+---------+
And when I continue (since we have 100% drive utilization), it get's worse very fast:
mysql> insert into widgetuser select * from widgetuser_tmp limit 9000000,1000000;flush tables;
Query OK, 1000000 rows affected (31.93 sec)
Records: 1000000 Duplicates: 0 Warnings: 0
Query OK, 0 rows affected (2.34 sec)
mysql> insert into widgetuser select * from widgetuser_tmp limit 10000000,1000000;flush tables;
Query OK, 1000000 rows affected (2 min 39.72 sec)
Records: 1000000 Duplicates: 0 Warnings: 0
Query OK, 0 rows affected (7.82 sec)
The primary key is checked to see whether the new entry is unique or not. As soon as the key does not fit into memory (key_buffer_size=512MB = ca. 8Mio entries), it needs to fetch the missing key parts from the drive(-cache) for checking it. Therefore we should see more reads and slower insert times - we don't see the slower reads since the key is buffered in disk cache. But my question: who is writing suddenly so much and where and why and how can I fix this? Any ideas are appreciated!
Futher ideas and insights:
since the 1MB/s random writes follow the finished statement, the unique validation is already passed
it is a software raid-1 with ahci on, disks are 93% free and capable of about 80wMB/s
the machine has 8GB ram, 5GB cache, 600MB taken by MySQL, 1,7GB free
MySQL 5.1.31-1ubuntu2-log
the delay_key_write does not change this behavior
myisam_sort_buffer_size = 2 GB (not used here, though?)
key_buffer_size = 512 MB
bin_log is off
Linux 2.6.28-15-server #52-Ubuntu SMP Wed Sep 9 11:34:09 UTC 2009 x86_64 GNU/Linux
It's not entirely clear from your question what behaviour you're expecting, or getting. Here are some things you might not know
FLUSH TABLES blows away the MyISAM key cache - it doesn't just write dirty blocks, it also discards clean ones so every index block must be fetched again to be modified
MyISAM uses a block size of 1k by default which is probably smaller than your filesystem blocks; this can create performance problems
If you intend any kind of durability (which you presumably don't, because you're using MyISAM), then you should use hardware raid with a battery-backed cache in the controller.
My guess is that either the index no longer fits in the key buffer, or that it's having to do a lot more writes, which trigger reads because they're unbuffered writes off the block-size boundaries.
Try changing myisam_block_size to 4k or higher and rebuild the table (this is a my.cnf-only option which only takes effect on new tables after a restart).
You can examine the block size on a table with myisamchk -dv
i'm using mariadb5528,if usage of key_buffer_size >90% ,it seems that delay_key_write don't work
so enlarge the key_buffer_size to 2G.