I have a MyISAM table, that contains just one field, a SMALL INT. That field has an index on it, and there are 5.6 million records.
So in theory 5.6mil * 2 bytes (smallint) = 11MB (approx), but the data file of the table is 40MB, why so different?
The index file takes up 46MB, would would it be bigger than the data file?
Here is the create table:
CREATE TABLE `key_test` (
`key2` smallint(5) unsigned NOT NULL DEFAULT '0',
KEY `key2` (`key2`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1
There is some overhead.
First, some controllable 'variables':
myisam_data_pointer_size defaults to 6 (bytes).
myisam_index_pointer_size defaults to 1 less than that.
For the data (.MYD):
N bytes for up to 8*N NULLable columns. (N=0 for your table.)
1 byte for "deleted". You do have this.
DELETEd rows leave gaps.
When a record is deleted, the gap is filled in with a data_pointer to the next record. This implies that the smallest a row can be is 6 bytes.
So: 1 + MAX(row_length, 6) = 7 bytes per row.
If you had 3 SMALLINTs, the table would be the same size.
For the index (.MYI):
BTree organization has some overhead; if randomly built, it settles in on about 69% full.
A 6-byte pointer (byte offset into .MYD, DYNAMIC) is needed in each leaf row.
Links within the BTree are a 5-byte row that is controlled by an unlisted setting (myisam_index_pointer_size).
So: row_length + 6 per record, plus some overhead. 46M sounds like the data was sorted so that the index was built "in order".
Beyond that, my memory of MyISAM details is fading.
Related
I've recently noticed a significant increase in variance of time needed to complete simple INSERT statements. While these statements on average take around 11ms, they can sometimes take 10-30 seconds, and I even noticed them taking over 5 minutes to execute.
MySQL version is 8.0.24, running on Windows Server 2016. The server's resources are never overloaded as far as I can tell. There is an ample amount of cpu overhead for the server to use, and 32GB of ram is allocated to it.
This is the table I'm working with:
CREATE TABLE `saved_segment` (
`recording_id` bigint unsigned NOT NULL,
`index` bigint unsigned NOT NULL,
`start_filetime` bigint unsigned NOT NULL,
`end_filetime` bigint unsigned NOT NULL,
`offset_and_size` bigint unsigned NOT NULL DEFAULT '18446744073709551615',
`storage_id` tinyint unsigned NOT NULL,
PRIMARY KEY (`recording_id`,`index`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
The table has no other indices or foreign keys, nor is it used as a reference for a foreign key in any other table. The entire table size is approximately 20GB with around 281M rows, which doesn't strike me as too large.
The table is used almost entirely read-only, with up to 1000 reads per second. All of these reads happen in simple SELECT queries, not in complex transactions, and they utilize the primary key index efficiently. There are very few, if any, concurrent writes to this table. This has been done intentionally in order to try to figure out if it would help with slow inserts, but it didn't. Before that there were up to 10 concurrent inserts going on at all times. UPDATE or DELETE statements are never executed on this table.
The queries that I'm having trouble with are all structured like this. They never appear in a transaction. While the inserts are definitely not append-only according to the clustered primary key, the queries almost alwayas insert 1 to 20 adjacent rows into the table:
INSERT IGNORE INTO saved_segment
(recording_id, `index`, start_filetime, end_filetime, offset_and_size, storage_id) VALUES
(19173, 631609, 133121662986640000, 133121663016640000, 20562291758298876, 10),
(19173, 631610, 133121663016640000, 133121663046640000, 20574308942546216, 10),
(19173, 631611, 133121663046640000, 133121663076640000, 20585348350688128, 10),
(19173, 631612, 133121663076640000, 133121663106640000, 20596854568114720, 10),
(19173, 631613, 133121663106640000, 133121663136640000, 20609723363860884, 10),
(19173, 631614, 133121663136640000, 133121663166640000, 20622106425668780, 10),
(19173, 631615, 133121663166640000, 133121663196640000, 20634653501528448, 10),
(19173, 631616, 133121663196640000, 133121663226640000, 20646967172721148, 10),
(19173, 631617, 133121663226640000, 133121663256640000, 20657773176227488, 10),
(19173, 631618, 133121663256640000, 133121663286640000, 20668825200822108, 10)
This is the output for an EXPLAIN statement of the above query:
id
select_type
table
partitions
type
possible_keys
key
key_len
ref
rows
filtered
Extra
1
INSERT
saved_segment
NULL
ALL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
These problems are relatively recent and weren't apparent while the table was around twice as small.
I tried reducing the number of concurrent inserts into the table, from around 10 to 1. I also deleted foreign keys on some columns (recording_id) in order to speed up inserts farther. ANALYZE TABLE and schema profiling didn't yield any actionable information.
One solution I had in mind was to remove the clustered primary key, add an AUTOINCREMENT primary key and a regular index on the (recording_id, index) columns. In my mind this would help by making inserts 'append only'. I'm open to any and all suggestions, thanks in advance!
EDIT:
I'm going to address some points and questions raised in the comments and answers:
autocommit is set to ON
the value of innodb_buffer_pool_size is 21474836480, and the value of innodb_buffer_pool_chunk_size is 134217728
One comment raised a concern about contention between the read-locks used by reads, and the exclusive locks used by writes. The table is question is used somewhat like a cache, and I don't need reads to always reflect the most up to date state of the table, if that would mean increased performance. The table should however remain durable even in cases of server crashes and hardware failures. Is this possible to achieve with a more relaxed transaction isolation level?
The schema could definitely be optimized; recording_id could be 4 byte integer, end_filetime could instead be an elapsed value, and start_filetime could also probably be smaller. I'm afraid that these changes would just push the issue back for a while until the table grows in size to compensate for the space savings.
INSERTs into the table are always sequential
SELECTs executed on the table look like this:
SELECT TRUE
FROM saved_segment
WHERE recording_id = ? AND `index` = ?
SELECT index, start_filetime, end_filetime, offset_and_size, storage_id
FROM saved_segment
WHERE recording_id = ? AND
start_filetime >= ? AND
start_filetime <= ?
ORDER BY `index` ASC
The second type of query could definitely be improved with an index, but I'm afraid this would further degrade INSERT performance.
Another thing that I forgot to mention is the existence of a very similar table to this one. It is queried and inserted into in exactly the same manner, but might further contribute to IO starvation.
EDIT2:
Results of SHOW TABLE STATUS for the table saved_segment, and a very similar table saved_screenshot (this one has an aditional INDEX on an bigint unsigned not null column).
Name
Engine
Version
Row_format
Rows
Avg_row_length
Data_length
Max_data_length
Index_length
Data_free
Auto_increment
Create_time
Update_time
Check_time
Collation
Checksum
Create_options
Comment
saved_screenshot
InnoDB
10
Dynamic
483430208
61
29780606976
0
21380464640
6291456
NULL
"2021-10-21 01:03:21"
"2022-11-07 16:51:45"
NULL
utf8mb4_0900_ai_ci
NULL
saved_segment
InnoDB
10
Dynamic
281861164
73
20802699264
0
0
4194304
NULL
"2022-11-02 09:03:05"
"2022-11-07 16:51:22"
NULL
utf8mb4_0900_ai_ci
NULL
I'll go out on a limb with this Answer.
Assuming that
The value of innodb_buffer_pool_size is somewhat less than 20MB, and
Those 1K Selects/second randomly reach int various parts of the table, then
The system recently become I/O bound because the 'next' block needed for the next Select is becoming more often not cached in the buffer_pool.
The simple solution is to get more RAM and up the setting of that tunable. But the table will only grow to whatever next limit you purchase.
Instead, here are some partial solutions.
If the numbers don't get too big, the first two columns could be INT UNSIGNED (4 bytes instead of 8) or maybe even MEDIUMINT UNSIGNED (3 bytes). Caution the ALTER TABLE would lock the table for a long time.
Those start and end times look like timestamps with fractional seconds that are always ".000". DATETIME and TIMESTAMP take 5 bytes (versus 8).
Your sample shows 0 elapsed time. If (end-start) is usually very small, then storing elapsed instead of endtime would further shrink the data. (But make it messy to use the endtime).
The sample data you presented looks "consecutive". That is about as efficient as an autoincrement. Is that the norm? If not, the INSERTs could be part of the I/O thrashing.
Your suggestion of adding AI, plus a secondary index, sort of doubles the effort for Inserts; so I do not recommend it.
More
just push the issue back for a while until the table grows in size
Yes, that will be the case.
Both of your queries are optimally helped by this as an INDEX or, even better, as the start of the PRIMARY KEY:
(recording_id, index)
Re:
SELECT TRUE
FROM saved_segment
WHERE recording_id = ? AND `index` = ?
If that is used to control some other SQL, consider adding this to that other SQL:
... EXISTS ( SELECT 1
FROM saved_segment
WHERE recording_id = ? AND `index` = ? ) ...
That query (in either form) needs what you already have
PRIMARY KEY(recording_id, index)
Your other query needs
INDEX(recording_id, start_filetime)
So, add that INDEX, or...
Even better... This combination would be better for both SELECTs:
PRIMARY KEY(recording_id, start_filetime, index).
INDEX(recording_id, index)
With that combo,
The single-row existence check would be performed "Using index" because it is "covering".
And the other query would find all the relevant rows clustered together on the PK.
(The PK has those 3 columns because it needs to be Unique. And they are in that order to benefit your second query. And it is the PK, not just an INDEX so it does not need to bounce between the index's BTree and the data's BTree.)
The "clustering" may help your performance by cutting down on the number of disk blocks needed for such queries. This leads to less "thrashing" in the buffer_pool, hence less need to increase RAM.
My index suggestions are mostly orthogonal to my datatype suggestions.
Background
The MySQL documentation states the following:
In contrast to CHAR, VARCHAR values are stored as a 1-byte or 2-byte length prefix plus data. The length prefix indicates the number of bytes in the value. A column uses one length byte if values require no more than 255 bytes, two length bytes if values may require more than 255 bytes.
To put this to the test myself, I created two tables:
CREATE TABLE `varchar_length_test_255` (
`characters` varchar(255) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
CREATE TABLE `varchar_length_test_256` (
`characters` varchar(256) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
I then inserted 10,000 rows into each table, each row with values having the maximum length for the characters column.
Since I am using a character set that has a maximum byte length of one byte per character (latin1), I expected to see a difference of 20,000 bytes in storage size between the two tables, derived from the following:
Each row in the varchar_length_test_256 table contains an additional character than the rows in the varchar_length_test_255 table. With the latin1 character set, that adds up to 10,000 bytes, since there are 10,000 rows in each table.
Based on the MySQL documentation, VARCHAR values exceeding 255 bytes require an additional "length" byte. Since each row in the varchar_length_test_256 table contains a value in the characters column that has a length of 256, which equates to 256 bytes for each value since the latin1 character set is used, that adds up to another 10,000 bytes utilized.
Problem
When issuing a query to retrieve the size of each table, it appears that the tables are the same size! I used the following query (based on off of this SO post) to determine the size of each table:
SELECT
table_name AS `Table`,
(data_length + index_length) `Size in Bytes`
FROM
information_schema.TABLES
WHERE
table_schema = "test";
which yielded this output:
+-------------------------+---------------+
| Table | Size in Bytes |
+-------------------------+---------------+
| varchar_length_test_255 | 4734976 |
| varchar_length_test_256 | 4734976 |
+-------------------------+---------------+
2 rows in set (0.00 sec)
What am I missing here?
Am I correctly understanding the MySQL documentation?
Is there something wrong with my test that is preventing the expected outcome?
Is the query I am using to calculate the size of the tables correct?
How could I correctly observe the information communicated in the MySQL documentation?
Check he data_free column too.
InnoDB stores data on so called 'pages' which are 16KB in size (by default). When a page is almost full, and you insert a new record, but it can't fit on the page, MySQL will open a new page leaving the leftover space empty.
It is my assumption, that MySQL reports the number of pages times the page size as data/index sizes.
This is the effective size used on the OS to store the table's data, not the actual size stored on those pages.
Update: https://mariadb.com/kb/en/library/information-schema-tables-table/
On this page (even if it is MariaDB, but the storage engine is the same) the descrtiption of data_lenght is the following:
For InnoDB/XtraDB, the index size, in pages, multiplied by the page
size. For Aria and MyISAM, length of the data file, in bytes. For
MEMORY, the approximate allocated memory.
Edit (some calculations)
16 KB = 16384 B
Storage (B) # of record # of pages
on a page
---------------------------------------------------
varchar(255) 256 64 156.25
varchar(256) 258 63.5 158.73
As you see the raw data (with the length marker) can be stored on almost the same amount of pages.
Due to the fact that a page is not necessary filled to 100% (however innodb_fill_factor defaults to 100) and there is some overhead in the row structure, this little difference won't necessarily visible.
The database files are not like a csv file, but they have to handle multiple things such as NULL values, row size when it is varying, etc which takes up additional space.
More about the InnoDB Row Structure: https://dev.mysql.com/doc/refman/5.5/en/innodb-physical-record.html
I was reading through various forums and stack over flow questions but couldn't find my answer.
I'm trying to find out the number of keys which will be stored in 16KB InnoDb Database page.
As you can see in this forum they mentioned how to calculate the number of keys for MyISAM in a single page . I would like to do the same for InnoDb. I don't understand how these calculations were made.
Im comparing a int which is 4KB and a VARCHAR (200). It would be great if I can get this calculation.
An index is structured as a BTree.
InnoDB BTrees are initially filled only 15/16th full.
After a lot of churn, a BTree averages 69% full.
There is a significant amount of overhead for each 'row' in a index entry.
The PRIMARY KEY (in InnoDB) is "clustered" with the data. So only the non-leaf nodes take extra blocks.
Secondary indexes contain all the column(s) of the PRIMARY KEY; that is how they 'point' to the record.
Based on the above two items, it is not meaningful to have just an INT in an the BTree for an index.
I use the simple Rule of Thumb: 100 'rows' per BTree.
Corollary: A million-row BTree is about 3 levels deep; a billion-row table is about 5 level deep.
Let's take this:
CREATE TABLE x (
id INT ...,
num INT ...,
str VARCHAR(200) CHARACTER SET utf8,
PRIMARY KEY (id),
INDEX i_num (num),
INDEX i_str (str)
) ENGINE=InnoDB;
For i_num, note that there are two INTs. You might get 300-400 'rows' per block. 10 million rows would take 3 levels.
For i_str, let's assume an average of 100 Korean characters -- that's 300 bytes for the text. You might get 25-35 'rows' per block. 10M rows would take 5 levels.
ALTER and OPTIMIZE may or may not defragment indexes.
There are information_schema tables that give some details about each BTree and its levels. Percona and MySQL have different tables for such.
Bottom line: The computations are too fuzzy to be exact. I hope my hand-waving has given you some better handles.
I have table storing phone numbers with 800M rows.
column
region_code_id smallint(4) unsigned YES
local_number mediumint(7) unsigned YES
region_id smallint(4) unsigned YES
operator_id smallint(4) unsigned YES
id int(10) unsigned NO PRI auto_increment
I need find number.id where region_code_id = 119 and localnumber = 1234567
select * from numbers where numbers.region_code_id = 119 and numbers.local_number = 1234567;
this query execute over 600 second.
How can I improve it ?
UPDATE
Thank for unswer, i understand i need index for this column, i try this as soon as I get the server with more SSD, now i have free 1GB SSD space. How i can to find out how much space the index will occupy?
Consider adding INDEX on columns which you use in WHERE clause.
Start with:
ALTER TABLE `numbers`
ADD INDEX `region_code_id_local_number`
(`region_code_id`, `local_number`);
Note : it can take some time for index to build.
Before and after change, execute explain plan to compare:
EXPLAIN EXTENDED select * from numbers where numbers.region_code_id = 119 and numbers.local_number = 1234567;
References:
How MySQL uses indexes
For this query:
select *
from numbers
where numbers.region_code_id = 119 and
numbers.local_number = 1234567;
You want an index on numbers(region_code_id, local_number) or numbers(local_number, region_code_id). The order of the columns doesn't matter because the conditions are equality for both columns.
create index idx_numbers_region_local on numbers(region_code_id, local_number);
I agree that INDEX(region_code_id, local_number) (in either order) is mandatory for this problem, but I am sticking my nose in to carry it a step further. Isn't that pair "unique"? Or do you have duplicate numbers in the table? If it is unique, then get rid of id and make that pair PRIMARY KEY(region_code_id, local_number). The table will possibly be smaller after the change.
Back to your question of "how big". How big is the table now? Perhaps 40GB? A secondary index (as originally proposed) would probably add about 20GB. And you would need 20-60GB of free disk space to perform the ALTER. This depends on whether adding the index can be done "inplace" in that version.
Changing the PK (as I suggest) would result in a little less than 40GB for the table. It will take 40GB of free space to perform the ALTER.
In general (and pessimistically), plan on an ALTER needing the both the original table and the new table sitting on disk at one time. That includes full copies of the data and index(es).
(A side question: Are you sure local_number is limited to 7 digits everywhere?)
Another approach to the question... For calculating the size of a table or index in InnoDB, add up the datatype sizes (3 bytes for MEDIUMINT, some average for VARCHAR, etc). Then multiply by the number of rows. Then multiply by 4; this will give you the approximate disk space needed. (Usually 2-3 is sufficient for the last multiplier.)
When changing the PK, do it in one step:
ALTER TABLE foo
DROP PRIMARY KEY,
ADD PRIMARY KEY(region_code_id, local_number);
Changing the PK cannot be done "inplace".
Edit (mostly for other readers)
#berap points out that id is needed for other purposes. Hence, dropping id and switching the PK is not an option.
However, this is sometimes an option (perhaps not in this case):
ALTER TABLE foo
DROP PRIMARY KEY,
ADD PRIMARY KEY(region_code_id, local_number),
ADD INDEX(id);
Notes:
The id..AUTO_INCREMENT will continue to work even with just INDEX.
The SELECT in question will be more efficient because it is the PK.
SELECT .. WHERE id = ... will be less efficient because id is a secondary key.
The table will be the same size either way; the secondary key would also be the same size either way -- because every secondary key contains the PK columns, too. (This note is InnoDB-specific.)
I'm trying to figure out storage requirements for different storage engines. I have this table:
CREATE TABLE `mytest` (
`num1` int(10) unsigned NOT NULL,
KEY `key1` (`num1`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
When I insert some values and then run show table status; I get the following:
+----------------+--------+---------+------------+---------+----------------+-------------+------------------+--------------+-----------+----------------+---------------------+---------------------+------------+-------------------+----------+----------------+---------+
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment |
+----------------+--------+---------+------------+---------+----------------+-------------+------------------+--------------+-----------+----------------+---------------------+---------------------+------------+-------------------+----------+----------------+---------+
| mytest | InnoDB | 10 | Compact | 1932473 | 35 | 67715072 | 0 | 48840704 | 4194304 | NULL | 2010-05-26 11:30:40 | NULL | NULL | latin1_swedish_ci | NULL | | |
Notice avg_row_length is 35. I am baffled that InnoDB would not make better use of space when I'm just storing a non-nullable integer.
I have run this same test on myISAM and by default myISAM uses 7 bytes per row on this table. When I run
ALTER TABLE mytest MAX_ROWS=50000000, AVG_ROW_LENGTH = 4;
causes myISAM to finally correctly use 5-byte rows.
When I run the same ALTER TABLE statement for InnoDB the avg_row_length does not change.
Why would such a large avg_row_length be necessary when only storing a 4-byte unsigned int?
InnoDB tables are clustered, that means that all data are contained in a B-Tree with the PRIMARY KEY as a key and all other columns as a payload.
Since you don't define an explicit PRIMARY KEY, InnoDB uses a hidden 6-byte column to sort the records on.
This and overhead of the B-Tree organization (with extra non-leaf-level blocks) requires more space than sizeof(int) * num_rows.
Here is some more info you might find useful.
InnoDB allocates data in terms of 16KB pages, so 'SHOW TABLE STATUS' will give inflated numbers for row size if you only have a few rows and the table is < 16K total. (For example, with 4 rows the average row size comes back as 4096.)
The extra 6 bytes per row for the "invisible" primary key is a crucial point when space is a big consideration. If your table is only one column, that's the ideal column to make the primary key, assuming the values in it are unique:
CREATE TABLE `mytest2`
(`num1` int(10) unsigned NOT NULL primary key)
ENGINE=InnoDB DEFAULT CHARSET=latin1;
By using a PRIMARY KEY like this:
No INDEX or KEY clause is needed, because you don't have a secondary index. The index-organized format of InnoDB tables gives you fast lookup based on the primary key value for free.
You don't wind up with another copy of the NUM1 column data, which is what happens when that column is indexed explicitly.
You don't wind up with another copy of the 6-byte invisible primary key values. The primary key values are duplicated in each secondary index. (That's also the reason why you probably don't want 10 indexes on a table with 10 columns, and you probably don't want a primary key that combines several different columns or is a long string column.)
So overall, sticking with just a primary key means less data associated with the table + indexes. To get a sense of overall data size, I like to run with
set innodb_file_per_table = 1;
and examine the size of the data/database/*table*.ibd files. Each .ibd file contains the data for an InnoDB table and all its associated indexes.
To quickly build up a big table for testing, I usually run a statement like so:
insert into mytest
select * from mytest;
Which doubles the amount of data each time. In the case of the single-column table using a primary key, since the values had to be unique, I used a variation to keep the values from colliding with each other:
insert into mytest2
select num1 + (select count(*) from mytest2) from mytest2;
This way, I was able to get average row size down to 25. The space overhead is based on the underlying assumption that you want to have fast lookup for individual rows using a pointer-style mechanism, and most tables will have a column whose values serve as pointers (i.e. the primary key) in addition to the columns with real data that gets summed, averaged, and displayed.
IN addition to Quassnoi's very fine answer, you should probably try it out using a significant data set.
What I'd do is, load 1M rows of simulated production data in, then measure the table size and use that as a guide.
That's what I've done in the past anyway
MyISAM
MyISAM, except in really old versions, uses a 7-byte "pointer" for locating a row, and a 6-byte pointer inside indexes. These defaults lead to a huge max table size. More details: http://mysql.rjweb.org/doc.php/limits#myisam_specific_limits . The kludgy way to change those involves the ALTER .. MAX_ROWS=50000000, AVG_ROW_LENGTH = 4 that you discovered. The server multiplies those values together to compute how many bytes the data pointer needs to be. Hence, you stumbled on how to shrink the avg_row_length.
But you actually needed to declare a table with fewer than 7 bytes to hit it! The pointer size shows in multiple places:
Free space links in the .MYD default to 7 bytes. So, when you delete a row, a link is provided to the next free spot. That link needs to be 7 bytes (by default), hence the row size was artificially extended from the 4-byte INT to make room for it! (There are more details having to do with whether the column is NULLable , etc.
FIXED vs DYNAMIC row -- When the table is FIXED size, the "pointer" is a row number. For DYNAMIC, it is a byte offset into the .MYD.
Index entries must also point to data rows with a pointer. So your ALTER should have shrunk the .MYI file as well!
There are more details, but MyISAM is likely to go away, so this ancient history is not likely to be of concern to anyone.
InnoDB
https://stackoverflow.com/a/64417275/1766831