I was reading through various forums and stack over flow questions but couldn't find my answer.
I'm trying to find out the number of keys which will be stored in 16KB InnoDb Database page.
As you can see in this forum they mentioned how to calculate the number of keys for MyISAM in a single page . I would like to do the same for InnoDb. I don't understand how these calculations were made.
Im comparing a int which is 4KB and a VARCHAR (200). It would be great if I can get this calculation.
An index is structured as a BTree.
InnoDB BTrees are initially filled only 15/16th full.
After a lot of churn, a BTree averages 69% full.
There is a significant amount of overhead for each 'row' in a index entry.
The PRIMARY KEY (in InnoDB) is "clustered" with the data. So only the non-leaf nodes take extra blocks.
Secondary indexes contain all the column(s) of the PRIMARY KEY; that is how they 'point' to the record.
Based on the above two items, it is not meaningful to have just an INT in an the BTree for an index.
I use the simple Rule of Thumb: 100 'rows' per BTree.
Corollary: A million-row BTree is about 3 levels deep; a billion-row table is about 5 level deep.
Let's take this:
CREATE TABLE x (
id INT ...,
num INT ...,
str VARCHAR(200) CHARACTER SET utf8,
PRIMARY KEY (id),
INDEX i_num (num),
INDEX i_str (str)
) ENGINE=InnoDB;
For i_num, note that there are two INTs. You might get 300-400 'rows' per block. 10 million rows would take 3 levels.
For i_str, let's assume an average of 100 Korean characters -- that's 300 bytes for the text. You might get 25-35 'rows' per block. 10M rows would take 5 levels.
ALTER and OPTIMIZE may or may not defragment indexes.
There are information_schema tables that give some details about each BTree and its levels. Percona and MySQL have different tables for such.
Bottom line: The computations are too fuzzy to be exact. I hope my hand-waving has given you some better handles.
Related
I've recently noticed a significant increase in variance of time needed to complete simple INSERT statements. While these statements on average take around 11ms, they can sometimes take 10-30 seconds, and I even noticed them taking over 5 minutes to execute.
MySQL version is 8.0.24, running on Windows Server 2016. The server's resources are never overloaded as far as I can tell. There is an ample amount of cpu overhead for the server to use, and 32GB of ram is allocated to it.
This is the table I'm working with:
CREATE TABLE `saved_segment` (
`recording_id` bigint unsigned NOT NULL,
`index` bigint unsigned NOT NULL,
`start_filetime` bigint unsigned NOT NULL,
`end_filetime` bigint unsigned NOT NULL,
`offset_and_size` bigint unsigned NOT NULL DEFAULT '18446744073709551615',
`storage_id` tinyint unsigned NOT NULL,
PRIMARY KEY (`recording_id`,`index`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
The table has no other indices or foreign keys, nor is it used as a reference for a foreign key in any other table. The entire table size is approximately 20GB with around 281M rows, which doesn't strike me as too large.
The table is used almost entirely read-only, with up to 1000 reads per second. All of these reads happen in simple SELECT queries, not in complex transactions, and they utilize the primary key index efficiently. There are very few, if any, concurrent writes to this table. This has been done intentionally in order to try to figure out if it would help with slow inserts, but it didn't. Before that there were up to 10 concurrent inserts going on at all times. UPDATE or DELETE statements are never executed on this table.
The queries that I'm having trouble with are all structured like this. They never appear in a transaction. While the inserts are definitely not append-only according to the clustered primary key, the queries almost alwayas insert 1 to 20 adjacent rows into the table:
INSERT IGNORE INTO saved_segment
(recording_id, `index`, start_filetime, end_filetime, offset_and_size, storage_id) VALUES
(19173, 631609, 133121662986640000, 133121663016640000, 20562291758298876, 10),
(19173, 631610, 133121663016640000, 133121663046640000, 20574308942546216, 10),
(19173, 631611, 133121663046640000, 133121663076640000, 20585348350688128, 10),
(19173, 631612, 133121663076640000, 133121663106640000, 20596854568114720, 10),
(19173, 631613, 133121663106640000, 133121663136640000, 20609723363860884, 10),
(19173, 631614, 133121663136640000, 133121663166640000, 20622106425668780, 10),
(19173, 631615, 133121663166640000, 133121663196640000, 20634653501528448, 10),
(19173, 631616, 133121663196640000, 133121663226640000, 20646967172721148, 10),
(19173, 631617, 133121663226640000, 133121663256640000, 20657773176227488, 10),
(19173, 631618, 133121663256640000, 133121663286640000, 20668825200822108, 10)
This is the output for an EXPLAIN statement of the above query:
id
select_type
table
partitions
type
possible_keys
key
key_len
ref
rows
filtered
Extra
1
INSERT
saved_segment
NULL
ALL
NULL
NULL
NULL
NULL
NULL
NULL
NULL
These problems are relatively recent and weren't apparent while the table was around twice as small.
I tried reducing the number of concurrent inserts into the table, from around 10 to 1. I also deleted foreign keys on some columns (recording_id) in order to speed up inserts farther. ANALYZE TABLE and schema profiling didn't yield any actionable information.
One solution I had in mind was to remove the clustered primary key, add an AUTOINCREMENT primary key and a regular index on the (recording_id, index) columns. In my mind this would help by making inserts 'append only'. I'm open to any and all suggestions, thanks in advance!
EDIT:
I'm going to address some points and questions raised in the comments and answers:
autocommit is set to ON
the value of innodb_buffer_pool_size is 21474836480, and the value of innodb_buffer_pool_chunk_size is 134217728
One comment raised a concern about contention between the read-locks used by reads, and the exclusive locks used by writes. The table is question is used somewhat like a cache, and I don't need reads to always reflect the most up to date state of the table, if that would mean increased performance. The table should however remain durable even in cases of server crashes and hardware failures. Is this possible to achieve with a more relaxed transaction isolation level?
The schema could definitely be optimized; recording_id could be 4 byte integer, end_filetime could instead be an elapsed value, and start_filetime could also probably be smaller. I'm afraid that these changes would just push the issue back for a while until the table grows in size to compensate for the space savings.
INSERTs into the table are always sequential
SELECTs executed on the table look like this:
SELECT TRUE
FROM saved_segment
WHERE recording_id = ? AND `index` = ?
SELECT index, start_filetime, end_filetime, offset_and_size, storage_id
FROM saved_segment
WHERE recording_id = ? AND
start_filetime >= ? AND
start_filetime <= ?
ORDER BY `index` ASC
The second type of query could definitely be improved with an index, but I'm afraid this would further degrade INSERT performance.
Another thing that I forgot to mention is the existence of a very similar table to this one. It is queried and inserted into in exactly the same manner, but might further contribute to IO starvation.
EDIT2:
Results of SHOW TABLE STATUS for the table saved_segment, and a very similar table saved_screenshot (this one has an aditional INDEX on an bigint unsigned not null column).
Name
Engine
Version
Row_format
Rows
Avg_row_length
Data_length
Max_data_length
Index_length
Data_free
Auto_increment
Create_time
Update_time
Check_time
Collation
Checksum
Create_options
Comment
saved_screenshot
InnoDB
10
Dynamic
483430208
61
29780606976
0
21380464640
6291456
NULL
"2021-10-21 01:03:21"
"2022-11-07 16:51:45"
NULL
utf8mb4_0900_ai_ci
NULL
saved_segment
InnoDB
10
Dynamic
281861164
73
20802699264
0
0
4194304
NULL
"2022-11-02 09:03:05"
"2022-11-07 16:51:22"
NULL
utf8mb4_0900_ai_ci
NULL
I'll go out on a limb with this Answer.
Assuming that
The value of innodb_buffer_pool_size is somewhat less than 20MB, and
Those 1K Selects/second randomly reach int various parts of the table, then
The system recently become I/O bound because the 'next' block needed for the next Select is becoming more often not cached in the buffer_pool.
The simple solution is to get more RAM and up the setting of that tunable. But the table will only grow to whatever next limit you purchase.
Instead, here are some partial solutions.
If the numbers don't get too big, the first two columns could be INT UNSIGNED (4 bytes instead of 8) or maybe even MEDIUMINT UNSIGNED (3 bytes). Caution the ALTER TABLE would lock the table for a long time.
Those start and end times look like timestamps with fractional seconds that are always ".000". DATETIME and TIMESTAMP take 5 bytes (versus 8).
Your sample shows 0 elapsed time. If (end-start) is usually very small, then storing elapsed instead of endtime would further shrink the data. (But make it messy to use the endtime).
The sample data you presented looks "consecutive". That is about as efficient as an autoincrement. Is that the norm? If not, the INSERTs could be part of the I/O thrashing.
Your suggestion of adding AI, plus a secondary index, sort of doubles the effort for Inserts; so I do not recommend it.
More
just push the issue back for a while until the table grows in size
Yes, that will be the case.
Both of your queries are optimally helped by this as an INDEX or, even better, as the start of the PRIMARY KEY:
(recording_id, index)
Re:
SELECT TRUE
FROM saved_segment
WHERE recording_id = ? AND `index` = ?
If that is used to control some other SQL, consider adding this to that other SQL:
... EXISTS ( SELECT 1
FROM saved_segment
WHERE recording_id = ? AND `index` = ? ) ...
That query (in either form) needs what you already have
PRIMARY KEY(recording_id, index)
Your other query needs
INDEX(recording_id, start_filetime)
So, add that INDEX, or...
Even better... This combination would be better for both SELECTs:
PRIMARY KEY(recording_id, start_filetime, index).
INDEX(recording_id, index)
With that combo,
The single-row existence check would be performed "Using index" because it is "covering".
And the other query would find all the relevant rows clustered together on the PK.
(The PK has those 3 columns because it needs to be Unique. And they are in that order to benefit your second query. And it is the PK, not just an INDEX so it does not need to bounce between the index's BTree and the data's BTree.)
The "clustering" may help your performance by cutting down on the number of disk blocks needed for such queries. This leads to less "thrashing" in the buffer_pool, hence less need to increase RAM.
My index suggestions are mostly orthogonal to my datatype suggestions.
I'm running a couple tests on MySQL Clustered vs Non Clustered indexes where I have a table 100gb_table which contains ~60 million rows:
100gb_table schema:
CREATE TABLE 100gb_table (
id int PRIMARY KEY NOT NULL AUTO_INCREMENT,
c1 int,
c2 text,
c3 text,
c4 blob NOT NULL,
c5 text,
c6 text,
ts timestamp NOT NULL default(CURRENT_TIMESTAMP)
);
and I'm executing a query that only reads the clustered index:
SELECT id FROM 100gb_table ORDER BY id;
I'm seeing that it takes almost an ~55 min for this query to complete which is strangely slow. I modified the table by adding another index on top of the Primary Key column and ran the following query which forces the non-clustered index to be used:
SELECT id FROM 100gb_table USE INDEX (non_clustered_key) ORDER BY id;
This finished in <10 minutes, much faster than reading with the clustered index. Why is there such a large discrepancy between these two? My understanding is that both indexes store the index column's values in a tree structure, except the clustered index contains table data in the leaf nodes so I would expect both queries to be similarly performant. Could the BLOB column possibly be distorting the clustered index structure?
The answer comes in how the data is laid out.
The PRIMARY KEY is "clustered" with the data; that is, the data is order ed by the PK in a B+Tree structure. To read all of the ids, the entire BTree must be read.
Any secondary index is also in a B+Tree structure, but it contains (1) the columns of the index, and (2) any other columns in the PK.
In your example (with lots of [presumably] bulky columns), the data BTree is a lot bigger than the secondary index (on just id). Either test probably required reading all the relevant blocks from the disk.
A side note... This is not as bad as it could be. There is a limit of about 8KB on how big a row can be. TEXT and BLOB columns, when short enough, are included in that 8KB. But when one is bulky, it is put in another place, leaving behind a 'pointer' to the text/blob. Hence, the main part of the data BTree is smaller than it might be if all the text/blob data were included directly.
Since SELECT id FROM tbl is a mostly unnecessary query, the design of InnoDB does not worry about the inefficiency you discovered.
Tack on ORDER BY or WHERE, etc, and there are many different optimizations that could into play. You might even find that INDEX(c1) will let your query run in not much more than 10 minutes. (I think I have given you all the clues for 'why'.)
Also, if you had done SELECT * FROM tbl, it might have taken much longer than 55 minutes. This is because of having extra [random] fetches to get the texts/blobs from the "off-record" storage. And from the network time to shovel far more data.
I have a MyISAM table, that contains just one field, a SMALL INT. That field has an index on it, and there are 5.6 million records.
So in theory 5.6mil * 2 bytes (smallint) = 11MB (approx), but the data file of the table is 40MB, why so different?
The index file takes up 46MB, would would it be bigger than the data file?
Here is the create table:
CREATE TABLE `key_test` (
`key2` smallint(5) unsigned NOT NULL DEFAULT '0',
KEY `key2` (`key2`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1
There is some overhead.
First, some controllable 'variables':
myisam_data_pointer_size defaults to 6 (bytes).
myisam_index_pointer_size defaults to 1 less than that.
For the data (.MYD):
N bytes for up to 8*N NULLable columns. (N=0 for your table.)
1 byte for "deleted". You do have this.
DELETEd rows leave gaps.
When a record is deleted, the gap is filled in with a data_pointer to the next record. This implies that the smallest a row can be is 6 bytes.
So: 1 + MAX(row_length, 6) = 7 bytes per row.
If you had 3 SMALLINTs, the table would be the same size.
For the index (.MYI):
BTree organization has some overhead; if randomly built, it settles in on about 69% full.
A 6-byte pointer (byte offset into .MYD, DYNAMIC) is needed in each leaf row.
Links within the BTree are a 5-byte row that is controlled by an unlisted setting (myisam_index_pointer_size).
So: row_length + 6 per record, plus some overhead. 46M sounds like the data was sorted so that the index was built "in order".
Beyond that, my memory of MyISAM details is fading.
My problem: imagine a table with millions of rows, like
CREATE TABLE a {
id INT PRIMARY KEY,
column2..,
column3..,
many other columns..
..
INDEX (column2);
and a query like this:
SELECT id FROM a WHERE column2 > 10000 LIMIT 1000 OFFSET 5000;
My question: does mysql only use the index "column2" (so the primary key id is implicitly stored as a reference in other indexes), or does it have to fetch all rows to get also the id, which is selected for output? In that case the query should be much faster with a key declared as:
INDEX column2(column2, id)
Short answer: No.
Long answer:
MyISAM, unlike InnoDB, has a "pointer" to the data in the leaf node of each index, including that PRIMARY KEY.
So, INDEX(col2) is essentially INDEX(col2, ptr). Ditto for INDEX(id) being INDEX(id, ptr).
The "pointer" is either a byte offset into the .MYD file (for DYNAMIC) or record number (for FIXED). In either case, the pointer leads to a "seek" into the .MYD file.
The pointer defaults to a 6-byte number, allowing for a huge number of rows. It can be changed by a setting, either for saving space or allowing an even bigger number of rows.
For your particular query, INDEX(col2, id) is optimal and "covering". It is better than INDEX(col2) for MyISAM, but they are equivalent for InnoDB, since InnoDB implicitly has the PK in each secondary index.
The query will have to scan at least 5000+1000 rows, at least in the index's BTree.
Note that InnoDB's PRIMARY KEY is clustered with the data, but MyISAM's PRIMARY KEY is a separate BTree, just like other secondary indexes.
You really should consider moving to InnoDB; there is virtually no reason to use MyISAM today.
An index on column2 is required. Your suggestion with id in the index will prevent table scans and should be very efficient.
Further more it is faster to do this assuming that column2 is a continuous sequence:
SELECT id FROM a WHERE column2 > 15000 LIMIT 1000;
This is because to work with the offset it would just have to scan the next 5000 records (MySQL does not realize that you are actually offsetting column2).
I have table storing phone numbers with 800M rows.
column
region_code_id smallint(4) unsigned YES
local_number mediumint(7) unsigned YES
region_id smallint(4) unsigned YES
operator_id smallint(4) unsigned YES
id int(10) unsigned NO PRI auto_increment
I need find number.id where region_code_id = 119 and localnumber = 1234567
select * from numbers where numbers.region_code_id = 119 and numbers.local_number = 1234567;
this query execute over 600 second.
How can I improve it ?
UPDATE
Thank for unswer, i understand i need index for this column, i try this as soon as I get the server with more SSD, now i have free 1GB SSD space. How i can to find out how much space the index will occupy?
Consider adding INDEX on columns which you use in WHERE clause.
Start with:
ALTER TABLE `numbers`
ADD INDEX `region_code_id_local_number`
(`region_code_id`, `local_number`);
Note : it can take some time for index to build.
Before and after change, execute explain plan to compare:
EXPLAIN EXTENDED select * from numbers where numbers.region_code_id = 119 and numbers.local_number = 1234567;
References:
How MySQL uses indexes
For this query:
select *
from numbers
where numbers.region_code_id = 119 and
numbers.local_number = 1234567;
You want an index on numbers(region_code_id, local_number) or numbers(local_number, region_code_id). The order of the columns doesn't matter because the conditions are equality for both columns.
create index idx_numbers_region_local on numbers(region_code_id, local_number);
I agree that INDEX(region_code_id, local_number) (in either order) is mandatory for this problem, but I am sticking my nose in to carry it a step further. Isn't that pair "unique"? Or do you have duplicate numbers in the table? If it is unique, then get rid of id and make that pair PRIMARY KEY(region_code_id, local_number). The table will possibly be smaller after the change.
Back to your question of "how big". How big is the table now? Perhaps 40GB? A secondary index (as originally proposed) would probably add about 20GB. And you would need 20-60GB of free disk space to perform the ALTER. This depends on whether adding the index can be done "inplace" in that version.
Changing the PK (as I suggest) would result in a little less than 40GB for the table. It will take 40GB of free space to perform the ALTER.
In general (and pessimistically), plan on an ALTER needing the both the original table and the new table sitting on disk at one time. That includes full copies of the data and index(es).
(A side question: Are you sure local_number is limited to 7 digits everywhere?)
Another approach to the question... For calculating the size of a table or index in InnoDB, add up the datatype sizes (3 bytes for MEDIUMINT, some average for VARCHAR, etc). Then multiply by the number of rows. Then multiply by 4; this will give you the approximate disk space needed. (Usually 2-3 is sufficient for the last multiplier.)
When changing the PK, do it in one step:
ALTER TABLE foo
DROP PRIMARY KEY,
ADD PRIMARY KEY(region_code_id, local_number);
Changing the PK cannot be done "inplace".
Edit (mostly for other readers)
#berap points out that id is needed for other purposes. Hence, dropping id and switching the PK is not an option.
However, this is sometimes an option (perhaps not in this case):
ALTER TABLE foo
DROP PRIMARY KEY,
ADD PRIMARY KEY(region_code_id, local_number),
ADD INDEX(id);
Notes:
The id..AUTO_INCREMENT will continue to work even with just INDEX.
The SELECT in question will be more efficient because it is the PK.
SELECT .. WHERE id = ... will be less efficient because id is a secondary key.
The table will be the same size either way; the secondary key would also be the same size either way -- because every secondary key contains the PK columns, too. (This note is InnoDB-specific.)