SELECT nar.name, nar.reg, stat.lvl
FROM members AS nar
JOIN stats AS stat
ON stat.id = nar.id
WHERE nar.ref = 9
I have indexes on id in both tables and I have index referavo either. But still, it checks all rows in stats table (I use Explain to get this information), but in members table it checks only one row how it supposed to be. What's wrong with stats table? Thank you very much.
CREATE TABLE `members` (
`id` int(11) NOT NULL
`ref` int(11) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT
CREATE TABLE `stats` (
`id` int(11) NOT NULL AUTO_INCREMENT
PRIMARY KEY (`id`),
) ENGINE=InnoDB AUTO_INCREMENT=37 DEFAULT CHARSET=utf8 ROW_FORMAT=DYNAMIC
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE stat ALL PRIMARY NULL NULL NULL 22
1 SIMPLE nar eq_ref PRIMARY PRIMARY 4 table_nme.stat.id 1 Using where
Your tables are ridiculously small - just 23 rows is tiny.
MySQL chooses different query plans depending on how many rows there are in the table and based on how many it estimates will be selected (from the statistics). You should performance test your queries with realistic data - both the amount of data and the distribution of values in the data should be as realistic as possible. Otherwise the query plan MySQL chooses in testing might not be the same the actual query plan for your live system.
Your tables are so small that using an index could be slower than just checking the table directly. Remember that checking data that is already in memory is fast, but reads are slow. Accessing an index can require an extra read - first the index has to be fetched and read to find which rows to select, then if your index isn't a covering index the relevant rows in the table have to be fetched and read to get the values that aren't in the index. MySQL is perfectly entitled to not use an index even if one is available if it believes that doing so will result in a slower plan.
Put some more rows in your table (thousands) and try running EXPLAIN again. You will probably find that when you have more rows that the PRIMARY KEY index will be used for the join.
MySQL can use only one index at a time per table, thus it sees the member row using the index, and then performs a sequential search for the ID.
You have to create a multi columns index for the members table
CREATE INDEX idref ON members(id,ref);
please try the reverse one as well if it doesn't get better (first: drop index idref on members)
CREATE INDEX idref ON members(ref,id);
(I cannot try it myself now)
Related
Recently,I have reviwed the basic of SQL and found A question about index.
My Working environment is as follows:
os: Centos 7
mysql: 5.7.39
database: sakila
table: customer
AND My question is why Innodb uses idx_fk_store_id instead of the primary index when I use select count(customer_id) from customer
mysql> explain select count(customer_id) from customer;
AND Result:
type
key
Extra
index
idx_fk_store_id
Using index
The code to create this table is as follow:
CREATE TABLE `customer`(
`customer_id` smallint(5) unsigned NOT NULL AUTO_INCREMENT,
`store_id` tinyint(3) unsigned NOT NULL,
PRIMARY KEY(`customer_id`),
KEY `idx_fk_store_id`(`store_id`)
) ENGINE=InnoDE AUTO_INCREMENT=600 DEFAULT CHARSET=utf8mb4
I've considered that it caused by MYSQL's Optimizer , even if it's hard to understand.
The type: index in the EXPLAIN report indicates it is doing an index-scan. That is, reading every entry in the index idk_fk_store_id.
It can get the values of the primary key from a secondary index, so it can count them.
The alternative of using the primary key would be a table-scan, which reads every row of the table.
The primary key index in InnoDB is the clustered index. It stores all the columns of the table.
The secondary index stores only the values of store_id plus the primary key values of rows where a given store_id value occurs.
So it will be able to get the same answer by doing an index-scan, by reading fewer pages than doing the table-scan.
Saying COUNT(x) is common mistake.
COUNT(*) counts the number of rows
COUNT(x) counts the number of rows where x is NOT NULL
So, if you try SELECT COUNT(*) FROM customer, the Explain will again say that it is using that index. But for a different reason than Bill gave. This time the Optimizer's logic goes this way:
find the "smallest" index. That will happen to be the same store_id index.
count the "rows" in that index. But no need to test whether customer_id IS NOT NULL.
Another note: In MySQL, the PRIMARY KEY is required to be NOT NULL, so customer_id cannot be NULL. Also you declared it to be NOT NULL. Hence COUNT(customer_id) is necessarily the same as COUNT(*).
TMI.
Recently,I have reviwed the basic of SQL and found A question about index.
My Working environment is as follows:
os: Centos 7
mysql: 5.7.39
database: sakila
table: customer
AND My question is why Innodb uses idx_fk_store_id instead of the primary index when I use select count(customer_id) from customer
mysql> explain select count(customer_id) from customer;
AND Result:
type
key
Extra
index
idx_fk_store_id
Using index
The code to create this table is as follow:
CREATE TABLE `customer`(
`customer_id` smallint(5) unsigned NOT NULL AUTO_INCREMENT,
`store_id` tinyint(3) unsigned NOT NULL,
PRIMARY KEY(`customer_id`),
KEY `idx_fk_store_id`(`store_id`)
) ENGINE=InnoDE AUTO_INCREMENT=600 DEFAULT CHARSET=utf8mb4
I've considered that it caused by MYSQL's Optimizer , even if it's hard to understand.
The type: index in the EXPLAIN report indicates it is doing an index-scan. That is, reading every entry in the index idk_fk_store_id.
It can get the values of the primary key from a secondary index, so it can count them.
The alternative of using the primary key would be a table-scan, which reads every row of the table.
The primary key index in InnoDB is the clustered index. It stores all the columns of the table.
The secondary index stores only the values of store_id plus the primary key values of rows where a given store_id value occurs.
So it will be able to get the same answer by doing an index-scan, by reading fewer pages than doing the table-scan.
Saying COUNT(x) is common mistake.
COUNT(*) counts the number of rows
COUNT(x) counts the number of rows where x is NOT NULL
So, if you try SELECT COUNT(*) FROM customer, the Explain will again say that it is using that index. But for a different reason than Bill gave. This time the Optimizer's logic goes this way:
find the "smallest" index. That will happen to be the same store_id index.
count the "rows" in that index. But no need to test whether customer_id IS NOT NULL.
Another note: In MySQL, the PRIMARY KEY is required to be NOT NULL, so customer_id cannot be NULL. Also you declared it to be NOT NULL. Hence COUNT(customer_id) is necessarily the same as COUNT(*).
TMI.
I have a MySQL 8 database table accounts that has the following columns:
id (primary)
city_id (foreign key)
province_id (foreign key)
country_id (foreign key)
school_id (foreign key)
age (indexed)
EDIT: See bottom for complete table structure.
Now, imagine the following SQL query:
SELECT
COUNT(`id`) AS AGGREGATE
FROM
`accounts`
WHERE
`city_id` = 1
AND
`country_id` = 7
AND
`age` = 3
At 1 million records, this query becomes slow (~200ms).
When running EXPLAIN, I receive the following output:
id
select_type
table
partitions
type
possible_keys
key
key_len
ref
rows
filtered
Extra
1
SIMPLE
accounts
NULL
index_merge
accounts_city_id_foreign accounts_country_id_foreign accounts_age_index
accounts_city_id_foreign accounts_country_id_foreign accounts_age_index
9,2,9
NULL
15542
100.00
Using intersect(accounts_city_id_foreign, accounts_country_id_foreign, accounts_age_index); Using where; Using index
Given that MySQL appears to be using the indexes, I'm not sure what I can do to bring the execution time down. Does anyone have any ideas?
EDIT: In the future, the table will include more columns that will make it impossible to use a composite index as it will exceed the 16 column limit.
EDIT: Here's the complete table structure:
CREATE TABLE `accounts` (
`id` bigint unsigned NOT NULL AUTO_INCREMENT,
`city_id` bigint unsigned DEFAULT NULL,
`school_id` bigint unsigned DEFAULT NULL,
`country_id` bigint unsigned DEFAULT NULL,
`province_id` bigint unsigned DEFAULT NULL,
`age` tinyint unsigned DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `accounts_city_id_foreign` (`city_id`),
KEY `accounts_school_id_foreign` (`school_id`),
KEY `accounts_country_id_foreign` (`country_id`),
KEY `accounts_province_id_foreign` (`province_id`),
KEY `accounts_age_index` (`age`),
CONSTRAINT `accounts_city_id_foreign` FOREIGN KEY (`city_id`) REFERENCES `cities` (`id`) ON DELETE SET NULL,
CONSTRAINT `accounts_country_id_foreign` FOREIGN KEY (`country_id`) REFERENCES `countries` (`id`) ON DELETE SET NULL,
CONSTRAINT `accounts_province_id_foreign` FOREIGN KEY (`province_id`) REFERENCES `provinces` (`id`) ON DELETE SET NULL,
CONSTRAINT `accounts_school_id_foreign` FOREIGN KEY (`school_id`) REFERENCES `schools` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB AUTO_INCREMENT=1000002 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
Try creating a composite index on all three columns, e.g. CREATE INDEX idx_city_country_age ON table (city_id, country_id, age)
Indexes are to help your querying. So as suggested by Marko and agreed by others, having an index on (city_id, country_id, age) should significantly help. Now, yes, you will add other columns to the table, but are you trying to filter on 16+ criteria??? I doubt it. And of the queries you would be running, even if you have multiple composite indexes to help optimize those queries, how many columns might you need at any single time? 4, 5, 6? After that, I mean how granular do you plan on getting with your data. Country, State/Province, City, Town, Village, Neighborhood, Street, House? and by the time you are that low in the data, you would be at the page level data anyhow, wouldn't you?
So, your query of Country = 7, that already chops off a ton of stuff. Then to a given city within that country? Great, now you are at a finite level.
if you are going do be doing queries against large data that requires any aggregations, and the data is rather fixed from a historical perspective, maybe having pre-aggregated tables by some common elements might help long term.
FEEDBACK
The performance of querying is not necessarily where you will be hit, it would be in the inserts, updates, deletes as whatever may change has to update all the indexes on the table - single or composite. If you are getting more than 5 columns in an index, ask yourself, really??? How granular is it that you need for the index to be optimized. Querying out the data should be very fast with proper indexes. Updating indexes is also quick, but if you are dealing with millions of inserts in a month, quarter, year? The user doing theirs may have a slight delay ( 1/4 second?) but adding up a million seconds starts to get delay. But again, over what period of time would insert/update/delete be done anyhow.
You asked what will bring the query time down, and using a composite index will do that. Searching a single composite index is faster than searching several single-column indexes and performing an intersection merge on the results.
You commented that you will be adding more columns in the future, and there will eventually be more than 16 columns.
You don't have to add ALL the columns to the composite index!
Index design is not magic. It follows rules. You will create indexes designed to support specific queries that you need to run. You don't add add columns to an index unless they help the given query. You may have multiple composite indexes in the table, created to help different queries.
You might like my presentation How to Design Indexes, Really (or the video).
Re your comment:
I won't know every possible query combination ahead of time.
Yes, that's true. You can only create indexes for queries that you know. Other queries will not be optimized. If you need to optimize queries in the future, you might need to add new indexes to support them.
In my experience, this happens regularly, and I address this in the presentation. You will review your queries from time to time, because of course your application code changes and the queries you need change. You may add new indexes, or replace an index with a different index, or drop indexes that are no longer needed.
I have a table definition:
CREATE TABLE `k_timestamps` (
`id` bigint(20) NOT NULL,
`k_timestamp` datetime NULL DEFAULT NULL,
`data1` smallint(6) NOT NULL,
KEY `k_timestamp_key` (`k_timestamp`,`id`) USING BTREE,
CONSTRAINT `k_time_fk` FOREIGN KEY (`id`) REFERENCES `data` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
Basically, I have a whole lot of id and data1 key-value pairs, and every few hours I either add new key-value pairs not seen before to the list, or the value of a previous id has changed. I want to track what all the values were for every id in time. Thus, the id column can contain duplicate id's and is not the primary key.
Side note, k_time_fk points to another, much smaller table that has common information for a particular id regardless of what the current time is or value it currently holds.
(id, k_timestamp) should be thought of as the (composite) primary key of the table.
For example,
id k_timestamp data1
1597071247 2012-11-15 12:25:47 4
1597355222 2012-11-15 12:25:47 4
1597201376 2012-11-15 12:25:47 4
1597071243 2012-11-15 13:25:47 4
1597071247 2012-11-15 13:25:47 3
1597071249 2012-11-15 13:25:47 3
Anyways, I ran this query:
SELECT concat(table_schema,'.',table_name),
concat(round(table_rows/1000000,2),'M') rows,
concat(round(data_length/(1024*1024*1024),2),'G') DATA,
concat(round(index_length/(1024*1024*1024),2),'G') idx,
concat(round((data_length+index_length)/(1024*1024*1024),2),'G') total_size,
round(index_length/data_length,2) idxfrac
FROM information_schema.TABLES ORDER BY data_length+index_length DESC LIMIT 20;
To pull space info on my table:
rows Data idx total_size idxfrac
11.25M 0.50G 0.87G 1.36G 1.76
I'm not really sure I understand this, how can the index be taking up so much space? Is there something obvious I did wrong here, or is this normal? I'm looking to try to reduce to footprint of this table if possible. I'm not even really sure what that k_timestamp_key really buys for me, can it be safely deleted?
The index is bigger because InnoDB tables will assign a 6 byte primary key when you have no unique column that it can treat as a unique index. All other indexes in the table also contain the primary key... see 14.2.3.12.2. Clustered and Secondary Indexes from the manual
Firstly, yes, this is pretty normal behaviour, as innvo writes.
Secondly, you can optimize the table and its index using OPTIMIZE TABLE. As your primary key is likely to be "fragmented" - i.e. it's not safe to assume that an inserted row is physically next to the previous row - there may be some gains there.
Finally, you may not need a primary key on the table, but you almost certainly need an index if you're querying across millions of rows...
I am sorry if this is a dumb question (cause it sounds unlikely).
I have a table that is 20 Million rows. However, only about 300K of these rows get accessed regularly, and they can be identified in a column condition called "app_user=1"
Is there anyway i can just index those rows, and when I call a select, i will be sure to pass in the condition as well?
I would recommend splitting the table into two separate tables. But in case you don't want to do that, the highest performance way to do this if you're always going to include "where app_user=1" in your queries is to create a primary key on the table that includes the app_user column as the first part of the key. InnoDB will use this as a clustered index which saves you a few extra disk accesses. You can create the table like this:
create table testTable (
app_user tinyint UNSIGNED default 0,
id int UNSIGNED NOT NULL,
name varchar(255) default '',
PRIMARY KEY k1(app_user, id)
) ENGINE=InnoDB;
A friend wrote this article on clustered indexes in InnoDB a while back:
http://www.joehruska.com/?p=6
Add a column called app_user and index on that, then pass in "WHERE app_user = 1" in your query.
You could go further to partition your table based on that column.