delete by primary key takes forever mysql - mysql

I am trying to delete by primary key from Table (300 rows) and it takes up to max query execution time and at the end returns ERROR 2013: 2013: Lost connection to MySQL server during query. This table has foreign key to the large table (200k rows). What can be an issue?
Query: DELETE FROM Table Where table_id=x
EDIT:
There are no triggers associated with this DELETE statement. DELETE/INSERT/UPDATE statements in some database tables work really slow while SELECT statements in whole database work perfectly fine.
EDIT#2:
Additional information from innodb trx table for the query:
trx_lock_structs 429704
trx_lock_memmory_bytes 34698792
trx_rows_locked 214938
trx_isolation_level REPEATABLE READ
trx_unique_checks 1
trx_foreign_key_checks 1
This query deletes 1 row and doesn't have child rows, why locked rows value is so high?
EDIT#3
Investigating situation further I have determined that tables that have slow insert/update/delete operations are the tables that have foreign key with the big table (200k). Is it necessary to remove this foreign keys or data integrity is more important? Although 200k rows is not that much what can be reasons for this slow operations?
EDIT#4
SHOW CREATE TABLE:
CREATE TABLE `Table` (
`table_id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`tableb_id` bigint(20) unsigned NOT NULL,
`tablec_id` bigint(20) unsigned DEFAULT NULL,
`bigtable_id` bigint(20) unsigned NOT NULL,
PRIMARY KEY (`table_id`),
KEY `fk_tableb_id_idx` (`tableb_id`),
KEY `fk_bigtable_id_idx` (`bigtable_id`),
KEY `fk_tablec_id_idx` (`tablec_id`),
CONSTRAINT `fk_bigtable_id` FOREIGN KEY (`bigtable_id`) REFERENCES `Bigtable`
(`bigtable_id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `fk_tableb_id` FOREIGN KEY (`tableb_id`) REFERENCES `tablebs`
(`tableb_id`) ON DELETE CASCADE ON UPDATE NO ACTION,
CONSTRAINT `fk_tablec_id` FOREIGN KEY (`tablec_id`) REFERENCES `tablecs`
(`tablec_id`) ON DELETE CASCADE ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=271 DEFAULT CHARSET=utf8
BigTable is a typical Users table id and additional information.
EDIT#5
EXPLAIN DELETE:
select_type : SIMPLE,
table : Table,
type : range,
possible_keys : PRIMARY,
key : PRIAMRY,
key_len : 8,
ref : const,
rows : 1,
Extra : Using where

The reason is cascade on big table. Do you understand the complexity of this computation? it is not just a delete operation on 300 rows. It is basically 300 rows * 200k. Each delete would go and pass through the big table and would perform delete operation on the corresponding rows (based on the id).
Just follow the below mentioned steps:
1) Remove the references of this primary id of this table from other tables (if any)
2) Alter this table and add NO ACTION to the UPDATE & CASCADE of big table foreign key
OR
remove all the foreign key contraints
3) delete from tableName

I may suggest an empirical approach...
Delete ops with cascade impacts on indexes, slow execution time may depend on time needed to elaborate fKey indexes, btw your tables aren't huge. You say you have no triggers, functions or procedure, so you have to benchmark your query to find what's part is taking so much time...
I assume that you have already verified indexes efficency.
So to benchmark your delete query you should handy write "inner deletion" queries and benchmark each one, using BENCHMARK().
This way you will find which part of that deletion took so long.
What if each single part is reasonably fast?
You may have some misconfiguration in your my.cnf, so you could try checking with https://tools.percona.com/wizard recommendations...
Maybe you have some memory allocation limit or some thread/memory limit.
You can find many tutorials on mysql optimization like http://www.codingpedia.org/ama/optimizing-mysql-server-settings/
You can also find some scripts that may help you in finding mysql optimizations.
Without knowing your architecture, mysql configuration and your database structure, it's quite hard to give a 100% solution, but I hope you can find the way, I'll be glad to read about your findings. If something more will come to my mind I'll keep you posted.

Related

Optimising a query that uses index merge by intersection

I have a MySQL 8 database table accounts that has the following columns:
id (primary)
city_id (foreign key)
province_id (foreign key)
country_id (foreign key)
school_id (foreign key)
age (indexed)
EDIT: See bottom for complete table structure.
Now, imagine the following SQL query:
SELECT
COUNT(`id`) AS AGGREGATE
FROM
`accounts`
WHERE
`city_id` = 1
AND
`country_id` = 7
AND
`age` = 3
At 1 million records, this query becomes slow (~200ms).
When running EXPLAIN, I receive the following output:
id
select_type
table
partitions
type
possible_keys
key
key_len
ref
rows
filtered
Extra
1
SIMPLE
accounts
NULL
index_merge
accounts_city_id_foreign accounts_country_id_foreign accounts_age_index
accounts_city_id_foreign accounts_country_id_foreign accounts_age_index
9,2,9
NULL
15542
100.00
Using intersect(accounts_city_id_foreign, accounts_country_id_foreign, accounts_age_index); Using where; Using index
Given that MySQL appears to be using the indexes, I'm not sure what I can do to bring the execution time down. Does anyone have any ideas?
EDIT: In the future, the table will include more columns that will make it impossible to use a composite index as it will exceed the 16 column limit.
EDIT: Here's the complete table structure:
CREATE TABLE `accounts` (
`id` bigint unsigned NOT NULL AUTO_INCREMENT,
`city_id` bigint unsigned DEFAULT NULL,
`school_id` bigint unsigned DEFAULT NULL,
`country_id` bigint unsigned DEFAULT NULL,
`province_id` bigint unsigned DEFAULT NULL,
`age` tinyint unsigned DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `accounts_city_id_foreign` (`city_id`),
KEY `accounts_school_id_foreign` (`school_id`),
KEY `accounts_country_id_foreign` (`country_id`),
KEY `accounts_province_id_foreign` (`province_id`),
KEY `accounts_age_index` (`age`),
CONSTRAINT `accounts_city_id_foreign` FOREIGN KEY (`city_id`) REFERENCES `cities` (`id`) ON DELETE SET NULL,
CONSTRAINT `accounts_country_id_foreign` FOREIGN KEY (`country_id`) REFERENCES `countries` (`id`) ON DELETE SET NULL,
CONSTRAINT `accounts_province_id_foreign` FOREIGN KEY (`province_id`) REFERENCES `provinces` (`id`) ON DELETE SET NULL,
CONSTRAINT `accounts_school_id_foreign` FOREIGN KEY (`school_id`) REFERENCES `schools` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB AUTO_INCREMENT=1000002 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
Try creating a composite index on all three columns, e.g. CREATE INDEX idx_city_country_age ON table (city_id, country_id, age)
Indexes are to help your querying. So as suggested by Marko and agreed by others, having an index on (city_id, country_id, age) should significantly help. Now, yes, you will add other columns to the table, but are you trying to filter on 16+ criteria??? I doubt it. And of the queries you would be running, even if you have multiple composite indexes to help optimize those queries, how many columns might you need at any single time? 4, 5, 6? After that, I mean how granular do you plan on getting with your data. Country, State/Province, City, Town, Village, Neighborhood, Street, House? and by the time you are that low in the data, you would be at the page level data anyhow, wouldn't you?
So, your query of Country = 7, that already chops off a ton of stuff. Then to a given city within that country? Great, now you are at a finite level.
if you are going do be doing queries against large data that requires any aggregations, and the data is rather fixed from a historical perspective, maybe having pre-aggregated tables by some common elements might help long term.
FEEDBACK
The performance of querying is not necessarily where you will be hit, it would be in the inserts, updates, deletes as whatever may change has to update all the indexes on the table - single or composite. If you are getting more than 5 columns in an index, ask yourself, really??? How granular is it that you need for the index to be optimized. Querying out the data should be very fast with proper indexes. Updating indexes is also quick, but if you are dealing with millions of inserts in a month, quarter, year? The user doing theirs may have a slight delay ( 1/4 second?) but adding up a million seconds starts to get delay. But again, over what period of time would insert/update/delete be done anyhow.
You asked what will bring the query time down, and using a composite index will do that. Searching a single composite index is faster than searching several single-column indexes and performing an intersection merge on the results.
You commented that you will be adding more columns in the future, and there will eventually be more than 16 columns.
You don't have to add ALL the columns to the composite index!
Index design is not magic. It follows rules. You will create indexes designed to support specific queries that you need to run. You don't add add columns to an index unless they help the given query. You may have multiple composite indexes in the table, created to help different queries.
You might like my presentation How to Design Indexes, Really (or the video).
Re your comment:
I won't know every possible query combination ahead of time.
Yes, that's true. You can only create indexes for queries that you know. Other queries will not be optimized. If you need to optimize queries in the future, you might need to add new indexes to support them.
In my experience, this happens regularly, and I address this in the presentation. You will review your queries from time to time, because of course your application code changes and the queries you need change. You may add new indexes, or replace an index with a different index, or drop indexes that are no longer needed.

How to speed up a highly active big data table (MySQL)?

I'll begin to try and explain my problem and what I meant with the title.
Currently I have got a table with around ~8 million rows.
This table is highly active, what this means is there's constant updates, inserts and deletes.
These are caused by users (it's like a collecting game). Meaning I also need to make sure the data is accurately displayed.
I've looked so far into:
indexing
partitioning
sharding
mapreduce
optimize
I applied indexing, however I'm not sure if I applied this method correctly and it doesn't seem to help much more than I thought.
As I said, my table is highly active, meaning that if I'd add partitioning to this table, it would mean there are going to be additional inserts/deletes and make this process way more complex than I can understand. I do not have that much experience with databases.
Sharding this database is way too complex for me and I only have one service I can run this database on, so this option is a no-go.
As for mapreduce, I am not entirely sure what this does, but as far as I understood, it mainly has to do more so with the code, than with the database.
I applied optimize, but it didn't really seem to have too much effect neither as I experienced.
I have tried to not use the * in SELECT statements, I made sure to get rid of most DISTINCT, COUNT and other functionalities of SQL alike, so that these wouldn't affect the speed of the database.
However even after narrowing down the data in each table and specifically this table, it's currently slower than it was before this.
This table consists of:
CREATE TABLE `claim` (
`global_id` bigint NOT NULL AUTO_INCREMENT,
`fk_user_id` bigint NOT NULL,
`fk_series_id` smallint NOT NULL,
`fk_character_id` smallint NOT NULL,
`fk_image_id` int NOT NULL,
`fk_gif_id` smallint DEFAULT NULL,
`rarity` smallint NOT NULL,
`emoji` varchar(31) DEFAULT NULL,
PRIMARY KEY (`global_id`),
UNIQUE KEY `global_id_UNIQUE` (`global_id`),
KEY `fk_claim_character_id` (`fk_character_id`),
KEY `fk_claim_image_id` (`fk_image_id`),
KEY `fk_claim_series_id` (`fk_series_id`),
KEY `fk_claim_user_id` (`fk_user_id`) /*!80000 INVISIBLE */,
KEY `fk_claim_gif_id` (`fk_gif_id`) /*!80000 INVISIBLE */,
KEY `fk_claim_rarity` (`rarity`) /*!80000 INVISIBLE */,
KEY `fk_claim_emoji` (`emoji`),
CONSTRAINT `fk_claim_character_id` FOREIGN KEY (`fk_character_id`) REFERENCES `character` (`character_id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `fk_claim_image_id` FOREIGN KEY (`fk_image_id`) REFERENCES `image` (`image_id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `fk_claim_series_id` FOREIGN KEY (`fk_series_id`) REFERENCES `series` (`series_id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `fk_claim_user_id` FOREIGN KEY (`fk_user_id`) REFERENCES `user` (`user_id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=7622452 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
Is there possibly another solution to speed up the database? If so, how? I'm currently at wits end and stuck on it. The database needs to respond preferably within 300ms.
EXAMPLE SLOW QUERIES:
SELECT PK FROM <table> WHERE fk_user_id = ?;
SELECT PK FROM <table> WHERE fk_user_id = ? GROUP BY fk_character_id HAVING MAX(fk_character_id) = 1;
SELECT PK, fk_user_id, fk_character_id, etc, etc, etc FROM <table> WHERE fk_user_id = ? ORDER BY PK ASC LIMIT 0, 20
Redundant
PRIMARY KEY (`global_id`),
UNIQUE KEY `global_id_UNIQUE` (`global_id`),
A PRIMARY KEY, in MySQL, is a UNIQUE KEY. So the UNIQUE KEY is redundant, wastes disk space, and slows down INSERT.
Need VISIBLE index starting with user_id for Q1 and Q2
Replace this
KEY `fk_claim_user_id` (`fk_user_id`) /*!80000 INVISIBLE */,
with
INDEX(fk_user_id, fk_character_id)
in that order -- this will help with your first 2 queries.
Query 3
The 3rd query may still need (in the given order)
INDEX(fk_user_id, global_id)
If you need some of the DISTINCTs/COUNTs, let's see them. Changing indexes may help.
Strange query
As for
SELECT PK FROM <table> WHERE fk_user_id = ?;
Why would you just want the PK? Is global_id useful by itself? Or is it useful only for looking up something else? If the latter, let's see it; it is often more practical to optimize a single, complex, query than two queries that are artificially split.
Tuning
How much RAM is available to MySQL? What is the value of innodb_buffer_pool_size? 30s for 50K rows -- sounds like being I/O-bound. Maybe that setting is too low.
In some cases, DISTINCT speeds up a query -- if for no other reason that less data is shoveled back to the client.
Redesign PK
Based on the names "claim" and "user_id" and the test for "user_id" in all 3 queries, I deduce that you are frequently looking up stuff for a single "user"? What, if anything, is global_id needed for outside this table?
If you need need global_id elsewhere or nothing else could be used for uniqueness, do
PRIMARY KEY(user_id, global_id), -- for locality of reference
INDEX(global_id) -- to keep AUTO_INCREMENT happy
If (user_id, xx) is known to be unique (for some column(s) xx), toss global_id and change to
PRIMARY KEY(user_id, xx)
In either case, these go away:
PRIMARY KEY (`global_id`),
UNIQUE KEY `global_id_UNIQUE` (`global_id`),
KEY `fk_claim_user_id` (`fk_user_id`) /*!80000 INVISIBLE */,
InnoDB stores the data in PK order. By having the PK start with user_id, all the rows for one user are "adjacent" on the disk, thereby more readily cached in RAM (in the buffer_pool).
Given a user with 100 claims, I am restructuring the table so that the data is found in a couple of consecutive blocks (16KB unit of storage by InnoDB) instead of upwards of 100 scattered blocks.

How to use index in my table correctly?

I realized, that when I am creating foreign keys in table, indexes are adding automatically.
In my table:
CREATE TABLE `SupplierOrderGoods` (
`shopOrder_id` INT(11) NOT NULL,
`supplierGood_id` INT(11) NOT NULL,
`count` INT(11) NOT NULL,
PRIMARY KEY (`shopOrder_id`, `supplierGood_id`),
CONSTRAINT `FK_SupplierOrderGoods_ShopOrders` FOREIGN KEY (`shopOrder_id`) REFERENCES `shoporders` (`id`),
CONSTRAINT `FK_SupplierOrderGoods_SupplierGoods` FOREIGN KEY (`supplierGood_id`) REFERENCES `suppliergoods` (`id`)
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB;
Index
INDEX `FK_SupplierOrderGoods_SupplierGoods` (`supplierGood_id`)
have been created automatically.
It is okay, that index have been created as I found in another post. I was looking what indexes are used for and found, that they are used for optimizing search in tables.
Now, I know, that I have to use indexes to optimize work with database.
Also, I found, that indexes can be complex (not on one field, but on some fields). In that case, I want to ask should I use complex index:
INDEX `FK_ShopOrders_SupplierGoods` (`shopOrder_id`, `supplierGood_id`),
or two simple indexes?:
INDEX `FK_SupplierOrderGoods_SupplierGoods` (`supplierGood_id`),
INDEX `FK_SupplierOrderGoods_ShopOrders` (`shopOrder_id`),
I'm still earning about indexes myself but I believe it's going to depend on what kind of data you will be querying the DB for.
For example, if you have a report for a certain record that will be ran a lot you'll want an index on it. If the report pulls just one column then make a one column index, if it's comprised of two, like a first name and a last name record, you'll probably want one for both.
You do not want to put an index on everything though as that can have performance issues as both the record and the index need to be updated. As such, tables that have a high amount of inserts or updating done on them you'll want to think about whether an index hurts or helps.
Lot of information to cover with indexes.

After converting MySQL key columns to a FOREIGN Columns the site slowed down

I never used FOREIGN KEYS before. I always placed the id it a key column but it is not necessary a foreign key. so it acts like a foreign key but it is not.
so if I have the following 2 tables
CREATE TABLE `accounts` (
`account_id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(60) NOT NULL,
`owner_id` int(11) unsigned NOT NULL,
PRIMARY KEY (`account_id`),
KEY `owner_id` (`owner_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
CREATE TABLE `users` (
`user_id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(60) NOT NULL
PRIMARY KEY (`user_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
the column account.owner_id is linked to users.user_id but without setting a foreign key relationship.
so I can do something like this
SELECT
a.name AS account_name,
u.name AS user_name FROM
accounts AS a INNER JOIN users AS u ON u.user_id = a.owner_id
So after learning what FOREIGN KEYS are and what they do I have created a foreign key like so.
ALTER TABLE accounts
ADD CONSTRAINT fk_owner_id FOREIGN KEY(owner_id)
REFERENCES users(user_id) ON DELETE NO ACTION ON UPDATE CASCADE;
However, I notices a performance change as they system slowed down a lot. I am not sure if adding FOREIGN KEYs will reuse performance or no? or is there a performance reduction when adding a foreign keys to a table?
Please note that that there are lots of tables and columns in my database. so I do many INNER/LEFT/RIGHT JOINS and all the columns where i identified them as FOREIGN KEYS are indexed. Notices that I did not add any indexes to this columns as all of they already exists prior adding the FOREIGN keys to my database.
my question, will FOREIGN KEYs reduce performance on UPDATE/INSERT/DELETE/SELECT?
Also is there a benefit of adding a FOREIGN KEY constraint when we specified ON DELETE NO ACTION ON UPDATE NO ACTION?
Thanks
The foreign key constraint means that any insert to the FK column has to check if the value exists in the referenced column of users. There could be some overhead to this, but it's an index lookup by definition (probably a PK lookup) so the cost shouldn't be high.
Foreign keys also create a shared lock on the parent table during some updates on the child table. This can get in the way of concurrent updates against that table, and make it seem like the system has slower performance. See http://www.mysqlperformanceblog.com/2006/12/12/innodb-locking-and-foreign-keys/
The foreign key also implicitly created an index on the FK column, if no index already exists. Every insert, update, delete has to modify all the indexes of a table at the time of the change, so there is a bit of overhead. For this reason, some people say indexes "hurt" performance of insert, update, delete. But it's not that simple -- an index that supports conditions in a WHERE clause can make an update or delete run faster by finding the affected rows more efficiently.
Re your comment:
Yes, any update to the child tables creates an exclusive lock on the modified row in the child table. You probably expected this. But this action also creates a shared lock on the referenced row in clients. Any number of sessions may independently create shared locks on the same row (hence the name shared). But an exclusive lock requires that there be no lock of any type. So if there are frequently shared locks outstanding on a row in clients, then direct updates to that row in clients can't get the required exclusive lock.
The purpose of these shared locks is basically so that the row in clients doesn't get removed or modified while someone is updating a row that depends on it. In other words, "don't delete my parent." But depending on the frequency of updates and the duration of transactions, it could make it hard to perform updates to a parent row.
One way to mitigate this is to try to make locks live for shorter periods of time, by finishing the work for a transactions promptly, and then commit.

Partitioning mySQL tables that has foreign keys?

What would be an appropriate way to do this, since mySQL obviously doesnt enjoy this.
To leave either partitioning or the foreign keys out from the database design would not seem like a good idea to me. I'll guess that there is a workaround for this?
Update 03/24:
http://opendba.blogspot.com/2008/10/mysql-partitioned-tables-with-trigger.html
How to handle foreign key while partitioning
Thanks!
It depends on the extent to which the size of rows in the partitioned table is the reason for partitions being necessary.
If the row size is small and the reason for partitioning is the sheer number of rows, then I'm not sure what you should do.
If the row size is quite big, then have you considered the following:
Let P be the partitioned table and F be the table referenced in the would-be foreign key. Create a new table X:
CREATE TABLE `X` (
`P_id` INT UNSIGNED NOT NULL,
-- I'm assuming an INT is adequate, but perhaps
-- you will actually require a BIGINT
`F_id` INT UNSIGNED NOT NULL,
PRIMARY KEY (`P_id`, `F_id`),
CONSTRAINT `Constr_X_P_fk`
FOREIGN KEY `P_fk` (`P_id`) REFERENCES `P`.`id`
ON DELETE CASCADE ON UPDATE RESTRICT,
CONSTRAINT `Constr_X_F_fk`
FOREIGN KEY `F_fk` (`F_id`) REFERENCES `F`.`id`
ON DELETE RESTRICT ON UPDATE RESTRICT
) ENGINE=INNODB CHARACTER SET ascii COLLATE ascii_general_ci
and crucially, create a stored procedure for adding rows to table P. Your stored procedure should make certain (use transactions) that whenever a row is added to table P, a corresponding row is added to table X. You must not allow rows to be added to P in the "normal" way! You can only guarantee that referential integrity will be maintained if you keep to using your stored procedure for adding rows. You can freely delete from P in the normal way, though.
The idea here is that your table X has sufficiently small rows that you should hopefully not need to partition it, even though it has many many rows. The index on the table will nevertheless take up quite a large chunk of memory, I guess.
Should you need to query P on the foreign key, you will of course query X instead, as that is where the foreign key actually is.
I would strongly suggest sharding using Date as the key for archiving data to archive tables. If you need to report off multiple archive tables, you can use Views, or build the logic into your application.
However, with a properly structured DB, you should be able to handle tens of millions of rows in a table before partitioning, or sharding is really needed.