My Magento(Vesion 1.8.0.0) website was running without any problem when I was using MySQL5.5 database for years. After moving to mariadb-server-10.2 recently(exported data using sqldump and then imported on new mariadb server), many times PHP child process stays in waiting states until max_execution_time expires and finally php-fpm stops responding.
On investigation I found that all the php child processes are waiting on database queries. I checked the processlist on mariadb when I encountered the problem recently and output is below. Any idea how I could fix this? Thanks in advance.
MariaDB [(none)]> show processlist;
+--------+-------------+-------------------------------+---------+---------+------+------------------------------+---------------------------------------------------------------------- --------------------------------+----------+
| Id | User | Host | db | Command | Time | State | Info | Progress |
+--------+-------------+-------------------------------+---------+---------+------+------------------------------+---------------------------------------------------------------------- --------------------------------+----------+
| 1 | system user | | NULL | Daemon | NULL | | NULL | 0.000 |
| 2 | system user | | NULL | Daemon | NULL | | NULL | 0.000 |
| 4 | system user | | NULL | Daemon | NULL | | NULL | 0.000 |
| 3 | system user | | NULL | Daemon | NULL | | NULL | 0.000 |
| 5 | system user | | NULL | Daemon | NULL | InnoDB shutdown handler | NULL | 0.000 |
| 109822 | catalog | mymagentositexxxxxx.com:56405 | catalog | Query | 985 | update | INSERT INTO `cataloginventory_stock_status` (`product_id`,`website_id `,`stock_id`,`qty`,`stock_statu | 0.000 |
| 109850 | catalog | mymagentositexxxxxx.com:56436 | catalog | Query | 985 | update | INSERT INTO `cataloginventory_stock_status` (`product_id`,`website_id `,`stock_id`,`qty`,`stock_statu | 0.000 |
| 109859 | catalog | mymagentositexxxxxx.com:56448 | catalog | Query | 985 | update | INSERT INTO `cataloginventory_stock_status` (`product_id`,`website_id `,`stock_id`,`qty`,`stock_statu | 0.000 |
| 109931 | catalog | mymagentositexxxxxx.com:56554 | catalog | Query | 986 | Waiting for table level lock | DELETE FROM `catalogsearch_fulltext` WHERE (store_id=1) AND (product_ id IN ('26515')) | 0.000 |
| 109966 | catalog | mymagentositexxxxxx.com:56606 | catalog | Query | 984 | Waiting for table level lock | INSERT INTO `catalogsearch_result` SELECT 33218 AS `query_id`, `s`.`p roduct_id`, 0 AS `relevance` FR | 0.000 |
| 109993 | catalog | mymagentositexxxxxx.com:56645 | catalog | Query | 934 | update | INSERT INTO `sales_flat_quote_item` (`quote_id`, `created_at`, `updat ed_at`, `product_id`, `store_id | 0.000 |
| 109997 | catalog | mymagentositexxxxxx.com:56650 | catalog | Query | 987 | Sending data | INSERT INTO `catalogsearch_result` SELECT 33215 AS `query_id`, `s`.`p roduct_id`, 0 AS `relevance` FR | 0.000 |
| 110040 | catalog | mymagentositexxxxxx.com:56709 | catalog | Query | 983 | Waiting for table level lock | INSERT INTO `catalogsearch_result` SELECT 33215 AS `query_id`, `s`.`p roduct_id`, 0 AS `relevance` FR | 0.000 |
| 110218 | catalog | mymagentositexxxxxx.com:56925 | catalog | Query | 940 | Waiting for table level lock | INSERT INTO `catalogsearch_result` SELECT 6656 AS `query_id`, `s`.`pr oduct_id`, 0 AS `relevance` FRO | 0.000 |
| 110248 | catalog | mymagentositexxxxxx.com:56956 | catalog | Query | 932 | Waiting for table level lock | INSERT INTO `catalogsearch_result` SELECT 33220 AS `query_id`, `s`.`p roduct_id`, 0 AS `relevance` FR | 0.000 |
| 110380 | catalog | mymagentositexxxxxx.com:57095 | catalog | Query | 906 | Waiting for table level lock | INSERT INTO `catalogsearch_result` SELECT 33214 AS `query_id`, `s`.`p roduct_id`, 0 AS `relevance` FR | 0.000 |
| 110392 | catalog | mymagentositexxxxxx.com:57107 | catalog | Query | 905 | Waiting for table level lock | INSERT INTO `catalogsearch_result` SELECT 33215 AS `query_id`, `s`.`p roduct_id`, 0 AS `relevance` FR | 0.000 |
| 110810 | catalog | mymagentositexxxxxx.com:57554 | catalog | Query | 825 | Waiting for table level lock | INSERT INTO `catalogsearch_result` SELECT 6656 AS `query_id`, `s`.`pr oduct_id`, 0 AS `relevance` FRO | 0.000 |
| 110865 | catalog | mymagentositexxxxxx.com:57616 | catalog | Query | 808 | Waiting for table level lock | INSERT INTO `catalogsearch_result` SELECT 21581 AS `query_id`, `s`.`p roduct_id`, 0 AS `relevance` FR | 0.000 |
| 110968 | catalog | mymagentositexxxxxx.com:57782 | catalog | Query | 48 | update | INSERT INTO `sales_flat_quote_item` (`quote_id`, `created_at`, `updat ed_at`, `product_id`, `store_id | 0.000 |
| 110969 | catalog | mymagentositexxxxxx.com:57783 | catalog | Query | 48 | update | INSERT INTO `sales_flat_quote_item` (`quote_id`, `created_at`, `updat ed_at`, `product_id`, `store_id | 0.000 |
| 110989 | catalog | mymagentositexxxxxx.com:57803 | catalog | Query | 39 | update | INSERT INTO `sales_flat_quote_item` (`quote_id`, `created_at`, `updat ed_at`, `product_id`, `store_id | 0.000 |
| 110999 | catalog | mymagentositexxxxxx.com:57813 | catalog | Query | 34 | update | INSERT INTO `sales_flat_quote_item` (`quote_id`, `created_at`, `updat ed_at`, `product_id`, `store_id | 0.000 |
| 111012 | catalog | mymagentositexxxxxx.com:57867 | catalog | Query | 32 | update | INSERT INTO `sales_flat_quote_item` (`quote_id`, `created_at`, `updat ed_at`, `product_id`, `store_id | 0.000 |
| 111067 | catalog | mymagentositexxxxxx.com:57959 | catalog | Query | 18 | update | INSERT INTO `sales_flat_quote_item` (`quote_id`, `created_at`, `updat ed_at`, `product_id`, `store_id | 0.000 |
| 111077 | catalog | mymagentositexxxxxx.com:57974 | catalog | Query | 15 | update | INSERT INTO `sales_flat_quote_item` (`quote_id`, `created_at`, `updat ed_at`, `product_id`, `store_id | 0.000 |
| 111078 | root | localhost | NULL | Query | 0 | init | show processlist | 0.000 |
| 111117 | catalog | mymagentositexxxxxx.com:58015 | catalog | Query | 10 | Waiting for table level lock | INSERT INTO `catalogsearch_result` SELECT 13226 AS `query_id`, `s`.`p roduct_id`, 0 AS `relevance` FR | 0.000 |
| 111147 | moodle | mymagentositexxxxxx.com:58046 | moodle | Sleep | 0 | | NULL | 0.000 |
+--------+-------------+-------------------------------+---------+---------+------+------------------------------+---------------------------------------------------------------------- --------------------------------+----------+
29 rows in set (0.00 sec)
#update2
'Show create table output' of table's in waiting state is as below
MariaDB [catalog]> SHOW CREATE TABLE catalogsearch_fulltext;
+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| catalogsearch_fulltext | CREATE TABLE `catalogsearch_fulltext` (
`product_id` int(10) unsigned NOT NULL COMMENT 'Product ID',
`store_id` smallint(5) unsigned NOT NULL COMMENT 'Store ID',
`data_index` longtext DEFAULT NULL COMMENT 'Data index',
`fulltext_id` int(10) unsigned NOT NULL AUTO_INCREMENT COMMENT 'Entity ID',
PRIMARY KEY (`fulltext_id`),
UNIQUE KEY `UNQ_CATALOGSEARCH_FULLTEXT_PRODUCT_ID_STORE_ID` (`product_id`,`store_id`),
FULLTEXT KEY `FTI_CATALOGSEARCH_FULLTEXT_DATA_INDEX` (`data_index`)
) ENGINE=MyISAM AUTO_INCREMENT=912741 DEFAULT CHARSET=utf8 COMMENT='Catalog search result table' |
+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
MariaDB [catalog]> SHOW CREATE TABLE catalogsearch_result;
+----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| catalogsearch_result | CREATE TABLE `catalogsearch_result` (
`query_id` int(10) unsigned NOT NULL COMMENT 'Query ID',
`product_id` int(10) unsigned NOT NULL COMMENT 'Product ID',
`relevance` decimal(20,4) NOT NULL DEFAULT 0.0000 COMMENT 'Relevance',
PRIMARY KEY (`query_id`,`product_id`),
KEY `IDX_CATALOGSEARCH_RESULT_QUERY_ID` (`query_id`),
KEY `IDX_CATALOGSEARCH_RESULT_PRODUCT_ID` (`product_id`),
CONSTRAINT `FK_CATALOGSEARCH_RESULT_QUERY_ID_CATALOGSEARCH_QUERY_QUERY_ID` FOREIGN KEY (`query_id`) REFERENCES `catalogsearch_query` (`query_id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `FK_CATSRCH_RESULT_PRD_ID_CAT_PRD_ENTT_ENTT_ID` FOREIGN KEY (`product_id`) REFERENCES `catalog_product_entity` (`entity_id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Catalog search result table' |
+----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
The process with the most time (#109997) is touching that MyISAM table? After doing SHOW FULL PROCESSLIST, I see
INSERT
INTO `catalogsearch_result`
SELECT 11474 AS `query_id`, `s`.`product_id`, 0 AS `relevance`
FROM `catalogsearch_fulltext` AS `s`
INNER JOIN `catalog_product_entity` AS `e` ON e.entity_id = s.product_id
WHERE (s.store_id = 1)
AND ((`s`.`data_index` LIKE '%Informaatika%'
OR `s`.`data_index` LIKE '%8.c%')
) ON DUPLICATE KEY
UPDATE `relevance` = VALUES(`relevance`)
That is a slow SELECT that only recently got started. Since we are talking about MyISAM, and there writes going on, the queries will happen one at a time (mostly).
It is slow because of both the OR and the leading wildcard in the LIKE test. Do you have any way to avoid both of those issues? With them there, the SELECT must scan the entire table, which apparently takes minutes. (How big is the table?)
Changing timeouts may get you past one instance, but then simply keep you in a never-ending sequence of queries waiting for other queries to finish.
The best solution is to get away from MyISAM. More tips on that: http://mysql.rjweb.org/doc.php/myisam2innodb
(Even after converting, there could be other issues.)
Related
So I made a basic struct of a database, and mysql filled some rows on a table and I don't know why.
Maybe I can explain myself with the tables
mysql> show fields
-> from products;
+--------------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------------------+--------------+------+-----+---------+-------+
| ID_PRODUCT | decimal(8,0) | NO | PRI | NULL | |
| ID_CATEGORY | decimal(8,0) | NO | MUL | NULL | |
| NAME | text | NO | | NULL | |
| BRAND | text | NO | | NULL | |
| PICTURE | longblob | YES | | NULL | |
| OBSERVATIONS | text | YES | | NULL | |
| QUANT_UNIT | text | NO | | NULL | |
+--------------------+--------------+------+-----+---------+-------+
8 rows in set (0.00 sec)
So this is a table with products, and I have one other table with the prices so I can save the history of prices
mysql> show fields from prices;
+------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------+--------------+------+-----+---------+-------+
| ID_PRODUCT | decimal(8,0) | NO | MUL | NULL | |
| PRICE | text | NO | | NULL | |
| TIME | timestamp | NO | | NULL | |
+------------+--------------+------+-----+---------+-------+
3 rows in set (0.00 sec)
And if I insert the values into the product and prices tables
mysql> insert into products (id_product, id_category, name, brand, quant_unit)
-> values (1, 0, 'rice', 'XPTO', 'kg');
Query OK, 1 row affected (0.01 sec)
mysql> select * from products;
+------------+--------------+--------+----------+----------------+-------------+--------------------+
| ID_PRODUCT | ID_CATEGORY | NAME | BRAND | PICTURE | OBSERVATION | QUANT_UNIT |
+------------+--------------+--------+----------+----------------+-------------+--------------------+
| 0 | 0 | mug | no_brand | 0x | favorite | un |
| 1 | 0 | rice | xpto | 0x | NULL | kg |
+------------+--------------+--------+----------+----------------+-------------+--------------------+
2 rows in set (0.00 sec)
mysql> insert into prices (id_product, price, time)
-> values (0, '5', CURRENT_TIMESTAMP);
Query OK, 1 row affected (0.01 sec)
mysql> select * from prices;
+------------+-------+---------------------+
| ID_PRODUCT | PRICE | TIME |
+------------+-------+---------------------+
| 0 | 7.30 | 2020-11-27 23:25:34 |
| 1 | 5 | 2020-11-27 23:29:12 |
+------------+-------+---------------------+
2 rows in set (0.00 sec)
all fine so far, but if I do a select with both of the tables
mysql> select pro.id_product, pro.id_category, pro.name, pro.brand, pro.quant_unit, pre.price
-> from products pro, prices pri;
+------------+--------------+--------+----------+-------------+-------+
| id_product | id_category | name | brand | quant_unit | price |
+------------+--------------+--------+----------+-------------+-------+
| 0 | 0 | mug | no_brand | un | 7.30 |
| 1 | 0 | rice | xpto | kg | 7.30 |
| 0 | 0 | mug | no_brand | un | 5 |
| 1 | 0 | rice | xpto | kg | 5 |
+------------+--------------+--------+----------+-------------+-------+
4 rows in set (0.00 sec)
those are my constraints to connect the tables
alter table PRICES add constraint FK_PRICES_COST_PRODUCTS foreign key (ID_PRODUCT)
references PRODUCTS (ID_PRODUCT) on delete restrict on update restrict;
alter table PRODUCTS add constraint FK_PRODUCTS_PERTENCE_CATEGORY foreign key (ID_CATEGORY)
references CATEGORY (ID_CATEGORY) on delete restrict on update restrict;
I can't even search the problem because it doesn't make sense to me... But I'm new at SQL, so I think I'm doing something wrong...
I will appreciate any feedback, thanks!
I advise against using comma-join but in your query you didn't specify what to match between those two tables. You can simply fix that by adding WHERE in your query like this:
select pro.id_product, pro.id_category, pro.name, pro.brand, pro.quant_unit, pre.price
from products pro, prices pri
where pro.id_product=pri.id_product;
But nowadays, most people uses JOIN instead of comma-join, hence:
select pro.id_product, pro.id_category, pro.name, pro.brand, pro.quant_unit, pre.price
from products pro JOIN prices pri
ON pro.id_product=pri.id_product;
There's a lot of JOIN types in MySQL and you can refer to this documentation.
From https://stackoverflow.com/a/51181742/3284469
If the table has no PRIMARY KEY or suitable UNIQUE index, InnoDB
internally generates a hidden clustered index named GEN_CLUST_INDEX on
a synthetic column containing row ID values. The rows are ordered by
the ID that InnoDB assigns to the rows in such a table. The row ID is
a 6-byte field that increases monotonically as new rows are inserted.
Thus, the rows ordered by the row ID are physically in insertion
order.
My mysql version is:
$ mysql --version
mysql Ver 8.0.11 for Linux on x86_64 (MySQL Community Server - GPL)
I followed the commands there to verify the internal index is created, but the last command doesn't show any index has been created. Why is that? Thanks.
Note that I changed the last command a little bit, because the original command gives me Unknown table 'INNODB_INDEX_STATS' in information_schema error.
# Create the table
create table test.check_table (id int, description varchar(10)) ENGINE = INNODB;
# Verify that there is no primary or unique column
desc test.check_table;
+-------------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------+-------------+------+-----+---------+-------+
| id | int(11) | YES | | NULL | |
| description | varchar(10) | YES | | NULL | |
+-------------+-------------+------+-----+---------+-------+
# Insert some values
insert into test.check_table values(1, 'value-1');
insert into test.check_table values(2, 'value-2');
insert into test.check_table values(null, 'value-3');
insert into test.check_table values(4, null);
insert into test.check_table values(1, 'value-1');
# Verify table
select * from test.check_table;
+------+-------------+
| id | description |
+------+-------------+
| 1 | value-1 |
| 2 | value-2 |
| NULL | value-3 |
| 4 | NULL |
| 1 | value-1 |
+------+-------------+
# Verify that the GEN_CLUST_INDEX index is auto-created.
select * from INFORMATION_SCHEMA.INNODB_INDEX_STATS where TABLE_SCHEMA='test' and TABLE_NAME = 'check_table';
ERROR 1109 (42S02): Unknown table 'INNODB_INDEX_STATS' in information_schema
SELECT DISTINCT TABLE_NAME, COLUMN_NAME , INDEX_NAME FROM INFORMATION_SCHEMA.STATISTICS WHERE TABLE_SCHEMA = 'test' AND TABLE_NAME='check_table';
Empty set (0.00 sec)
In all versions of MySQL that I've been able to find, the INNODB_INDEX_STATS table is located in the mysql database, not INFORMATION_SCHEMA. This appears to be an error in the post you're referencing.
mysql> select * from mysql.innodb_index_stats where table_name = 'check_table';
+---------------+-------------+-----------------+---------------------+--------------+------------+-------------+-----------------------------------+
| database_name | table_name | index_name | last_update | stat_name | stat_value | sample_size | stat_description |
+---------------+-------------+-----------------+---------------------+--------------+------------+-------------+-----------------------------------+
| test | check_table | GEN_CLUST_INDEX | 2018-07-10 11:34:01 | n_diff_pfx01 | 5 | 1 | DB_ROW_ID |
| test | check_table | GEN_CLUST_INDEX | 2018-07-10 11:34:01 | n_leaf_pages | 1 | NULL | Number of leaf pages in the index |
| test | check_table | GEN_CLUST_INDEX | 2018-07-10 11:34:01 | size | 1 | NULL | Number of pages in the index |
+---------------+-------------+-----------------+---------------------+--------------+------------+-------------+-----------------------------------+
This index isn't a "real" index from the perspective of SQL (it doesn't appear in the output of DESCRIBE, and can't be modified or dropped), so it isn't shown in INFORMATION_SCHEMA.STATISTICS.
For version 8.0.11, the table innodb_index_stats is located in mysql schema in lieu of INFORMATION_SCHEMA. Following the commands, the last query gives result as below:
mysql> select VERSION();
+-----------+
| VERSION() |
+-----------+
| 8.0.11 |
+-----------+
mysql> select * from mysql.innodb_index_stats where database_name='test' and table_name = 'check_table';
+---------------+-------------+-----------------+---------------------+--------------+------------+-------------+-----------------------------------+
| database_name | table_name | index_name | last_update | stat_name | stat_value | sample_size | stat_description |
+---------------+-------------+-----------------+---------------------+--------------+------------+-------------+-----------------------------------+
| test | check_table | GEN_CLUST_INDEX | 2018-07-10 18:57:45 | n_diff_pfx01 | 5 | 1 | DB_ROW_ID |
| test | check_table | GEN_CLUST_INDEX | 2018-07-10 18:57:45 | n_leaf_pages | 1 | NULL | Number of leaf pages in the index |
| test | check_table | GEN_CLUST_INDEX | 2018-07-10 18:57:45 | size | 1 | NULL | Number of pages in the index |
+---------------+-------------+-----------------+---------------------+--------------+------------+-------------+-----------------------------------+
Also, the post referenced in the question creates a second table with a primary key specified. The index verification for that query gives:
mysql> create table test.check_table_2 (id int, description varchar(10), PRIMARY KEY(id)) ENGINE = INNODB;
mysql> desc check_table_2;
+-------------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------+-------------+------+-----+---------+-------+
| id | int(11) | NO | PRI | NULL | |
| description | varchar(10) | YES | | NULL | |
+-------------+-------------+------+-----+---------+-------+
mysql> select * from mysql.innodb_index_stats where database_name='test' and table_name = 'check_table_2';
+---------------+---------------+------------+---------------------+--------------+------------+-------------+-----------------------------------+
| database_name | table_name | index_name | last_update | stat_name | stat_value | sample_size | stat_description |
+---------------+---------------+------------+---------------------+--------------+------------+-------------+-----------------------------------+
| test | check_table_2 | PRIMARY | 2018-07-10 19:00:39 | n_diff_pfx01 | 0 | 1 | id |
| test | check_table_2 | PRIMARY | 2018-07-10 19:00:39 | n_leaf_pages | 1 | NULL | Number of leaf pages in the index |
| test | check_table_2 | PRIMARY | 2018-07-10 19:00:39 | size | 1 | NULL | Number of pages in the index |
+---------------+---------------+------------+---------------------+--------------+------------+-------------+-----------------------------------+
I'm looking to improve the speed of queries on a very large MySQL analytics table that I have. This table is tracking playercount on gameservers and the structure looks as so:
`server_tracker` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`ip` int(10) unsigned NOT NULL,
`port` smallint(5) unsigned NOT NULL,
`date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`players` tinyint(3) unsigned NOT NULL,
`map` varchar(28) NOT NULL,
`portjoin` smallint(5) NOT NULL,
PRIMARY KEY (`id`),
KEY `idx_tracking_ip_port` (`ip`,`port`)
) ENGINE=InnoDB AUTO_INCREMENT=310729056 DEFAULT CHARSET=utf8 ROW_FORMAT=FIXED |
This table is inserted into very frequently, with 10k+ servers being tracked 10+ times an hour. However, every hour the data is taken and averaged out, and put into an "averaged" table with basically the same structure.
Currently I have the IP/port setup as key. However - sometimes it can be a tad slow when doing that hourly averaging - so I am curious if it would be worth putting an index on the timestamp, which is frequently used to select data from a certain timeframe like so:
SELECT `players`
FROM `server_tracker`
WHERE `ip` = x
AND `port` = x
AND `date` > NOW()
AND `date` < NOW() + INTERVAL 60 MINUTE
ORDER BY `id` DESC
This is the only type of query ran on this table. The table is only used for fetching the playercount from gameservers within a specific timeframe. The data is never updated or changed.
However, I am a bit new to all of this - and I am not sure if putting an index on the timestamp would do much of anything. Just looking for some friendly advice.
Results of EXPLAIN SELECT players FROM server_tracker WHERE ip = x AND port = x AND date > NOW() AND date < NOW() + INTERVAL 60 MINUTE ORDER BY id DESC
+----+-------------+-----------------+------+----------------------+----------------------+---------+-------------+-------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-----------------+------+----------------------+----------------------+---------+-------------+-------+-------------+
| 1 | SIMPLE | server_tracker | ref | idx_tracking_ip_port | idx_tracking_ip_port | 6 | const,const | 15354 | Using where |
+----+-------------+-----------------+------+----------------------+----------------------+---------+-------------+-------+-------------+
One of the most important information in MySQL and scripts is to know the MySQL to very few exceptions, always just ONE INDEX can be used in a query.
So it does not use much depending on an index ever to set a Column when all 4 are used verfelder in the where clause.
Only a combined index hilt over these fields.
The order of the fields is very important for this index can also be used for other queries.
An example:
An index on field1, field2 and field3 is used when you have the WHERE FIELD1 or FIELD1 and FIELD2 or field1, field2 and FIELD3. This index is not used if you in the WHERE FIELD2 or used FIELD3 or FIELD2 and field. 3 So always use the first field.
Too easy to find out if un like the QUERY works you can just run your query and EXPALIN and beommst directly the information whether and which index is used. If there are several lines you can as an indicator, the individual values under rows muliplizieren together. The smaller this number is the better performs your query.
MariaDB [tmp]> EXPLAIN select * from content;
+------+-------------+---------+------+---------------+------+---------+------+------+-------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+---------+------+---------------+------+---------+------+------+-------+
| 1 | SIMPLE | content | ALL | NULL | NULL | NULL | NULL | 13 | |
+------+-------------+---------+------+---------------+------+---------+------+------+-------+
1 row in set (0.00 sec)
MariaDB [tmp]>
Anternativ you can check out the profiler how long the QUERY in what capacity depends and about optimizing server
An example:
MariaDB [(none)]> use tmp
Database changed
MariaDB [tmp]> SET PROFILING=ON;
Query OK, 0 rows affected (0.00 sec)
MariaDB [tmp]>
MariaDB [tmp]> SELECT * FROM content;
+----+------+---------------------+--------+------+--------------+------+------+------+------+
| id | Wert | Zeitstempel | WertID | aaa | d | e | wwww | n | ddd |
+----+------+---------------------+--------+------+--------------+------+------+------+------+
| 1 | 10 | 2001-01-01 00:00:00 | 1 | NULL | 1.5000 | NULL | NULL | 1 | NULL |
| 2 | 12.3 | 2001-01-01 00:01:00 | 2 | NULL | 2.5000 | NULL | NULL | 2 | NULL |
| 3 | 17.4 | 2001-01-01 00:02:00 | 3 | NULL | 123456.1250 | NULL | NULL | 3 | NULL |
| 4 | 10.9 | 2001-01-01 01:01:00 | 1 | NULL | 1000000.0000 | NULL | NULL | 4 | NULL |
| 5 | 15.4 | 2001-01-01 01:02:00 | 2 | NULL | NULL | NULL | NULL | 5 | NULL |
| 6 | 20.9 | 2001-01-01 01:03:00 | 3 | NULL | NULL | NULL | NULL | 6 | NULL |
| 7 | 22 | 2001-01-02 00:00:00 | 1 | NULL | NULL | NULL | NULL | 7 | NULL |
| 8 | 12.3 | 2001-01-02 00:01:00 | 2 | NULL | NULL | NULL | NULL | 8 | NULL |
| 9 | 17.4 | 2001-01-02 00:02:00 | 3 | NULL | NULL | NULL | NULL |
+----+------+---------------------+--------+------+--------------+------+------+------+------+
13 rows in set (0.00 sec)
MariaDB [tmp]>
MariaDB [tmp]> SHOW PROFILE;
+----------------------+----------+
| Status | Duration |
+----------------------+----------+
| starting | 0.000031 |
| checking permissions | 0.000005 |
| Opening tables | 0.000036 |
| After opening tables | 0.000004 |
| System lock | 0.000003 |
| Table lock | 0.000002 |
| After opening tables | 0.000005 |
| init | 0.000013 |
| optimizing | 0.000006 |
| statistics | 0.000013 |
| preparing | 0.000010 |
| executing | 0.000002 |
| Sending data | 0.000073 |
| end | 0.000003 |
| query end | 0.000003 |
| closing tables | 0.000006 |
| freeing items | 0.000003 |
| updating status | 0.000012 |
| cleaning up | 0.000003 |
+----------------------+----------+
19 rows in set (0.00 sec)
MariaDB [tmp]>
I am trying to add a unique key to a column and am getting an self explanatory error of
MariaDB [dnd]> ALTER TABLE ability
-> ADD UNIQUE INDEX UK_ability_shortname (shortname);
ERROR 1062 (23000): Duplicate entry '' for key 'UK_ability_shortname'
But the thing is that I cannot see any duplicates
MariaDB [INFORMATION_SCHEMA]> select * from INNODB_SYS_INDEXES WHERE NAME LIKE 'UK_%';
+----------+---------------------------------+----------+------+----------+---------+-------+
| INDEX_ID | NAME | TABLE_ID | TYPE | N_FIELDS | PAGE_NO | SPACE |
+----------+---------------------------------+----------+------+----------+---------+-------+
| 1733 | UK_food_food | 563 | 2 | 1 | 4 | 558 |
| 1737 | UK_type_type | 565 | 2 | 1 | 4 | 560 |
| 2077 | UK_invite_status_status | 763 | 2 | 1 | 4 | 752 |
| 2586 | UK_tool_class_name | 1093 | 2 | 1 | 4 | 1082 |
| 2591 | UK_food_drink_lodging | 1097 | 2 | 2 | 4 | 1086 |
| 2595 | UK_lifestyle_expenses_lifestyle | 1100 | 2 | 1 | 4 | 1089 |
| 2691 | UK_weapon_name | 1173 | 2 | 1 | 4 | 1162 |
| 2692 | UK_weapon_property_name | 1174 | 2 | 1 | 4 | 1163 |
| 2696 | UK_ability_name | 1183 | 2 | 1 | 4 | 1172 |
+----------+---------------------------------+----------+------+----------+---------+-------+
9 rows in set (0.00 sec)
Any ideas what else I can check to find out why the DB thinks there is a conflict in name or am I exceeding some index name limit?
The table definition that I am trying to alter.
CREATE TABLE `ability` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(50) NOT NULL,
`shortname` char(3) NOT NULL,
`description` text,
PRIMARY KEY (`id`),
UNIQUE KEY `UK_ability_name` (`name`)
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8 COMMENT='Page 175'
Using: Server version: 10.0.15-MariaDB MariaDB Server
Let me know if I can provide any other information.
The error message means there are at least two rows that have the same value for the column shortname ('': the empty string). MariaDB consequently cannot create a unique index on that column. Search for any (other) duplicate values in the column with this query:
SELECT
shortname,
GROUP_CONCAT(id)
FROM
ability
GROUP BY
shortname
HAVING
COUNT(0) > 1
I'm stuck with the following problem and hope, that someone help me to solve it...
Mysql version 5.6.19-0ubuntu0.14.04.1-log.
CREATE TABLE `dr` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`start_date` bigint(20) unsigned NOT NULL,
`end_date` bigint(20) unsigned NOT NULL,
UNIQUE KEY `id` (`id`,`start_date`),
KEY `start_date` (`start_date`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 ROW_FORMAT=COMPRESSED;
CREATE TABLE `map_dr2pdu` (
`drtable_id` int(10) unsigned NOT NULL,
`dr_id` bigint(20) unsigned NOT NULL,
`packet_location` bigint(20) unsigned NOT NULL,
`sctp_chunk` smallint(5) unsigned DEFAULT NULL,
KEY `dr_id` (`dr_id`),
KEY `drtable_id` (`drtable_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 ROW_FORMAT=COMPRESSED;
CREATE TABLE `dr_hot` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`start_date` bigint(20) unsigned NOT NULL,
`end_date` bigint(20) unsigned NOT NULL,
UNIQUE KEY `id` (`id`,`start_date`)
) ENGINE=MEMORY;
CREATE TABLE `map_dr2pdu_hot` (
`drtable_id` int(10) unsigned NOT NULL,
`dr_id` bigint(20) unsigned NOT NULL,
`packet_location` bigint(20) unsigned NOT NULL,
`sctp_chunk` smallint(5) unsigned DEFAULT NULL,
KEY `dr_id` (`dr_id`),
KEY `drtable_id` (`drtable_id`)
) ENGINE=MEMORY;
dr and map_dr2pdu have partitions (if it matters).
dr_hot.id <===> map_dr2pdu_hot.dr_id
There is a client C1 (auto commit off)
writes into dr_hot
obtains LAST_INSERTED_ID and inserts 1-n records into map_dr2pdu_hot.
C1 commit politics: 500 inserts or 1 sec (which came first)
Memory tables are introduced solely to support high insertion rate. Actual data must be store in dr and map_dr2pdu (INNODB) tables. To make this "rotation"
there is second client C2 (auto commit off) which
locks _hot table,
stores its autoincrement id
dumps its data into file in memory,
truncates _hot table (faster, than delete)
updates its autoincrement id
commit
unlocks the table
[loads data from the file to INNODB table]
Problem appears when C2 begins its work:
mysql> show processlist;
+--------+-----------------+-----------------+-----------+---------+-------+----------------+---------------+----+
| Id | User | Host | db | Command | Time | State | Info |
+--------+-----------------+-----------------+-----------+---------+-------+---------------------------------+----+
| 1 | event_scheduler | localhost | NULL | Daemon | 1 | Waiting for next activation | NULL |
| 57 | riva | localhost | riva | Sleep | 283 | | NULL |
| 115748 | root | localhost | riva | Query | 0 | init | show processlist |
| 117538 | riva | localhost | riva | Sleep | 10889 | | NULL |
| 120150 | riva | localhost | riva | Sleep | 30 | | NULL |
| 120155 | riva | localhost | riva | Sleep | 7 | | NULL |
| 120158 | riva | localhost | riva | Sleep | 5 | | NULL |
| 120161 | riva | localhost | riva | Sleep | 5 | | NULL |
| 120164 | riva | localhost | riva | Sleep | 5 | | NULL |
| 120167 | riva | localhost | riva | Sleep | 7 | | NULL |
| 120170 | riva | localhost | riva | Query | 22 | Waiting for table metadata lock | ALTER TABLE dr_hot AUTO_INCREMENT=1152885790 |
| 120178 | riva | localhost | riva | Sleep | 5 | | NULL |
| 120179 | riva | localhost | riva | Query | 23 | Waiting for table metadata lock | LOCK TABLES map_dr2pdu_hot LOW_PRIORITY WRITE |
| 120243 | riva | localhost | riva | Sleep | 24 | | NULL |
| 120244 | riva | localhost:40934 | riva | Sleep | 24 | | NULL |
| 120245 | riva | localhost:40935 | riva | Sleep | 24 | | NULL |
| 120246 | riva | localhost:40936 | riva | Sleep | 22 | | NULL |
| 120247 | riva | localhost:40937 | riva | Sleep | 24 | | NULL |
| 120248 | riva | localhost:40938 | riva | Sleep | 24 | | NULL |
| 120249 | riva | localhost:40939 | riva | Sleep | 24 | | NULL |
| 120250 | riva | localhost:40940 | riva | Sleep | 24 | | NULL |
| 120251 | riva | localhost:40941 | riva | Sleep | 22 | | NULL |
| 120252 | riva | localhost:40942 | riva | Sleep | 22 | | NULL |
| 120253 | riva | localhost:40943 | riva | Execute | 22 | Waiting for table metadata lock | INSERT INTO map_dr2pdu_hot (xdrtable_id, xdr_id, packet_location) VALUES (25, 6078, |
+--------+-----------------+-----------------+-----------+---------+-------+---------------------------------+------+
24 rows in set (0.00 sec)
mysql general_log + processlist:
119177 Execute INSERT INTO map_dr2pdu_hot (drtable_id, dr_id, packet_location) VALUES (26, 131287232, 606
4773029186237937)
119177 Query COMMIT
119453 Query LOCK TABLES map_dr2pdu_hot WRITE
processlist:
| 119453 | rivasense | localhost | rivasense | Query | 164 | Waiting for table metadata lock | LOCK TABLES map_dr2pdu_hot WRITE
119177 Query START TRANSACTION
119177 Execute INSERT INTO dr_hot ()
Show engine innodb status, does not show any deadlock, just like
mysql> select * from INNODB_LOCKS;
Empty set (0.04 sec)
mysql> select * from INNODB_LOCK_WAITS;
Empty set (0.00 sec)
As I understoo : C1 acquires lock for insert and seems not commits, but it DOES according to log. therefore, C2 can't lock table.
Please help to find the root cause.
update on 06/10/14:
Basically problem reproducible when C1 tries to write something into map_dr2pdu_hot and C2 simultaneously tries to lock the same table. This treated as "almost" deadlock (not listed among deadlocks) and released with lock_wait_timeout expired.
Could you suggest how to resolve such a problem?
Finally we had to apply WA - modify algorithm of C2 to:
stores _hot tbl autoincrement id
dumps its data into file in memory,
deletes from _hot table (where id less than auto inc id from step 1)
loads data from the file to INNODB table
of course map_dr2pdu table altered to have id auto-increment column.