Query too big and slow on multiple tables, how to optimize that? - mysql

Question

I have a store system with orders, which have status for both the order and it's items. I have these six tables:
Order (pedido).
Order item (pedido_item).
Order status (status_pedido).
Order item status (status).
Order status log (pedido_status_pedido).
Order item status log (pedido_item_status).
So, when I have to update the status from the order, I insert a new row on pedido_status_pedido and I update the status_pedido_id on table pedido. Same goes for items.
One order status have many item status associated, for example, the order status "Pending" is related to the item statuses "Waiting for file", "File with error" and "File approved".
The current order status is based on it's current items status, on the most "delayed" status, for example, and item "File with error" is behind an "Finished production" item. For that there are ordenation columns both on order and item status.
So, if I have 3 items on an order, with the status "File with error", "In production", "Finished production", the order status is "Pending", because it's the correspondent order status for the order item status "File with error", which is further behind.
The problem is when I have to update an specific order status. I came up with a query that got so complex I need to SET SQL_BIG_SELECTS=1 to make it run. Obviously the query is so slow it's making my entire site slow (it's called every 10 minutes, for a large number of orders)
Here is my query, explained:
INSERT INTO pedido_status_pedido (pedido_id, status_pedido_id) VALUES ({$this->pedido_id}, ( --Insert into order status log the id, and the status id
SELECT sp.status_pedido_id FROM `status` s --Subquery for the order status id, get it from the relationship inside the item status
LEFT JOIN status_pedido sp ON sp.status_pedido_id = s.status_pedido_id
WHERE s.status_id = ( --Subquery for the further behind item status
SELECT s.status_id FROM pedido_item_status p1
LEFT JOIN `status` s ON s.status_id = p1.status_id
LEFT JOIN pedido_item ON pedido_item.pedido_item_id = p1.pedido_item_id
INNER JOIN ( --Get the LATEST status of each item and compare
SELECT MAX( si.sta_ordem ) AS maxordem, pedido_item_id FROM pedido_item_status pi
LEFT JOIN status si ON pi.status_id = si.status_id
WHERE pi.excluido IS NULL AND pi.pedido_id = {$this->pedido_id}
GROUP BY pi.pedido_item_id
) p2 ON ( s.sta_ordem = p2.maxordem ) AND p1.excluido IS NULL AND p1.pedido_item_id = p2.pedido_item_id
WHERE p1.pedido_id = {$this->pedido_id}
ORDER BY s.sta_ordem ASC
LIMIT 1
)
)
And here are the tables definitions (sorry, it's a bit large):
CREATE TABLE `pedido` (
`pedido_id` int(11) NOT NULL AUTO_INCREMENT,
`cliente_id` int(11) NULL DEFAULT NULL,
`forma_envio_id` int(11) NULL DEFAULT NULL,
`balcao_retirada_id` int(11) NULL DEFAULT NULL,
`ped_responsavel_retirada` varchar(255) CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL DEFAULT NULL,
`ped_codigo_rastreio` varchar(50) CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL DEFAULT NULL,
`ped_prazo_entrega` int(11) NULL DEFAULT NULL,
`ped_prazo_producao` int(11) NULL DEFAULT 0,
`ped_data` datetime(0) NULL DEFAULT '0000-00-00 00:00:00',
`ped_data_producao` date NULL DEFAULT NULL,
`ped_data_entrega` date NULL DEFAULT NULL,
`ped_notificado` char(1) CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL DEFAULT 'N',
`ped_transacao_pagarme` varchar(255) CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL DEFAULT NULL,
`ped_cmd` varchar(255) CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL DEFAULT '',
`ped_data_confirmacao_pagamento` datetime(0) NULL DEFAULT NULL,
`forma_pagamento_id` int(11) NULL DEFAULT NULL,
`ped_vencimento_boleto` date NULL DEFAULT NULL,
`ped_comprovante` varchar(255) CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL DEFAULT NULL,
`ped_pago` char(1) CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL DEFAULT 'N',
`status_id` int(11) NOT NULL DEFAULT 0,
`status_pedido_id` int(11) NOT NULL DEFAULT 0,
`ped_valor_adicionais` decimal(10, 2) NULL DEFAULT NULL,
`ped_valor_frete` decimal(10, 2) NULL DEFAULT NULL,
`ped_valor_produtos` decimal(10, 2) NULL DEFAULT 0.00,
`ped_valor_desconto` decimal(10, 2) NULL DEFAULT 0.00,
`ped_valor_total` decimal(10, 2) NULL DEFAULT 0.00,
`excluido` datetime(0) NULL DEFAULT NULL,
`cadastrado` timestamp(0) NULL DEFAULT CURRENT_TIMESTAMP,
`atualizado` timestamp(0) NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP(0),
PRIMARY KEY (`pedido_id`) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 15876 CHARACTER SET = latin1 COLLATE = latin1_swedish_ci ROW_FORMAT = Compact;
CREATE TABLE `pedido_item` (
`pedido_item_id` int(11) NOT NULL AUTO_INCREMENT,
`pedido_id` int(11) NULL DEFAULT NULL,
`pei_indice` int(11) NULL DEFAULT NULL,
`produto_id` int(11) NULL DEFAULT NULL,
`produto_variacao_id` int(11) NULL DEFAULT NULL,
`produto_preco_id` int(11) NULL DEFAULT NULL,
`tipo_arquivo_id` int(11) NULL DEFAULT NULL,
`pei_arquivo` varchar(255) CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL DEFAULT NULL,
`pei_arquivo_nome` varchar(255) CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL DEFAULT NULL,
`pei_nome` varchar(255) CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL DEFAULT NULL,
`pei_quantidade` int(11) NULL DEFAULT NULL,
`pei_valor_unitario` decimal(10, 2) NULL DEFAULT NULL,
`pei_valor_total` decimal(10, 2) NULL DEFAULT NULL,
`pei_valor_frete` decimal(10, 2) NULL DEFAULT NULL,
`pei_codigo_preco` int(11) NULL DEFAULT NULL,
`pei_codigo_interno` varchar(40) CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL DEFAULT NULL,
`status_id` int(11) NOT NULL DEFAULT 0,
`excluido` timestamp(0) NULL DEFAULT NULL,
`cadastrado` timestamp(0) NULL DEFAULT CURRENT_TIMESTAMP,
`atualizado` timestamp(0) NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP(0),
`pei_producao_finalizada` char(1) CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL DEFAULT 'N',
`pei_arquivo_erro` char(1) CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL DEFAULT NULL,
PRIMARY KEY (`pedido_item_id`) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 17528 CHARACTER SET = latin1 COLLATE = latin1_swedish_ci ROW_FORMAT = Compact;
CREATE TABLE `pedido_item_status` (
`pedido_item_status_id` int(11) NOT NULL AUTO_INCREMENT,
`pedido_id` int(11) NOT NULL,
`pedido_item_id` int(11) NOT NULL,
`status_id` int(11) NOT NULL,
`pis_texto` text CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL,
`pis_notificar_cliente` char(1) CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL DEFAULT 'S',
`excluido` datetime(0) NULL DEFAULT NULL,
`cadastrado` timestamp(0) NULL DEFAULT CURRENT_TIMESTAMP,
`atualizado` timestamp(0) NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP(0),
PRIMARY KEY (`pedido_item_status_id`) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 35743 CHARACTER SET = latin1 COLLATE = latin1_swedish_ci ROW_FORMAT = Compact;
CREATE TABLE `pedido_status_pedido` (
`pedido_status_pedido_id` int(11) NOT NULL AUTO_INCREMENT,
`pedido_id` int(11) NULL DEFAULT NULL,
`status_pedido_id` int(11) NULL DEFAULT NULL,
`psp_notificar_cliente` char(1) CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL DEFAULT 'S',
`psp_texto` text CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL,
`excluido` timestamp(0) NULL DEFAULT NULL,
`cadastrado` timestamp(0) NULL DEFAULT CURRENT_TIMESTAMP,
`atualizado` timestamp(0) NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP(0),
PRIMARY KEY (`pedido_status_pedido_id`) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 38216 CHARACTER SET = latin1 COLLATE = latin1_swedish_ci ROW_FORMAT = Compact;
CREATE TABLE `status` (
`status_id` int(11) NOT NULL AUTO_INCREMENT,
`sta_nome` varchar(255) CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL DEFAULT NULL,
`sta_observacao` varchar(255) CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL DEFAULT NULL,
`sta_ordem` int(11) NULL DEFAULT NULL,
`status_pedido_id` int(11) NULL DEFAULT NULL,
`excluido` datetime(0) NULL DEFAULT NULL,
`cadastrado` datetime(0) NULL DEFAULT NULL,
`atualizado` datetime(0) NULL DEFAULT NULL,
`sta_cor` varchar(255) CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL DEFAULT NULL,
`sta_icon` varchar(50) CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL DEFAULT NULL,
`sta_alert` varchar(50) CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL DEFAULT NULL,
PRIMARY KEY (`status_id`) USING BTREE,
INDEX `idx_1`(`excluido`, `status_id`) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 14 CHARACTER SET = latin1 COLLATE = latin1_swedish_ci ROW_FORMAT = Compact;
INSERT INTO `status` VALUES (1, 'Aguardando pagamento', NULL, 1, 1, NULL, NULL, NULL, '#CACACA', 'fa-clock-o', 'alert-warning');
INSERT INTO `status` VALUES (2, 'Aguardando arquivo', NULL, 2, 2, NULL, NULL, NULL, '#FAC08C', 'fa-file-image-o', 'alert-warning');
INSERT INTO `status` VALUES (3, 'Arquivo em análise', NULL, 4, 2, NULL, NULL, NULL, '#8CBCFA', 'fa-spinner', 'alert-info');
INSERT INTO `status` VALUES (4, 'Produção finalizada', NULL, 9, 4, NULL, NULL, NULL, '#DCBCA5', 'check-square-o', 'alert-info');
INSERT INTO `status` VALUES (5, 'Arquivo com erro', NULL, 5, 2, NULL, NULL, NULL, '#FF8C8C', 'fa-exclamation-circle', 'alert-danger');
INSERT INTO `status` VALUES (6, 'Em produção', NULL, 7, 3, NULL, NULL, NULL, '#8CBCFA', 'fa-cogs', 'alert-info');
INSERT INTO `status` VALUES (7, 'Em transporte', NULL, 11, 5, NULL, NULL, NULL, '#DCA5A5', 'fa-truck', 'alert-info');
INSERT INTO `status` VALUES (8, 'Entregue', NULL, 12, 6, NULL, NULL, NULL, '#5CCE90', 'fa-check-circle-o', 'alert-success');
INSERT INTO `status` VALUES (9, 'Cancelado', NULL, 13, 7, NULL, NULL, NULL, '#FF7979', 'fa-times-circle-o', 'alert-danger');
INSERT INTO `status` VALUES (10, 'Pronto para retirada', NULL, 10, 4, NULL, NULL, NULL, '#FFD24D', 'fa-check-circle-o', 'alert-info');
INSERT INTO `status` VALUES (11, 'Produto com defeito', NULL, 8, 3, NULL, NULL, NULL, '#FF8C8C', 'fa-exclamation-circle', 'alert-danger');
INSERT INTO `status` VALUES (12, 'Arquivo aprovado', NULL, 6, 20, NULL, NULL, NULL, '#C0ED85', 'fa-check-circle-o', 'alert-info');
INSERT INTO `status` VALUES (13, 'Em Espera', NULL, 3, 1, NULL, NULL, NULL, '#8CBCFA', 'fa-spinner', 'alert-info');
CREATE TABLE `status_pedido` (
`status_pedido_id` int(11) NOT NULL AUTO_INCREMENT,
`stp_nome` varchar(255) CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL DEFAULT NULL,
`stp_observacao` varchar(255) CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL DEFAULT NULL,
`stp_ordem` int(11) NULL DEFAULT NULL,
`excluido` datetime(0) NULL DEFAULT NULL,
`cadastrado` datetime(0) NULL DEFAULT NULL,
`atualizado` datetime(0) NULL DEFAULT NULL,
`stp_cor` varchar(255) CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL DEFAULT NULL,
`stp_icon` varchar(50) CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL DEFAULT NULL,
`stp_alert` varchar(50) CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL DEFAULT NULL,
PRIMARY KEY (`status_pedido_id`) USING BTREE,
INDEX `idx_1`(`excluido`, `status_pedido_id`) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 21 CHARACTER SET = latin1 COLLATE = latin1_swedish_ci ROW_FORMAT = Compact;
INSERT INTO `status_pedido` VALUES (1, 'Aguardando pagamento', NULL, 1, NULL, NULL, NULL, NULL, NULL, NULL);
INSERT INTO `status_pedido` VALUES (2, 'Pendente', NULL, 2, NULL, NULL, NULL, NULL, NULL, NULL);
INSERT INTO `status_pedido` VALUES (3, 'Em produção', NULL, 4, NULL, NULL, NULL, NULL, NULL, NULL);
INSERT INTO `status_pedido` VALUES (4, 'Pronto', NULL, 5, NULL, NULL, NULL, NULL, NULL, NULL);
INSERT INTO `status_pedido` VALUES (5, 'Em transporte', NULL, 6, NULL, NULL, NULL, NULL, NULL, NULL);
INSERT INTO `status_pedido` VALUES (6, 'Entregue', NULL, 7, NULL, NULL, NULL, NULL, NULL, NULL);
INSERT INTO `status_pedido` VALUES (7, 'Cancelado', NULL, 8, NULL, NULL, NULL, NULL, NULL, NULL);
INSERT INTO `status_pedido` VALUES (20, 'Aprovado', NULL, 3, NULL, NULL, NULL, NULL, NULL, NULL);
The query itself works fine, but I need a way to make it faster. Perhaps it would be better if I break it down on smaller queries? Or I'm using too much unnecessary data?
EDIT:
Following the answer, the new query now looks like this:
SELECT s.status_pedido_id AS status_pedido_atual, p.status_pedido_id AS status_pedido_anterior FROM `status` s
LEFT JOIN pedido p ON p.pedido_id = {$_POST['pedido_id']}
WHERE s.sta_ordem =
(
SELECT MAX( si.sta_ordem ) AS max_ordem
FROM pedido_item_status pis
LEFT JOIN `status` si ON si.status_id = pis.status_id
WHERE pis.pedido_id = {$_POST['pedido_id']}
AND pis.excluido IS NULL
GROUP BY pis.pedido_item_id
ORDER BY max_ordem ASC
LIMIT 1
)
And I insert the order status log on another query using the result. I got much faster. Also the were other problems with the logic of the whole process I solved.

Answer 1

Well, my guess is to run EXPLAIN ... and run it for your INSERT ... query
+---+----------+----------------------+--+--------+-------------+-------------+--+--+---+--------+----------------------------------------------+
| 1 | INSERT | pedido_status_pedido | | ALL | | | | | | | |
+---+----------+----------------------+--+--------+-------------+-------------+--+--+---+--------+----------------------------------------------+
| 2 | SUBQUERY | | | | | | | | | | no matching row in const table |
| 3 | SUBQUERY | p1 | | ALL | | | | | 1 | 100.00 | Using where; Using temporary; Using filesort |
| 3 | SUBQUERY | s | | eq_ref | PRIMARY | PRIMARY | | | 1 | 100.00 | Using where |
| 3 | SUBQUERY | pedido_item | | eq_ref | PRIMARY | PRIMARY | | | 1 | 100.00 | Using index |
| 3 | SUBQUERY | <derived4> | | ref | <auto_key0> | <auto_key0> | | | 2 | 100.00 | Using index |
| 4 | DERIVED | pi | | ALL | | | | | 1 | 100.00 | Using where; Using temporary; Using filesort |
| 4 | DERIVED | si | | eq_ref | PRIMARY | PRIMARY | | | 1 | 100.00 | |
+---+----------+----------------------+--+--------+-------------+-------------+--+--+---+--------+----------------------------------------------+
I ran with all your data and found that (possibly) you're having 3 of 7 subqueries with no possible key for searching
I would start de-compositing your query and checking them with EXPLAIN for possible bottlenecks.
Otherwise I ran this query (previosly I've filled some fake data in tables of course) and it performed for 0.016sec. That's nuisance. And that's why another guess is in backend code AND/OR volume of your data / your server capabilities.
So, at the end:
Check query with EXPLAIN
Decompose your complex query, check inner SELECTs for possible bottleneck
Change your logic to avoid bottlenecks - break complex queries into simple ones. For example, you can use trigger and set variable to store MAX( si.sta_ordem ) without recalculating it for every query. If you have constantly running service - you can use temporary tables for caching purposes.
If your MySQL query is ok, check you backend code (PHP). Maybe you're having a deadlock here.
Check you data volume.
If it's possible, reboot server and check how it's performing under load. Performance counters are available for all OSes