Issue with Query using Aurora-MySQL works with MySQL - mysql

We have a simple query that checks if a record exists in one table that has the id of another. If so it returns the id. This works for small data sets such as the working example below in Aurora.
However, if we have a lot of ids (like 1000) the first query returns 0 rows and the second succeeds in Aurora but works on MySQL.
We have been unable to see any optimizations or other gaps in Aurora MySQL that would explain this behavior. Any advice or guidance is appreciated.
Working example:
CREATE TABLE `test_table_1` (
`id` char(36) COLLATE utf8mb4_unicode_ci NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
CREATE TABLE `test_table_2` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`test_table_1_id` char(36) COLLATE utf8mb4_unicode_ci NOT NULL,
`another_id` char(36) COLLATE utf8mb4_unicode_ci NOT NULL,
PRIMARY KEY (`id`),
KEY `test_table_1_id_foreign` (`test_table_1_id`),
KEY `another_id_index` (`another_id`),
CONSTRAINT `test_table_1_id_foreign` FOREIGN KEY (`test_table_1_id`) REFERENCES `test_table_1` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
INSERT INTO `test_table_1` (`id`)
VALUES
('e54b3097-21ff-4d7d-8bee-5aa62e66d9b6'),
('4fada427-ef5a-455d-ad55-831de6192dc1');
INSERT INTO `test_table_2` (`test_table_1_id`, `another_id`)
VALUES
('e54b3097-21ff-4d7d-8bee-5aa62e66d9b6', 'aeef668e-1365-41e8-a4a7-026d9a021935'),
('4fada427-ef5a-455d-ad55-831de6192dc1', '61e7b307-e356-4537-bcc5-927c3c217992');
SELECT * FROM `test_table_1`
WHERE `id` = 'e54b3097-21ff-4d7d-8bee-5aa62e66d9b6'
AND EXISTS (
SELECT * FROM `test_table_2`
WHERE `test_table_2`.`another_id` = 'aeef668e-1365-41e8-a4a7-026d9a021935'
AND `test_table_2`.`test_table_1_id` = `test_table_1`.`id`);
SELECT * FROM `test_table_1`
WHERE `id` = '4fada427-ef5a-455d-ad55-831de6192dc1'
AND EXISTS (
SELECT * FROM `test_table_2`
WHERE `test_table_2`.`another_id` = '61e7b307-e356-4537-bcc5-927c3c217992'
AND `test_table_2`.`test_table_1_id` = `test_table_1`.`id`);

Related

Very Slow result when use WHERE and ORDER BY condition in MYSQL Query

I am facing issue of very slow result.
I am sharing table structure as and results also.
if you have any suggestion please update soon if possible for you.
=================================================================
Table Structure
-- https://www.phpmyadmin.net/
--
-- Host: localhost:3306
-- Generation Time: Mar 25, 2019 at 11:48 AM
-- Server version: 5.5.61-cll
-- PHP Version: 7.2.7
SET SQL_MODE = "NO_AUTO_VALUE_ON_ZERO";
SET AUTOCOMMIT = 0;
START TRANSACTION;
SET time_zone = "+00:00";
--
-- Database: `energe3c_lms`
--
-- --------------------------------------------------------
--
-- Table structure for table `user_material`
--
CREATE TABLE `user_material` (
`id` int(11) NOT NULL,
`user_course_id` int(11) NOT NULL,
`user_id` int(11) NOT NULL,
`material_id` int(11) NOT NULL,
`attempt` int(11) NOT NULL,
`another_attempt` tinyint(1) DEFAULT '0',
`status` varchar(255) DEFAULT NULL,
`complete` tinyint(1) DEFAULT NULL,
`percent` float DEFAULT '0',
`time` varchar(255) DEFAULT NULL,
`marking_time` varchar(255) DEFAULT NULL,
`marked_by` int(11) DEFAULT NULL,
`feedback` text,
`submitted_date` datetime DEFAULT NULL,
`marking_date` datetime DEFAULT NULL,
`created` datetime DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
--
-- Indexes for dumped tables
--
--
-- Indexes for table `user_material`
--
ALTER TABLE `user_material`
ADD PRIMARY KEY (`id`),
ADD KEY `user_material-user` (`user_id`),
ADD KEY `user_material-material` (`material_id`),
ADD KEY `user_material-marking-user` (`marked_by`),
ADD KEY `user_course-user_material` (`user_course_id`);
--
-- AUTO_INCREMENT for dumped tables
--
--
-- AUTO_INCREMENT for table `user_material`
--
ALTER TABLE `user_material`
MODIFY `id` int(11) NOT NULL AUTO_INCREMENT;
--
-- Constraints for dumped tables
--
--
-- Constraints for table `user_material`
--
ALTER TABLE `user_material`
ADD CONSTRAINT `user_course-user_material` FOREIGN KEY (`user_course_id`) REFERENCES `user_course` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
ADD CONSTRAINT `user_material-marking-user` FOREIGN KEY (`marked_by`) REFERENCES `user` (`id`) ON DELETE SET NULL ON UPDATE CASCADE,
ADD CONSTRAINT `user_material-material` FOREIGN KEY (`material_id`) REFERENCES `material` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
ADD CONSTRAINT `user_material-user` FOREIGN KEY (`user_id`) REFERENCES `user` (`id`) ON DELETE CASCADE ON UPDATE CASCADE;
COMMIT;
Showing rows 0 - 24 (25586268 total, Query took 0.0007 seconds.)
SELECT * FROM user_material
=================================================================
Showing rows 0 - 24 (77 total, Query took 22.9434 seconds.)
SELECT * FROM user_material where status='submitted'
=================================================================
Showing rows 0 - 24 (34026300 total, Query took 24.4978 seconds.)
[submitted_date: ... - ...]
SELECT * FROM user_material ORDER BY submitted_date ASC
=================================================================
If You have any suggestion, please update me.
I am using this query in yii2 framework
For this query:
SELECT *
FROM user_material
WHERE status = 'submitted';
You want an index on user_material(status).
For this query:
SELECT *
FROM user_material
ORDER BY submitted_date ASC
You want an index on user_material(submitted_date).
In both these cases, additional columns can be in the index after the one specified as the first column. I do not see these indexes in your schema.
Adding to the recommendation from #Gordon Linoff, you should consider normalizing your table further, to make the SQL more efficient. Use an very short code or int for status_id and store a unique list of status codes and descriptions in a lookup table. Then simply join the lookup table in your SQL. See this article for a detailed explanation on normalization.
I also recommend either changing the column name or the data type for the time column, because it doesn't make sense. What time value requires 255 characters?
If you are concerned about performance, then the first thing to focus your attention on is removing unnecessary bytes, sorts and filters from your query. This approach deals with most performance issues. How you join tables is important as well. As the data volume and complexity increases, the tuning effort becomes increasingly more complex as well.
Getting to know EXPLAIN will help your effort. Refer to the EXPLAIN documentation for more information.
-- Create your lookup table first
CREATE TABLE `material_status` (
`id` int(11) NOT NULL,
`status` varchar(255) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
ALTER TABLE `material_status`
ADD PRIMARY KEY (`id`),
ALTER TABLE `material_status`
MODIFY `id` int(11) NOT NULL AUTO_INCREMENT;
-- Then create your user_material table
CREATE TABLE `user_material` (
`id` int(11) NOT NULL,
`user_course_id` int(11) NOT NULL,
`user_id` int(11) NOT NULL,
`material_id` int(11) NOT NULL,
`attempt` int(11) NOT NULL,
`another_attempt` tinyint(1) DEFAULT '0',
`status_id` int(11) DEFAULT NULL,
`complete` tinyint(1) DEFAULT NULL,
`percent` float DEFAULT '0',
`time` varchar(255) DEFAULT NULL,
`marking_time` varchar(255) DEFAULT NULL,
`marked_by` int(11) DEFAULT NULL,
`feedback` text,
`submitted_date` datetime DEFAULT NULL,
`marking_date` datetime DEFAULT NULL,
`created` datetime DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
-- Create your extra index here
CREATE INDEX ix-submitted_date ON user_material(submitted_date);
-- Add the status to the keys and constraints
ALTER TABLE `user_material`
ADD PRIMARY KEY (`id`),
ADD KEY `user_material-user` (`user_id`),
ADD KEY `user_material-material` (`material_id`),
ADD KEY `user_material-marking-user` (`marked_by`),
ADD KEY `user_course-user_material` (`user_course_id`),
ADD KEY `user_material-status` (`status_id`);
ALTER TABLE `user_material`
MODIFY `id` int(11) NOT NULL AUTO_INCREMENT;
ALTER TABLE `user_material`
ADD CONSTRAINT `user_course-user_material` FOREIGN KEY (`user_course_id`) REFERENCES `user_course` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
ADD CONSTRAINT `user_material-marking-user` FOREIGN KEY (`marked_by`) REFERENCES `user` (`id`) ON DELETE SET NULL ON UPDATE CASCADE,
ADD CONSTRAINT `user_material-material` FOREIGN KEY (`material_id`) REFERENCES `material` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
ADD CONSTRAINT `user_material-user` FOREIGN KEY (`user_id`) REFERENCES `user` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
ADD CONSTRAINT `user_material-status` FOREIGN KEY (`status_id`) REFERENCES `material_status` (`id`) ON DELETE CASCADE ON UPDATE CASCADE;
The queries will now look like this:
SELECT
um.id,
um.user_course_id,
um.user_id,
um.material_id,
um.attempt,
um.another_attempt,
ms.status,
um.complete,
um.percent,
um.time,
um.marking_time,
um.marked_by,
um.feedback,
um.submitted_date,
um.marking_date,
um.created
FROM user_material um
INNER JOIN material_status ms on ms.id = um.status_id
WHERE ms.status = 'submitted';
SELECT
um.id,
um.user_course_id,
um.user_id,
um.material_id,
um.attempt,
um.another_attempt,
ms.status,
um.complete,
um.percent,
um.time,
um.marking_time,
um.marked_by,
um.feedback,
um.submitted_date,
um.marking_date,
um.created
FROM user_material um
INNER JOIN material_status ms on ms.id = um.status_id
ORDER BY um.submitted_date ASC;
Please try this.
SELECT
um.id,
um.user_course_id,
um.user_id,
um.material_id,
um.attempt,
um.another_attempt,
ms.status,
um.complete,
um.percent,
um.time,
um.marking_time,
um.marked_by,
um.feedback,
um.submitted_date,
um.marking_date,
um.created
FROM user_material um
INNER JOIN material_status ms on ms.id = um.status_id
AND ms.status = 'submitted';
And also create indexing on where and order field.

Not able to store emojis on a utf8mb mysql table

Having this table using mysql 5.7:
CREATE TABLE `emails` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`subject` varchar(191) COLLATE utf8mb4_bin DEFAULT NULL,
PRIMARY KEY (`id`),
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin;
When I try to insert some emojis:
INSERT INTO `emails` (`from_address`, `subject`) VALUES (1, 'A😀B C👨🏽‍🎨D')
I receive:
Incorrect string value: '\xF0\x9F\x98\x80B ...' for column 'subject' at row 1
Why? if i'm using utfmb?
Is your connection also utf8mb4? Detailed explanation of this can be found at: https://mathiasbynens.be/notes/mysql-utf8mb4

Error Code: 1052 Column 'admin_id' in field list is ambiguous

Hi I have tried to create a time sheet view in my database but I'm having trouble with the admin_id column. I'm reusing this code from another assignment which works so I'm confused t why it doesn't work. Please Help me!!!
Select Statement
SELECT timesheet_id, class, day, hour, week, admin_id, date_added FROM timesheet, day, classes, admin
WHERE timesheet_id AND
classes.class_id = timesheet.class_id AND
day.day_id = timesheet.day_id AND
admin.admin_id = timesheet.admin_id ORDER BY timesheet.timesheet_id.;
Database code
'CREATE DATABASE /*!32312 IF NOT EXISTS*/`timesheet` /*!40100 DEFAULT CHARACTER SET latin1 */;
USE `timesheet`;
/*Table structure for table `admin` */
DROP TABLE IF EXISTS `admin`;
CREATE TABLE `admin` (
`admin_id` int(100) NOT NULL AUTO_INCREMENT,
`username` varchar(10) DEFAULT NULL,
`password` char(30) DEFAULT NULL,
PRIMARY KEY (`admin_id`)
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1;
/*Data for the table `admin` */
insert into `admin`(`admin_id`,`username`,`password`) values (1,'1627724','troll1'),(2,'1627406','troll2');
/*Table structure for table `classes` */
DROP TABLE IF EXISTS `classes`;
CREATE TABLE `classes` (
`class_id` int(11) NOT NULL AUTO_INCREMENT,
`class` varchar(30) DEFAULT NULL,
PRIMARY KEY (`class_id`)
) ENGINE=InnoDB AUTO_INCREMENT=9 DEFAULT CHARSET=latin1;
/*Data for the table `classes` */
insert into `classes`(`class_id`,`class`) values (1,'Validate and Test'),(2,'Complex Web'),(3,'Advanced OO Web'),(4,'Project Management'),(5,'Project Web'),(6,'Meeting'),(7,'Study'),(8,'Software Development');
/*Table structure for table `day` */
DROP TABLE IF EXISTS `day`;
CREATE TABLE `day` (
`day_id` int(11) NOT NULL AUTO_INCREMENT,
`day` varchar(15) NOT NULL,
PRIMARY KEY (`day_id`)
) ENGINE=InnoDB AUTO_INCREMENT=8 DEFAULT CHARSET=latin1;
/*Data for the table `day` */
insert into `day`(`day_id`,`day`) values (1,'Monday'),(2,'Tuesday'),(3,'Wednesday'),(4,'Thursday'),(5,'Friday'),(6,'Saturday'),(7,'Sunday');
/*Table structure for table `menu` */
DROP TABLE IF EXISTS `menu`;
CREATE TABLE `menu` (
`Menu_id` int(100) NOT NULL AUTO_INCREMENT,
`Menu` char(10) DEFAULT NULL,
PRIMARY KEY (`Menu_id`)
) ENGINE=InnoDB AUTO_INCREMENT=5 DEFAULT CHARSET=latin1;
/*Data for the table `menu` */
insert into `menu`(`Menu_id`,`Menu`) values (1,'index'),(2,'contact us'),(3,'project'),(4,'timesheets');
/*Table structure for table `timesheet` */
DROP TABLE IF EXISTS `timesheet`;
CREATE TABLE `timesheet` (
`timesheet_id` int(11) NOT NULL AUTO_INCREMENT,
`class_id` int(11) NOT NULL,
`day_id` int(11) NOT NULL,
`hour` float DEFAULT NULL,
`week` varchar(8) NOT NULL,
`admin_id` int(11) NOT NULL,
`date_added` date NOT NULL,
PRIMARY KEY (`timesheet_id`),
KEY `class_fk` (`class_id`),
KEY `day_fk` (`day_id`),
KEY `admin_fk` (`admin_id`),
CONSTRAINT `admin_fk` FOREIGN KEY (`admin_id`) REFERENCES `admin` (`admin_id`),
CONSTRAINT `class_fk` FOREIGN KEY (`class_id`) REFERENCES `classes` (`class_id`),
CONSTRAINT `day_fk` FOREIGN KEY (`day_id`) REFERENCES `day` (`day_id`)
) ENGINE=InnoDB AUTO_INCREMENT=6 DEFAULT CHARSET=latin1;
/*Data for the table `timesheet`'
insert into `timesheet`(`timesheet_id`,`class_id`,`day_id`,`hour`,`week`,`admin_id`,`date_added`) values (1,1,1,1,'week1',2,'2013-10-14'),(2,2,5,6,'week1',2,'2013-10-14'),(3,1,3,5,'week1',2,'2013-10-14'),(4,5,6,2,'week1',2,'2013-10-14'),(5,8,6,4,'week1',2,'2013-10-14');
This means that there is more than one column called admin_id in the tables being accessed in your query so mysql doesn't know which to return results from.
Change your select statement to include the table alias (either admin or timesheet) as so: SELECT timesheet_id, class, day, hour, week, timesheet.admin_id, date_added FROM timesheet
Since admin_id is present in 2 tables (admin and timesheet) you can not use it in SELECT list without table name or its alias.

Nested "select ... in" performance is slow - how to fix?

Here I have a simple join query. If first two queries get results, the whole query can be done in 0.3 secs, but if the first 2 select doesn't fetch any result, the whole query will cost more than half a minute. What causes this difference? How to fix this problem and improve the performance?
SELECT * FROM music WHERE id IN
(
SELECT id FROM music_tag_map WHERE tag_id IN
(
SELECT id FROM tag WHERE content ='xxx'
)
)
LIMIT 10
Here's the table structure:
CREATE TABLE `tag` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`content` varchar(20) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `index2` (`content`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE `music` (
`id` int(7) NOT NULL AUTO_INCREMENT,
`name` varchar(500) NOT NULL,
`othername` varchar(200) DEFAULT NULL,
`player` varchar(3000) DEFAULT NULL,
`genre` varchar(100) DEFAULT NULL,
`sounds` text,
`create_time` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `player` (`player`(255)),
KEY `name` (`othername`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE `music_tag_map` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`music_id` int(7) NOT NULL,
`tag_id` int(7) NOT NULL,
`times` int(11) DEFAULT '1',
PRIMARY KEY (`id`),
KEY `music_id` (`music_id`),
KEY `tag_id` (`tag_id`),
CONSTRAINT `music_tag_map_ibfk_1` FOREIGN KEY (`id`) REFERENCES `music` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `music_tag_map_ibfk_2` FOREIGN KEY (`tag_id`) REFERENCES `tag` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
There are no joins in that query; there are two sub-selects.
A joined query would be:
SELECT *
FROM music
JOIN music_tag_map ON music.id=music_tag_map.id
JOIN tag ON music_tag_map.tag_id=tag.id
WHERE tag.content = ?
LIMIT 10;
An EXPLAIN applied to each will show you why the join performs better than the sub-select: the sub-select will scan the entire music table (the primary query), while the optimizer can pick the order of tables to scan for the joins, allowing MySQL to use indices to get only the needed rows from all the tables.

Optimize Join sentence with foreign keys, and show records with nulls

I have the following structure
SET SQL_MODE="NO_AUTO_VALUE_ON_ZERO";
CREATE TABLE IF NOT EXISTS `sis_param_tax` (
`id` int(5) NOT NULL auto_increment,
`description` varchar(50) NOT NULL,
`code` varchar(5) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=7;
CREATE TABLE IF NOT EXISTS `sis_param_city` (
`id` int(4) NOT NULL auto_increment,
`name` varchar(100) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=3 ;
CREATE TABLE IF NOT EXISTS `sis_supplier` (
`id` int(15) NOT NULL auto_increment,
`name` varchar(200) NOT NULL,
`address` varchar(200) default NULL,
`phone` varchar(30) NOT NULL,
`fk_city` int(11) default NULL,
`fk_tax` int(11) default NULL,
PRIMARY KEY (`id`),
KEY `fk_city` (`fk_city`),
KEY `fk_tax` (`fk_tax`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=2 ;
ALTER TABLE `sis_supplier`
ADD CONSTRAINT `sis_supplier_ibfk_4` FOREIGN KEY (`fk_tax`) REFERENCES `sis_param_tax` (`id`) ON DELETE SET NULL ON UPDATE CASCADE,
ADD CONSTRAINT `sis_supplier_ibfk_3` FOREIGN KEY (`fk_city`) REFERENCES `sis_param_city` (`id`) ON DELETE SET NULL ON UPDATE CASCADE;
My questions are
1. This structure allows me to have a supplier with city and tax fields = null (in case user didn't set these values). Right?
2. If I delete "X" city, supplier's fk_city with city="X" are set to null, same with fk_tax. Right?
3. I want to optimize (IF POSSIBLE) the following join sentence, so I can show suppliers whom have fk_city and/or fk_tax = NULL
SELECT DISTINCT
sis_supplier.id,
sis_supplier.name,
sis_supplier.telefono,
sis_supplier.address,
sis_supplier.phone,
sis_supplier.cuit,
sis_param_city.name AS city,
sis_param_tax.description AS tax,
sis_supplier.fk_city,
sis_supplier.fk_tax
FROM
sis_supplier
LEFT OUTER JOIN sis_param_city
ON
sis_supplier.`fk_city` = sis_param_city.id
LEFT OUTER JOIN `sis_param_tax`
ON
sis_supplier.`fk_tax` = `sis_param_tax`.`id`
Thanks a lot in advance,
Yes.
Yes.
Yes, it's good to optimize. The query you showed looks fine. How is it not working for you?
Have you analyzed the query with EXPLAIN? This can help you tell when you have a query that isn't using indexes effectively. In fact, all of Chapter 7 Optimization would be recommended reading.
if you want to show records with nulls than use RIGHT or LEFT JOIN
depend on your needs