So I am trying to make an app with "code first" approach and active record ORM. I am willing to work with MySQL and SQLite (both) in this app. Here is how I am creating a database:
(this is a mockup table I came up with solely for this question)
$this->int("id", self::INT_MEDIUM)->unSigned()->primaryKey()->autoIncrement();
$this->string("hash", 64, self::STR_FIXED)->unique();
$this->enum("status", "available","sold","pending")->defaultValue("available");
$this->int("category", self::INT_MEDIUM)->unSigned()->defaultValue(0);
$this->string("name")->unique();
$this->text("descr")->nullable();
$this->decimal("price", 10, 4)->defaultValue(1.234);
$this->uniqueKey("store_np", "name", "price");
$this->foreignKey("category", "categories", "id");
and then my code generates CREATE TABLE statement for me, here are the results:
MySQL:
CREATE TABLE `products` (
`id` mediumint UNSIGNED PRIMARY KEY auto_increment NOT NULL,
`hash` char(64) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL,
`status` enum('available','sold','pending') CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL default 'available',
`category` mediumint UNSIGNED NOT NULL default 0,
`name` varchar(255) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL,
`descr` TEXT CHARACTER SET utf8 COLLATE utf8_unicode_ci default NULL,
`price` decimal(10,4) NOT NULL default 1.234,
UNIQUE KEY (`hash`),
UNIQUE KEY (`name`),
UNIQUE KEY `store_np` (`name`,`price`),
FOREIGN KEY (`category`) REFERENCES `categories`(`id`)
) ENGINE=InnoDB;
SQLite:
CREATE TABLE `products` (
`id` INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
`hash` TEXT UNIQUE NOT NULL,
`status` TEXT CHECK(status in ('available','sold','pending') ) NOT NULL default 'available',
`category` INTEGER UNSIGNED NOT NULL default 0,
`name` TEXT UNIQUE NOT NULL,
`descr` TEXT default NULL,
`price` REAL NOT NULL default 1.234,
CONSTRAINT `store_np` UNIQUE (`name`,`price`),
CONSTRAINT `cnstrnt_category_frgn` FOREIGN KEY (`category`) REFERENCES `categories`(`id`)
);
I executed both queries in phpMyAdmin and phpLiteAdmin, and both seemed to worked fine.
Here are my concerns:
For example, I didn't know I cannot use "UNSIGNED" with "PRIMARY KEY AUTOINCREMENT" in SQLite, which gave me hard time to figure out. Is there anything else like that I should be concerned about?
Even though both statements were executed successfully, are they going to work as expected? Especially the constraints in SQLite
Please check "status" column in SQLite, is it appropriate method to use as an alternative to MySQL's enum?
Related
I'm running a count query which is very slow, how can improve this?
I've got the following query, but it takes around 1.33 seconds:
select
count(*) as aggregate
from
`tickets`
inner join `orders` on `orders`.`id` = `tickets`.`order_id`
where
`orders`.`status` = 'paid' and
`tickets`.`created_at` > '2023-01-01 00:00:00'
The tickets table has around 650000 rows and the order table has around 320000 rows.
This is the result of SHOW CREATE TABLE tickets:
CREATE TABLE `tickets` (
`id` bigint unsigned NOT NULL AUTO_INCREMENT,
`tickettype_id` int unsigned NOT NULL,
`order_id` int unsigned NOT NULL,
`variant_id` bigint unsigned DEFAULT NULL,
`seat_id` bigint unsigned DEFAULT NULL,
`barcode` varchar(191) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`first_name` varchar(191) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`last_name` varchar(191) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`email` varchar(191) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`telephone` varchar(191) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`birthday` date DEFAULT NULL,
`age` int unsigned DEFAULT NULL,
`gender` enum('m','f') CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`price` double(10,2) DEFAULT NULL,
`extra_info` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`created_at` timestamp NULL DEFAULT NULL,
`updated_at` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `tickets_barcode_unique` (`barcode`),
KEY `tickets_tickettype_id_foreign` (`tickettype_id`),
KEY `tickets_order_id_foreign` (`order_id`),
KEY `tickets_order_id_index` (`order_id`),
KEY `tickets_tickettype_id_index` (`tickettype_id`),
KEY `tickets_seat_id_foreign` (`seat_id`),
KEY `tickets_variant_id_foreign` (`variant_id`),
CONSTRAINT `tickets_ibfk_1` FOREIGN KEY (`order_id`) REFERENCES `orders` (`id`) ON DELETE CASCADE,
CONSTRAINT `tickets_seat_id_foreign` FOREIGN KEY (`seat_id`) REFERENCES `seatplan_seats` (`id`) ON DELETE SET NULL,
CONSTRAINT `tickets_tickettype_id_foreign` FOREIGN KEY (`tickettype_id`) REFERENCES `tickets_types` (`id`) ON DELETE CASCADE,
CONSTRAINT `tickets_variant_id_foreign` FOREIGN KEY (`variant_id`) REFERENCES `ticket_variants` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB AUTO_INCREMENT=2945088 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
How can I improve the speed?
The performance of your query depends on several factors. Such as,
The table size
Performance of your machine
Indexing etc.
If you dont have indices created for status, order_id and created_at, better create them. Which can significantly improve the query performance.
CREATE INDEX order_id_index ON tickets(order_id);
CREATE INDEX status_index ON orders(status);
CREATE INDEX created_at_index ON tickets(created_at);
Additionally, if you are using PostgreSQL, try running VACUUM on your tables which removes the dead tuples and improves performance.
First of all
You need to add two indexes :
CREATE INDEX order_id_idx ON tickets(order_id);
// composite index since you are using both columns in where
CREATE INDEX status_created_at_idx ON tickets(status, created_at);
The query optimizer uses the composite indexes for queries that test all columns in the index, or queries that test the first columns, the first two columns, and so on.
More informations regarding composite can be found here
my app get stuck for hours on simple queries like :
SELECT COUNT(*) FROM `item`
Context :
This table is around 200Gb+ and 50M+ rows.
We have a RDS on AWS with 2CPU and 16GiB RAM (db.r6g.large).
This is the table structure SQL dump :
/*
Target Server Type : MySQL
Target Server Version : 80023
File Encoding : 65001
*/
SET NAMES utf8mb4;
SET FOREIGN_KEY_CHECKS = 0;
DROP TABLE IF EXISTS `item`;
CREATE TABLE `item` (
`id` bigint unsigned NOT NULL AUTO_INCREMENT,
`status` tinyint DEFAULT '1',
`source_id` int unsigned DEFAULT NULL,
`type` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`url` varchar(2048) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`title` varchar(500) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`sku` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`price` decimal(20,4) DEFAULT NULL,
`price_bc` decimal(20,4) DEFAULT NULL,
`price_original` decimal(20,4) DEFAULT NULL,
`currency` varchar(10) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`description` text CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci,
`image` varchar(1024) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`time_start` datetime DEFAULT NULL,
`time_end` datetime DEFAULT NULL,
`block_update` tinyint(1) DEFAULT '0',
`status_api` tinyint(1) DEFAULT '1',
`data` json DEFAULT NULL,
`created_at` int unsigned DEFAULT NULL,
`updated_at` int unsigned DEFAULT NULL,
`retailer_id` int DEFAULT NULL,
`hash` char(32) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`count_by_hash` int DEFAULT '1',
`item_last_update` int DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `sku_retailer_idx` (`sku`,`retailer_id`),
KEY `updated_at_idx` (`updated_at`),
KEY `time_end_idx` (`time_end`),
KEY `retailer_id_idx` (`retailer_id`),
KEY `hash_idx` (`hash`),
KEY `source_id_hash_idx` (`source_id`,`hash`) USING BTREE,
KEY `count_by_hash_idx` (`count_by_hash`) USING BTREE,
KEY `created_at_idx` (`created_at`) USING BTREE,
KEY `title_idx` (`title`),
KEY `currency_idx` (`currency`),
KEY `price_idx` (`price`),
KEY `retailer_id_title_idx` (`retailer_id`,`title`) USING BTREE,
KEY `source_id_idx` (`source_id`) USING BTREE,
KEY `source_id_count_by_hash_idx` (`source_id`,`count_by_hash`) USING BTREE,
KEY `status_idx` (`status`) USING BTREE,
CONSTRAINT `fk-source_id` FOREIGN KEY (`source_id`) REFERENCES `source` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=1858202585 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
SET FOREIGN_KEY_CHECKS = 1;
does partitioning the table could help on a simple query like this ?
do I need to increase the RAM of the RDS ? If yes what configuration do I need ?
is NoSQL more compatible to this kind of structure ?
Do you have any advices/solutions/fixes so the app can run those queries (we would like to keep all the data and not erase it if possible..) ?
"SELECT COUNT(*) FROM item" needs to scan an index. The smallest index is about 200MB, so that seems like it should not take "minutes".
There are probably multiple queries that do full table scans. Such will bump out all the cached data from the ~11GB of cache (the buffer_pool) and do that about 20 times. That's a lot of I/O and a lot of elapsed time. Meanwhile, most other queries will run slowly because their cached data is being bumped out.
The resolution:
Locate these naughty queries. RDS probably gives you access to the "slowlog".
Grab the slowlog and run pt-query-digest or mysqldumpslow -s t to find the "worst" queries.
Then we can discuss them.
There are some redundant indexes; removing them won't solve the problem. A rule: If you have INDEX(a), INDEX(a,b), you don't need the former.
If hash is some kind of scrambled value, it is likely that a single-row lookup (or update) will require a disk hit (and bump something else out of the cache).
decimal(20,4) takes 10 bytes and allows values up to 9,999,999,999,999,999.9999; that seems excessive. (Shrinking it won't save much space; something to keep in mind for the future.)
I see that AUTO_INCREMENT has reached 1.8 billion. If there are only 50M rows, does the processing do a lot of DELETEs? Or maybe REPLACE``? IODKU is better than REPLACE`.
Thanks for all the advices here, but the problem was that we were using the MySQL json type for a very heavy column. Removing this column or even changing it to varchar made the COUNT(id) around 1000x faster (also adding WHERE id > 1 helped..)
Note : it was impossible to just delete the column as it was, we had to change it to varchar before.
I got the following error message whenever I try to import this sql file to my localhost
(#1005 - Can't create table hostel_management_system.Hostel_Manager (errno: 150 "Foreign key constraint is incorrectly formed"))
Here is the sql:
DROP TABLE IF EXISTS `Hostel_Manager`;
CREATE TABLE `Hostel_Manager` (
`Hostel_man_id` int(10) NOT NULL AUTO_INCREMENT,
`Username` varchar(255) NOT NULL,
`Fname` varchar(255) NOT NULL,
`Lname` varchar(255) NOT NULL,
`Mob_no` varchar(255) NOT NULL,
`Hostel_id` int(10) NOT NULL,
`Pwd` LONGTEXT NOT NULL,
`Isadmin` tinyint(1) DEFAULT '0',
PRIMARY KEY (`Hostel_man_id`),
UNIQUE (`Username`),
KEY `Hostel_id` (`Hostel_id`),
CONSTRAINT `Hostel_Manager_ibfk_1` FOREIGN KEY (`Hostel_id`) REFERENCES `Hostel` (`Hostel_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
LOCK TABLES `Hostel_Manager` WRITE;
UNLOCK TABLES;
Hostel table :
CREATE TABLE `Hostel` (
`Hostel_id` int(10) NOT NULL AUTO_INCREMENT,
`Hostel_name` varchar(255) NOT NULL,
`current_no_of_rooms` varchar(255) DEFAULT NULL,
`No_of_rooms` varchar(255) DEFAULT NULL,
`No_of_students` varchar(255) DEFAULT NULL,
PRIMARY KEY (`Hostel_id`)
)
Why do I get that error and how to fix it?
Both the parent and children column have the same datatype and length, and the parent column is a primary colum, so that's not the issue here.
I suspect that the problem is the charset. Hostel_Manager specifies a default charset, while Hostel does not. If the default charset of your database is something else than latin1, then the foreign is malformed.
I would recommend explictly aligning the charset so it is the same for both tables (either remove both, or declare the same value).
Note that both tables also need to have the same storage engine. InnoDB is the default, so that should not be an issue, but you might want to explictly align that as well (in case the default of your database is MyISAM).
I'm trying to set up two tables in a database, and add a foreign key between them. They're declared as follows:
CREATE TABLE `clothing` (
`name` varchar(26) COLLATE utf8_bin NOT NULL,
`image` varchar(64) COLLATE utf8_bin NOT NULL,
`localized_name` longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL CHECK (json_valid(`localized_name`)),
`main` varchar(18) COLLATE utf8_bin DEFAULT NULL,
`stars` tinyint(3) unsigned NOT NULL,
`id` tinyint(3) unsigned NOT NULL,
`splatnet` smallint(5) unsigned NOT NULL,
PRIMARY KEY (`splatnet`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
CREATE TABLE `abilities` (
`name` varchar(18) COLLATE utf8_bin DEFAULT NULL,
`image` varchar(48) COLLATE utf8_bin NOT NULL,
`id` tinyint(3) unsigned NOT NULL,
`localized_name` longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL CHECK (json_valid(`localized_name`))
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
I want to create a foreign key on clothing that references abilities with the following command:
ALTER TABLE `abilities` ADD FOREIGN KEY (`name`) REFERENCES `clothing` (`main`);
However, attempting to do this raises this error in return:
Source and target columns must have the same data type, there must be an index on the target columns and referenced data must exist.
Can't create table `prismarine_rusted`.`abilities` (errno: 150 "Foreign key constraint is incorrectly formed")
I'm not entirely sure what's causing this, and unless I'm overlooking something really obvious, main and name have the same type, and therefore, should be able to be tied together via a foreign key. I'm using MariaDB v10.4.12, with SQL mode set to TRADITIONAL.
Although the foreign and primary key columns involved here are the same type, you are trying to reference clothing.main, which is not a unique or primary key column. From the MariaDB documentation:
The referenced columns must be a PRIMARY KEY or a UNIQUE index.
Note that this differs from InnoDB on MySQL, where a foreign key column can in fact reference a non unique column in another table.
One way to remedy this error would be to make clothing.main a unique column:
ALTER TABLE clothing ADD UNIQUE (main);
Note that doing this might only make logical sense if the values in main are already unique. If not, then perhaps you would have to revisit your data model.
It might be because there is a value in abilities.name that has no match in the referenced table.
I have the following schema:
CREATE TABLE `news` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`news_category_id` int(10) unsigned NOT NULL,
`news_type_id` int(10) unsigned NOT NULL,
`news_pictures_main_id` int(10) unsigned DEFAULT NULL,
`title` tinytext COLLATE latin1_general_ci,
`body` text COLLATE latin1_general_ci,
`tmstp` timestamp NULL DEFAULT NULL,
`subcategory` varchar(64) COLLATE latin1_general_ci DEFAULT NULL,
`source` varchar(128) COLLATE latin1_general_ci DEFAULT NULL,
`old_id` int(10) unsigned DEFAULT NULL,
`tags` text COLLATE latin1_general_ci,
PRIMARY KEY (`id`),
KEY `news_time_idx` (`tmstp`),
KEY `fk_news_news_pictures1` (`news_pictures_main_id`),
KEY `fk_news_news_category1` (`news_category_id`),
KEY `fk_news_news_type1` (`news_type_id`),
CONSTRAINT `fk_news_news_category1` FOREIGN KEY (`news_category_id`) REFERENCES `news_category` (`id`) ON UPDATE CASCADE,
CONSTRAINT `fk_news_news_pictures1` FOREIGN KEY (`news_pictures_main_id`) REFERENCES `news_pictures` (`id`) ON DELETE SET NULL ON UPDATE CASCADE,
CONSTRAINT `fk_news_news_type1` FOREIGN KEY (`news_type_id`) REFERENCES `news_type` (`id`) ON UPDATE CASCADE
) ENGINE=InnoDB
CREATE TABLE `news_pictures` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`path` text COLLATE latin1_general_ci,
`description` text COLLATE latin1_general_ci,
`author` varchar(45) COLLATE latin1_general_ci DEFAULT NULL,
`news_id` int(10) unsigned DEFAULT NULL,
`temp_id` varchar(40) COLLATE latin1_general_ci DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `filename_old_id_unq` (`path`(20),`temp_id`(6)),
KEY `fk_news_pictures_news1` (`news_id`),
KEY `temp_id_idx` (`temp_id`(8)),
CONSTRAINT `fk_news_pictures_news1` FOREIGN KEY (`news_id`) REFERENCES `news` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB
CREATE TABLE `news_category` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(45) COLLATE latin1_general_ci DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB
CREATE TABLE `news_type` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(45) COLLATE latin1_general_ci DEFAULT NULL,
`slug` varchar(45) COLLATE latin1_general_ci DEFAULT NULL,
PRIMARY KEY (`id`)
KEY `news_type_slug_idx` (`slug`)
) ENGINE=InnoDB
From that, there is derived the following view:
CREATE OR REPLACE VIEW `news_full` AS select `n`.`id` AS `id`,
`n`.`title` AS `title`,
`n`.`body` AS `body`,
`n`.`tmstp` AS `tmstp`,
`n`.`subcategory` AS `subcategory`,
`n`.`source` AS `source`,
`n`.`old_id` AS `old_id`,
`n`.`news_type_id` AS `news_type_id`,
`n`.`tags` AS `tags`,
`nt`.`name` AS `news_type_name`,
`nt`.`slug` AS `news_type_slug`,
`n`.`news_pictures_main_id` AS `news_pictures_main_id`,
`np`.`path` AS `news_pictures_main_path`,
`np`.`description` AS `news_pictures_main_description`,
`np`.`author` AS `news_pictures_main_author`,
`np`.`temp_id` AS `news_pictures_main_temp_id`,
`n`.`news_category_id` AS `news_category_id`,
`nc`.`name` AS `news_category_name`
from (((`news` `n`
left join `news_pictures` `np` on((`n`.`news_pictures_main_id` = `np`.`id`)))
join `news_category` `nc` on((`n`.`news_category_id` = `nc`.`id`)))
join `news_type` `nt` on((`n`.`news_type_id` = `nt`.`id`)));
However, if I try to run the following query:
select * from news_full order by tmstp limit 100
I get the following execution plan (please click on the image to expand it):
Notice the Using temporary; Using filesort field in the first step. But this is weird, because tmstp field is indexed on the base table.
First I thought this was due the left join on the view, but I've changed it to inner join and I got the same results.
Edit
As #Michael-sqlbot cleverly noticed, the query optimizer is inverting the order of the base tables, putting news_category (nc) first.
If I change the query that creates the view to use only LEFT JOINs it seems to work:
The execution times, as expected, as blatantly different:
Not satisfied, I created another view with the original query, adding the STRAIGHT_JOIN statement. So, the query plan comes as follows:
So, it's not using the index.
However, if I run the plan for the base query adding the same ORDER BY and LIMIT clauses, it does uses the index:
(Not an answer, but some other issues to bring up...)
UNIQUE KEY `filename_old_id_unq` (`path`(20),`temp_id`(6))
That constrains the first 20 characters of path, together with the first 6 characters of temp_id to be unique across the table. Did you really want that?
I suspect the optimizer will never use both columns of that index. (In general, prefixing is useless.)
And...
`title` tinytext COLLATE latin1_general_ci
Change to VARCHAR(255). There are disadvantages of TINYTEXT and perhaps no advantages.