It may sound similar but,I am working on partitioning on some table...the table looks like
mysql> DESC SHOPS;
+-------------------+-------------+------+-----+-------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+-------------------+-------------+------+-----+-------------------+-----------------------------+
| SHOP_ID | int(255) | NO | PRI | NULL | |
| SHOP_NAME | varchar(50) | YES | | NULL | |
| SHOP_CREATED_DATE | timestamp | NO | | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
+-------------------+-------------+------+-----+-------------------+-----------------------------+
3 rows in set (0.00 sec)
so i have search feature where people can search only by shop name so table have around 1 million records so i wanted to RANGE partitioning on shop name alphabetically but i cant do since i have primary key shop_id and shop name can be same...and getting error
ERROR 1503 (HY000): A PRIMARY KEY must include all columns in the
table's partitioning function
Solution:
ALTER TABLE SHOPS ADD CONSTRAINT T UNIQUE (SHOP_ID,SHOP_NAME);
And do partitioning ...i cant do this because it does not make sure shop_id is unique(Primary Key)
You can, and you must. Assuming you always let AUTO_INCREMENT do its thing, shop_id will always be unique, and any index starting with shop_id is all you need.
int(255) -- The (255) means nothing. An INT (SIGNED, by default) has a range of -2 billion to +2 billion and occupies 4 bytes, regardless of the (...) after it..
There is probably no performance advantage (or any other advantage) of Partitioning this table. If you think otherwise, please show us a query that you think will benefit.
Please use SHOW CREATE TABLE; it is more descriptive than DESCRIBE.
Related
I have the following table (it has more data columns, removed them because it would be a long post):
CREATE TABLE `members` (
`memberid` int(11) NOT NULL AUTO_INCREMENT,
`firstname` varchar(45) COLLATE utf8_unicode_ci DEFAULT NULL,
`lastname` varchar(45) COLLATE utf8_unicode_ci DEFAULT NULL,
PRIMARY KEY (`memberid`),
KEY `members_lname_ix` (`lastname`)
) ENGINE=InnoDB AUTO_INCREMENT=1019 DEFAULT CHARSET=utf8
COLLATE=utf8_unicode_ci;
By default, a user only ever accesses 10-20 rows from this table at a time and it is usually sorted by the lastname column, it's all paginated server side. so I decided to add an index to lastname to help with sorting, however the index does not seem to be working like I would expect it to. when I run EXPLAIN SELECT * FROM members ORDER BY lastname ASC I get:
id | select_type | table | type | possible_keys | key | key_len | ref | rows | extra
1 | simple | members | ALL | null | null | null | null | 711 | using filesort
I can at least confirm the index exists because if I run SHOW INDEX FROM members I get:
Table | Non_Unique | Key_name | Seq_in_ix | Col_name | Collation | Cardinality | Sub part | Packed | Null | Ix type
members | 0 | PRIMARY | 1 | memberid | A | 711 | null | null | (blank) | BTREE
members | 1 | members_lname_ix | 1 | lastname | A | 711 | null | null | YES | BTREE
if I add USE INDEX (members_lname_ix) both possible_keys and key will remain null. However if I add FORCE INDEX (members_lname_ix) possible_keys remains null and key shows members_lname_ix. This is my first time trying to apply indexing but to me this doesn't seem very intuitive - it feels like mysql should know that I created an index for lastname, no? I can't quite figure out what I'm doing wrong here unless I am misunderstanding something. Is the solution here to just keep using FORCE INDEX?
There are two ways to perform that query:
Plan A (as you were expecting):
Scan through the index sequentially, reading the entire (estimated) 711 rows.
Randomly look up each row in the data BTree. This involves reading the entire dataset.
Deliver the data in order.
Plan B (what it does):
Scan through the data, reading all 711 rows.
Sort the data
Deliver the sorted data.
Plan B does not touch the index at all; this was deemed to be a bigger savings than not having to sort the data.
In a table as tiny as yours, it would be hard to see a difference in speed. (In my test case, it took under 10 milliseconds either way.) In huge tables, the difference could be significant.
For optimal pagination, see http://mysql.rjweb.org/doc.php/pagination
I am trying to make a order system, but I am stuck right now.
In the mysql tabel right now, I am using varchar(255) in a column named "bestillinger", but it can only store 255 chars. So I searched a bit, and remembered that i could use longtext or just text, but now when i try to do it, i get a error saying:
#1170 - BLOB/TEXT column 'bestilling' used in key specification without a key length
I have tried to search here and in Google, but got no luck with me.
My MySQL tabel is:
CREATE TABLE IF NOT EXISTS `bestillinger` (
`id` int(11) NOT NULL,
`bestilling` TEXT NOT NULL PRIMARY KEY,
`accepted` varchar(255) DEFAULT NULL,
UNIQUE KEY `id_bestilling` (`id`,`bestilling`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
I am using UNIQUE KEY because I am using "ON DUPLICATE KEY UPDATE" in my PHP part. But don't mind that.
Thanks in advance.
you can't set a text as a primary key. Mysql only can Index some characters, and type text could be too big to index.
take a look here:
http://www.mydigitallife.info/mysql-error-1170-42000-blobtext-column-used-in-key-specification-without-a-key-length/
MySQL error: key specification without a key length
From your definition above, and the verbose monologue in this answer below I suggest the following revision:
CREATE TABLE IF NOT EXISTS `bestillinger` (
`id` INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY,
`bestilling` TEXT NOT NULL,
`hashcode` CHAR(32) AS (MD5(bestilling)),
`accepted` VARCHAR(255) DEFAULT NULL,
UNIQUE KEY `id_bestilling` (hashcode,bestilling(333))
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
Specific to MySQL, the default max length for a single-column index key is 1000 bytes long (767 bytes for InnoDB tables, unless you have innodb_large_prefix set). The same length limit applies to any index key prefix (the first N characters for CHAR, VARCHAR, TEXT or first N bytes for BINARY, VARBINARY, BLOB). Note the character versus byte difference; assuming a UTF-8 character set and the maximum of 3 bytes for each character, you might hit this limit with a column prefix index of more than 333 characters on a TEXT or VARCHAR column. In practice, 333 tends to be a good max for me.
Similarly in SQL Server, there is a 900-byte limit for the maximum total size of all index key columns.
But that only uses the first characters of your TEXT as your key, with the obvious collisions imminent.
In the accepted answer, the suggestion is to use a FULLTEXT index. A FULLTEXT index is specially designed for text searching, and it has not-so-good performance for INSERT/DELETE since it maintains an N-gram over the vocabulary of the column records and stores the resulting vector. This would work if every operation were to use text search functions in a where clause...but not for a unique index. There is also both a primary key and a unique key defined on 'id', which seems redundant or I miss the intent.
Instead, I suggest a computed hash. you'd be correct to point out there is a chance (Birthday Paradox) that there will be a collision with a hash, so a UNIQUE index alone isn’t enough. We'll couple it with a column prefix index, as described above, to give us faith in the uniqueness.
I gather the intent of your ID column is to allow proper foreign key referencing from other tables. Quite right, but then that would be the more useful primary key for this table. As well, int(11) refers to the display width of the column, not the underlying value. I put an unsigned auto_increment on 'id' as well, to clarify its role for the larger audience.
And that brings us to the proposed design above.
use this
CREATE TABLE IF NOT EXISTS `bestillinger` (
`id` int(11) NOT NULL,
`bestilling` TEXT NOT NULL PRIMARY KEY,
`accepted` varchar(255) DEFAULT NULL,
UNIQUE KEY `id_bestilling` (`id`,`bestilling`(767))
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
767 is the limit in mysql while dealing with blob/text indexes
Ref : http://dev.mysql.com/doc/refman/5.7/en/innodb-restrictions.html
I think the following example will best explain this problem.
I got the problem because I was trying to set a text field to UNIQUE. I fixed the problem by changing data type of email(TEXT) to email(VARCHAR(254)).
mysql> desc users;
+-------------+-------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+-------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| fname | varchar(50) | NO | | NULL | |
| lname | varchar(50) | NO | | NULL | |
| uname | varchar(20) | NO | | NULL | |
| email | text | NO | | NULL | |
| contact | bigint(12) | NO | | NULL | |
| profile_pic | text | NO | | NULL | |
| password | varchar(20) | NO | | admin | |
+-------------+-------------+------+-----+---------+----------------+
8 rows in set (0.00 sec)
mysql> ALTER TABLE users ADD UNIQUE(email);
ERROR 1170 (42000): BLOB/TEXT column 'email' used in key specification without a key length
mysql>
mysql> ALTER TABLE users MODIFY email VARCHAR(254);
Query OK, 9 rows affected (0.02 sec)
Records: 9 Duplicates: 0 Warnings: 0
mysql> ALTER TABLE users ADD UNIQUE(email);
Query OK, 0 rows affected (0.02 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> desc users;
+-------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| fname | varchar(50) | NO | | NULL | |
| lname | varchar(50) | NO | | NULL | |
| uname | varchar(20) | NO | | NULL | |
| email | varchar(254) | YES | UNI | NULL | |
| contact | bigint(12) | NO | | NULL | |
| profile_pic | text | NO | | NULL | |
| password | varchar(20) | NO | | admin | |
+-------------+--------------+------+-----+---------+----------------+
8 rows in set (0.00 sec)
mysql>
Try to use all column length in varchar. Text should be not allowed because of size.
I got a weird problem of MySQL index. I have a table views_video:
CREATE TABLE `views_video` (
`video_id` smallint(5) unsigned NOT NULL,
`record_date` date NOT NULL,
`region` char(2) NOT NULL DEFAULT '',
`views` mediumint(8) unsigned NOT NULL
PRIMARY KEY (`video_id`,`record_date`,`region`),
KEY `video_id` (`video_id`)
)
The table contains 3.4 million records.
I run the EXPLAIN on this query:
SELECT video_id, views FROM views_video where video_id <= 156
I got:
+----+-------------+-------------+-------+------------------+----------+---------+------+--------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------------+-------+------------------+----------+---------+------+--------+-------------+
| 1 | SIMPLE | views_video | range | PRIMARY,video_id | video_id | 2 | NULL | 587984 | Using where |
+----+-------------+-------------+-------+------------------+----------+---------+------+--------+-------------+
But when I run the EXPLAIN on this query:
SELECT video_id, views FROM views_video where video_id <= 157
I got:
+----+-------------+-------------+------+------------------+------+---------+------+---------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------------+------+------------------+------+---------+------+---------+-------------+
| 1 | SIMPLE | views_video | ALL | PRIMARY,video_id | NULL | NULL | NULL | 3412892 | Using where |
+----+-------------+-------------+------+------------------+------+---------+------+---------+-------------+
video_id is from 1 to 1034. There is nothing special between 156 and 157.
What happens here?
* update *
I have added more data into the database. Now video_id is from 1 to 1064. And the table now has 3.8M records. And the difference become 114 and 115.
I'm guessing that with 3.4 million records, and only 1064 possible entries for your key, your selectivity is very low. (In other words, there are many duplicates, which makes it far less useful as a key.) The optimizer is taking its best guess if it is more efficient to use the key or not. You've found a threshold for that decision.
It might be the key population
Run these
SELECT (COUNT(1)/20) INTO #FivePctOfData FROM views_video;
SELECT COUNT(1) videpidcount,video_id FROM FROM views_video
WHERE id <= 157 GROUP BY video_id;
The query optimizer proabably took a vacation when one one of the key hit the 5% threshold.
You said there are 3.4 million rows. 5% would be 170,000. Perhaps this number was exceeded at some point in the query optimizer's life cycle on your query.
If you've added/deleted substantial data since creating the table, it's worthwhile to try ANALYZE TABLE on it. It frequently solves a lot of phantom indexing issues, and it's very fast even on large tables.
Update: Also, the unique index values are very low compared to the number of rows in the table. MySQL won't use indexes when a single index value points to too many rows. Try constraining the query further with another column that's part of the primary key.
I've been messing around all day trying to find why my query performance is terrible. It is extremely simple, yet can take over 15 minutes to execute (I abort the query at that stage). I am joining a table with over 2 million records.
This is the select:
SELECT
audit.MessageID, alerts.AlertCount
FROM
audit
LEFT JOIN (
SELECT MessageID, COUNT(ID) AS 'AlertCount'
FROM alerts
GROUP BY MessageID
) AS alerts ON alerts.MessageID = audit.MessageID
This is the EXPLAIN
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
| 1 | PRIMARY | AL | index | NULL | IDX_audit_MessageID | 4 | NULL | 2330944 | 100.00 | Using index |
| 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 124140 | 100.00 | |
| 2 | DERIVED | alerts | index | NULL | IDX_alerts_MessageID | 5 | NULL | 124675 | 100.00 | Using index |
This is the schema:
# Not joining, just showing types
CREATE TABLE messages (
ID int NOT NULL AUTO_INCREMENT,
MessageID varchar(255) NOT NULL,
PRIMARY KEY (ID),
INDEX IDX_messages_MessageID (MessageID)
);
# 2,324,931 records
CREATE TABLE audit (
ID int NOT NULL AUTO_INCREMENT,
MessageID int NOT NULL,
LogTimestamp timestamp NOT NULL,
PRIMARY KEY (ID),
INDEX IDX_audit_MessageID (MessageID),
CONSTRAINT FK_audit_MessageID FOREIGN KEY(MessageID) REFERENCES messages(ID)
);
# 124,140
CREATE TABLE alerts (
ID int NOT NULL AUTO_INCREMENT,
AlertLevel int NOT NULL,
Text nvarchar(4096) DEFAULT NULL,
MessageID int DEFAULT 0,
PRIMARY KEY (ID),
INDEX IDX_alert_MessageID (MessageID),
CONSTRAINT FK_alert_MessageID FOREIGN KEY(MessageID) REFERENCES messages(ID)
);
A few very important things to note - the MessageID is not 1:1 in either 'audit' or 'alerts'; The MessageID can exist in one table, but not the other, or may exist in both (which is the purpose of my join); In my test DB, none of the MessageID exist in both. In other words, my query will return 2.3 million records with 0 as the count.
Another thing to note is that the 'audit' and 'alert' tables used to use MessageID as varchar(255). I created the 'messages' table expecting that it would fix the join. It actually made it worse. Previously, it would take 78 seconds, now, it never returns.
What am I missing about MySQL?
Subqueries are very hard for the MySQL engine to optimize. Try:
SELECT
audit.MessageID, COUNT(alerts.ID) AS AlertCount
FROM
audit
LEFT JOIN alerts ON alerts.MessageID = audit.MessageID
GROUP BY audit.MessageID
You're joining to a subquery.
The subquery results are effectively a temporary table - note the <derived2> in the query execution plan. As you can see there, they're not indexed, since they're ephemeral.
You should execute the query as a single unit with a join, rather than joining to the results of a second query.
EDIT: Andrew has posted an answer with one example of how to do your work in a normal join query, instead of in two steps.
I am new to database, and run into a time issue when querying a table where no field is indicated as primary key.
for example
+-----------+----------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+----------+------+-----+---------+-------+
| a_id | char(10) | NO | | | |
| media | char(3) | YES | | NULL | |
| label | char(37) | YES | | NULL | |
As you can see, no of the field have any key specification. When I do a query like
"select label from table where a_id=?", the query is extremely slow. Is this caused by the lack of primary key?
thanks,
No, it is caused by lack of indexes. A primary key must contain a unique and non-null value. If you add an index on the column, it may contain duplicates, but your query will still be faster.
If you want the field to be unique, but you don't want it to be a primary key (although you should wonder why), you can even add a unique index. It will force the value to be unique, but it's no primary key.
That can be convenient when you want the table to have an id and a description that should both be unique. Usually you'll make the id the primary key, and the description just unique.