I have following table - 'element'
CREATE TABLE `element` (
`eid` bigint(22) NOT NULL AUTO_INCREMENT,
`tag_name` varchar(45) COLLATE utf8_bin DEFAULT NULL,
`text` text COLLATE utf8_bin,
`depth` tinyint(2) DEFAULT NULL,
`classes` tinytext COLLATE utf8_bin,
`webarchiver_uniqueid` int(11) DEFAULT NULL,
`created` datetime DEFAULT NULL,
`updated` datetime DEFAULT NULL,
`rowstatus` char(1) COLLATE utf8_bin DEFAULT 'A',
PRIMARY KEY (`eid`)
) ENGINE=InnoDB AUTO_INCREMENT=12090 DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
Column details and current index details are given above. Almost 90% of queries on this table are like:
select * from element
where tag_name = 'XXX'
and text = 'YYYY'
and depth = 20
and classes = 'ZZZZZ'
and rowstatus = 'A'
What would be the most optimal way to create index on this table? The table has around 60k rows.
Change classes from TINYTEXT to VARCHAR(255) (or some more reasonable size), then have
INDEX(tag_name, depth, classes)
with the columns in any order. I left out rowstatus because it smells like a column that is likely to change. (Anyway, a flag does not add much to an index.)
You can't include TEXT or BLOB columns in an index. And it is not worth it to do a 'prefix' index.
Since a PRIMARY KEY is a UNIQUE key, DROP INDEX eid_UNIQUE.
Is there some reason for picking "binary" / "utf8_bin" for all the character fields?
Related
I have a table that contains all translations of words:
CREATE TABLE `localtexts` (
`Id` int(11) NOT NULL,
`Lang` char(2) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT 'pe',
`Text` varchar(300) DEFAULT NULL,
`ShortText` varchar(100) NOT NULL,
`DbVersion` timestamp NOT NULL DEFAULT current_timestamp(),
`Status` int(11) NOT NULL DEFAULT 1
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
As example there is a table that refers to localtexts:
CREATE TABLE `composes` (
`Status` int(11) NOT NULL DEFAULT 1,
`Id` int(11) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
The table above has foreign key Id to localtexts.Id. And when I need to get word on English I do:
SELECT localtexts.text,
composes.status
FROM composes
LEFT JOIN localtexts ON composes.Id = localtexts.Id
WHERE localtexts.Lang = 'en'.
I'm concerned in performance this decision when there are a lot of tables for join with localtexts.
You might find that adding the following index to the localtexts table would speed up the query:
CREATE INDEX idx ON localtexts (Lang, id, text);
This index covers the WHERE clause, join, and SELECT.
Here is my posts table:
CREATE TABLE `posts` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`date` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
`img` varchar(255) COLLATE utf8_croatian_ci NOT NULL,
`vid` varchar(255) COLLATE utf8_croatian_ci NOT NULL,
`title` varchar(255) COLLATE utf8_croatian_ci NOT NULL,
`subtitle` varchar(255) COLLATE utf8_croatian_ci NOT NULL,
`auth` varchar(54) COLLATE utf8_croatian_ci NOT NULL,
`story` longtext COLLATE utf8_croatian_ci NOT NULL,
`tags` varchar(255) COLLATE utf8_croatian_ci NOT NULL,
`status` varchar(100) COLLATE utf8_croatian_ci NOT NULL,
`moder` varchar(50) COLLATE utf8_croatian_ci NOT NULL,
`rec` varchar(50) COLLATE utf8_croatian_ci NOT NULL,
`pos` varchar(50) COLLATE utf8_croatian_ci NOT NULL,
`inde` int(11) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=117 DEFAULT CHARSET=utf8 COLLATE=utf8_croatian_ci
I want to make two partitions in order to improve query performances.
First partition should contain all non-archive rows.
Second partition - all archive rows.
ALTER TABLE posts
PARTITION BY LIST COLUMNS (status)
(
PARTITION P1 VALUES IN ('admin', 'moder', 'public', 'rec'),
PARTITION P2 VALUES IN ('archive')
);
phpmyadmin error:
Static analysis:
1 errors were found during analysis.
Unrecognized alter operation. (near "" at position 0)
MySQL said:
#1503 - A PRIMARY KEY must include all columns in the table's partitioning function
Any help?
What queries are you trying to speed up? Since the only index you currently have, WHERE id=... or WHERE id BETWEEN ... AND ... are the only queries that will be fast. And the partitioning you suggest will not help much for other queries.
You seem to have only dozens of rows; don't consider partitioning unless you expect to have at least a million rows.
status has only 5 values? Then make it ENUM('archive', 'admin', 'moder', 'public', 'rec') NOT NULL. That will take 1 byte instead of lots.
If you will be querying on date and/or status and/or auth, then let's talk about indexes, especially 'composite' indexes on such. And, to achieve the "archive" split you envision, put status as the first column in the index.
The following throws slow query log.
APN::Notification.
select('apn_notifications.*, devices.device_uid').
joins('INNER JOIN apn_devices ON (apn_notifications.device_id = apn_devices.id) INNER JOIN devices ON (apn_devices.device_id = devices.id)').
where(['apn_notifications.sent_at IS NULL AND apn_notifications.badge > 0 AND devices.customer_id = ? AND devices.device_type IN (?)', customer.id, Object::Device.platform_device_types('ios')])
The output of EXPLAIN
EXPLAIN for: SELECT apn_notifications.*, devices.device_uid FROM `apn_notifications` INNER JOIN apn_devices ON (apn_notifications.device_id = apn_devices.id) INNER JOIN devices ON (apn_devices.device_id = devices.id) WHERE (apn_notifications.disabled_at IS NULL) AND (apn_notifications.sent_at IS NULL AND apn_notifications.badge > 0 AND devices.customer_id = 1 AND devices.device_type IN ('iphone4','ipad','iphone3'))
The Output of 'show create table apn_notifications'
| apn_notifications | CREATE TABLE `apn_notifications` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`device_id` int(11) DEFAULT NULL,
`errors_nb` int(11) DEFAULT NULL,
`device_language` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`sound` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`alert` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`badge` int(11) DEFAULT NULL,
`custom_properties` text COLLATE utf8_unicode_ci,
`sent_at` datetime DEFAULT NULL,
`disabled_at` datetime DEFAULT NULL,
`created_at` datetime DEFAULT NULL,
`updated_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_apn_notifications_on_device_id` (`device_id`)
) ENGINE=InnoDB AUTO_INCREMENT=12984412 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci |
the apn_notifications table has 1.5 million records. So if I try to add index, it is taking much longer time. What is the best way to remove this from slow query?
Also From Mysql 5.6, adding index will not result in any downtime. Am I right?
Often "composite" indexes are better:
apn_notifications: INDEX(device_id, sent_at, badge)
apn_notifications: INDEX(sent_at, badge)
devices: INDEX(customer_id, device_type)
apn_devices is a many-to-many mapping? If so, check that it is following the guidelines here .
I added the following indexes and that reduced lots of time.
add_index :apn_notifications, :sent_at
add_index :apn_notifications, :badge
NOTE: Already indexes were added for foreign keys
Using MySQL 5.6 and the following table structure:
CREATE TABLE `dataitem` (
`AI` int(11) unsigned NOT NULL AUTO_INCREMENT,
`ID` binary(16) NOT NULL,
`OwnerID` binary(16) NOT NULL,
`DataItemTimeUtc` datetime NOT NULL,
`DataItemTimeLocal` datetime NOT NULL,
`DataItemTimeMicroSeconds` int(11) NOT NULL,
`DataItemArrivalTimeUtc` datetime NOT NULL DEFAULT '0001-01-01 00:00:00',
`DataItemTimeTimeZoneID` binary(16) NOT NULL,
`QuestionID` binary(16) NOT NULL,
`QuestionHistoryID` binary(16) DEFAULT NULL,
`QuestionAbsolutePositionID` varchar(1000) COLLATE utf8_unicode_ci DEFAULT NULL,
`GroupSessionIDString` varchar(250) COLLATE utf8_unicode_ci DEFAULT NULL,
`DataItemType` int(11) NOT NULL,
`DataEntryDevice` varchar(250) COLLATE utf8_unicode_ci DEFAULT NULL,
`DataEntryDeviceCradle` varchar(250) COLLATE utf8_unicode_ci DEFAULT NULL,
`DataItemXml` longtext COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`AI`),
UNIQUE KEY `dataitem_ID_UQ_Idx` (`ID`),
KEY `dataitem_OwnerID_Idx` (`OwnerID`),
KEY `dataitem_DataItemTimeUtc_Idx` (`DataItemTimeUtc`),
KEY `dataitem_QuestionID_Idx` (`QuestionID`),
KEY `dataitem_QuestionHistoryID_Idx` (`QuestionHistoryID`),
KEY `dataitem_QuestionAbsolutePositionID_Idx` (`QuestionAbsolutePositionID`(255)),
KEY `dataitem_DataItemType_Idx` (`DataItemType`)
) ENGINE=InnoDB AUTO_INCREMENT=23467 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
I am experiencing something that I am struggling to understand. The following query causes a fatal error because it is taking forever to execute:
Guid patientid = new Guid("cfed2acf-acbd-4ab2-8c23-7ab0b3a8cfa3");
var latestRecord = (from f in QueryHelper.GetEntityTable<DataItem>()
where
f.OwnerID == patientid
&& f.QuestionAbsolutePositionID == "5867FF5EC08B9C0422EFD1359B2802B29A8E167952D381EC70AE53CE6D4C9318"
orderby f.DataItemTimeUtc descending
select f.ID).FirstOrDefault();
However if I change .FirstOrDefault() to .ToArray() the query runs like a flash and returs 2 results. Can someone explain this?
SQL Query generated from .ToArray():
SELECT t0.`ID`
FROM `DataItem` AS t0
WHERE ((t0.`OwnerID` = #p0) AND (t0.`QuestionAbsolutePositionID` = #p1))
ORDER BY t0.`DataItemTimeUtc` DESC
-- p0 = [cfed2acf-acbd-4ab2-8c23-7ab0b3a8cfa3]
-- p1 = [5867FF5EC08B9C0422EFD1359B2802B29A8E167952D381EC70AE53CE6D4C9318]
SQL query generated from .FirstOrDefault():
SELECT t0.`ID`
FROM `DataItem` AS t0
WHERE ((t0.`OwnerID` = #p0) AND (t0.`QuestionAbsolutePositionID` = #p1))
ORDER BY t0.`DataItemTimeUtc` DESC
LIMIT 0, 1
-- p0 = [cfed2acf-acbd-4ab2-8c23-7ab0b3a8cfa3]
-- p1 = [5867FF5EC08B9C0422EFD1359B2802B29A8E167952D381EC70AE53CE6D4C9318]
First, figure out why QuestionAbsolutePositionID needs to be 1000 characters long. It it can be less than 256, make it so. If no, then ask yourself whether it can be changed to CHARACTER SET ascii. It looks like hex, which works fine with ascii. (Rarely do "ids" include accented letters, Cyrillic, Japanese, etc.) If neither of those 'fixes' are possible, can you upgrade to MySQL 5.7?
Once you have fixed the problem of index size (above), add this 'composite' (and 'covering') index; it should speed up the query:
INDEX(OwnerID, QuestionAbsolutePositionID, DataItemTimeUtc, ID)
(The first two columns can be in either order.)
If it does not help, then we need to discuss the #variables.
Have the following table structure:
CREATE TABLE `PARSER_U_R_L` (
`PARSER_ID` varchar(20) COLLATE latin1_general_cs NOT NULL,
`URL_MD5` varchar(255) COLLATE latin1_general_cs NOT NULL,
`ENTRY_POINT_ID` varchar(20) COLLATE latin1_general_cs DEFAULT NULL,
`TYPE_ID` varchar(20) COLLATE latin1_general_cs DEFAULT NULL,
`STATUS_ID` varchar(20) COLLATE latin1_general_cs DEFAULT NULL,
`INDEXED_TIME` datetime DEFAULT NULL,
PRIMARY KEY (`PARSER_ID`,`URL_MD5`),
KEY `PURL_PARSER` (`PARSER_ID`))
ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_general_cs;
As you see PRIMARY KEY in the Parser_id and URL_MD5.
When I try a simple select
EXPLAIN
SELECT *
FROM `PARSER_U_R_L`
WHERE `URL_MD5` IN ( ids )
In EXPLAIN of this select I have possible keys = Null.
What can be problem ?
The problem is that the URL_MD5 is not the first column in your primary key. Since you have an index with multiple columns, the optimizer won't use that index unless you supply a value for the first column as well.
If you supply a value for just the first column, the optimizer will use it,
so try reversing the columns in the index.