MySQL inconsistent results with Date() select - mysql

What's going on here? BTW, MySQL Server version: 5.0.45-log Source distribution.
mysql> select count(*)
from notes
where date(updated_at) > date('2010-03-25');
+----------+
| count(*) |
+----------+
| 0 |
+----------+
1 row in set (0.59 sec)
mysql> select count(*)
from notes
where message like'%***%'
and date(updated_at) > date('2010-03-25');
+----------+
| count(*) |
+----------+
| 26 |
+----------+
1 row in set (1.30 sec)
mysql> explain select count(*)
from notes
where date(updated_at) > date('2010-03-25');
+----+-------------+-------+------+---------------+------+---------+------+--------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+---------------+------+---------+------+--------+-------------+
| 1 | SIMPLE | notes | ALL | NULL | NULL | NULL | NULL | 588106 | Using where |
+----+-------------+-------+------+---------------+------+---------+------+--------+-------------+
1 row in set (0.07 sec)
mysql> explain select updated_at
from notes
where message like'%***%'
and date(updated_at) > date('2010-03-25');
+----+-------------+-------+------+---------------+------+---------+------+--------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+---------------+------+---------+------+--------+-------------+
| 1 | SIMPLE | notes | ALL | NULL | NULL | NULL | NULL | 588106 | Using where |
+----+-------------+-------+------+---------------+------+---------+------+--------+-------------+
1 row in set (0.09 sec)
mysql>
Here's the table schema.
CREATE TABLE `notes` (
`id` int(11) NOT NULL auto_increment,
`status` varchar(255) default NULL,
`message` text,
`noteable_id` int(11) default NULL,
`noteable_type` varchar(255) default NULL,
`deleted_at` datetime default NULL,
`creator_id` int(11) default NULL,
`updater_id` int(11) default NULL,
`deleter_id` int(11) default NULL,
`created_at` datetime default NULL,
`updated_at` datetime default NULL,
`public` tinyint(1) default '0',
`forced` tinyint(1) default '0',
`agent_created_at` datetime default NULL,
PRIMARY KEY (`id`),
KEY `noteable_id` (`noteable_id`),
KEY `deleted_at` (`deleted_at`),
KEY `noteable_type` (`noteable_type`(10)),
KEY `creator_id` (`creator_id`),
KEY `status` (`status`),
KEY `created_at` (`created_at`)
) ENGINE=InnoDB AUTO_INCREMENT=613168 DEFAULT CHARSET=latin1

If the table is small, try dumping & reloading on a fresh server on another box (with the same version). If the problem goes away, there is some internal corruption and you will need to either reload the table on the existing server, or reinit the entire database from a dump.
If the behaviour is reproducible on a clean database and nobody can explain it (after you post the schema etc), then raise a bug.

It turns out, this particular instance was not caused by database corruption, but a bug in the Date function for MySQL version 5.0.45 (+?).
http://bugs.mysql.com/bug.php?id=32159
We don't see a need to react to this situation immediately, but we will be migrating to a higher version of MySQL (either 5.0.77+ or 5.1)

Related

indexes and constraints not created in CREATE TABLE statement

I am using MySQL/MariaDB, and I create an employees table:
CREATE TABLE employees(
id INT AUTO_INCREMENT,
name VARCHAR(40) NOT NULL,
description VARCHAR(50) DEFAULT 'No Description',
random_assignment_id INT UNIQUE,
birth_date DATE,
salary DECIMAL(5,2),
supervisor_id INT,
branch_id INT NOT NULL,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT random_assignment_check CHECK (LENGTH(random_assignment_id) = 5),
INDEX(random_assignment_id, supervisor_id, branch_id),
PRIMARY KEY(id)
)
Then I confirm the table is created as expected:
SHOW CREATE TABLE employees\G;
*************************** 1. row ***************************
Table: employees
Create Table: CREATE TABLE `employees` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(40) NOT NULL,
`description` varchar(50) DEFAULT 'No Description',
`random_assignment_id` int(11) DEFAULT NULL,
`birth_date` date DEFAULT NULL,
`salary` decimal(5,2) DEFAULT NULL,
`supervisor_id` int(11) DEFAULT NULL,
`branch_id` int(11) NOT NULL,
`created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`updated_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
UNIQUE KEY `random_assignment_id` (`random_assignment_id`),
KEY `random_assignment_id_2` (`random_assignment_id`,`supervisor_id`,`branch_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
1 row in set (0.01 sec)
I don't see the random_assignment_check constraint listed, and I expected it to index random_assignment_id, supervisor_id and branch_id, but it does not:
DESCRIBE employees;
+----------------------+--------------+------+-----+-------------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+----------------------+--------------+------+-----+-------------------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| name | varchar(40) | NO | | NULL | |
| description | varchar(50) | YES | | No Description | |
| random_assignment_id | int(11) | YES | UNI | NULL | |
| birth_date | date | YES | | NULL | |
| salary | decimal(5,2) | YES | | NULL | |
| supervisor_id | int(11) | YES | | NULL | |
| branch_id | int(11) | NO | | NULL | |
| created_at | timestamp | NO | | CURRENT_TIMESTAMP | |
| updated_at | timestamp | NO | | CURRENT_TIMESTAMP | |
+----------------------+--------------+------+-----+-------------------+----------------+
There are no MUL flags under key.
Note that I read MariaDB now supports Constraints; according to homebrew, I am using:
brew info mariadb
mariadb: stable 10.3.12 (bottled)
Drop-in replacement for MySQL
What am I doing wrong?
MySQL treats KEY as a synonym for INDEX. Your index INDEX(random_assignment_id, supervisor_id, branch_id) became KEY random_assignment_id_2 (random_assignment_id,supervisor_id,branch_id). The index name was generated by MySQL, but logically they're the same index.
When I tested your CREATE TABLE statement and then use DESC to display it, I also see no MUL indicators. This fits the documentation:
If Key is MUL, the column is the first column of a nonunique index in which multiple occurrences of a given value are permitted within the column.
So in your output, the Key field for random_assignment_id is UNI because it's a unique key, in addition to part of a multi-column key.
MySQL doesn't support CHECK constraints. It parses them, then ignores them. They are not stored with your table, and subsequently using SHOW CREATE TABLE doesn't show them.
MariaDB has implemented CHECK constraints in 10.2.1 according to https://mariadb.com/kb/en/library/constraint/#check-constraints (I don't use MariaDB, so I'll trust their doc).
It's not clear from your question if you tested your CHECK constraint on MySQL Community Edition or MariaDB. CHECK constraint will not be saved on MySQL, and on MariaDB it seems okay per the doc, but I have never tested it.
We should all stop thinking of MariaDB a drop-in replacement for MySQL. The two products have been diverging for nearly 10 years, and they can no longer be assumed to be compatible.

mysql indexing makes group by slow

Please refer the table strcuture below.
CREATE TABLE `oarc` (
`ID` bigint(20) NOT NULL AUTO_INCREMENT,
`zID` int(11) NOT NULL,
`cID` int(11) NOT NULL,
`bID` int(11) NOT NULL,
`rtype` char(1) COLLATE utf8_unicode_ci NOT NULL,
`created` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
PRIMARY KEY (`ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=1821039 ;
Other than the PRIMARY KEY, I have not set any index on this, and when I run the following query
select COUNT(oarc.ID) as total
from `oarc` where`oarc`.`rtype` = 'v'
group
by `oarc`.`zID`
I am getting the result in less than 1 second. But if I add index to zID it is taking more than 5 seconds.
Please see below explain result :
id | select_type | table | type | possible_keys | key | key_len | ref | row | Extra
--------------------------------------------------------------------------------------------------------
1 | SIMPLE | oarc | index | NULL | zone_ID | 4 | NULL | 1909387 | Using where
Currently the table have more than 1821039 records in it and it will increase on a hourly basis. What are the things I need to do in order to reduce the query execution time. I am expecting only something at the table and query level, nothing on my.cnf or server side because I can not do anything there.
Thanks in advance.
Is this better?
CREATE TABLE `oarc` (
`ID` bigint(20) NOT NULL AUTO_INCREMENT,
`zID` int(11) NOT NULL,
`cID` int(11) NOT NULL,
`bID` int(11) NOT NULL,
`rtype` char(1) COLLATE utf8_unicode_ci NOT NULL,
`created` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
PRIMARY KEY (`ID`),
KEY(rtype,zid)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=1821039 ;
explain
select COUNT(oarc.ID) as total
from `oarc` where`oarc`.`rtype` = 'v'
group
by `oarc`.`zID`
+----+-------------+-------+------+---------------+-------+---------+-------+------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+---------------+-------+---------+-------+------+--------------------------+
| 1 | SIMPLE | oarc | ref | rtype | rtype | 3 | const | 1 | Using where; Using index |
+----+-------------+-------+------+---------------+-------+---------+-------+------+--------------------------+

Query perfoming full table scan

My query is not using any indexes. Query performing full table scan. What can I do to avoid this?
explain select * from
timed_delivery_messages
where start_time <= '06:39'
and end_time > '06:39'
and mode='Active'
and rotation_weight like '%,45,%'
and substr(day_of_week, 2, 1) = 'T'
limit 1;
Explain Plan
+----+-------------+-------------------------+------+---------------+------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------------------------+------+---------------+------+---------+------+------+-------------+
| 1 | SIMPLE | timed_delivery_messages | ALL | NULL | NULL | NULL | NULL | 22 | Using where |
+----+-------------+-------------------------+------+---------------+------+---------+------+------+-------------+
1 row in set (0.00 sec)
Tables:
mysql> show create table timed_delivery_messages\G
*************************** 1. row ***************************
Table: timed_delivery_messages
Create Table: CREATE TABLE `timed_delivery_messages` (
`row_create` datetime DEFAULT NULL,
`row_mod` datetime DEFAULT NULL,
`rule_id` int(11) NOT NULL,
`start_time` time DEFAULT NULL,
`end_time` time DEFAULT NULL,
`day_of_week` varchar(7) DEFAULT NULL,
`rotation_weight` varchar(50) DEFAULT NULL,
`mode` varchar(10) DEFAULT 'active',
`long_message` varchar(256) DEFAULT NULL,
`short_message` varchar(256) DEFAULT NULL,
PRIMARY KEY (`rule_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
1 row in set (0.00 sec)
You must create one group index with columns
start_time,
end_time,
mode
And also do experiment to include day_of_week into this index. Maybe it will speed up your system

How can I optimize a Mysql query that searches for rows in a certain date range

Here is the query:
select timespans.id as timespan_id, count(*) as num
from reports, timespans
where timespans.after_date >= '2011-04-13 22:08:38' and
timespans.after_date <= reports.authored_at and
reports.authored_at < timespans.before_date
group by timespans.id;
Here are the table defs:
CREATE TABLE `reports` (
`id` int(11) NOT NULL auto_increment,
`source_id` int(11) default NULL,
`url` varchar(255) default NULL,
`lat` decimal(20,15) default NULL,
`lng` decimal(20,15) default NULL,
`content` text,
`notes` text,
`authored_at` datetime default NULL,
`created_at` datetime default NULL,
`updated_at` datetime default NULL,
`data` text,
`title` varchar(255) default NULL,
`author_id` int(11) default NULL,
`orig_id` varchar(255) default NULL,
PRIMARY KEY (`id`),
KEY `index_reports_on_title` (`title`),
KEY `index_content_on_reports` (`content`(128))
CREATE TABLE `timespans` (
`id` int(11) NOT NULL auto_increment,
`after_date` datetime default NULL,
`before_date` datetime default NULL,
`after_offset` int(11) default NULL,
`before_offset` int(11) default NULL,
`is_common` tinyint(1) default NULL,
`created_at` datetime default NULL,
`updated_at` datetime default NULL,
`is_search_chunk` tinyint(1) default NULL,
`is_day` tinyint(1) default NULL,
PRIMARY KEY (`id`),
KEY `index_timespans_on_after_date` (`after_date`),
KEY `index_timespans_on_before_date` (`before_date`)
And here is the explain:
+----+-------------+-----------+-------+--------------------------------------------------------------+-------------------------------+---------+------+--------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-----------+-------+--------------------------------------------------------------+-------------------------------+---------+------+--------+----------------------------------------------+
| 1 | SIMPLE | timespans | range | index_timespans_on_after_date,index_timespans_on_before_date | index_timespans_on_after_date | 9 | NULL | 84 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | reports | ALL | NULL | NULL | NULL | NULL | 183297 | Using where |
+----+-------------+-----------+-------+--------------------------------------------------------------+-------------------------------+---------+------+--------+----------------------------------------------+
And here is the explain after I create an index on authored_at. As you can see, the index is not actually getting used (I think...)
+----+-------------+-----------+-------+--------------------------------------------------------------+-------------------------------+---------+------+--------+------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-----------+-------+--------------------------------------------------------------+-------------------------------+---------+------+--------+------------------------------------------------+
| 1 | SIMPLE | timespans | range | index_timespans_on_after_date,index_timespans_on_before_date | index_timespans_on_after_date | 9 | NULL | 86 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | reports | ALL | index_reports_on_authored_at | NULL | NULL | NULL | 183317 | Range checked for each record (index map: 0x8) |
+----+-------------+-----------+-------+--------------------------------------------------------------+-------------------------------+---------+------+--------+------------------------------------------------+
There are about 142k rows in the reports table, and far fewer in the timespans table.
The query is taking about 3 seconds now.
The strange thing is that if I add an index on reports.authored_at, it actually makes the query far slower, about 20 seconds. I would have thought it would do the opposite, since it would make it easy to find the reports at either end of the range, and throw the rest away, rather than having to examine all of them.
Can someone clarify? I'm stumped.
Instead of two separate indexes for the timespan table, try merging them into a single multi-column index with before_date and after_date in a single index. Then add that index to authored_at as well.
i rewrite you query like this:
select t.id, count(*) as num from timespans t
join reports r where t.after_date >= '2011-04-13 22:08:38'
and r.authored_at >= '2011-04-13 22:08:38'
and r.authored_at < t.before_date
group by t.id order by null;
and change indexes of tables
alter table reports add index authored_at_idx(authored_at);
You can used partition feature of database on column after_date. It will help u a lot.

MySQL structure help for joins ( large tables)

I currently have 2 tables that are used for a select query with a simple join. The first table houses around 6-9 million rows, and this gets used as the join. The primary table is anywhere from 1mil to 300mil rows. However, I notice when I join above 10mil rows on the primary table the select query goes from instant to very slow (3+ seconds and grows).
Here is my table structure and queries.
CREATE TABLE IF NOT EXISTS `links` (
`link_id` int(10) unsigned NOT NULL,
`domain_id` mediumint(7) unsigned NOT NULL,
`parent_id` int(11) unsigned DEFAULT NULL,
`hash` int(10) unsigned NOT NULL,
`url` text NOT NULL,
`type` enum('html','pdf') DEFAULT NULL,
`processed` enum('N','Y') NOT NULL DEFAULT 'N',
UNIQUE KEY `hash` (`hash`),
KEY `idx_processed` (`processed`),
KEY `domain_id` (`domain_id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 ROW_FORMAT=COMPACT;
CREATE TABLE IF NOT EXISTS `domains` (
`domain_id` mediumint(7) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(170) NOT NULL,
`blocked` enum('N','Y') NOT NULL DEFAULT 'N',
`count` mediumint(6) NOT NULL DEFAULT '0',
`mcount` mediumint(3) NOT NULL,
PRIMARY KEY (`domain_id`),
KEY `name` (`name`),
KEY `blocked` (`blocked`),
KEY `mcount` (`mcount`),
KEY `count` (`count`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=10834389 ;
Query:
(SELECT link_id, url, hash FROM links, domains WHERE links.domain_id = domains.domain_id and mcount > 1 and processed='N' limit 200)
UNION
(SELECT link_id, url, hash FROM links where processed='N' and type='html' limit 200)
Explain select:
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------+------------+-------+-------------------------+--------------- +---------+---------------------------+---------+-------------+
| 1 | PRIMARY | domains | range | PRIMARY,mcount | mcount | 3 | NULL | 257673 | Using where |
| 1 | PRIMARY | links | ref | idx_processed,domain_id | domain_id | 3 | crawler.domains.domain_id | 1 | Using where |
| 2 | UNION | links | ref | idx_processed | idx_processed | 1 | const | 7090017 | Using where |
| NULL | UNION RESULT | <union1,2> | ALL | NULL | NULL | NULL | NULL | NULL | |
+----+--------------+------------+-------+-------------------------+---------------+---------+---------------------------+---------+-------------+
Right now, I'm trying a partition with 20 partitions on links using domain_id as the key.
Any other options would be greatly appreciated.
A single SELECT statement would replace your entire UNION statement:
SELECT link_id, url, hash
FROM links, domains
WHERE links.domain_id = domains.domain_id
AND mcount > 1
AND processed='N'
AND type='html'
This may not be THE answer you are looking for, but it should help you simplify your question.
When things suddenly slow down you might want to check the size of your indexes (used in the query execution) vs size of various mysql buffers.