I have table with following create statement
CREATE TABLE `table1` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`workname` varchar(25) COLLATE utf8_bin DEFAULT NULL,
`setup_dt` timestamp NULL DEFAULT CURRENT_TIMESTAMP,
`setup_new_dt` timestamp NULL DEFAULT CURRENT_TIMESTAMP,
`check_type` varchar(20) COLLATE utf8_bin DEFAULT NULL,
`work_id` int(21) DEFAULT NULL,
`workstat` varchar(25) COLLATE utf8_bin DEFAULT NULL,
`data` longtext COLLATE utf8_bin,
`a_id` int(20) DEFAULT NULL,
`workstat2` varchar(20) COLLATE utf8_bin DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1031 DEFAULT CHARSET=utf8 COLLATE=utf8_bin
My select query is select * from table
My select query is taking quite a long time to execute.
What changes (such as indexing) can i make in the table to make sure that select queries are no more slow.
Is it taking long because of ``datalongtext COLLATE utf8_bin
Can I do some indexing to resolve this slowness?
When I do explain in front of query I get following output
# id, select_type, table, type, possible_keys, key, key_len, ref, rows, Extra
'1', 'SIMPLE', 'table1', 'ALL', NULL, NULL, NULL, NULL, '8756', NULL
Related
CREATE TABLE `tvnotif` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`pingId` int(11) DEFAULT NULL,
`token` varchar(45) COLLATE utf8_bin DEFAULT NULL,
`summary` varchar(45) COLLATE utf8_bin DEFAULT NULL,
`startTime` int(11) DEFAULT NULL,
`endTime` int(11) DEFAULT NULL,
`processed` int(1) DEFAULT '0',
`created` timestamp NULL DEFAULT CURRENT_TIMESTAMP,
`modified` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `processedIndex` (`processed`),
KEY `summaryIndex` (`summary`),
KEY `tokenIndex` (`token`)
) ENGINE=InnoDB AUTO_INCREMENT=18297898 DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
CREATE TABLE `vv_us` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`athleteid` int(11) DEFAULT NULL,
`token` varchar(45) COLLATE utf8_bin DEFAULT NULL,
`secret` varchar(45) COLLATE utf8_bin DEFAULT NULL,
`active` int(1) DEFAULT '1',
`created` timestamp NULL DEFAULT CURRENT_TIMESTAMP,
`modified` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`type` varchar(45) COLLATE utf8_bin DEFAULT 'mc',
`step` varchar(45) COLLATE utf8_bin DEFAULT NULL,
`host` varchar(45) COLLATE utf8_bin DEFAULT NULL,
`server` mediumblob,
`tempcreds` mediumblob,
PRIMARY KEY (`id`),
KEY `activeIndex` (`active`),
KEY `typeIndex` (`type`)
) ENGINE=InnoDB AUTO_INCREMENT=33888 DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
I am running a query which is mostly based on the table tvnotif which has at least 2 million rows of data in it, all the other tables are having less amount only. I added the index for the tables before that the query took 20 minutes to run and now its taking 160 secs.
EXPLAIN SELECT tvu.secret,COALESCE(php_timezone,"America/Los_Angeles") AS userTz,tn.*,tvu.athleteid,tvu.type FROM tvnotif AS tn
LEFT JOIN vv_us AS tvu ON tvu.token = tn.token
LEFT JOIN tbl_ath_pro AS tap ON tap.athleteid = tvu.athleteid
LEFT JOIN timezones AS tz ON tz.tz_id = tap.tz_id
WHERE tvu.active = 1 AND tn.summary = 'dailies' AND tn.processed = 0
LIMIT 300
The problem is probably your indexes... You have indexes on each field individually. What you need is a composite index on ALL 3 parts as a single index. Without, it can't pick the best one as you have 3 parts of the where clause.
Build a SINGLE index on ( processed, summary, token )
This way the query can jump directly to the processed records, directly to the summary value and then get those records and be done.
Additionally, your VV_US table should have an index on ( token, active ) so the join will be optimized on BOTH parts.
query is simple, as below:
select count(1) from ec_account a join ec_card b on a.id = b.AccountId
there are 2.5 million rows in either ec_account and ec_card.(InnoDB)
here is the execution plan:
execution plan
as you see,
it already added index and used it, but the query still costed almost 60 seconds, is there any way could optimize it except changing database(mariadb has no such choke point as far as i know).
here is table DDL,ec_ccount:
CREATE TABLE `ec_account` (
`Id` varchar(64) NOT NULL,
`AccountType` varchar(32) NOT NULL,
`Name` varchar(32) NOT NULL,
`Status` tinyint(3) unsigned NOT NULL,
`IDCardType` varchar(32) DEFAULT NULL,
`IDCardNo` varchar(64) DEFAULT NULL,
`Password` varchar(256) DEFAULT NULL,
`PasswordHalt` varchar(128) DEFAULT NULL,
`Sex` varchar(8) DEFAULT NULL,
`BirthDay` datetime NOT NULL,
`Mobile` varchar(16) DEFAULT NULL,
`Address` varchar(64) DEFAULT NULL,
`Linkman` varchar(32) DEFAULT NULL,
`LinkmanRelation` varchar(16) DEFAULT NULL,
`LinkmanTel` varchar(16) DEFAULT NULL,
`Remark` varchar(128) DEFAULT NULL,
`Nationality` varchar(32) DEFAULT NULL,
`Nation` varchar(32) DEFAULT NULL,
`MaritalStatus` varchar(8) DEFAULT NULL,
`NativePlace` varchar(64) DEFAULT NULL,
`Occupation` varchar(32) DEFAULT NULL,
`BloodType` varchar(8) DEFAULT NULL,
`Education` varchar(8) DEFAULT NULL,
`LinkmanAddress` varchar(64) DEFAULT NULL,
`HomeAddress` varchar(128) DEFAULT NULL,
`Email` varchar(64) DEFAULT NULL,
`CompanyName` varchar(64) DEFAULT NULL,
`CompanyAddress` varchar(128) DEFAULT NULL,
`CompanyTel` varchar(16) DEFAULT NULL,
`Creator` char(36) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`CreateTime` datetime NOT NULL,
`LastModifier` char(36) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`LastModifyTime` datetime DEFAULT NULL,
`Avatar` longblob,
PRIMARY KEY (`Id`),
KEY `IX_Name` (`Name`) USING HASH,
KEY `Idx_IDCard_Account` (`IDCardType`,`IDCardNo`) USING HASH,
KEY `Idx_Mobile` (`Mobile`) USING HASH,
KEY `Idx_CreateTime` (`CreateTime`) USING BTREE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
and ec_card :
CREATE TABLE `ec_card` (
`Id` char(36) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT '',
`AccountId` varchar(64) NOT NULL,
`CardType` varchar(32) NOT NULL,
`CardNo` varchar(32) NOT NULL,
`Status` tinyint(3) unsigned NOT NULL,
`IsPasswordAuth` tinyint(1) NOT NULL,
PRIMARY KEY (`Id`),
UNIQUE KEY `Idx_Unique_AccountId_CardType` (`AccountId`,`CardType`) USING HASH,
UNIQUE KEY `Idx_Unique_CardType_CardNo` (`CardType`,`CardNo`) USING HASH,
KEY `Idx_Uniques_AccountId` (`AccountId`) USING BTREE,
CONSTRAINT `FK_ec_card_ec_account_AccountId` FOREIGN KEY (`AccountId`) REFERENCES `ec_account` (`Id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Not without fundamentally changing the query.
There are no conditions on your query! It selects all 2.5 million rows from ec_card, as well as every matching row from ec_account. Reading all this data from disk and sending it over the network is the bottleneck; there is no way to change that without changing what the query does.
Here is a workaround for you. I think it would run much faster, and get the same result.
Calculate the total count of ec_account:
SELECT count(1) AS total_count FROM ec_account;
Calculate the amount of records those existed in ec_account but not existed in ec_card:
SELECT count(1) AS missing_count
FROM ec_account a LEFT JOIN ec_card b on a.id = b.AccountId
WHERE b.AccountId IS NULL;
Matched count = total_count - missing_count
The core problem here is that you combined two large table together, it requires a lot of memory and it apparently needs a lot of time to finish.
try it using correlated subquery. This might help:
select count(1) from ec_account a where exists (select * from ec_card b
where b.AccountId=a.id)
Also, other than indexing following strategies generally help:
- Denormalization
- Caching results
- Using a NoSQL database
I'm trying to create a report and running 4 queries, but performance is so terrible.
I'm using 2 tables
This one has 2500 items in it
CREATE TABLE `bolt_accounts` (
`id` int(11) NOT NULL,
`slug` varchar(128) COLLATE utf8_unicode_ci NOT NULL,
`datecreated` datetime NOT NULL,
`datechanged` datetime NOT NULL,
`datepublish` datetime DEFAULT NULL,
`datedepublish` datetime DEFAULT NULL,
`username` varchar(32) COLLATE utf8_unicode_ci DEFAULT '',
`ownerid` int(11) DEFAULT NULL,
`status` varchar(32) COLLATE utf8_unicode_ci NOT NULL,
`templatefields` longtext COLLATE utf8_unicode_ci COMMENT '(DC2Type:json_array)',
`managerid` varchar(128) COLLATE utf8_unicode_ci DEFAULT '',
`parentid` varchar(256) COLLATE utf8_unicode_ci DEFAULT '',
`name` varchar(256) COLLATE utf8_unicode_ci DEFAULT '',
`qualify` varchar(256) COLLATE utf8_unicode_ci DEFAULT '',
`regdate` date DEFAULT NULL,
`city` varchar(256) COLLATE utf8_unicode_ci DEFAULT '',
`phone` varchar(256) COLLATE utf8_unicode_ci DEFAULT '',
`passhash` varchar(256) COLLATE utf8_unicode_ci DEFAULT '',
`cookie` varchar(256) COLLATE utf8_unicode_ci DEFAULT '',
`resettoken` varchar(256) COLLATE utf8_unicode_ci DEFAULT '',
`block` tinyint(1) NOT NULL DEFAULT '0',
`blocksms` tinyint(1) NOT NULL DEFAULT '0',
`birthday` date DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
ALTER TABLE `bolt_accounts`
ADD PRIMARY KEY (`id`),
ADD KEY `IDX_9C703491989D9B62` (`slug`),
ADD KEY `IDX_9C703491AFBA6FD8` (`datecreated`),
ADD KEY `IDX_9C703491BE74E59A` (`datechanged`),
ADD KEY `IDX_9C703491A5131421` (`datepublish`),
ADD KEY `IDX_9C703491B7805520` (`datedepublish`),
ADD KEY `IDX_9C7034917B00651C` (`status`),
ADD KEY `IDX_9C703491C13A5CC2` (`managerid`),
ADD KEY `IDX_9C703491856A684C` (`parentid`(255)),
ADD KEY `IDX_9C7034911E6AC3AE` (`regdate`),
ADD KEY `IDX_9C7034914709B432` (`birthday`);
and another one with all statistics, it has more than 1 400 000 items in it
CREATE TABLE `bolt_statistics` (
`id` int(11) NOT NULL,
`slug` varchar(128) COLLATE utf8_unicode_ci NOT NULL,
`datecreated` datetime NOT NULL,
`datechanged` datetime NOT NULL,
`datepublish` datetime DEFAULT NULL,
`datedepublish` datetime DEFAULT NULL,
`username` varchar(32) COLLATE utf8_unicode_ci DEFAULT '',
`ownerid` int(11) DEFAULT NULL,
`status` varchar(32) COLLATE utf8_unicode_ci NOT NULL,
`templatefields` longtext COLLATE utf8_unicode_ci COMMENT '(DC2Type:json_array)',
`managerid` varchar(256) COLLATE utf8_unicode_ci DEFAULT '',
`statdate` datetime DEFAULT NULL,
`lopv` double NOT NULL DEFAULT '0',
`gope` double NOT NULL DEFAULT '0',
`gopv` double NOT NULL DEFAULT '0'
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
ALTER TABLE `bolt_statistics`
ADD PRIMARY KEY (`id`),
ADD KEY `IDX_BE38DFD2989D9B62` (`slug`),
ADD KEY `IDX_BE38DFD2AFBA6FD8` (`datecreated`),
ADD KEY `IDX_BE38DFD2BE74E59A` (`datechanged`),
ADD KEY `IDX_BE38DFD2A5131421` (`datepublish`),
ADD KEY `IDX_BE38DFD2B7805520` (`datedepublish`),
ADD KEY `IDX_BE38DFD27B00651C` (`status`),
ADD KEY `IDX_BE38DFD2C13A5CC2` (`managerid`(255));
So the problem is, when I join this tables together, performance become low...
SELECT ba.managerid,name,replace(phone,'+','') as phone,passhash, date_format(ba.datepublish,'%d.%m.%Y %H:%i') as datepublish, max(bs.lopv) as lopv, max(bs.gopv) as gopv
FROM bolt_accounts ba
LEFT JOIN bolt_statistics bs ON ba.managerid=bs.managerid
WHERE (parentid='007-645930')
AND (date(ba.datechanged)=('2018-01-06'))
AND (date(bs.datecreated)=('2018-01-06'))
GROUP BY ba.managerid
ORDER BY gopv desc
this query will run for 360-450ms ~0,3 of a sec..
It will return all managerids that has parentid=007-645930
some thing like that:
managerid
007-663360
007-677590
007-697191
007-1526400
007-1155884
007-1842169
077-1564660
007-1883072
007-777143
007-1865946
007-1875083
007-1753407
007-1322124
007-1100631
007-1603795
007-1171656
007-1890892
007-1166247
007-1564611
007-1882959
007-1145375
007-1878383
007-1128857
007-1762655
007-1346877
007-1714252
007-1709538
007-1319044
007-1698517
007-1316756
007-1679094
007-1298984
007-1905146
007-1675451
007-1287166
007-1899632
007-1629224
007-1190862
007-1894824
007-1616741
007-1171665
007-1894330
Than I take 1 id from that list, and run 3 queries
SELECT max(s.lopv) as lopv, max(s.gopv) as gopv
FROM bolt_statistics s WHERE (managerid='007-663360')
AND (datecreated between DATE_FORMAT('2018-01-06' - INTERVAL 1 MONTH,'%Y-%m-28 23:00:00') and DATE_FORMAT(LAST_DAY('2018-01-06' - INTERVAL 1 MONTH),'%Y-%m-%d 23:59:59'))
execution time 20-25ms
SELECT max(s.lopv) as lopv, max(s.gopv) as gopv
FROM bolt_statistics s
WHERE (managerid='007-663360')
AND (date(datecreated) = date('2018-01-06' -INTERVAL 1 day))
execution time 15-20ms
SELECT max(s.lopv) as lopv, max(s.gopv) as gopv
FROM bolt_statistics s
WHERE (managerid='007-663360')
AND (date(datecreated) = date('2018-01-06' -INTERVAL 2 day))
execution time 15-20ms
When all executions are over, it took 1,5 sec (1500ms) to render the php report.
I know, that I'm not quite good at mysql querying ;)) but I wonder, how can I improve performance on that queries?
Will it be much faster if I union all this queries in 1?
Do those fields really need a full 256 characters? Change them to a reasonable number, then get rid of the prefixing on ADD KEY IDX_BE38DFD2C13A5CC2 (managerid(255)), etc. (Prefix indexes are often useless.)
Don't hid columns inside functions (date(ba.datechanged)). Instead:
AND ba.datechanged >= '2018-01-06' - INTERVAL 2 DAY
AND ba.datechanged < '2018-01-06' - INTERVAL 3 DAY
Note: The above pattern works fine regardless of what datatype datechanged is -- DATE, DATETIME, DATETIME(6), TIMESTAMP. And the Optimizer can make use of an index such as ...
After that, have the composite INDEX(managerid, datecreated) for significant performance improvement.
Use a derived table instead of LEFT JOIN plus GROUP BY. This is likely to improve speed a bunch.
What is status? Why VARCHAR(32)? If it is a simple, low-cardinality, value, don't index it by itself; the index won't be used.
(There may be more tips, but this should get you started.)
Here is my posts table:
CREATE TABLE `posts` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`date` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
`img` varchar(255) COLLATE utf8_croatian_ci NOT NULL,
`vid` varchar(255) COLLATE utf8_croatian_ci NOT NULL,
`title` varchar(255) COLLATE utf8_croatian_ci NOT NULL,
`subtitle` varchar(255) COLLATE utf8_croatian_ci NOT NULL,
`auth` varchar(54) COLLATE utf8_croatian_ci NOT NULL,
`story` longtext COLLATE utf8_croatian_ci NOT NULL,
`tags` varchar(255) COLLATE utf8_croatian_ci NOT NULL,
`status` varchar(100) COLLATE utf8_croatian_ci NOT NULL,
`moder` varchar(50) COLLATE utf8_croatian_ci NOT NULL,
`rec` varchar(50) COLLATE utf8_croatian_ci NOT NULL,
`pos` varchar(50) COLLATE utf8_croatian_ci NOT NULL,
`inde` int(11) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=117 DEFAULT CHARSET=utf8 COLLATE=utf8_croatian_ci
I want to make two partitions in order to improve query performances.
First partition should contain all non-archive rows.
Second partition - all archive rows.
ALTER TABLE posts
PARTITION BY LIST COLUMNS (status)
(
PARTITION P1 VALUES IN ('admin', 'moder', 'public', 'rec'),
PARTITION P2 VALUES IN ('archive')
);
phpmyadmin error:
Static analysis:
1 errors were found during analysis.
Unrecognized alter operation. (near "" at position 0)
MySQL said:
#1503 - A PRIMARY KEY must include all columns in the table's partitioning function
Any help?
What queries are you trying to speed up? Since the only index you currently have, WHERE id=... or WHERE id BETWEEN ... AND ... are the only queries that will be fast. And the partitioning you suggest will not help much for other queries.
You seem to have only dozens of rows; don't consider partitioning unless you expect to have at least a million rows.
status has only 5 values? Then make it ENUM('archive', 'admin', 'moder', 'public', 'rec') NOT NULL. That will take 1 byte instead of lots.
If you will be querying on date and/or status and/or auth, then let's talk about indexes, especially 'composite' indexes on such. And, to achieve the "archive" split you envision, put status as the first column in the index.
I am using MySQL database. I have a table that consist up to 75K rows of data. I am using simple query to fetch data:
select * from mytable
It works fine, shows 75k rows in few seconds. I wanted to fetch some of these data I had used limit 10000. It got stuck every time. I need to optimize MySQL query for 10k records.
I am using query like this:
select * from mytable limit 10000
Give me some solution how to execute my query fast.
my database structure is like that:
CREATE TABLE IF NOT EXISTS `mytable` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`col1` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`col2` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`col3` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`col4` varchar(255) COLLATE utf8_unicode_ci NOT NULL DEFAULT '',
`col5` int(11) NOT NULL,
`col6` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`col7` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`col8` int(11) NOT NULL,
`col9` int(11) NOT NULL,
`col11` int(11) NOT NULL,
`col12` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`col13` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`col15` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`col16` enum('0','1') COLLATE utf8_unicode_ci NOT NULL DEFAULT '0',
`col17` int(11) NOT NULL,
`col18` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`col19` enum('0','1') COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`id`),
KEY `id` (`id`,`col2`,`col3`,`col4`,`col5`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=75530 ;
With this solution I can fetch larger data in few seconds:
SELECT l.id,l.col1,l.col2,l.col3,l.col3,l.col4,l.col5,l.col6,l.col7
FROM (
SELECT id
FROM mytable
WHERE removed='0'
ORDER BY id
LIMIT 10000
) o
JOIN mytable ON l.id = o.id
ORDER BY l.id