I am using couchbase:community-6.0.0 in my Spring application. I have like 250.000 records in database. My database query works very fast without using COUNT query command.
SELECT app.*, META(app).id AS id FROM app WHERE ( deleted = FALSE OR
deleted IS MISSING ) AND _class =
“com.myexample.app.device.data.model.DeviceEntity” AND appId =
“something” AND dp.language = “somelanguage” LIMIT 100 OFFSET 0
This query works very well and fast…Response time smaller than 50ms.
However
SELECT COUNT(*) AS count FROM app WHERE ( deleted = FALSE OR deleted
IS MISSING ) AND _class =
“com.myexample.app.device.data.model.DeviceEntity” AND appId =
“something”
It takes 1 minute. I can not reduce.
Indexes
primary
CREATE INDEX class_appId_idx ON app(_class,appId)
CREATE INDEX ix1 ON app(_class,appId,ifmissing(deleted, false))
What is the solution of this ? I think index does not work with a count ? Any advice please, how can I achieve this?
Note : I tried with EE edition, did not work.
The system isn't able to match the index to the query. Sometimes the optimizer isn't all that bright. Try this:
create index ix_test on test(_class, appId) WHERE deleted = FALSE OR deleted IS MISSING
That will use the index.
Generally speaking, because of how we build the indexes, we have trouble with IS MISSING clauses. But putting that bit in the WHERE clause of the index makes it work. But this is a very specialized index. Consider changing your data so the "deleted" field is always present.
It works in miliseconds with using enterprise-6.0.0
Related
I have an app which get the data from PHP backend API.
now there are more and more people online and have chats, the messages loading time need so much time per messages(take over 10s). I would like to know how can I decrease the loading time of the loading of the message.
codes are as following, even I change the data from 20 to 5, although it show 5 messages per time, still same long loading time.
Can you guys help me check this PHP code? can I edit something to fix this issue? Thanks
$stmt = $this->db->prepare("SELECT * FROM messages
WHERE chatId = (:chatId)
AND id < (:msgId)
AND removeAt = 0
ORDER BY id DESC LIMIT 20");
code seems too long and not allow post here..please check in this link: paste.ofcode.org/i6rHbdVjhfscaMVVva9E9L
database table structure:
https://www.imageupload.net/image/msg.OIVDM
https://www.imageupload.net/image/database.OIgHh
You need to learn about indexes. If you set up index on column (or columns) MYSQL can search and sort questies much faster. Try adding few indexes.
I would start with columns removeAt, chatId, fromUserId, toUserId. You can also experiment with complex indexes (removeAt + chatId <-- in this order or fromUserId + toUserId).
After adding indexes run EXPLAIN command on your queries (eg. EXPLAIN SELECT * FROM messages WHERE chatId = 5 AND id < 10 AND removeAt = 0 ORDER BY id DESC LIMIT 20). In results you should see which indexes where possible and which index was used by MySQL.
One thing to note - do not add too many indexes ("just in case") because they will slow down adding new rows to database.
To add index go to table view in phpMyAdmin (structure) and select column on which you ant to add index. Then press index below columns:
You need to add indexes one by one. If you select more than one column it will create single complex index.
INDEX(chatId, removedAt, -- in either order
id) -- third
This index with be "covering" and perfect for the WHERE and ORDER BY, hence able to also handle LIMIT 20.
That is, it will look at only 20 rows, whereas any other index will require looking at all the rows.
More on optimal indexes: http://mysql.rjweb.org/doc.php/index_cookbook_mysql
i have a problem.
Within the time, i get much rows in mysql.
To Filter i want to get some informations.
This is my query
SELECT COUNT(*) AS cases,
SUM(`item_price`) AS preturile
FROM cases
WHERE opened = 1 AND
trade_id = '1234'
It is very slow... needs 1.5 secs or something
If i kick out opened = 1, so it looks like that
SELECT COUNT(*) AS cases,
SUM(`item_price`) AS preturile
FROM cases
WHERE trade_id = '1234'
The speed is fast and good! But i need that opened 1 in there... But why is that so slow?
opened is int(11) and has INDEX.
I dont know what i can do there, its so slow...
be sure you have a proper index eg a composite index on the columns trade_id, opened
create index myidx on cases ( trade_id, opened )
I'm not that well versed in indexing in MySQL, and am having a hard time trying to understand how the EXPLAIN output works, and how it can be read to know if my query is optimised or not.
I have a fairly large table (1.1M records) and I am executing the below query:
SELECT * FROM `Member` this_ WHERE (this_._Temporary_Flag = 0 or this_._Temporary_Flag
is null) and (this_._Deleted = 0 or this_._Deleted is null) and
(this_.Username = 'XXXXXXXX' or this_.Email = 'XXXXXXXX')
ORDER BY this_.Priority asc;
It takes a very long time to execute, between 30 - 60 seconds most of the times. The output of the EXPLAIN query is as below:
id select_type table type possible_keys key key_len ref rows Extra
----------------------------------------------------------------------------------------------------------------------------------
1 SIMPLE this_ ref_or_null _Temporary_Flag,_Deleted,username,email _Temporary_Flag 2 const 33735 Using where; Using filesort
What does this statement exactly mean? Does it mean that this query can be optimised? The table has mostly single-column indexes. What are the important output from the EXPLAIN query which I should use?
It is saying that the index it has chosen to use is the one called _Temporary_Flag (which I assume is on the _Temporary_Flag column). This is not a great index to use (it still leaves it looking at 33k records), but the best it can use in the situation. It might be worth adding an index covering both the _Temporary_Flag and the _Deleted columns.
However I doubt that narrows things down much.
One issue is that MySQL can only use a single index on a table within a query. It is likely that the best indexes to use would be on Username and another on Email, but as your query has an OR there it would have to chose one or the other.
A way round this restriction on indexes is to use 2 queries unioned together, something like this:-
SELECT *
FROM `Member` this_
WHERE (this_._Temporary_Flag = 0
or this_._Temporary_Flag is null)
and (this_._Deleted = 0
or this_._Deleted is null)
and this_.Email = 'XXXXXXXX'
UNION
SELECT *
FROM `Member` this_
WHERE (this_._Temporary_Flag = 0
or this_._Temporary_Flag is null)
and (this_._Deleted = 0
or this_._Deleted is null)
and this_.Username = 'XXXXXXXX'
ORDER BY this_.Priority asc;
http://dev.mysql.com/doc/refman/5.5/en/explain-output.html
Explain tells you what MySQL is doing, it doesn't necessarily tell you or even imply what can be done to make things better.
That said, there are a few warning signs that generally imply that you can optimize a query; the biggest one in this case is the occurrence of Using filesort in the Extra column.
The documentation explains what happens in that case:
MySQL must do an extra pass to find out how to retrieve the rows in
sorted order. The sort is done by going through all rows according to
the join type and storing the sort key and pointer to the row for all
rows that match the WHERE clause. The keys then are sorted and the
rows are retrieved in sorted order.
Another warning sign in your case is the key that is used. While not necessarily true in your case, a well normalized structure will generally require unique values for Username and Email.
So, why does it take so long when you are specifying those two things? Shouldn't the optimizer be able to just go straight to those rows? Probably not, because you are specifying them with an OR, which makes it difficult for the optimizer to use indexes to find those rows.
Instead, the optimizer decided to _Temporary_Flag to look through all the results, which probably didn't narrow the result set much, especially given that the Explain says that approximately 33735 rows were looked at.
So, working on the assumption that email and username will be much more selective than this key, you could try rewriting your query as a UNION.
SELECT * FROM `Member` this_
WHERE
(this_._Temporary_Flag = 0 or this_._Temporary_Flag
is null)
and
(this_._Deleted = 0 or this_._Deleted is null)
and this_.Email = 'XXXXXXXX'
UNION
SELECT * FROM `Member` this_
WHERE (this_._Temporary_Flag = 0 or this_._Temporary_Flag
is null)
and (this_._Deleted = 0 or this_._Deleted is null)
and
this_.Username = 'XXXXXXXX'
ORDER BY this_.Priority asc;
So, those are a couple of warning signs from EXPLAIN: Look for Using filesort and strange key choices as indicators that you can probably improve things.
My SQL Query with all the filters applied is returning 10 lakhs (one million) records . To get all the records it is taking 76.28 seconds .. which is not acceptable . How can I optimize my SQL Query which should take less time.
The Query I am using is :
SELECT cDistName , cTlkName, cGpName, cVlgName ,
cMmbName , dSrvyOn
FROM sspk.villages
LEFT JOIN gps ON nVlgGpID = nGpID
LEFT JOIN TALUKS ON nGpTlkID = nTlkID
left JOIN dists ON nTlkDistID = nDistID
LEFT JOIN HHINFO ON nHLstGpID = nGpID
LEFT JOIN MEMBERS ON nHLstID = nMmbHhiID
LEFT JOIN BNFTSTTS ON nMmbID = nBStsMmbID
LEFT JOIN STATUS ON nBStsSttsID = nSttsID
LEFT JOIN SCHEMES ON nBStsSchID = nSchID
WHERE (
(nMmbGndrID = 1 and nMmbAge between 18 and 60)
or (nMmbGndrID = 2 and nMmbAge between 18 and 55)
)
AND cSttsDesc like 'No, Eligible'
AND DATE_FORMAT(dSrvyOn , '%m-%Y') < DATE_FORMAT('2012-08-01' , '%m-%Y' )
GROUP BY cDistName , cTlkName, cGpName, cVlgName ,
DATE_FORMAT(dSrvyOn , '%m-%Y')
I have searched on the forum and outside and used some of the tips given but it hardly makes any difference . The joins that i have used in above query is left join all on Primary Key and Foreign key . Can any one suggest me how can I modify this sql to get less execution time ....
You are, sir, a very demanding user of MySQL! A million records retrieved from a massively joined result set at the speed you mentioned is 76 microseconds per record. Many would consider this to be acceptable performance. Keep in mind that your client software may be a limiting factor with a result set of that size: it has to consume the enormous result set and do something with it.
That being said, I see a couple of problems.
First, rewrite your query so every column name is qualified by a table name. You'll do this for yourself and the next person who maintains it. You can see at a glance what your WHERE criteria need to do.
Second, consider this search criterion. It requires TWO searches, because of the OR.
WHERE (
(MEMBERS.nMmbGndrID = 1 and MEMBERS.nMmbAge between 18 and 60)
or (MEMBERS.nMmbGndrID = 2 and MEMBERS.nMmbAge between 18 and 55)
)
I'm guessing that these criteria match most of your population -- females 18-60 and males 18-55 (a guess). Can you put the MEMBERS table first in your list of LEFT JOINs? Or can you put a derived column (MEMBERS.working_age = 1 or some such) in your table?
Also try a compound index on (nMmbGndrID,nMmbAge) on MEMBERS to speed this up. It may or may not work.
Third, consider this criterion.
AND DATE_FORMAT(dSrvyOn , '%m-%Y') < DATE_FORMAT('2012-08-01' , '%m-%Y' )
You've applied a function to the dSrvyOn column. This defeats the use of an index for that search. Instead, try this.
AND dSrvyOn >= '2102-08-01'
AND dSrvyOn < '2012-08-01' + INTERVAL 1 MONTH
This will, if you have an index on dSrvyOn, do a range search on that index. My remark also applies to the function in your ORDER BY clause.
Finally, as somebody else mentioned, don't use LIKE to search where = will do. And NEVER use column LIKE '%something%' if you want acceptable performance.
You claim yourself you base your joins on good and unique indexes. So there is little to be optimized. Maybe a few hints:
try to optimize your table layout, maybe you can reduce the number of joins required. That probably brings more performance optimization than anything else.
check your hardware (available memory and things) and the server configuration.
use mysqls explain feature to find bottle necks.
maybe you can create an auxilliary table especially for this query, which is filled by a background process. That way the query itself runs faster, since the work is done before the query in background. That usually works if the query retrieves data that must not neccessarily be synchronous with every single change in the database.
check if an RDBMS is really the right type of database. For many purposes graph databases are much more efficient and offer better performance.
Try adding an index to nMmbGndrID, nMmbAge, and cSttsDesc and see if that helps your queries out.
Additionally you can use the "Explain" command before your select statement to give you some hints on what you might do better. See the MySQL Reference for more details on explain.
If the tables used in joins are least use for updates queries, then you can probably change the engine type from INNODB to MyISAM.
Select queries in MyISAM runs 2x faster then in INNODB, but the updates and insert queries are much slower in MyISAM.
You can create Views in order to avoid long queries and time.
Your like operator could be holding you up -- full-text search with like is not MySQL's strong point.
Consider setting a fulltext index on cSttsDesc (make sure it is a TEXT field first).
ALTER TABLE articles ADD FULLTEXT(cSttsDesc);
SELECT
*
FROM
table_name
WHERE MATCH(cSttsDesc) AGAINST('No, Eligible')
Alternatively, you can set a boolean flag instead of cSttsDesc like 'No, Eligible'.
Source: http://devzone.zend.com/26/using-mysql-full-text-searching/
This SQL has many things that are redundant that may not show up in an explain.
If you require a field, it shouldn't be in a table that's in a LEFT JOIN - left join is for when data might be in the joined table, not when it has to be.
If all the required fields are in the same table, it should be the in your first FROM.
If your text search is predictable (not from user input) and relates to a single known ID, use the ID not the text search (props to Patricia for spotting the LIKE bottleneck).
Your query is hard to read because of the lack of table hinting, but there does seem to be a pattern to your field names.
You require nMmbGndrID and nMmbAge to have a value, but these are probably in MEMBERS, which is 5 left joins down. That's a redundancy.
Remember that you can do a simple join like this:
FROM sspk.villages, gps, TALUKS, dists, HHINFO, MEMBERS [...] WHERE [...] nVlgGpID = nGpID
AND nGpTlkID = nTlkID
AND nTlkDistID = nDistID
AND nHLstGpID = nGpID
AND nHLstID = nMmbHhiID
It looks like cSttsDesc comes from STATUS. But if the text 'No, Eligible' matches exactly one nBStsSttsID in BNFTSTTS then find out the value and use that! If it is 7, take out LEFT JOIN STATUS ON nBStsSttsID = nSttsID and replace AND cSttsDesc like 'No, Eligible' with AND nBStsSttsID = '7'. This would see a massive speed improvement.
I am running the be query
SELECT packages.id, packages.title, subcat.id, packages.weight
FROM packages ,provider, packagestosubcat,
packagestocity, subcat, usertosubcat,
usertocity, usertoprovider
WHERE packages.endDate >'2011-03-11 06:00:00' AND
usertosubcat.userid = 1 AND
usertocity.userid = 1 AND
packages.providerid = provider.id AND
packages.id = packagestosubcat.packageid AND
packages.id = packagestocity.packageid AND
packagestosubcat.subcatid = subcat.id AND
usertosubcat.subcatid = packagestosubcat.subcatid AND
usertocity.cityid = packagestocity.cityid AND
(
provider.providertype = 'reg' OR
(
usertoprovider.userid = 1 AND
provider.providertype != 'reg' AND
usertoprovider.providerid = provider.ID
)
)
GROUP BY packages.title
ORDER BY subcat.id, packages.weight DESC
When i run explain, everything seems to look ok except for the scan on the usertoprovider table, which doesn't seem to be using table's keys:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE usertocity ref user,city user 4 const 4 Using temporary; Using filesort
1 SIMPLE packagestocity ref city,packageid city 4 usertocity.cityid 419
1 SIMPLE packages eq_ref PRIMARY,enddate PRIMARY 4 packagestocity.packageid 1 Using where
1 SIMPLE provider eq_ref PRIMARY,providertype PRIMARY 4 packages.providerid 1 Using where
1 SIMPLE packagestosubcat ref subcatid,packageid packageid 4 packages.id 1 Using where
1 SIMPLE subcat eq_ref PRIMARY PRIMARY 4 packagestosubcat.subcatid 1
1 SIMPLE usertosubcat ref userid,subcatid subcatid 4 const 12 Using where
1 SIMPLE usertoprovider ALL userid,providerid NULL NULL NULL 3735 Using where
As you can see in the above query, the condition itself is:
provider.providertype = 'reg' OR
(
usertoprovider.userid = 1 AND
provider.providertype != 'reg' AND
usertoprovider.providerid = provider.ID
)
Both tables, provider and usertoprovider, are indexed. provider has indexes on providerid and providertype while usertoprovider has indexes on userid and providerid
The cardinality of the keys is:
provider.id=47, provider.type=1, usertoprovider.userid=1245, usertoprovider.providerid=6
So its quite obvious that the indexes are not used.
Further more, to test it out, i went ahead and:
Duplicated the usertoprovider table
Inserted all the provider values that have providertype='reg' into the cloned table
Simplified the condition to (usertoprovider.userid = 1 AND usertoprovider.providerid = provider.ID)
The query execution time changed from 8.1317 sec. to 0.0387 sec.
Still, provider values that have providertype='reg' are valid for all the users and i would like to avoid inserting these values into the usertoprovider table for all the users since this data is redundant.
Can someone please explain why MySQL still runs a full scan and doesn't use the keys? What can be done to avoid it?
It seems that provider.providertype != 'reg' is redundant (always true) unless provider.providertype is nullable and you want the query to fail on NULL.
And shouldn't != be <> instead to be standard SQL, although MySQL may allow !=?
On cost of table scans
It is not necessarily that a full table scan is more expensive than walking an index, because walking an index still requires multiple page accesses. In many database engines, if your table is small enough to fit inside a few pages, and the number of rows are small enough, it will be cheaper to do a table scan. Database engines make this type of decision based on the data and index statistics of the table.
This case
However, in your case, it might also be because of the other leg in your OR clause: provider.providertype = 'reg'. If providertype is "reg", then this query joins in ALL the rows of usertoprovider (most likely not what you want) since it is a multi-table cross join.
The database engine is correct in determining that you'll likely need all the table rows in usertoprovider anyway (unless none of the providertype's is "reg", but the engine also may know!).
The query hides this fact because you are grouping on the (MASSIVE!) result set later on and just returning the package ID, so you won't see how many usertoprovider rows have been returned. But it will run very slowly. Get rid of the GROUP BY clause to find out how many rows you are actually forcing the database engine to work on!!!
The reason you see a massive speed improvement if you fill out the usertoprovider table is because then every row participates in a join, and there is no full cross join happening in the case of "reg". Before, if you have 1,000 rows in usertoprovider, every row with type="reg" expands the result set 1,000 times. Now, that row joins with only one row in usertoprovider, and the result set is not expanded.
If you really want to pass anything with providertype='reg', but not in your many-to-many mapping table, then the easiest way may be to use a sub-query:
Remove usertoprovider from your FROM clause
Do the following:
provider.providertype='reg' OR EXISTS (SELECT * FROM usertoprovider WHERE userid=1 AND providerid = provider.ID)
Another method is to use an OUTER JOIN on the usertoprovider -- any row with "reg" which is not in the table will come back with one row of NULL instead of expanding the result set.
Hmm, I know that MySQL does funny things with grouping. In any other RDBMS, your query won't even be executed. What does that even mean,
SELECT packages.id
[...]
GROUP BY packages.title
ORDER BY subcat.id, packages.weight DESC
You want to group by title. Then in standard SQL syntax, this means you can only select title and aggregate functions of the other columns. MySQL magically tries to execute (and probably guess) what you may have meant to execute. So what would you expect to be selected as packages.id ? The First matching package ID for every title? Or the last? And what would the ORDER BY clause mean with respect to the grouping? How can you order by columns that are not part of the result set (because only packages.title really is)?
There are two solutions, as far as I can see:
You're on the right track with your query, then remove the ORDER BY clause, because I don't think it will affect your result, but it may severely slow down your query.
You have a SQL problem, not a performance problem