MySQL simple fulltext search is getting slower as table size grows.
When I run a query like below using fulltext index, it takes about 90 seconds to execute.
SELECT * FROM project_fulltext_indices WHERE match(search_text) against ('abcdefghijklmnopq') limit 1;
The tables have about 4G rows, and the size is about 9.4GB.
The table mainly contains source code(English).
It used to be much faster when the table is much smaller.
Is there any idea how to improve the performance ?
You can use the mysql indexes.
It is like a placing a bookmark in a book.
Create an index in the project_fulltext_indices
take note. avoid using mysql functions in querying a large data for faster result.
If I am correct mysql indexes doesn't working then mysql function is used.
I created the copy of table by creating the same schema, inserting all the rows, and creating the fullt-text index. The rename the copied table to original table.
After that, the speed of full-text search becomes 50ms from 90seconds.(more than 1000 times faster.)
I also tried to run "OPTIMIZE TABLE project_fulltext_indices", but it takes long time. I waited more than 1 hour, and gave up. And worse, while optimizing the table, the table looks being locked and the running web services stopped working.
Related
We have a large MySQL 5.5 database in which many rows are inserted daily and never deleted or updated. There are also users querying the live database. Tables are MyISAM.
But it is effectively impossible to run ANALYZE TABLES because it takes way too long. And so the query optimizer will often pick the wrong index. (15 hours, and sometimes crashes the tables.)
We want to try switching to all InnoDB. Will we need to run ANALYZE TABLES or not?
The MySQL docs say:
The cardinality (the number of different key values) in every index of a table
is calculated when a table is opened, at SHOW TABLE STATUS and ANALYZE TABLE and
on other circumstances (like when the table has changed too much).
But that begs the question: when is a table opened? If that means accessed during a connection then we need do nothing special. But I do not think that that is the case for InnoDB.
So what is the best approach? Run ANALYZE TABLE periodically? Perhaps with an increased dive count?
Or will it all happen automatically?
The query users use apps to get the data, so each run is a separate connection. They generally do NOT expect the rows to be up-to-date within just minutes.
I have a table in MySQL having 700 million rows. I want to change its storage engine from InnoDB to MyISAM but the SQL query to do so is taking a lot of time to execute.
Below is the query :-
ALTER TABLE table ENGINE=MyISAM
In phpMyAdmin and MySQL workbench this query is getting timed out.
Is this query supposed to take a lot of time, given the fact that I have very large amount of data, if so what I need to do to make it successfully execute?
Note :- I have decided to switch to MyISAM because there will be more reads and very few writes on this table.
Maybe You can try first of all to create new MyISAM empty table and then insert data from another table?
I dont have so big table to test this in that scale, in my biggest table it takes almost the same.
It's important to look at indexes, because rebulding then sometimes takes sooo looonngggg.
I have a query that is running very slowly. The table is was querying has about 100k records and no indexes on most of the columns used in the where clause. I just added indexes on those columns but the query hasn't gotten any faster.
I think this is because when a column is indexed, it's value is written in the index at the time of insertion. I just added the indexes now after all those records were added. So is there a way to "re-run the indexes" on the table?
Edit
Here is the query and explain result:
Oddly enough when I copy the query and run in directly in my SQL manager tool it runs quite fast so may bye the problem is in my application code and not in the query itself.
Mysql keeps consistent indexes. It does not matter if the data is added first, the index is added first, or the data is changed at any time. The same final index will result (assuming the same final data and index type).
Your slow query is not caused by adding the index later. There will be some other reason.
This is an extremely common problem.
Use MySQL explain http://dev.mysql.com/doc/refman/5.0/en/using-explain.html
When you precede a SELECT statement with the keyword EXPLAIN, MySQL displays information from the optimizer about the query execution plan. That is, MySQL explains how it would process the statement, including information about how tables are joined and in which order.
Using these results... verify the index you created is functioning the way you expected.
If not, you will want to tweak your index until you have it working as expected.
You might want to create a new table, create indexes, then insert all elements from old table to new while testing this. It's easier than dropping and re-adding indices a million times.
I have a question when I try to remove over 100,000 rows from a mysql table the server freezes and non of its websites can be accessed anymore!
I waited 2 hours and then restarted the server and restored the account.
I used following query:
DELETE FROM `pligg_links` WHERE `link_id` > 10000
-
SELECT* FROM `pligg_links` WHERE `link_id` > 10000
works perfectly
Is there a better way to do this?
You could delete the rows in smaller sets. A quick script that deletes 1000 rows at a time should see you through.
"Delete from" can be very expensive for large data sets.
I recommend using partitioning.
This may be done slightly differently in PostgreSQL and MySQL, but in PostgreSQL you can create many tables that are "partitions" of the larger table or on a partition. Queries and whatnot can be run on the larger table. This can greatly increase the speed with which you can query given you partition correctly. Also, you can delete a partition by simply dropping it. This happens very very quickly because it is somewhat equivalent to dropping a table.
Documentation for table partitioning can be found here:
http://www.postgresql.org/docs/8.3/static/ddl-partitioning.html
Make sure you have an index on link_id column.
And try to delete with chunks like 10.000 in a time.
Deleting from table is very costy operation.
I have a database that I am trying to index
The index I have created is as follows:
CREATE INDEX <name> ON geoplanet_places(name(15));
When I run the following query:
SELECT * FROM geoplanet_places WHERE name LIKE "vancouver%";
The result is returned in less than 1 seconds
When I run this query (note the aditional '%' wild card):
SELECT * FROM geoplanet_places WHERE name LIKE "%vancouver%";
The result return time is greatly increased, sometimes clocking in at over 9 seconds. This is about the same amount of time it took before the database was indexed.
The database has over 5 million records, I understand why it is slowing down. What i'd like to know is if there is anyway to do the wild card before the name without taken such a huge performance hit in MySQL.
Thanks in advance.
MySQL indexes are created from the leading part of the column - the first query looks for 'vancouver' at the start of the column - entirely within the 15 chars of the index. However the second query looks for 'vancouver' anywhere within the column - there's no guarantee that it will be within the 15 char of the index (and I'd be very surprised if the index would be able to look somewhere other than the start of the indexed string section) - if you looked at the query plan you would probably see a tablescan where the engine is looking at all values in the column sequentially.
It looks a little as though you should investigate MySQL's FULLTEXT index - last time I looked at it it was not good enough to make a search engine, but it might solve your problem (it also looks as if modern MySQL supports FULLTEXT indexes on InnoDB tables as well as the MyISAM tables it was historically restricted to).