I have a site in shared plan with a mysql database. The database has a table with ~300000 rows. The table is ~250mb. In every page I call query:
select * from table order by added limit 0,30
In every row is field with 400 characters code which I need. Basically I need all fields.
Until few days ago everything is ok but slow with 500 visitors/day. Now my site is down because I have an alert about cpu abuse(with 1000 visitors/day). In my local server all goes very well with no big cpu usage (~10%).
What can I do to make best performance for my queries?
If I go to VPS plan everything will be ok or the real problem is my table?
Is "added" field indexed? MySQL will scan whole table every time if the table is not indexed by the field in ORDER BY. By indexing your table
ALTER TABLE `table_name` ADD INDEX `added` (`added`);
you'll dramatically reduce cpu usage.
Related
MySQL simple fulltext search is getting slower as table size grows.
When I run a query like below using fulltext index, it takes about 90 seconds to execute.
SELECT * FROM project_fulltext_indices WHERE match(search_text) against ('abcdefghijklmnopq') limit 1;
The tables have about 4G rows, and the size is about 9.4GB.
The table mainly contains source code(English).
It used to be much faster when the table is much smaller.
Is there any idea how to improve the performance ?
You can use the mysql indexes.
It is like a placing a bookmark in a book.
Create an index in the project_fulltext_indices
take note. avoid using mysql functions in querying a large data for faster result.
If I am correct mysql indexes doesn't working then mysql function is used.
I created the copy of table by creating the same schema, inserting all the rows, and creating the fullt-text index. The rename the copied table to original table.
After that, the speed of full-text search becomes 50ms from 90seconds.(more than 1000 times faster.)
I also tried to run "OPTIMIZE TABLE project_fulltext_indices", but it takes long time. I waited more than 1 hour, and gave up. And worse, while optimizing the table, the table looks being locked and the running web services stopped working.
I am running MariaDB on a vServer (8 CPU vCores, 32 GB RAM) with a few dozen database tables which aggregate data from external services around the web for efficient use across my collection of websites (basically an API layer with database caching and it's own API for easy use in all of my projects).
All but one of these database tables allow quick, basic queries such as
SELECT id, content FROM tablename WHERE date_added > somedate
(with "content" being some JSON data). I am using InnoDB as the storage engine to allow inserts without table locking, "id" is always the primary key in any table and most of these tables only have a few thousand or maybe a few hundred thousand entries, resulting in a few hundred MB.
One table where things don't work properly though has already >6 million entries (potentially heading to 100 million) and uses >60 GB including indexes. I can insert, update and select by "id" but anything more complex (e.g. involving a search in 1 or 2 additional fields or sorting the results) runs into infinity. Example:
SELECT id FROM tablename WHERE extra = ''
This query would select entries where "extra" is empty. There is an index on "extra" and
EXPLAIN SELECT id FROM tablename WHERE extra = ''
tells me it is just a SIMPLE query with the correct index automatically chosen ("Using where; Using index"). If I set a low LIMIT I am fine, selecting thousands of results though and the query never stops running. Using more than 1 field in my search even with a combined index and explicitly adding the name of that index to the query and I'm out of luck as well.
Since there is more than enough storage available on my vServer and MariaDB/InnoDB don't have such low size limits for tables there might be some settings or other limitations that would prevent me from running queries on larger database tables. Looking through all the settings of MariaDB I couldn't find anything appropriate though.
Would be glad if someone could point me into the right direction.
I have been asked to diagnose why a query looking something like this
SELECT COUNT(*) AS count
FROM users
WHERE first_digit BETWEEN 500 AND 1500
AND second_digit BETWEEN 5000 AND 45000;
went from taking around 0.3 seconds to execute suddenly is taking over 3 seconds. The system is MySQL running on Ubuntu.
The table is not sorted and contains about 1.5M rows. After I added a composite index I got the execution time down to about 0.2 seconds again, however this does not explain the root cause why all of a sudden the execution time increased exponentially.
How can I begin to investigate the cause of this?
Since your SQL query has not changed, and I interpret your description as the data set has not changed/grown - I suggest you take a look at the following areas, in order:
1) Have your removed the index and run your SQL query again?
2) Other access to the database. Are other applications or users running heavy queries on the same database? Larger data transfers, in particular to and from the database server in question.
A factor of 10 slowdown? A likely cause is going from entirely cached to not cached.
Please show us SHOW CREATE TABLE. EXPLAIN SELECT, RAM size, and the value of innodb_buffer_pool_size. And how big (GB) is the table?
Also, did someone happen to do a dump or ALTER TABLE or OPTIMIZE TABLE just before the slowdown.
The above info will either show what caused caching to fail, or show the need for more RAM.
INDEX(first_digit, second_digit) (in either order) will be "covering" for that query; this will be faster than without any index.
I have a table containing log entries for a single week for about a thousand web servers. Each server writes about 60,000 entries per day to the table, so there are 420,000 entries per week for each server. The table is truncated weekly. Each log entry contains the servername, which is a varchar (this cannot be changed).
The main operation is to select * from table where servername = 'particular', so as to retrieve the 420,000 records for a server, and a C# program then analyzes the data from that server once selected.
Should I create a clustered index on the servername column to speed up the read operation? (It currently takes over half an hour to execute the above SQL statement.)
Would partitioning help? The computer has only two physical drives.
The query is run for each server once per week. After the query is run for all servers, the table is truncated.
The "standard" ideal clustered key is something like an INT IDENTITY that keeps increasing and is narrow.
However, if your primary use for this table is the listed query, then I think a clustered index on servername makes sense. You will see a big increase in speed if the table is wide, since you will eliminate an expensive key/bookmark lookup that runs on a SELECT * from a nonclustered index (unless you include all the fields in the table).
EDIT:
KM pointed out this will slow down inserts, which is true. For this scenario you may want to consider a two-field key on servername, idfield where idfield is an INT Identity. This would still allow access based only on servername in your query but will insert new records at the end PER SERVER. You will still have fragmentation and reordering.
based on:
The query is run for each server once per week. After the query is run
for all servers, the table is truncated.
and
for about a thousand web servers
I'd change the c# program to just run a single query one time:
select * from table Order By servername,CreateDate
and have it handle "breaking" on a server name changes.
One table scan is better than 1,000. I would not slow down the main application's INSERTS into a log table (with a clustered index) just so your once a week queries run faster.
Yes, it would be a good idea to create a clustered index on servername column as now database has to do table scan to find out which records satisfy the criteria of servername = 'particular'.
Also horizontally partition the table by date would help the cause further. So, at a time the database would need to worry about only a day's data for all servers.
Then make sure that you fire date-based queries:
SELECT * FROM table
WHERE date BETWEEN '20110801' AND '20110808'
AND servername = 'particular'
Actually i queried optimize table query for one table. then i didn't do any operation on that table. then again i'm querying optimize table query at the end of every month. but the data in the table may be changed once in four or 8 months. is it create any problem in performance of the mysql query?
If you don't do DML operations on the table, OPTIMIZE TABLE is useless.
OPTIMIZE TABLE cleans the table of deleted records, sorts the index pages (brings the physical order of the pages in consistence to logical one) and recalculates the statistics.
For the duration of the command, the table is unavailable both for reading and writing, and the command may take long for large tables.
Did your read the manual about OPTIMIZE? And do you have a problem you want to solve using OPTIMIZE? If not, don't use this statement at all.
If the data doesn't quite change over a period of 4-8 months it should not create any issue with performance for the end of month report.
However if the count of rows that are changed in the 4-8 months period is huge then you would want to rebuild indexes/analyze the tables so that the queries run fine after the load.