I have a database with 200+ entries, and with a cronjob I'm updating the database every 5 minutes. All entries are unique.
My code:
for($players as $pl){
mysql_query("UPDATE rp_players SET state_one = '".$pl['s_o']."', state_two = '".$pl['s_t']."' WHERE id = '".$pl['id']."' ")
or die(mysql_error());
}
There are 200+ queries every 5 minute. I don't know what would happen if my database will have much more entries (2000... 5000+). I think the server will die.
Is there any solution (optimization or something...)?
I think you can't do much but make the cron to be executed every 10 minutes if it's getting slower and slower. Also, you can set X rule to delete X days old entries.
If id is your primary (and unique as you mentioned) key, updates should be fast and couldn't be optimised (since it's a primary key... if not, see if you can add an index).
The only problem which could occur (on my mind) is cronjob overlapping, due to slow updates: let's assume your job starts at 1:00am and isn't finished at 1:05am... this will mean that your queries will pile up, creating server load, slow response time, etc...
If this is your case, you should use rabbitmq in order to queue your update queries in order to process them in a more controlled way...
I would load all data that is to be updated into a temporary table using the LOAD DATA INFILE command: http://dev.mysql.com/doc/refman/5.5/en/load-data.html
Then, you could update everything with one query:
UPDATE FROM rp_players p
INNER JOIN tmp_players t
ON p.id = t.id
SET p.state_one = t.state_one
, p.state_two = t.state_two
;
This would be much more efficient because you would remove a lot of the back and forth to the server that you are incurring by running a separate query every time through a php loop.
Depending on where the data is coming from, you might be able to remove PHP from this process entirely.
Related
Update code
$stmt_update = $db->prepare( '
UPDATE table SET Column = 1
WHERE Id = 17
' );
$stmt_update->execute( );
Takes ~ 25 miliseconds
But
$stmt_select = $db->prepare( '
SELECT
`Text`,
`NameOfPerson`,
`EmailOne`,
`Phone`,
`EmailTwo`
FROM table_name
WHERE Id = ?
' );
$stmt_select->execute( array( trim( $_GET["two"] ) ) );
This takes ~ one millisecond.
Is it normal (such difference)? Any ideas how to make faster update (to execute faster)?
That makes sense. But, you need to learn a few things about measuring performance. The first query may be reading data into memory. This takes a bit of time. When you run the second query it is already there. Often, the second time you run exactly the same query is faster than the first time -- unless you fiddle with caching options on the server.
The update is going to be slower because databases have what are called ACID properties. That means that the update is not completed until the database is as sure as it can be that the change has been made. Typically, this means committing a log transaction to disk, so you are waiting for the disk write to be completed. It is not enough for the disk write to start -- it has to be completed. It also means that the update has acquire locks for the parts of the table being updated.
In addition, the database eventually has to write the actual modified data pages to disk. In MySQL, this probably depends on the storage engine. So, you might be waiting for that.
A select doesn't modify anything. It just reads. So there is some time for getting the data, but as soon as it is in memory, the query can process and finish.
In addition, updates may generate other work for the database engine -- such as updating indexes and running triggers. It is unclear if these are defined on your table.
So, I would expect an update to take longer than a select.
I'm running a forum on a VPS, running Percona DB, with PHP 5.5.8, Opcode caching, etc, it's all very speed orientated.
I'm also running New Relic, (yes I have the t-shirt).
As I'm tuning the application, optimising queries the forum is making to the DB for any query at the top of my time consumed list.
Right now, the most time consuming query I have, as it's the most frequently used is a simple hit counter on each topic.
So the query is:
UPDATE topics SET num_views = num_views + 1 WHERE id_topic = ?
I can't think of a simpler way to perform this, or if any of the various other ways might be quicker, and why.
Is there a way of writing this query to be even faster, or an index I can add to a field to aide speed?
Thanks.
Assuming id_topic is indexed, you're not going to get better. The only recommendation I would have is to look at the other indexes on this table and make sure you don't have redundant ones that include num_views in them. That would decrease update speed on this update.
For example if you had the following indexes
( some_column, num_views)
( some_column, num_views, another_column)
Index #1 would be extraneous and just add to the insert/update overhead
Not sure if that is an improvement, but you could check the following:
How about only adding a row for each page hit to the table instead of locking and updating the row?
And then using a count to get the results, and cache them instead of doing the count each time?
(And maybe compacting the table once per day?)
I am trying to delete more than 2 million records from a table by a mysql query (no joins). The table has around 80 million records.
I used set autocommit=0; and it is taking long time to complete. Will this be faster if I run the query with autocommit=1?
I'm assuming your table is InnoDB. For those 2 million rows, it needs to keep track of the undo log entries for each modification. This builds up in the memory and will eventually go into disk. That's why it's taking a long time. If you do it in chunks, that'll prevent it from going into disk and for MySQL to keep track of less undo log entries, making things more efficient.
The autocommit happens at the end of your query so it wouldn't do anything.
The best way to figure out what your chunk size should be is by experimenting. Something like
delete from table1 limit 1000;
Then keep doubling it until you come up with the best rows-deleted per time ratio.
I am assuming you are trying to run 2 million individual delete statements.
If you try bulk deletes using the primary key or ranges to delete 100-1000 records at a time it will be much much faster.
Examples:
DELETE FROM Table WHERE ID > 0 AND ID < 1000;
OR
DELETE FROM Table WHERE ID IN (1,2,3,4,5 .... 1000);
You can adjust the number of records to delete to your liking, increasing it quite a bit if you want. On high load production servers I usually run scripts with smaller ranges like this maybe 100 times before sleeping for a bit and then continuing with another loop.
I always have autocommit turned on for this type of thing. Managing a transaction to delete 2 million records would add a lot of overhead.
Also, please ensure the column you use for the bulk/range deleting is the either the primary key or has an index.
It won't be any faster with changing value for autocommit variable. MySQL always build old image even if autocommit is true, because if user interrupts query then it must have old image to rollback.
My application accesses a local DB where it inserts records into a table (+- 30-40 million a day). I have processes that run and process data and do these inserts. Part of the process involves selecting an id from an IDs table which is unique and this is done using a simple
Begin Transaction
Select top 1 #id = siteid from siteids WITH (UPDLOCK, HOLDLOCK)
delete siteids where siteid = #id
Commit Transaction
I then immediately delete that id with a separate statement from that very table so that no other process grabs it. This is causing tremendous timeout issues and with only 4 processes accessing it, I am surprised though. I also get timeout issues when checking my main post table to see if a record was inserted using the above id. It runs fast but with all the deadlocks and timeouts I think this indicates poor design and is a recipe for disaster.
Any advice?
EDIT
this is the actual statement that someone else here helped with. I then removed the delete and included it in my code as a separately executed statement. Will the order by clause really help here?
MySQL 5.1, Ubuntu 10.10 64bit, Linode virtual machine.
All tables are InnoDB.
One of our production machines uses a MySQL database containing 31 related tables. In one table, there is a field containing display values that may change several times per day, depending on conditions.
These changes to the display values are applied lazily throughout the day during usage hours. A script periodically runs and checks a few inexpensive conditions that may cause a change, and updates the display value if a condition is met. However, this lazy method doesn't catch all posible scenarios in which the display value should be updated, in order to keep background process load to a minimum during working hours.
Once per night, a script purges all display values stored in the table and recalculates them all, thereby catching all possible changes. This is a much more expensive operation.
This has all been running consistently for about 6 months. Suddenly, 3 days ago, the run time of the nightly script went from an average of 40 seconds to 11 minutes.
The overall proportions on the stored data have not changed in a significant way.
I have investigated as best I can, and the part of the script that is suddenly running slower is the last update statement that writes the new display values. It is executed once per row, given the (INT(11)) id of the row and the new display value (also an INT).
update `table` set `display_value` = ? where `id` = ?
The funny thing is, that the purge of all the previous values is executed as:
update `table` set `display_value` = null
And this statement still runs at the same speed as always.
The display_value field is not indexed. id is the primary key. There are 4 other foreign keys in table that are not modified at any point during execution.
And the final curve ball: If I dump this schema to a test VM, and execute the same script it runs in 40 seconds not 11 minutes. I have not attempted to rebuild the schema on the production machine, as that's simply not a long term solution and I want to understand what's happening here.
Is something off with my indexes? Do they get cruft in them after thousands of updates on the same rows?
Update
I was able to completely resolve this problem by running optimize on the schema. Since InnoDB doesn't support optimize, this forced a rebuild, and resolved the issue. Perhaps I had a corrupted index?
mysqlcheck -A -o -u <user> -p
There is a chance the the UPDATE statement won't use an index on id, however, it's very improbable (if possible at all) for a query like yours.
Is there a chance your table are locked by a long-running concurrent query / DML? Which engine does the table use?
Also, updating the table record-by-record is not efficient. You can load your values into a temporary table in a bulk manner and update the main table with a single command:
CREATE TEMPORARY TABLE tmp_display_values (id INT NOT NULL PRIMARY KEY, new_display_value INT);
INSERT
INTO tmp_display_values
VALUES
(?, ?),
(?, ?),
…;
UPDATE `table` dv
JOIN tmp_display_values t
ON dv.id = t.id
SET dv.new_display_value = t.new_display_value;