I am working on some game logic and I need a way to update a level when time runs out. Basically, there is a research table with columns as such:
research_time_id
research_started
research_end
active
level
I need the level to be set to level+1 when research_ended is sooner then NOW().
I know how to write the UPDATE statement. My question is should I run this through mysql eventscheduler? should I run it every second? Is that too often? What is optimum in a scalable environment? This table could have many rows in it.
Related
i have a very huge table "table1" from which i am continuously querying all day(24x7)
What happens is at the end of the day say at 12.AM, i run a query which would alter "table1" at row level. this activity takes around 3-4 hours till my updated "table1" is finished creating.
But till that time i wanted to still query from "table1".
SO i decided to create two tables. "table1_active" and "table1_passive"
normally during the day i will query from "table1_passive" and after i am updating "table1_active" i should switch my querying from "table1_passive"
to "table1_active"
and this switching should be done everyday, so that my all day querying should not hamper.
I dont know is there a better way to actually set a trigger or can anyone suggest me a method to do it?
In my experience, the use of a secondary table like table1_passive it's risky. You don't know exactly (as I understand) when the update process finish so you will don't know either when you should switch querying between table1_passive and table1_active.
There are several ways to improve the update process over your table but you have to keep in mind, those are temporary solutions if table1 grows constantly:
Use MyISAM as storage engine. Here is a very good article about improve the updates over a MyISAM table.
If you are updating table1 based on a where clause, you might use indexes to help database engine find which records has to update.
Consider use partitions to work with your table faster.
If you sill have those two tables, you can:
Create an Unique_Index on table1_active and set ON DUPLICATE KEY UPDATE
Update table1_passive.
Use bulk insert in table1_active to speed up the process, the database will make sure that there are not duplicate rows based on your criteria.
BUT, if you are querying all day, and the table grows constantly I suggest to use NoSql because the problem will be there, even if you optimize the update process now.
If we could know the table structure and the update query you are using maybe we can help you better ;)
Regards and good luck!
I have a MySQL server with many innodb tables.
I have a background script that does A LOT a delete/insert with one request : it deletes many millions of rows from table 2, then insert many millions of rows to table 2 using data from table 1 :
INSERT INTO table 2 (date)
SELECT date from table 1 GROUP BY date
(The request is actually more complex but it is to shown what kind of request I am doing).
At the same time, I am going to run a second background script, that does about a million INSERT or UPDATE requests, but separately (I mean, I execute the first update query, then I execute an insert query, etc...) in table 3.
My issue is that when a script is running, it is fast, like let's say it takes 30minutes each, so 1h total. But when the two scripts are running at the same time, it is VERY slow, like it will take 5h, instead of 1h.
So first, I would like to know what can cause this ? Is it because of IO performance ? (like mysql is writing in two different tables so it is slow to switch between the two ?)
And how could I fix this ? If I could say that the big INSERT query is paused while my second background script is running, it would be great, for example... But I can't find a way to do something like this.
I am not an expert at MySQL administration.. If you need more information, please let me know !
Thank you !!
30 minutes for million INSERT is not fast. Do you have an index on date column? (or whatever column you are using to pivot on)
Regarding your original question.It's difficult to say much without knowing the details of both your scripts and the table structures, but one possible reason why the scripts are running reasonably quickly separately is because you are doing similar kinds of SELECT queries, which might be getting cached by MySQL and then reused for subsequent queries. But if you are running two queries in parallel, then the SELECT's for the corresponding query might not stay in the cache (because there are two concurrent processes which send new queries all the time).
You might want to explicitly disable cache for some queries which you are sure you only run once (using SQL_NO_CACHE modifier) and see if it changes anything. But I'd look into indexing and into your table structure first, because 30 minutes seems to be extremely slow :) E.g. you might also want to introduce partitioning by date for your tables, if you know that you always choose entries in a given period (say by month). The exact tricks depend on your data.
UPDATE: Another issue might be that both your queries work with the same table (table 1), and the default transaction isolation level in MySQL is REPEATABLE READS afair. So it might be that one query is waiting until the other is done with the table to satisfy the transaction isolation level. You might want to lower the transaction isolation level if you are sure that your table 1 is not changed when scripts are working on it.
You can use an event scheduler so you can set mysql to launch this queries at different hours of the day, in another stackoverflow related question you have an exmaple of how to do it: MySQL Event Scheduler on a specific time everyday
Another thing to have in mind is to use the explain plan to see what could be the reason the query is that slow.
Currently i am using cron for this. I thought perhaps it is possible to implement some procedure that will remove all data from database that is older than one month, but i am not sure that this is the best way.
Problem is that we have many servers with many cron processes, that are controlled by very small amount of stuff, and we need to make it clear and easy-to-manage, that's why i don't want to have such cron process.
Data in table i want to delete - statistics, huge amount of this data is inserted every day, and if it will not be deleted - database will be unbeliaveable huge (about ~500M every day, for us it's quite big amount, 500M * 365 days is 182,5G per year)
Is it possible to delete data using some procedure in mysql (perhaps after new row is added) / and is that a good idea?
If you're intending on moving away from cron jobs, you could always create an event that runs at a scheduled frequency.
Whatever you do, it's a very bad idea to delete data every time a new row is added, as it'll slow down your insert and it's more likely to fragment your tables.
I've done some googling but can't really get much relevant information. I'm trying to set a date/time for certain rows to be deleted depending on activity. If active, the time would be bumped to a later time unless activated once again.. Otherwise it will be deleted. I've managed to sort the rows when activated (inserted/updated) in activity.
Thanks in advance.
Firstly do not put this update/delete in a trigger if you have millions of rows that needs to be deleted you are going to see a huge performance hit on inserts/updates. It is not the best place for it. You can create either a cron job as Filype suggested. Or if you want to keep it all in MySQL use the MySQL Event scheduler.
Go to this page to read more about scheduling events in MySQL:
http://dev.mysql.com/doc/refman/5.1/en/events.html
MySQL Event allows you to schedule things on MySQL on a regular basis.
The code would look something like
CREATE EVENT myevent
ON SCHEDULE AT CURRENT_TIMESTAMP + INTERVAL 1 HOUR
DO
DELETE FROM MyTable Where Expired< NOW();
Here is a suggestion, I haven't tried yet, you might think to update the row with deleted=1 instead of actually deleting the record.
CREATE TRIGGER deleteInactiveRecords AFTER UPDATE,INSERT ON myTable
FOR EACH ROW BEGIN
DELETE FROM myTable WHERE updated < (updated-((60)*60*24))
END;
I've come across a PHP forum software which updates its threads views each hour.
So each time you view a thread, a row is inserted to a threadviews table with the thread id, then a script runs once an hour and updates the actual views count in the thread table.
My question is, what's the logic behind this? Why not just update the thread table (i.e. views = views + 1)?
updates in general is way slower than inserts, you can think of update as delete and insert. updates might require locking to support ACID compliance of the DBMS, with inserts you dont have any locking.
Also, due to concurrency, you dont want to lock the row, and wait for the update to finish, think of this, what happen, while you are updating, you get a new visitor, you will lose that visitor. This is called the lost update. On the other hand, the cron job aggregates the visits and updates once an hour, since that row is read only, write lock wont affect your reads during the update.
Insert is likely always faster. Look at it like this.
An update is first a search for all posts to update. In some tables that could take a really long time but with god indices it should be fast. It is then an update of data and on each update it needs to check table constraints and possibly update indexes.
An insert is the same without the search. It's also always one row (or it could be more but then it's actually more than one insert... ) that has to be checked against constraints and update indices.