I don't have much experience with MySQL so not sure if it's issue with MySQL or my code.
I have a table lets say data and it has a created_at column of DATETIME type.
This table gets 20-30 new records per second, no updates at all.
I have a Cron job that runs every 15 minute and tries to get all records created in the last 15 minutes.
if it runs at 10:15:06am and the last run was at 10:00:03, it makes this query:
SELECT * FROM `data` WHERE (created_at >= '2021-07-30T10:00:03Z' AND created_at < '2021-07-30T10:15:06Z')
Current time is excluded, hence the created_at < current_time.
But the problem is, once in a while I get duplicate data error. That is it includes a few data rows from the current_time, which should have been excluded.
Like in this case, if 15 records were inserted at 10:15:06, the query result might have 4-5 records included in it. But it does not happen every time.
I am using Golang and for current time, I use time.Now(). Can this be because of millisecond or something else ? I am not making more than 1 database query, so I think it has to do something with DB, if I have extra records.
Related
I am just learning MySql (SQL in general) and I have a question. I ran a process to populate a table with 72 records. This was done, however, I needed to run the process again and this time it populated the table again with a second record for each user for a total now of 144 records. How can I isolate the newest records created today?
A simple solution is to use current_date to figure out today's date and date() to remove the time portion of your column. Then:
where current_date = date(createdTS)
This is fine for a small dataset as yours. As general solution, you'd need a query that won't need to manipulate every row, e.g.
where createdTS >= current_date and createdTS < current_date + interval 1 day
You just have to use your createdTS column, (assuming you know what was the timestamp of both runs).
SELECT * FROM `my_table` WHERE `createdTS` > '2019-07-25 15:00:00'
You could also RANK() over and get only the newest run for each user (something like this)
SELECT *
FROM LOGS
WHERE datetime > DATE_SUB(NOW(), INTERVAL 1 MONTH)
I have a big table LOGS (InnoDB). When I try to get last month's data, the query waits too long.
I created an index for column datetime but it seems not helping. How to speed up this query?
Since the database records are inserted in oldest to newest, you could create 2 calls. The first call requesting the ID of the oldest record:
int oldestRecordID = SELECT TOP 1 MIN(id)
FROM LOGS
WHERE datetime > DATE_SUB(NOW(), INTERVAL 1 MONTH)
Then with that ID just request all records where ID > oldestRecordID:
SELECT *
FROM LOGS
WHERE ID > oldestRecordID
It's multiple calls, but it could be faster however I am sure you could combine those 2 calls too.
Probably the only thing you can do is create a clustered index on datetime. This will ensure that the values are co-located.
However, I don't think this will solve your real problem. Why are you bringing back all records from a month. This is a lot of data.
In all likelihood, you could summarize the data in the database and only bring back the information you need rather than all the data.
Please consider the following query:
SELECT submitted_time FROM jobs WHERE timediff(NOW(), submitted_time) < '24:00:00'
My hope is for this to return all rows that have a "submitted_time" column containing a timestamp that was within the last 24 hours, However I am receiving the following results:
2017-01-18 14:58:34
2017-01-16 14:58:34
If I run the query SELECT NOW() I get 2017-01-25 18:58:32
Which appears to be correct.
What is stranger still is that I have more recent rows in the DB such as:
2017-01-24 15:17:13
Which are not being returned.
I hope I have made a glaringly obvious error that someone can point out, rather than beginning the descent into madness.
Just to be clear, the simplest and probably most performant way to handle this is (as per the link I provided in the comment)
SELECT submitted_time FROM jobs WHERE submitted_time > DATE_ADD(NOW(), INTERVAL -1 DAY);
This should be all jobs submitted literally within the last 24 hours at the moment the query is issued.
This might not be important to you for this query, but whenever you apply functions to columns in your table, any indexes you might have can not be used, because the database must run the function(s) on each value in the table before it can perform a comparison.
Using this method you figure out what the comparable datetime needs to be and mysql will use an index on submitted_time for the comparison, assuming that column is indexed appropriately.
I have a table that stores some sensitive information but I would like that information to change to NULL when it exceeds 24 hours. How can I do that? I have a column named "last_updated" and stores value like this "2014-02-26 16:25:58".
How can I compare the last_updated value with the current time and if it exceeds 24 hours, the other field will change to "NULL".
Should I put something like UPDATE table SET info=NULL WHERE last_updated > 24hour? I don't know how to compare the last_updated when its 24 hours later.
or is there a function inside MySQL to check automatically without running the query using phpmyadmin?
You can do this with a scheduled job that resets the data. Of course, the time span would be 24-48 hours to the change, if you run the job only once per day.
There is another option. That is to do all the data access via views. Then the view could say:
create view v_table as
select (case when last_updated > now() - interval 1 day then col1 end) as col1,
. . .
from table;
Then, you can then update the data at your leisure -- if you still find that necessary. Access to the data won't be dependent on a job and job scheduler. If all accesses to the data are through the view, then after 24 hours, the data will appear as NULL.
I have Rails application which using MYSQL as database. For some condition, I have to delete all the records from table which was stored exactly 2 hours before the current time.
My query is :
DELETE FROM TABLE_NAME WHERE (NOW() - created_at) > 7200;
Here create_at is datetime column type. Storing the value in the format "2012-12-04 06:39:44"
My problem is, the above query fetch the records even though the record created time is just 40 to 50 minutes and got deleted. The only problem is the record got delete after it reach 40 to 50 minx from it create time.
Can any one please correct my query. I want the MySQL solution. Please help me
You probably need this if you want to delete records created exactly 2 hours ago:
DELETE FROM TABLE_NAME WHERE created_at = NOW() - INTERVAL 2 HOUR
or this, that will delete all records created more than 2 hours ago:
DELETE FROM TABLE_NAME WHERE created_at < NOW() - INTERVAL 2 HOUR
Try this ::
DELETE FROM TABLE_NAME WHERE TIMEDIFF(NOW(),created_at) < '02:00:00';
Try:
DELETE FROM TABLE_NAME WHERE created_at<DATE_SUB(NOW(),INTERVAL 2 HOUR)
This query will delete everything created MORE THAN 2 hours ago. Putting an equal sign would mean EXACTLY 2 hours ago (in second). Of course you can format date to consider only minutes, but that would slow down the query.
If created_at is indexed (and I think it should be) don't perform any functions on it so it can use index to perform delete faster.
I understand you want to delete all records created within a time lapse. So, you shouldn't apply a "greater than" operator to the subtract operation. Instead you should try to specify an appropriated time frame.
You could also take a look to the timediff function http://dev.mysql.com/doc/refman/5.5/en/date-and-time-functions.html#function_timediff
Sorry I'm not able to post the right statement for you, since I don't have a mysql server at hand.