I have a table with a DATETIME field called date_created, and need to check for some data with this kind of query:
SELECT * FROM table WHERE UNIX_TIMESTAMP(date_created) + $number < UNIX_TIMESTAMP()
Is it possible to do this without the UNIX_TIMESTAMP() function? Maybe with NOW())?
And if it is, will it be faster?
Is it possible to do this without the UNIX_TIMESTAMP() function? Maybe with NOW())?
Not exactly. Whilst you could do the following, it won't achieve the exact same results:
SELECT * FROM `table` WHERE date_created + INTERVAL $number SECOND < NOW()
The reason is that, unlike TIMESTAMP, DATETIME is neither intended nor capable of representing a specific instance in time; rather, it effectively represents the display of a calendar/clock (not the same thing).
When using the above query, which does not go via a UTC timestamp, the comparison is merely a question of whether a clock right now would show a date/time later than that which was recorded in the database (plus $number seconds).
However, when converting to a UTC timestamp, the timezone of the respective clock displays become relevant and because many timezones are not constant (e.g. they often move around for daylight savings), multiple DATETIME values could give rise to the same UTC timestamp.
For example:
CREATE TABLE `table` (date_created DATETIME);
INSERT INTO `table` VALUES ('2012-10-28 01:00:00'), ('2012-10-28 02:00:00');
Then compare the results of the two queries when run at 2012-10-28 02:00:00 in the UK:
Your original query:
SET SESSION time_zone = 'Europe/London';
SELECT *
FROM `table`
WHERE UNIX_TIMESTAMP(date_created) + 100
< UNIX_TIMESTAMP('2012-10-28 02:00:00');
The alternative query above:
SELECT *
FROM `table`
WHERE date_created + INTERVAL 100 SECOND
< '2012-10-28 02:00:00';
And if it is, will it be faster?
Probably (a lexicographic order for the comparison will suffice, versus parsing & converting the DATETIME values to UTC according the session timezone's rules followed by subtraction and sign inspection), but I'd advise performing your own benchmarks.
Using eval expression in WHERE clause should be avoided if possible. It prevents correct utilization of indexes. If possible do the math in code and send the values as query parameters.
Related
I have a more than 10 million data from my table and needs to pull it in order to display in the report. The origin of data was extracted from csv and all of them are in text format. and here is how it looks from my table:
I tried to query with limit on 1000 only and it will display quickly however If I am going to have a date filter for e.g getting 1 day data and it will take around 25-30 secs:
SELECT STR_TO_DATE(SUBSTRING_INDEX(time, '_', 1), '%m/%d/%Y') FROM myTable
WHERE STR_TO_DATE(SUBSTRING_INDEX(time, '_', 1), '%m/%d/%Y') BETWEEN DATE('2019-9-3') AND DATE('2019-9-3');
I already tried to create an index on time column which I am using for filter but still got the same result:
Is there any suggestion/comments how can I improve the speed to pull the data. TIA
When you apply functions to a column as part of your search, it can't use an index, even if you define an index for that column.
You should also use a proper DATE or DATETIME data type for the column, which will require dates be stored in YYYY-MM-DD format, not a string column in MM/DD/YYYY format.
If you store the dates properly, you can do this:
SELECT DATE(time) FROM myTable
WHERE time >= '2019-09-03' AND time < '2019-09-04';
That will make use of the index.
You are storing your dates/timestamps as text, which is going to force you to doing suboptimal things like calling STR_TO_DATE all over the place. I suggest adding a new bona fide datetime column, and then indexing that column:
ALTER TABLE myTable ADD COLUMN time_dt DATETIME;
Then, populate it using STR_TO_DATE:
UPDATE myTable
SET time_dt = STR_TO_DATE(time, '%m/%d/%Y_%H:%i:%s.%f');
Then, add an index on time_dt:
CREATE INDEX idx ON myTable (time_dt);
And finally, rewrite your query so that the WHERE clause is sargable (i.e. so that it may use the above index):
SELECT DATE(time_dt)
FROM myTable
WHERE time_dt >= '2019-09-03' AND time_dt < '2019-09-04';
Side note: You need to use %H in the format mask with STR_TO_DATE, because your hours are in 24-hour clock mode.
I have the following SQL statement running against a MariaDB 10.1.26 with ~2.000 rows with instant results.
select value, datetime from Schuppen
where (value = (select min(value) from Schuppen where (measure = 'temp')
and datetime between '2018-11-01 00:00:00' and '2018-11-02 00:00:00'))
and datetime between '2018-11-01 00:00:00' and '2018-11-02 00:00:00';
When I use the following statement with variables for the datetime fields, the execution takes ~5.5 seconds.
set #startdate = cast('2018-11-01 00:00:00' as datetime);
set #enddate = cast('2018-11-02 00:00:00' as datetime);
select value, datetime from Schuppen
where (value = (select min(value) from Schuppen where (measure = 'temp')
and datetime between #startdate and #enddate))
and datetime between #startdate and #enddate;
The more data rows I have, the longer it takes to execute the statement. Seems like the variables change the behaviour of the statement somehow.
What's wrong here?
I use MySQL Workbench and #variables are very useful to query/search different tables for a given attribute. I ran into a similar issue. After scouring through different threads and trying different things, it worked well when I set the #variable to be of exactly the same type and same encoding as the column in the table(s) that I am searching for that variable.
For example:
SET #keyword = CONVERT(CAST("KEYWORD" AS CHAR(8)) USING ASCII);
In this case, the search column cname in my table customer is of type CHAR(8) and encoded using ASCII:
SELECT * FROM CUSTOMERS WHERE cname=#keyword;
If you have multiple tables to query, where cname is CHAR(10) in one and CHAR(8) in another, then you can do the following:
SET #keyword = "KEYWORD";
SELECT * FROM CUSTOMERS WHERE cname=CONVERT(CAST(#keyword AS CHAR(8)) USING ASCII);
SELECT * FROM EMPLOYEES WHERE cname=CONVERT(CAST(#keyword AS CHAR(10)) USING ASCII);
The problem is that the query optimizer does a bad job on finding a suitable index when using variables. This is a known issue.
If you use EXPLAIN on both queries, you will see the difference. Just try to avoid variables when not necessary.
For the first query, the optimizer "sees" the chosen values and decides an index can be perfectly used to satisfy the selected range more efficiently.
For the second query, the optimizer is unaware of the two values that define the range, and decides to fall back to a FULL SCAN instead.
in a database table I have made a date attribute but I have set it's type to varchar and not Date.
My question is, will I still be able to compare such dates in a SQL Query?
Dates in my DB are stored in this format:
dd/mm/yyyy hh:mm:ss
I have to do a SQL Query in PHP that looks something like this:
SELECT *
FROM DBtable
WHERE DBname='$name' AND date>='01/01/2015' AND date<='01/09/2015';
I would appreciate an example how to do this.
Thank you for your help.
You'll need to convert/cast to compare:
SELECT *
FROM DBtable
WHERE DBname='$name'
AND CAST(date AS DATETIME) >='2015-01-01'
AND CAST(date AS DATETIME)<='2015-01-09'
;
Much better to store values as the appropriate data types to avoid this inefficiency. You could also use DATE instead of DATETIME if you want to compare without the time component. Syntax and available datatypes vary by database, so the above may need adjustment.
Update: Since you're using MySQL, you can use the following:
SELECT *
FROM DBtable
WHERE DBname='$name'
AND STR_TO_DATE(`date`, '%d/%c/%Y') >= '2015-01-01'
AND STR_TO_DATE(`date`, '%d/%c/%Y') <= '2015-01-09'
;
Yes you can cast a Varchar to a Date. Here is an example:
SELECT
CAST(date_column AS DATETIME)
FROM
TABLE_NAME
In your case it might look like:
SELECT *
FROM DBtable
WHERE DBname='$name'
AND CAST(date AS DATETIME) >='01/01/2015'
AND CAST(date AS DATETIME) <='01/09/2015';
You can cast or convert a varchar to a date or datetime before you do any comparisons.
But you'd have to do it every single time you compare the date to something. That's because the following comparisons are all true if you compare them as varchar:
'2/1/2015' > '1/5/2016'
'25/1/2015' > '15/2/2015'
'11/1/2015' < '3/1/2015'
You'll also need to convert if you want to pull out some time-based aspect of the dates, such as any records where the hour was before 8:00 AM. There is no easy way to do that if your date is a varchar.
And that assumes that the value in your database can always be parsed into a date! If an empty string or some other kind of data gets in there, CONVERT(datetime, MyColumn) will fail.
So I would strongly recommend that you change your column to be a date or datetime. It will make your life much easier.
Help! One column in my database is for dates. All of my dates are unfortunately in the String form (YYYY.MM.DD). I have a MASSIVE database (300+GB) so ideally would like to avoid transformations.
Is there a way I can select rows for dates in between YYYY.MM.DD and YYYY.MM.DD? What would the script look like?
Thank you!
If the months and days are stored with leading zeroes, the BETWEEN operator will work as expected. So will ORDER BY.
create table your_table (
date_value varchar(10) not null
);
insert into your_table values
('2013.01.01'), ('2013.01.13'), ('2013.01.30'), ('2013.01.31'),
('2013.02.01'), ('2013.02.13'), ('2013.02.28'), ('2013.02.31'),
('2013.03.01'), ('2013.03.15'), ('2013.03.30'), ('2013.03.31');
select date_value
from your_table
where date_value between '2013.01.01' and '2013-01-31'
order by date_value;
2013.01.01
2013.01.13
2013.01.30
One of the main problems with your structure is that you lose type safety. Look at this query.
select date_value
from your_table
where date_value between '2013.02.01' and '2013.02.31'
order by date_value;
2013.02.01
2013.02.13
2013.02.28
2013.02.31
If you'd used a column of type date or datetime or timestamp, the dbms would not have allowed inserting the values '2013.02.31', because that's not a value in the domain of date. It is a value in the domain of varchar. (And so is "Arrrrgh!", unless you've got a CHECK constraint on that column that severely restricts the acceptable values.)
Not good solution, but works (cost much performance).
You have formated date in order year, month, day (good order to compare strings, without transformation to datetime), so you can try
SELECT * FROM Table WHERE StringDate > '2013.07.10' AND StringDate < '2013.07.14'
It returns bad results if there are dates before year 1000 without leading zero ('999.07.14').
But I dont know how it works on big database.
SQL Fiddle
Between in SQL is inclusive of both bounds. If that is what you want, you can just use between:
where col between 'YYYY.MM.DD' and 'YYYY.MM.DD'
Where the two constants are whatever values you are looking for.
If you have an index on the column, then between (as well as >, >=, and so on) will use the index. You do not need to transform the values. If your constants are dates of one form or another, then you can use date_format() to create a string in the right format. For instance, to get dates within the past week:
where col >= date_format(adddate(now(), -7), '%Y.%m.%d')
In MySQL, I need to write a query (if possible) that finds all rows of a table where the date_created is the same as last_updated. The rub is that I need to ignore the time. Basically, I'm looking for user rows that were created and activated the same day (we don't store an activation date). So presumably the dates would be the same but the times may be different.
You could use the DATE() function, which returns only the date portion of a datetime value. This allows you to compare just the date portion of the values:
SELECT * FROM table_name
WHERE DATE(date_created) = DATE(last_updated)
The timezone may be relevant here. So you may want to cast the datetime values to the user's timezone prior to using the DATE() function, using CONVERT_TZ().
Try this:
SELECT *
FROM table_name
WHERE DATE_FORMAT(date_created, '%Y-%m-%d') = DATE_FORMAT(last_updated, '%Y-%m-%d')
not pretty but works:
SELECT *
FROM table_name
WHERE day(date_created) = day(last_updated) and
month(date_created) = month(last_updated) and
year(date_created) = year(last_updated)