I have a more than 10 million data from my table and needs to pull it in order to display in the report. The origin of data was extracted from csv and all of them are in text format. and here is how it looks from my table:
I tried to query with limit on 1000 only and it will display quickly however If I am going to have a date filter for e.g getting 1 day data and it will take around 25-30 secs:
SELECT STR_TO_DATE(SUBSTRING_INDEX(time, '_', 1), '%m/%d/%Y') FROM myTable
WHERE STR_TO_DATE(SUBSTRING_INDEX(time, '_', 1), '%m/%d/%Y') BETWEEN DATE('2019-9-3') AND DATE('2019-9-3');
I already tried to create an index on time column which I am using for filter but still got the same result:
Is there any suggestion/comments how can I improve the speed to pull the data. TIA
When you apply functions to a column as part of your search, it can't use an index, even if you define an index for that column.
You should also use a proper DATE or DATETIME data type for the column, which will require dates be stored in YYYY-MM-DD format, not a string column in MM/DD/YYYY format.
If you store the dates properly, you can do this:
SELECT DATE(time) FROM myTable
WHERE time >= '2019-09-03' AND time < '2019-09-04';
That will make use of the index.
You are storing your dates/timestamps as text, which is going to force you to doing suboptimal things like calling STR_TO_DATE all over the place. I suggest adding a new bona fide datetime column, and then indexing that column:
ALTER TABLE myTable ADD COLUMN time_dt DATETIME;
Then, populate it using STR_TO_DATE:
UPDATE myTable
SET time_dt = STR_TO_DATE(time, '%m/%d/%Y_%H:%i:%s.%f');
Then, add an index on time_dt:
CREATE INDEX idx ON myTable (time_dt);
And finally, rewrite your query so that the WHERE clause is sargable (i.e. so that it may use the above index):
SELECT DATE(time_dt)
FROM myTable
WHERE time_dt >= '2019-09-03' AND time_dt < '2019-09-04';
Side note: You need to use %H in the format mask with STR_TO_DATE, because your hours are in 24-hour clock mode.
Related
i've issue I try this query, do not return any rows. just 0 rows. Even tho there is data matching the request..
select * from repairshop_reservations where date = DATE_FORMAT("11/06/2017 20:00:00", '%d/%m/%Y %H:%i:%s"');
Currently my content of the selected table look like this
The data value of column Date is datetime
you could use str_to_date in this way you can control the proper formatting of the date when you don't use the standard mysql format
select * from repairshop_reservations
where date = str_to_date('11/06/2017 20:00:00', '%d/%m/%Y %H:%i:%s');
You are not inserting a column in your table, so you won't have to define a data type for it. That means that, you are not making changes to the conceptual scheme of your database.
Considering that your table is implemented correctly, the SQL query you would need to give you the desirable result would be:
SELECT * FROM repairshop_reservations
WHERE date = "11/06/2017 20:00:00";
You use the WHERE clause, to filter your record and get an output with a
specified condition. In plain English, what you want to do is:
Select and print for me, every column from the repairshop_reservations table, that has listed date as "11/06/2017 20:00:00"
in a database table I have made a date attribute but I have set it's type to varchar and not Date.
My question is, will I still be able to compare such dates in a SQL Query?
Dates in my DB are stored in this format:
dd/mm/yyyy hh:mm:ss
I have to do a SQL Query in PHP that looks something like this:
SELECT *
FROM DBtable
WHERE DBname='$name' AND date>='01/01/2015' AND date<='01/09/2015';
I would appreciate an example how to do this.
Thank you for your help.
You'll need to convert/cast to compare:
SELECT *
FROM DBtable
WHERE DBname='$name'
AND CAST(date AS DATETIME) >='2015-01-01'
AND CAST(date AS DATETIME)<='2015-01-09'
;
Much better to store values as the appropriate data types to avoid this inefficiency. You could also use DATE instead of DATETIME if you want to compare without the time component. Syntax and available datatypes vary by database, so the above may need adjustment.
Update: Since you're using MySQL, you can use the following:
SELECT *
FROM DBtable
WHERE DBname='$name'
AND STR_TO_DATE(`date`, '%d/%c/%Y') >= '2015-01-01'
AND STR_TO_DATE(`date`, '%d/%c/%Y') <= '2015-01-09'
;
Yes you can cast a Varchar to a Date. Here is an example:
SELECT
CAST(date_column AS DATETIME)
FROM
TABLE_NAME
In your case it might look like:
SELECT *
FROM DBtable
WHERE DBname='$name'
AND CAST(date AS DATETIME) >='01/01/2015'
AND CAST(date AS DATETIME) <='01/09/2015';
You can cast or convert a varchar to a date or datetime before you do any comparisons.
But you'd have to do it every single time you compare the date to something. That's because the following comparisons are all true if you compare them as varchar:
'2/1/2015' > '1/5/2016'
'25/1/2015' > '15/2/2015'
'11/1/2015' < '3/1/2015'
You'll also need to convert if you want to pull out some time-based aspect of the dates, such as any records where the hour was before 8:00 AM. There is no easy way to do that if your date is a varchar.
And that assumes that the value in your database can always be parsed into a date! If an empty string or some other kind of data gets in there, CONVERT(datetime, MyColumn) will fail.
So I would strongly recommend that you change your column to be a date or datetime. It will make your life much easier.
I have a mysql column where the data is stored as VARCHAR though the data values are of datetime in the format of yyyy-mm-dd hh:mm:ss.
Now my task is to group by the date part i.e yyyy-mm-dd by converting VARCHAR to date-time and then just taking date part out of it
QUERY
SELECT SUM(value)
FROM table
GROUP BY name , [date part of the varchar field]
Please let me know if this is at all possible and if yes, how?
Assuming that your data in this varchar field is properly formatted, you can work with the left function, like this:
SELECT LEFT(mydate, 10) AS myval,
SUM(myvalue)
FROM mytable
GROUP BY myval;
If this isn't a big issue; I'd advise converting your varchar column to datetime or timestamp. If not only for the possibly better data storage usage, it'll be way easier to do work with date and time related functions.
Just use the left function. You can leave the date as a string:
SELECT left(datecol, 10) as YYYYMMDD, SUM(value)
FROM table
GROUP BY left(datecol, 10);
I removed name from the group by because it doesn't seem relevant to the question. You can, of course, add it back in.
By the way, MySQL understands this format for dates, so if you really, really want a date:
SELECT date(left(datecol, 10)) as RealDate, SUM(value)
FROM table
GROUP BY RealDate;
Help! One column in my database is for dates. All of my dates are unfortunately in the String form (YYYY.MM.DD). I have a MASSIVE database (300+GB) so ideally would like to avoid transformations.
Is there a way I can select rows for dates in between YYYY.MM.DD and YYYY.MM.DD? What would the script look like?
Thank you!
If the months and days are stored with leading zeroes, the BETWEEN operator will work as expected. So will ORDER BY.
create table your_table (
date_value varchar(10) not null
);
insert into your_table values
('2013.01.01'), ('2013.01.13'), ('2013.01.30'), ('2013.01.31'),
('2013.02.01'), ('2013.02.13'), ('2013.02.28'), ('2013.02.31'),
('2013.03.01'), ('2013.03.15'), ('2013.03.30'), ('2013.03.31');
select date_value
from your_table
where date_value between '2013.01.01' and '2013-01-31'
order by date_value;
2013.01.01
2013.01.13
2013.01.30
One of the main problems with your structure is that you lose type safety. Look at this query.
select date_value
from your_table
where date_value between '2013.02.01' and '2013.02.31'
order by date_value;
2013.02.01
2013.02.13
2013.02.28
2013.02.31
If you'd used a column of type date or datetime or timestamp, the dbms would not have allowed inserting the values '2013.02.31', because that's not a value in the domain of date. It is a value in the domain of varchar. (And so is "Arrrrgh!", unless you've got a CHECK constraint on that column that severely restricts the acceptable values.)
Not good solution, but works (cost much performance).
You have formated date in order year, month, day (good order to compare strings, without transformation to datetime), so you can try
SELECT * FROM Table WHERE StringDate > '2013.07.10' AND StringDate < '2013.07.14'
It returns bad results if there are dates before year 1000 without leading zero ('999.07.14').
But I dont know how it works on big database.
SQL Fiddle
Between in SQL is inclusive of both bounds. If that is what you want, you can just use between:
where col between 'YYYY.MM.DD' and 'YYYY.MM.DD'
Where the two constants are whatever values you are looking for.
If you have an index on the column, then between (as well as >, >=, and so on) will use the index. You do not need to transform the values. If your constants are dates of one form or another, then you can use date_format() to create a string in the right format. For instance, to get dates within the past week:
where col >= date_format(adddate(now(), -7), '%Y.%m.%d')
I am working with a MySQL database where dates are stored as varchar like this:
'2013-01-31' in column cl_223
I need to select only records from 2013 so I tried:
SELECT ..
FROM ....
Where cl_223 Like '2013'
But that does not seem to work.
Thanks for all help!
You must add % as a wildcard :
SELECT ..
FROM ....
WHERE cl_223 LIKE '2013%'
Storing a datettime value in a varchar column complicates some functionality on date time operations. But of course you can select your values writing such a query as follow
SELECT * FROM table_name WHERE cl_223 LIKE '2013%'
But if you don't have any performance issue you can convert the varchar column to a datetime value and write stronger typed query like this:
SELECT * FROM table_name WHERE STR_TO_DATE(cl_223,'%Y-%m-%d2') BETWEEN '2013-01-01' AND '2013-12-31'
But if you need a date time value as a date time in your process you'd better store it in a datetime column instead of a varchar column.
The query should be
SELECT ..
FROM ....
Where cl_223 Like '2013%'
However, the better solution would be to store the dates as DATE data types. If the dates in that column are always used in the format they're in now, the change would be backwards compatible. It would also allow for easier processing of the date values.