I'm using date search in query. However, size of table (over 5 million records) isn't optimal for this query. I've set up indexes on used fields, e.g.
log_timestamp_index for timestamp and action_id
log_object_type for object_type_id
'Explain select' shows that for date range 2022-08-19 - 2023-01-28 is used index log_timestamp_index.
Changing date range by 1 day to 2022-08-18 - 2023-01-28 sets key to FK_log_users, which has an index on field user_id, which makes query much longer.
Searching after first date for log_timestamp_index takes about 5 seconds in app and returns 'explain select' 750k rows + 'Using index condition; Using where; using temporary; Using filesort'. For FK_log_users and date greater than 1 'explain select' returns more than 5 million records + 'Using where' - this already throws a 504 error. This query takes more than 2 minutes.
Is it possible to speed up execution of this query or change it so that its call is optimal and doesn't return a 504 error?
Does the optimizer choose the most optimal solution here?
Why does optimizer say in this date range that it is better to use a different key?
$query = Logs::select("user_id")
->selectRaw("count(case when action_id = 1 AND content = 'Text 1' then 1 end) as text1")
->selectRaw("count(case when action_id = 2 AND content = 'Text 2' then 1 end) as text2")
->selectRaw("count(case when action_id = 3 AND content = 'Text 3' then 1 end) as text3")
->selectRaw("count(case when action_id = 4 AND content LIKE 'Text 4%' then 1 end) as text4")
->selectRaw("count(case when action_id = 2 AND content LIKE 'Text 5%' then 1 end) as text5")
->selectRaw("count(case when action_id = 2 AND content LIKE 'Text 6%' then 1 end) as text6")
->selectRaw("count(case when action_id = 2 AND content LIKE 'Text 7%' then 1 end) as text7")
->whereBetween('timestamp', ['2022-08-18', '2023-01-28'])
->where('object_type_id', 1)
->groupBy('user_id')
->get();
Related
My AVG(rank) will return 2 and 34 clicksSum in this case. But how can I get it to only count the first row of ranks? So that it will only take one rank of each keyword_id from each date in the interval.
$interval = \DB::table('keywords')
->selectRaw('keyword, url, AVG(rank) rankAvg, SUM(clicks) clicksSum ')
->join('ranks', 'keywords.id', '=', 'ranks.keyword_id')
->whereBetween('ranks.date', array('2018-07-20','2018-07-29'))
->orderBy('clicksSum', 'DESC')
->groupBy('keywords.id')
->paginate(50);
Table: Keywords
id keyword
1 test
Table: Ranks
id keyword_id rank url clicks date
1 1 2 /test/ 29 2018-07-28
1 1 4 /test2/ 5 2018-07-28
The goal I wish to achieve is getting rankAvg = 2 and clicksSum = 29.
There is a alot of data in the table, this is just a simple example.
You can use mysql max and order by it
$interval = \DB::table('keywords')
->selectRaw('keyword, url, AVG(rank) rankAvg, SUM(clicks) clicksSum, max(clicks) as maxClicks')
->join('ranks', 'keywords.id', '=', 'ranks.keyword_id')
->whereBetween('ranks.date', array('2018-07-20','2018-07-29'))
->groupBy('keywords.id')
->orderBy('maxClicks')
->paginate(50);
I have a database with roughly 6 million entries - and will grow - where I'm running queries to return for a HighCharts charting functionality. I need to read longitudinally over years, so I'm running queries like this:
foreach($states as $state_id) { //php code
SELECT //mysql psuedocode
sum(case when mydatabase.Year = '2003' then 1 else 0 end) Year_2003,
sum(case when mydatabase.Year = '2004' then 1 else 0 end) Year_2004,
sum(case when mydatabase.Year = '2005' then 1 else 0 end) Year_2005,
sum(case when mydatabase.Year = '2006' then 1 else 0 end) Year_2006,
sum(case when mydatabase.Year = '2007' then 1 else 0 end) Year_2007,
sum(case when mydatabase.Year = '$more_years' then 1 else 0 end) Year_$whatever_year,
FROM mytable
WHERE State='$state_id'
AND Sex IN (0,1)
AND Age_segment IN (5,4,3,2,1)
AND "other_filters IN (etc, etc, etc)
} //end php code
But for various state at once... So returning lets say 5 states, each with the above statement but a state ID is substituted. Meanwhile the years can be any number of years, the Sex (male/female/other) and Age segment and other modifiers keep changing based on filters. The queries are long (at minimum 30-40seconds) a piece. So a thought I had - unless I'm totally doing it wrong - is to actually store the above query in a second table with the results, and first check that "meta query" and see if it was "cached" and then return the results without reading the db (which won't be updated very often).
Is this a good method or are there potential problems I'm not seeing?
EDIT: changed to table, not db (duh).
Table structure is:
id | Year | Sex | Age_segment | Another_filter | Etc
Nothing more complicated than that and no joining anything else. There are keys on id, Year, Sex, and Age_segment right now.
Proper indexing is what is needed to speed up the query. Start by doing an "EXPLAIN" on the query and post the results here.
I would suggest the following to start off. This way avoids the for loop and returns the data in 1 query. Not knowing the number of rows and cardinality of each column I suggest a composite index on State and Year.
SELECT mytable.State,mytable.Year,count(*)
FROM mytable
AND Sex IN (0,1)
AND Age_segment IN (5,4,3,2,1)
AND "other_filters IN (etc, etc, etc)
GROUP BY mytable.State,mytable.Year
The above query can be further optimised by checking the cardinality of some of the columns. Run the following to get the cardinality:
SELECT Age_segment FROM mytable GROUP BY Age_segment;
Pseudo code...
SELECT Year
, COUNT(*) total
FROM my_its_not_a_database_its_a_table
WHERE State = $state_id
AND Sex IN (0,1)
AND Age_segment IN (5,4,3,2,1)
GROUP
BY Year;
Following is the query that i am using:
SELECT *
FROM (
SELECT
(
CASE WHEN product_name like '%word1%' THEN 1 ELSE 0 END +
CASE WHEN product_name like '%word2%' THEN 1 ELSE 0 END +
CASE WHEN product_name like '%word3%' THEN 1 ELSE 0 END
) AS numMatches
FROM products as p
) as derived
WHERE numMatches > 0
ORDER BY numMatches DESC
LIMIT 30,10
I added an index (BTREE) on product_name, there are 3 million records in the column, the query is executing in 3-5 seconds.
Explain says 'Using where; Using filesort' so i can figure out its not using the index.
No, it's not using the index.
For that, you would have to compare with 'word1%', 'word2%', etc.. but doesn't work when you use the joker at the beginning.
But, If your mysql version is relatively modern you can use fulltext indexes, which would serve for your query.
https://dev.mysql.com/doc/refman/5.6/en/innodb-fulltext-index.html
I have a MySQL table of many records. I am trying to find a way to show the records which meet more than one condition of a query. For example, if I had this table.
TABLE NAME: DATA
ID contactid flag flag_type
-----------------------------------
1 99 Volunteer 1
2 99 Uploaded 2
3 100 Via Import 3
4 100 Volunteer 1
5 100 Uploaded 2
with conditions such as:
WHERE (ID > 2) OR (flag = 'Uploaded') OR (flag_type = 1) ..etc..
The output would be where IDs 4 & 5 only would be returned.
You can count the number of conditions in MySQL and use this value:
where ((id > 2) +
(flag = 'Uploaded') +
(flag_type = 1)
) > 1
A boolean value of "true" is treated as 1 AND "false" is treated as 0. So, by adding up the values, you get the number of conditions that are met.
Often, you do this in an order by, to get the most matching first:
where id > 2 or flag = 'Uploaded' or flag_type = 1
order by ((id > 2) +
(flag = 'Uploaded') +
(flag_type = 1)
) desc;
http://sqlfiddle.com/#!2/572e1/11
This solution selects each of your successful conditions as 1 which is added to "factors". Then we show results with at least 2 factors:
SELECT * FROM
(select id, contactid, flag, flag_type,
(CASE WHEN id > 2 then 1 else 0 END) +
(CASE WHEN flag = 'Uploaded' then 1 else 0 END) +
(CASE WHEN flag_type = 1 then 1 else 0 END) AS factors
from DATA
) t
WHERE factors > 1
With your limited question, this is the most that can be provided.
WHERE `ID`>2 AND (`flag`='Uploaded' OR `flag_type`=1) ..etc
You are on the right path, but you need to read first about boolean logic. ORs are going to return records that match any of the conditions and ANDs will return records that match all of the conditions.
Bit of a newbie and probably a very basic question to you, but is driving me mad.
I have a mysql table. I have a record which has a record id, and six other fields which contain names of photographs.
So,
Record_id, photo1, photo2, photo3, photo4, photo5, photo6
I want have have the end result where I have a column called 'TOTAL' which shows the number of how many records have photographs in them as a TOTAL. So, Record 1 may have a total of 5 photographs, record 2 may have 3 photographs, and record 3 may have a total of NO records for that record etc?
Now some of the photographs names which are uploaded to the field may contain numbers, ie, photo244.jpg, or pic3993.jpg etc - so I don't want these numbers added together, I am just looking for whether the field has a name in it etc, and this would be counted as 1 to add to the total. So the end result in on the page is that record 13 (for example) has a total of 4 photographs etc - sorry if not a good explanation!
It may have something to do with SUM or COUNT, but have spent ages trying to get combination or correct key/syntax! Can anyone help please by giving an example of the mysql select statement - would be grateful!
I presume that if there is no photo, then the field photoX is null or is empty.
So use CASE for output 0 if the field is null or empty, 1 else.
SELECT Record_id
,
CASE WHEN (photo1 IS NULL OR photo1 = '') THEN 0 ELSE 1 END
+ CASE WHEN (photo2 IS NULL OR photo2 = '') THEN 0 ELSE 1 END
+ CASE WHEN (photo3 IS NULL OR photo3 = '') THEN 0 ELSE 1 END
+ CASE WHEN (photo4 IS NULL OR photo4 = '') THEN 0 ELSE 1 END
+ CASE WHEN (photo5 IS NULL OR photo5 = '') THEN 0 ELSE 1 END
+ CASE WHEN (photo6 IS NULL OR photo6 = '') THEN 0 ELSE 1 END
AS Total
FROM myTable
It will the best query:
SELECT
Record_id,
(6 - IFNULL(photo1,1) - IFNULL(photo2,1) - IFNULL(photo3,1) - IFNULL(photo4,1) - IFNULL(photo5,1) - IFNULL(photo6,1)) AS TOTAL
FROM
table
Something like this will give you a count by Photo1, Photo2, etc.:
SELECT
SUM(CASE WHEN photo1 IS NOT NULL THEN 1 END) AS Photo1Count,
SUM(CASE WHEN photo2 IS NOT NULL THEN 1 END) AS Photo2Count,
SUM(CASE WHEN photo3 IS NOT NULL THEN 1 END) AS Photo3Count,
SUM(CASE WHEN photo4 IS NOT NULL THEN 1 END) AS Photo4Count,
SUM(CASE WHEN photo5 IS NOT NULL THEN 1 END) AS Photo5Count
FROM MyTable
Something like this will count how many rows have at least one photograph:
SELECT SUM (
CASE WHEN
photo1 IS NOT NULL OR
photo2 IS NOT NULL OR
photo3 IS NOT NULL OR
photo4 IS NOT NULL OR
photo5 IS NOT NULL THEN 1 END)
FROM MyTable
Note that these queries are for grand totals. They don't show activity by row. If activity by row is needed, try the answers from Scorpi0 or Michael Sivolobov.