SQL query takes very long - mysql

I have a table, named Ads, which contains ~441.000 rows, and 21 columns.
I am trying to run the following query:
SELECT Ads.* FROM Ads WHERE Ads.countries_CountryId = 'FR'
I put an index, to the countries_CountryId field, which is of type char(2), but when I run the above query, it takes around 5-8 seconds to complete. This ammount of time looks immensly high for me, for such a medium sized table, and such a simple SQL query. Where should I look for the problem? Or is it normal for such a query to take so long?
I also tried to EXPLAIN the above query, and get the following results, but I don't know how to decipher this:(
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE Ads ref countries_CountryId countries_CountryId 2 const 24368 Using where
EDIT1: (response to Seth McClaine)
I've tried your suggestion, the way you suggested, returns an error, but if I run SELECT count(*) FROM Ads WHERE Ads.countries_CountryId = 'FR', it runs much much faster: 0.0052490234375
But the problem is, I am not printing out anything, I am using php to run the queries, and calculating their runtime for the moment, and at the end, outputting the queries run, and the time they took:
Array
(
[0] => Array
(
[query] => SET character_set_results = 'utf8', character_set_client = 'utf8', character_set_connection = 'utf8', character_set_database = 'utf8', character_set_server = 'utf8'
[duration] => 0.00481009483337
)
[1] => Array
(
[query] => SELECT * FROM Countries WHERE NameFormatted LIKE '%FRANCE%'
[duration] => 0.00234889984131
)
[2] => Array
(
[query] => SELECT Ads.* FROM Ads WHERE Ads.countries_CountryId = 'FR'
[duration] => 4.71820402145
)
[3] => Array
(
[query] => SELECT COUNT(*) FROM Ads WHERE Ads.countries_CountryId = 'FR'
[duration] => 0.0052490234375
)
)
Here's also the code snippet, that runs the queries:
public function query($query, $cacheit=true) {
if (!$this->isConnected()) {
$this->throwError("Querying <b>".$query."</b> failed, because MySQL is not connected!", self::MYSQL_NOT_CONNECTED, false);
return false;
}
$qstart=microtime(true);
$result=#mysql_query($query, $this->conn_resource);
$qduration=microtime(true) - $qstart;
if ($result===FALSE) $this->throwError("Querying <b>".$query."</b> failed!", self::MYSQL_QUERY_ERR);
$this->numQueries++;
$this->executedQueries[] = array(
'query' => $query,
'duration' => $qduration
);
return $result;
}

Have your tried executing the query in read-only mode?
As explained here in mysql documentation.
You may try starting your query with
START TRANSACTION READ ONLY
Also, you may want to try using explain statement to get more information about your query.
EXPLAIN SELECT Ads.* FROM ....
Hope this helps. Helped for me in several large queries.

Related

SQL query needed for a complex structure

I have a tricky SQL query that needs to be built to get the highest priority rule based on customer session and geo IP data.
I attached the following tables: rule, rule_attribute, rule_attribute_value.
rule - table where all rules are stored
Click here to see a screenshot of the 'rule' table
rule_attribute - table where all rule attributes are stored
Click here to see a screenshot of the 'rule_attribute' table
rule_attribute_value - table where all rule attribute values are stored
Click here to see a screenshot of the 'rule_attribute_value' table
When the customer logs in, I have access to all those attributes (customer_id, customer_group_id, country_id, subdivision_one_id, subdivision_two_id). Only customer_id and customer_group_id will always have values. The others are optional, but there is a dependency between them. We can't have subdivisions without selecting first a country. We can have a second subdivision without selecting a country and then the first subdivision.
What I would like to get is the highest priority rule that matches the session data in the most optimized way. I have a solution that involves some coding, but I want to see if it's possible directly through SQL.
Here are some examples of session data arrays:
Array
(
[customer_id] => 2
[customer_group_id] => 1
[current_store_id] => 0
[country_id] => 15
[subdivision_one_id] => 224
[subdivision_two_id] =>
)
Array
(
[customer_id] => 2
[customer_group_id] => 1
[current_store_id] => 0
[country_id] => 15
[subdivision_one_id] =>
[subdivision_two_id] =>
)
Array
(
[customer_id] => 3
[customer_group_id] => 2
[current_store_id] => 0
[country_id] =>
[subdivision_one_id] =>
[subdivision_two_id] =>
)
Without a better understanding of the rules and data this is the best I can come up with. It is based on your first array example -
SELECT `r`.*
FROM `rule_attribute_value` `rav`
INNER JOIN `rule` `r`
ON `rav`.`rule_id` = `r`.`rule_id`
INNER JOIN `rule_attribute` `ra`
ON `rav`.`attribute_id` = `ra`.`attribute_id`
WHERE
(`rav`.`store_id` = 0 AND `ra`.`attribute_code` = 'customer' AND `rav`.`value` = 2) OR
(`rav`.`store_id` = 0 AND `ra`.`attribute_code` = 'customer_group' AND `rav`.`value` = 1) OR
(`rav`.`store_id` = 0 AND `ra`.`attribute_code` = 'country' AND `rav`.`value` = 15) OR
(`rav`.`store_id` = 0 AND `ra`.`attribute_code` = 'subdivision_one' AND `rav`.`value` = 224)
GROUP BY `r`.`rule_id`
HAVING COUNT(DISTINCT `rav`.`attribute_id`) = 4 /* 4 IS THE NUMBER OF ATTRIBUTES BEING QUERIED */
ORDER BY `r`.`position` ASC
LIMIT 1;

WordPress query, meta_query, compare BETWEEN, no results

I have code for my filter. It worked well until I add new product in my database. I found the problem, but dont know what to do with that.
I have parameters "alc_min" and "alc_max" in my filter. I get these from crawling all products. After I send this filter, I fire this code:
$meta_query = array();
$b = "alc_min";
$c = "alc_max";
if (isset ( $data [$b] ) && isset ( $data [$c] )) {
$compare = "BETWEEN";
$a = array (
'key' => "alc",
'value' => array (
$data [$b],
$data [$c]
),
'compare' => $compare
);
array_push ( $meta_query, $a );
}
$items = new WP_Query ( array (
'post_type' => $type,
'posts_per_page' => $posts_per_page,
'order' => $order,
'meta_key' => $orderkey,
'orderby' => $orderby,
'post_status' => 'publish',
'meta_query' => $meta_query,
'paged' => $paged
) );
Until now, it worked well. No I add new product with "alc" <10 and I found, that if I have "alc_min" and "alc_max" <10 or >10, it is ok. But if "alc_min" is <10 and "alc_max" >10 I get no results at all.
Does anyone any idea what to check or fix?
After the clarification, I've suspected that the reason why selecting "alc_min" = 7 and "alc_max" = 13 doesn't yield any result is because of the column datatype. Consider this example:
CREATE TABLE table1 (
alc VARCHAR(50));
INSERT INTO table1 VALUES
('7'),
('9'),
('11'),
('13');
The table above is created with alc column datatype as VARCHAR instead of INTEGER (or numeric datatype). I've tested that running either one of the query below:
SELECT * FROM table1 WHERE alc BETWEEN '7' AND '9';
SELECT * FROM table1 WHERE alc BETWEEN '11' AND '13';
will return the expected result. However, with this query:
SELECT * FROM table1 WHERE alc BETWEEN '7' AND '13';
yields no result. This is because the values are treated as string instead of numbers and when that happens, 1 is always smaller than 7. See below what happen you run select query with order by on the data set above:
SELECT * FROM table1 ORDER BY alc;
+-----+
| alc |
+-----+
| 11 |
| 13 |
| 7 |
| 9 |
+-----+
As you can see, since the data is treated as string (according to the column datatype), then you could imagine this in alphabetical form as the following:
+-----+--------------+
| alc | alphabetical |
+-----+--------------+
| 11 | AA |
| 13 | AC |
| 7 | G |
| 9 | I |
+-----+--------------+
So, the condition of BETWEEN '7' AND '13' becomes BETWEEN 'G' AND 'AC'; which doesn't really make sense. And if you change to BETWEEN '11' AND '9' you'll get the correct result but that made the query even more confusing and not making sense at all.
Now, I've discovered that there are at least 3 workaround/solution for this:
One of the oldest way I can think of is by adding +0 to the column in the query. I didn't find any official docs about this but I assume that doing this will change the data value to numeric in the query:
SELECT * FROM table1
WHERE alc+0 BETWEEN '7' AND '13';
This is probably the same as above is just that I'm not sure if this is version specific or not. It turns out that in my testing, if you didn't wrap the searched value in quotes, you'll get the result as if the data is numeric:
SELECT * FROM table1
WHERE alc BETWEEN 7 AND 13;
This require a change of column datatype but afterwards any of the query with or without quotes on the searched value should work:
ALTER TABLE table1 CHANGE alc alc INT;
I hope that this is true and the issue is really about column datatype. As far as I know, this is the closest thing to what your situation is that I had experience with.
Here's a fiddle for reference

Most efficient way to sum overlapping datetimes in MySQL

I'm in the process of evaluating the proposed solutions on SO related to the sum of overlapping datetimes in MySQL. I wasn't able to find out a silver-bullet solution, so would like to know if any classic/industrial-grade algorithmic procedure is available or if a custom-made needs to be developed.
Total should be 8 hours (4+4).
Proposed solution through MySQL
function final_balance($teacher_id, $aa, $teaching_id=0) {
$dbo = $this->Attendance->getDataSource();
$years=$this->Attendance->Student->Year->find('list', array('fields' => array('anno', 'data_from')));
$filteraa='attendances.start>="'.$years[$aa].'"';
$this->query('SET #interval_id = 0');
$this->query('SET #interval_end = \'1970-01-01 00:00:00\'');
$sql='SELECT
MIN(start) as start,
MAX(end) as end
FROM (
SELECT
#interval_id := IF(attendances.start > #interval_end, #interval_id + 1, #interval_id) AS interval_id,
#interval_end := IF(attendances.start < #interval_end, GREATEST(#interval_end, attendances.end), attendances.end) AS interval_end,
attendances.start,
attendances.end
FROM attendances
INNER JOIN attendance_sheets ON (
attendance_sheet_id = attendance_sheets.id AND
attendance_sheets.teacher_id='.$teacher_id.' AND '.$filteraa.' AND
attendance_sheet_status_id = 2 AND
attendance_status_id!=3'.
($teaching_id?' AND attendances.teaching_id IN ('.$teaching_id.')':'').'
)
ORDER BY attendances.start,attendances.end
) intervals GROUP BY interval_id';
// final query to sum in the temp table
$finalStatement =array(
'table' => $dbo->expression('('.$sql.')')->value,
'alias' => 'Attendance',
'fields' => array(
'DATE_FORMAT(start, \'%d/%m/%Y\') as data',
'DATE_FORMAT(start, \'%m-%Y\') as datamese',
'DATE(start) as datasql',
$teacher_id.' AS teacher_id',
'DAY(start) as giorno',
'MONTH(start) as mese',
'YEAR(start) as anno',
'SEC_TO_TIME(SUM((TIME_TO_SEC(end) - TIME_TO_SEC(start)))) as ore',
),
'conditions' => array(),
'limit' => null,
'group' => array('CONCAT(YEAR(start),MONTH(start))', 'DATE(start) WITH ROLLUP'),
'order' => null
);
$finalQuery= $dbo->buildStatement($finalStatement, $this->Attendance);
return $this->Attendance->query($dbo->expression($finalQuery)->value);
}
References
Sum amount of overlapping datetime ranges in MySQL
performs a different task
MySQL: sum time ranges exluding overlapping ones
and
MySQL: sum datetimes without counting overlapping periods twice
both seems to me like not considering all the cases
GeeksForCode: Merge Overlapping Intervals
Depending on the circumstances, the following might be useful and efficient.
Create another table that has one row per hour. Inner join that table with your table while selecting only the new column and dedupe the rows.
You can keep increasing the resolution (eg. to minutes or seconds), but that might make your code run slow.

Mysql JSON column any index ( [*] ) not working

I have a JSON column named prices_and_tags in which there are values in JSON array:
[{'price' => 100, 'tag' => 'Watch'}, {'price' => 200, 'tag' => 'Book'}]
If I want to search prices_tags where tag = 'Watch' in any array index, it's not working.
Query is like this:
select * from products where prices_and_tags->"$[*].tag" = 'Watch'
this gives 0 results.
But this query works for first or second array index, like this:
select * from products where prices_and_tags->"$[0].tag" = 'Watch'
I referred to this: https://dev.mysql.com/doc/refman/5.7/en/json-path-syntax.html for query syntax.
Thanks.
Have you tried this
select * from products where prices_and_tags->"$.tag[*]" = 'Watch'
Ref: https://dev.mysql.com/doc/refman/5.7/en/json.html

How to use the table columns instead of variables in QueryExpression::addCase()

In CakePHPs new ORM, you can use the QueryBuilder to build (in theory) any query.
I want to select the value of one of two columns, depending on another value. In a regular query, that can be done as follows:
SELECT IF(from_id = 1, to_id, from_id) AS other_id FROM messages;
I am trying to archive the same query using the QueryBuilder and QueryExpression::addCase()
$messagesQuery = $this->Messages->find('all');
$messagesQuery->select([
'other_id' => $messagesQuery->newExpr()->addCase(
$messagesQuery->newExpr()->add(['from_id' => $this->authUser->id]),
['to_id', 'from_id'],
['integer', 'integer']
)
]);
This does not work, as the passed values are not integers, but rather table columns containing integers.
Through trial and error (using the method add() again), I got the following:
$messagesQuery = $this->Messages->find('all');
$messagesQuery->select([
'other_id' => $messagesQuery->newExpr()->addCase(
$messagesQuery->newExpr()->add(['from_id' => $this->authUser->id]),
[
$messagesQuery->newExpr()->add(['to_id']),
$messagesQuery->newExpr()->add(['from_id'])
],
['integer', 'integer']
)
]);
This results in the following query:
SELECT (CASE WHEN from_id = 1 THEN to_id END) AS `other_id` FROM messages Messages
Now, the ELSE part is missing, although the CakePHP book states:
Any time there are fewer case conditions than values, addCase will automatically produce an if .. then .. else statement.
The examples in the CakePHP book are not very helpful in this case, as they only use static integers or strings as values, for example:
#SELECT SUM(CASE published = 'Y' THEN 1 ELSE 0) AS number_published, SUM(CASE published = 'N' THEN 1 ELSE 0) AS number_unpublished FROM articles GROUP BY published
$query = $articles->find();
$publishedCase = $query->newExpr()->addCase($query->newExpr()->add(['published' => 'Y']), 1, 'integer');
$notPublishedCase = $query->newExpr()->addCase($query->newExpr()->add(['published' => 'N']), 1, 'integer');
$query->select([
'number_published' => $query->func()->sum($publishedCase),
'number_unpublished' => $query->func()->sum($unpublishedCase)
])
->group('published');
Is there a way to get the method addCase to use the two table columns as values instead of just static values?
As it turns out, I was just one logical step short of the solution in my previous edit.
As the CakePHP book correctly states:
Any time there are fewer case conditions than values, addCase will automatically produce an if .. then .. else statement.
For that to work though, both the conditions and values have to be an array, even if there is only one condition. (This the CakePHP book does not state.)
This code:
$messagesQuery = $this->Messages->find('all');
$messagesQuery->select([
'other_id' => $messagesQuery->newExpr()->addCase(
[
$messagesQuery->newExpr()->add(['from_id' => $this->authUser->id])
],
[
$messagesQuery->newExpr()->add(['to_id']),
$messagesQuery->newExpr()->add(['from_id'])
],
['integer', 'integer']
)
]);
results in this query:
SELECT (CASE WHEN from_id = 1 THEN to_id ELSE from_id END) AS `other_id` FROM messages Messages
Eureka