Related
I have generated a calendar table, containing every date from 2000-01-01 until 2050-12-31.
Apart from that I also have the user table, this table contains the following columns:
id, created, is_profile_public
And lastly, I have a table which links my users to 1 or many organisations (this is optional, not every user will be linked to an organisation). This table is called user_organisation.
I want to fetch data for statistical purposes where I get the data from the earliest create date of my user until yesterday. And missing dates should just contain 0 values in every column.
I have created this query:
SELECT c.datefield, DATE(u.created) AS created,
SUM(case when u.is_profile_public=1 AND uo.user_id is null then 1 else 0 end) as amount_public_volunteers,
SUM(case when u.is_profile_public=0 AND uo.user_id is null then 1 else 0 end) as amount_private_volunteers,
SUM(case when u.is_profile_public=1 AND uo.user_id is not null then 1 else 0 end) as amount_public_volunteers_admin,
SUM(case when u.is_profile_public=0 AND uo.user_id is not null then 1 else 0 end) as amount_private_volunteers_admin
FROM calendar AS c
LEFT OUTER JOIN user AS u ON c.datefield = DATE(u.created)
LEFT JOIN (select max(organisation_id), user_id from user_organisation group by user_id) AS uo on uo.user_id=u.id
WHERE u.id IN (87, 89, 172, 185, 186, 341, 342, 343, 344, 443, 444, 445,
446, 455, 459, 463, 20, 94, 61, 100, 101, 102, 109, 112,
113, 115, 132, 166, 184, 198, 199, 203, 205, 206, 207, 271,
272, 273, 274, 275, 276, 277, 280, 278, 279, 281, 284, 282,
283, 285, 288, 286, 287, 289, 292, 290, 291, 293, 294, 295,
302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313,
318, 314, 316, 315, 319, 317, 324, 325, 326, 328, 330, 332,
340, 358, 369, 383, 384, 391, 395, 396, 397, 398, 405, 399,
406, 400, 409, 401) AND (c.datefield BETWEEN (SELECT MIN(DATE(created)) FROM user) AND DATE(NOW()))
GROUP BY c.datefield
This shows me only the dates on which the users have been created. But it does not give me any rows back on the dates where no users were created.
I like to make counter for product filters like this one Online Shop , when you select any filter/option from GROUP X to count/update products into other GROUPS->filters/options but not in the current one
for example if this is frontend filters checkboxes
Size (group_id: 33)
10m (option_id: 52) (21 products)
20m (option_id: 51) (1 product)
Color (group_id: 32)
Green (option_id: 49) (22 products)
Black (option_id: 38) (1 product)
We are looking for result only from one category_id 127
Example of same group check counting
If option_id: 52 checked
Size (group_id: 33)
[x] 10m (option_id: 52) (21 products)
20m (option_id: 51) (1 product)
Color (group_id: 32)
Green (option_id: 49) (22 products)
Black (option_id: 38) (1 product)
Result:
option_id:38 0,
option_id:49 2,
option_id:51 1,
option_id:52 21
option_id:51 and 52 still have initial state
If option_id: 51 checked
Size (group_id: 33)
10m (option_id: 52) (21 products)
[x] 20m (option_id: 51) (1 product)
Color (group_id: 32)
Green (option_id: 49) (22 products)
Black (option_id: 38) (1 product)
Result:
38 0
49 1
51 1
52 21
option_id:51 and 52 still have initial state
Example of different group check counting
Size (group_id: 33)
[x] 10m (option_id: 52) (21 products)
20m (option_id: 51) (1 product)
Color (group_id: 32)
[x] Green (option_id: 49) (22 products)
Black (option_id: 38) (1 product)
Result:
38 0
49 2
51 1
52 2
all option should become updated and lose their initial state
When you select one or more option_id from same group_id logic of showing products will be
for example:
show products with size 10m and show products with 20m
if you select option_id:51 first it should not update option_id:52 becasue they are in same group but will update all option_id in group_id: 32 and so on
When you select option_id from different group_id logic of showing products will be
for example: show products with size 10m who also have color green (if is available)
#Akina has done most of the code for counting in this Topic
Working example of DB and Query
SELECT options.option_id,
COUNT(DISTINCT CASE WHEN filter_counter.option_id = options.option_id
THEN product_id
END) option_count
FROM filter_counter
CROSS JOIN ( SELECT DISTINCT option_id
FROM filter_counter ) options
JOIN ( SELECT DISTINCT product_id
FROM filter_counter
WHERE option_id IN (51) ) filter1 USING (product_id)
GROUP BY options.option_id;
CREATE TABLE `filter_counter` (
`id` int(11) NOT NULL,
`group_id` int(11) NOT NULL,
`option_id` int(11) NOT NULL,
`product_id` int(11) NOT NULL,
`category_id` int(11) NOT NULL,
`manufacturer_id` int(11) NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
INSERT INTO `filter_counter` (`id`, `group_id`, `option_id`, `product_id`, `category_id`, `manufacturer_id`) VALUES
(1, 33, 52, 5124, 65, 36),
(2, 33, 52, 5124, 127, 36),
(3, 33, 52, 5125, 65, 36),
(4, 33, 52, 5125, 127, 36),
(5, 33, 52, 5138, 65, 36),
(6, 33, 52, 5138, 127, 36),
(7, 33, 52, 5141, 65, 36),
(8, 33, 52, 5141, 127, 36),
(9, 33, 52, 5146, 65, 36),
(10, 33, 52, 5146, 127, 36),
(11, 33, 52, 5147, 65, 36),
(12, 33, 52, 5147, 127, 36),
(13, 33, 52, 5148, 65, 36),
(14, 33, 52, 5148, 127, 36),
(15, 33, 52, 5149, 65, 36),
(16, 33, 52, 5149, 127, 36),
(17, 33, 52, 5150, 65, 36),
(18, 33, 52, 5150, 127, 36),
(19, 33, 52, 5151, 65, 36),
(20, 33, 52, 5151, 127, 36),
(21, 33, 52, 5152, 65, 36),
(22, 33, 52, 5152, 127, 36),
(23, 33, 52, 5153, 65, 36),
(24, 33, 52, 5153, 127, 36),
(25, 33, 52, 5154, 65, 36),
(26, 33, 52, 5154, 127, 36),
(27, 33, 52, 5155, 65, 36),
(28, 33, 52, 5155, 127, 36),
(29, 33, 52, 5156, 65, 36),
(30, 33, 52, 5156, 127, 36),
(31, 33, 52, 5157, 65, 36),
(32, 33, 52, 5157, 127, 36),
(33, 33, 52, 7042, 65, 38),
(34, 33, 52, 7042, 127, 38),
(35, 33, 52, 7048, 65, 38),
(36, 33, 52, 7048, 127, 38),
(37, 33, 52, 7124, 65, 0),
(38, 33, 52, 7124, 127, 0),
(39, 32, 49, 7185, 65, 0),
(40, 32, 49, 7185, 127, 0),
(41, 32, 49, 7517, 65, 39),
(42, 32, 49, 7517, 127, 39),
(43, 32, 49, 7518, 65, 39),
(44, 32, 49, 7518, 127, 39),
(45, 32, 49, 7538, 65, 39),
(46, 32, 49, 7538, 127, 39),
(47, 32, 49, 7657, 65, 39),
(48, 32, 49, 7657, 127, 39),
(49, 32, 49, 7658, 65, 39),
(50, 32, 49, 7658, 127, 39),
(51, 32, 49, 7797, 65, 21),
(52, 32, 49, 7797, 127, 21),
(53, 32, 49, 7798, 65, 21),
(54, 32, 49, 7798, 127, 21),
(55, 32, 49, 7799, 65, 21),
(56, 32, 49, 7799, 127, 21),
(57, 32, 49, 7800, 65, 21),
(58, 32, 49, 7800, 127, 21),
(59, 32, 49, 7801, 65, 21),
(60, 32, 49, 7801, 127, 21),
(61, 32, 49, 7802, 65, 21),
(62, 32, 49, 7802, 127, 21),
(63, 32, 49, 7803, 65, 21),
(64, 32, 49, 7803, 127, 21),
(65, 32, 49, 7804, 65, 21),
(66, 32, 49, 7804, 127, 21),
(67, 32, 49, 7805, 65, 21),
(68, 32, 49, 7805, 127, 21),
(69, 32, 49, 7806, 65, 21),
(70, 32, 49, 7806, 127, 21),
(71, 32, 49, 7807, 65, 21),
(72, 32, 49, 7807, 127, 21),
(73, 32, 49, 7808, 65, 21),
(74, 32, 49, 7808, 127, 21),
(75, 32, 49, 7809, 65, 21),
(76, 32, 49, 7809, 127, 21),
(77, 32, 49, 7810, 65, 21),
(78, 32, 49, 7810, 127, 21),
(79, 32, 38, 7811, 65, 21),
(80, 32, 38, 7811, 127, 21),
(81, 32, 49, 8020, 65, 21),
(82, 32, 49, 8020, 127, 21),
(83, 33, 52, 8020, 65, 21),
(84, 33, 52, 8020, 127, 21),
(85, 32, 49, 8021, 65, 21),
(86, 32, 49, 8021, 127, 21),
(87, 33, 51, 8021, 65, 21),
(88, 33, 51, 8021, 127, 21),
(89, 33, 52, 8021, 65, 21),
(90, 33, 52, 8021, 127, 21);
(What is the "question"?)
I think the best way to implement such is to build a query based on the checkboxes, then process the data in a single pass in your application language.
Side issue: Switch from MyISAM to InnoDB.
Side issue: Shrink INT (which takes 4 bytes) to smaller datatypes.
The point behind these side issues is performance and space. Many of the generated queries will involve a full table scan, filtering as it goes. (That is, INDEXes will not be useful much of the time.)
The user can click multiple boxes in each grouping, correct? Then consider having, for example, TINYINT UNSIGNED, which has 8 bits (in 1 byte) to handle up all combinations of up to 8 choices. For example, instead of
AND size_id IN (0, 1) -- 0 means '65in+'; 1 means '50-65in'
do
AND ((size_opts & 0x3) = 0x3)
That is, the bottom bit of size_opts represents '65in+', etc.
This would shrink the dataset quite a bit and require different pre-processing to generate the queries.
Note that the value of your size_id is 0..4, my size_opts would have some or all of the bottom 5 bits on.
ORing together (1 << size_id) values gets you from size_id to size_opts.
(More)
There are 2 things going on:
Filtering (limiting the display by brand/price/color/whatever)
Counting how many are still in the running (by each remaining criteria)
To goals are needed to make it somewhat efficient:
Shrink the dataset as much as possible
Don't include in the table any items that are 'deleted' etc. Don't include any columns that are not relevant to the filtering and counting. They fight with the shrinkage goal
Shrink the categorization as much as possible. The "bit fiddling" I suggested may be optimal.
Build the query dynamically. Leave out unnecessary tests. (Example: When nothing is clicked for Brand, don't test for Brand.)
To rephrase, build a table just for this purpose. Focus on it; ignore all other aspects of the data. Even if the data in this table is redundant, you must optimize this table.
I have a query similar to:
SELECT
ANY_VALUE(name) AS `name`,
100 * SUM(score) / SUM(sum(score)) OVER (PARTITION BY date(scores.created_at)) AS `average_score`,
ANY_VALUE(DATE_FORMAT(scores.created_at, "%Y-%m-%d")) AS `shift_date`
FROM
`scores`
INNER JOIN `shifts` ON `shifts`.`id` = `scores`.`shift_id`
WHERE
`shifts`.`table_c_id` in(1, 2, 3, 4, 5, 6, 7, 8, 9, 10……)
AND date(`scores`.`created_at`) >= '2020-01-01'
GROUP BY
`name`,
date(scores.created_at)
ORDER BY
`shift_date` ASC;
The where in can be up to 2000 IDs which may not be sequential and the created_at where can be up to 14 months ago. Currently, at those levels, the execution time is 10-20 seconds.
I'm trying to optimise this. I've tried adding an index on created_at on the scores table but that had no effect. I also tried changing the date where clause to:
AND `scores`.`created_at` >= '2020-01-01 00:00:00
Which again made no difference.
Having read up on the topic, some recommended creating a temporary table but I can't see how this would have any benefit. I'm also not sure how to do this in one (is it even possible?) query.
The indexes on scores table are: shift_id, employee_id, name,created_at (used for another query). As I said, a created_at index didn't help this one.
The shifts table has indexes on table_c_id and created_at
Some sites suggest using WITH and CTEs, but again, I'm not sure how this would work or if the performance would actually improve.
The schema for scores and shifts is:
DROP TABLE IF EXISTS `scores`;
CREATE TABLE `scores` (
`id` bigint unsigned NOT NULL AUTO_INCREMENT,
`shift_id` int unsigned NOT NULL,
`hash` varchar(40) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL,
`name` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL,
`sscore` double(8,2) unsigned NOT NULL,
`created_at` timestamp NULL DEFAULT NULL
PRIMARY KEY (`id`),
KEY `scores_hash_index` (`hash`) USING BTREE,
KEY `scores_shift_id_index` (`shift_id`) USING BTREE,
KEY `scores_name_created_at_index` (`name`,`created_at`)
) ENGINE=InnoDB AUTO_INCREMENT=3140922 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
DROP TABLE IF EXISTS `shifts`;
CREATE TABLE `shifts` (
`id` bigint unsigned NOT NULL AUTO_INCREMENT,
`table_c_id` int unsigned NOT NULL,
`created_at` timestamp NULL DEFAULT NULL,
`updated_at` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `shifts_table_c_id_index` (`table_c_id`),
KEY `shifts_created_at_index` (`created_at`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=536392 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
Update
Using a lookup table for names:
names table: int unsigned, id, primary; varchar, name
SELECT
names.name AS `name`,
100 * SUM(score) / SUM(sum(score)) OVER (PARTITION BY date(scores.created_at)) AS `average_score`,
ANY_VALUE(DATE_FORMAT(scores.created_at, "%Y-%m-%d")) AS `shift_date`
FROM
`scores`
INNER JOIN `shifts` ON `shifts`.`id` = `scores`.`shift_id`
INNER JOIN `names` ON `names`.id = `scores`.`name_id`
WHERE
`shifts`.`table_c_id` in(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350, 351, 352, 353, 354, 355, 356, 357, 358, 359, 360, 361, 362, 363, 364, 365, 366, 367, 368, 369, 370, 371, 372, 373, 374, 375, 376, 377, 378, 379, 380, 381, 382, 383, 384, 385, 386, 387, 388, 389, 390, 391, 392, 393, 394, 395, 396, 397, 398, 399, 400, 401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 418, 419, 420, 421, 422, 423, 424, 425, 426, 427, 428, 429, 430, 431, 432, 433, 434, 435, 436, 437, 438, 439, 440, 441, 442, 443, 444, 445, 446, 447, 448, 449, 450, 451, 452, 453, 454, 455, 456, 457, 458, 459, 460, 461, 462, 463, 464, 465, 466, 467, 468, 469, 470, 471, 472, 473, 474, 475, 476, 477, 478, 479, 480, 481, 482, 483, 484, 485, 486, 487, 488, 489, 490, 491, 492, 493, 494, 495, 496, 497, 498, 499, 500, 501, 502, 503, 504, 505, 506)
AND `scores`.`created_at` >= '2019-04-03'
GROUP BY
`names`.`name`,
date(scores.created_at)
ORDER BY
`shift_date` ASC;
Has given no benefit. Also an index on scores table for shift_id, name_id and created_at hasn't helped.
Plan A: Avoid windowing functions (many of them are slower than one would think.)
SELECT
ANY_VALUE(brand_name),
100 * SUM(score) / init.tot AS `average_score`,
DATE(scores.created_at) AS `shift_date`
FROM
`scores`
INNER JOIN `shifts` ON `shifts`.`id` = `scores`.`shift_id`
JOIN ( SELECT SUM(score) AS tot FROM shifts
WHERE table_c_id IN (...)
AND `created_at` >= '2020-01-01' ) AS init
WHERE
`shifts`.`table_c_id` in(1, 2, 3, 4, 5, 6, 7, 8, 9, 10……)
AND `scores`.`created_at` >= '2020-01-01'
GROUP BY
shift_date,
`brand_name`
ORDER BY
`shift_date` ASC;
Notes:
several changes with the state syntax.
I assumed that name and brand_name were the same
By flipping the GROUP BY order, it may avoid a second sort.
I used a derived table to compute the grand total, thereby obviating the need for OVER.
This composite, covering, index on scores may help:
INDEX(created_at, shift_id)
Plan B: Use a CTE to compute SUM(score), then finish the query.
I have a problem, i have a query that just simply displays the user id in set,for retreieving the user id i am calling the function and it gives me the below list as string
SELECT u.user_id
FROM user u
WHERE u.user_id
IN (
'2, 3, 4, 5, 6, 7, 22, 33, 44, 55, 66, 77, 13, 23, 43, 53, 63, 73'
)
but when i execute this query it displays only the first user_id ie: 2 and all the user id are present in the database
So any help is deeply appreciated
Your code:
SELECT u.user_id FROM user u WHERE u.user_id IN ( '2, 3, 4, 5, 6, 7, 22, 33, 44, 55, 66, 77, 13, 23, 43, 53, 63, 73' )
http://forums.mysql.com/read.php?10,217174
Im almost surprised you had any match.. each of the numbers in your in list, need to be individual strings, eg '1','2','3' etc.
Remove single quotes like this and try code again-
SELECT u.user_id FROM user u WHERE u.user_id IN ( 2, 3, 4, 5, 6, 7, 22, 33, 44, 55, 66, 77, 13, 23, 43, 53, 63, 73);
Remove the single quotes:
IN ( 2, 3, 4, 5, 6, 7, 22, 33, 44, 55, 66, 77, 13, 23, 43, 53, 63, 73 )
This is probably something very simple, so forgive my blonde moment :)
I have a table 'album'
* albumId
* albumOwnerId (who created)
* albumCSD (create stamp date)
Now what I am trying to do is to select the top 10 most recently updated albums. But, I don't want 10 albums from the same person coming back - I only want one album per unique person. I.E 10 albums from 10 different people.
So, this is what I have below, but it is not working properly and I just can't figure out why. Any ideas?
Thanks
SELECT DISTINCT(albumOwnerId), albumId
FROM album
ORDER BY albumCSD DESC
LIMIT 0,10
Here is some example data, followed by what I am trying to get. Hope this makes it clearer.
DATA:
albumOwnerID, albumId, albumCSD
18, 194, '2010-10-23 11:02:30'
23, 193, '2010-10-22 11:39:59'
22, 192, '2010-10-12 21:48:16'
21, 181, '2010-10-12 20:34:11'
21, 178, '2010-10-12 20:20:16'
19, 168, '2010-10-12 18:31:55'
18, 167, '2010-10-11 21:06:55'
20, 166, '2010-10-11 21:01:47'
18, 165, '2010-10-11 21:00:32'
20, 164, '2010-10-11 20:50:06'
17, 145, '2010-10-10 18:54:24'
17, 144, '2010-10-10 18:49:28'
17, 143, '2010-10-10 18:48:08'
17, 142, '2010-10-10 18:46:54'
16, 130, '2010-10-10 16:17:57'
16, 129, '2010-10-10 16:17:26'
16, 128, '2010-10-10 16:07:21'
15, 119, '2010-10-10 15:24:28'
15, 118, '2010-10-10 15:24:11'
14, 100, '2010-10-09 18:22:49'
14, 99, '2010-10-09 18:18:46'
11, 98, '2010-10-09 15:50:13'
11, 97, '2010-10-09 15:44:09'
11, 96, '2010-10-09 15:42:28'
11, 95, '2010-10-09 15:37:25'
DESIRED DATA:
18, 194, '2010-10-23 11:02:30'
23, 193, '2010-10-22 11:39:59'
22, 192, '2010-10-12 21:48:16'
21, 181, '2010-10-12 20:34:11'
19, 168, '2010-10-12 18:31:55'
17, 145, '2010-10-10 18:54:24'
16, 130, '2010-10-10 16:17:57'
15, 119, '2010-10-10 15:24:28'
14, 100, '2010-10-09 18:22:49'
11, 98, '2010-10-09 15:50:13'
I get results, you want to have, with this query
SELECT albumOwnerID, albumId, albumCSD
FROM album
WHERE albumCSD in
(SELECT Max(album.albumCSD) AS MaxvonalbumCSD
FROM album
GROUP BY album.albumOwnerID);
However in MS Access
select albumOwnerID, albumID
from album
Group by albumOwnerID, albumID
Order by albumcsd desc
LIMIT 0,10
EDIT:
select albumOwnerID, albumID
from album
where albumOwnerID in (select distinct albumOwnerID from album order by albumCSD )
LIMIT 0,10