MySQL complex subquery formulation - mysql

I have two tables - books and images. books has columns like id, name, releasedate, purchasecount. images has bookid (which is same as the id in books, basically one book can have multiple images. Although I haven't set any foreign key constraint), bucketid, poster (each record points to an image file in a certain bucket, for a certain bookid).
Table schema:
poster is unique in images, hence it is a primary key.
Covering index on books: (name, id, releasedate)
Covering index on images: (bookid,poster,bucketid)
My query is, given a name, find the top ten books (sorted by number of purchasecount) from the books table whose name matches that name, and for that book, return any (preferably the first) record (bucketid and poster) from the images table.
Obviously this can be solved by two queries by running the first, and using its results to query the images table, but that will be slow, so I want to use 'join' and subquery to do it in one go. However, what I am trying is not giving me correct results:
select books.id,books.name,year(releasedate),purchasecount,bucketid,poster from books
inner join (select bucketid,bookid, poster from images) t on
t.bookid = books.id where name like "%foo%" order by purchasecount desc limit 2;
Can anybody suggest an optimal query to fetch the result set as desired here (including any suggestion to change the table schema to improve search time) ?
Updated fiddle: http://sqlfiddle.com/#!9/17c5a8/1.
The example query should return two results - fooe and fool, and one (any of the multiple posters corresponding to each book) poster for each result. However I am not getting correct results. Expected:
fooe - 1973 - 459 - 11 - swt (or fooe - 1973 - 459 - 11 - pqr)
fool - 1963 - 456 - 12 - xxx (or fool - 1963 - 456 - 111 - qwe)

I agree with Strawberry about the schema. We can discuss ideas for better performance and all that. But here is my take on how to solve this after a few chats and changes to the question.
Note below the data changes to deal with various boundary conditions which include books with no images in that table, and tie-breaks. Tie-breaks meaning using the max(upvotes). The OP changed the question a few times and added a new column in the images table.
Modified quetion became return 1 row make per book. Scratch that, always 1 row per book even if there are no images. The image info to return would be the one with max upvotes.
Books table
create table books
( id int primary key,
name varchar(1000),
releasedate date,
purchasecount int
) ENGINE=InnoDB;
insert into books values(1,"fool","1963-12-18",456);
insert into books values(2,"foo","1933-12-18",11);
insert into books values(3,"fooherty","1943-12-18",77);
insert into books values(4,"eoo","1953-12-18",678);
insert into books values(5,"fooe","1973-12-18",459);
insert into books values(6,"qoo","1983-12-18",500);
Data Changes from original question.
Mainly the new upvotes column.
The below includes a tie-break row added.
create table images
( bookid int,
poster varchar(150) primary key,
bucketid int,
upvotes int -- a new column introduced by OP
) ENGINE=InnoDB;
insert into images values (1,"xxx",12,27);
insert into images values (5,"pqr",11,0);
insert into images values (5,"swt",11,100);
insert into images values (2,"yyy",77,65);
insert into images values (1,"qwe",111,69);
insert into images values (1,"blah_blah_tie_break",111,69);
insert into images values (3,"qwqqe",14,81);
insert into images values (1,"qqawe",8,45);
insert into images values (2,"z",81,79);
Visualization of a Derived Table
This is just to assist in visualizing an inner piece of the final query. It demonstrates the gotcha for tie-break situations, thus the rownum variable. That variable is reset to 1 each time the bookid changes otherwise it increments. In the end (our final query) we only want rownum=1 rows so that max 1 row is returned per book (if any).
Final Query
select b.id,b.purchasecount,xDerivedImages2.poster,xDerivedImages2.bucketid
from books b
left join
( select i.bookid,i.poster,i.bucketid,i.upvotes,
#rn := if(#lastbookid = i.bookid, #rn + 1, 1) as rownum,
#lastbookid := i.bookid as dummy
from
( select bookid,max(upvotes) as maxup
from images
group by bookid
) xDerivedImages
join images i
on i.bookid=xDerivedImages.bookid and i.upvotes=xDerivedImages.maxup
cross join (select #rn:=0,#lastbookid:=-1) params
order by i.bookid
) xDerivedImages2
on xDerivedImages2.bookid=b.id and xDerivedImages2.rownum=1
order by b.purchasecount desc
limit 10
Results
+----+---------------+---------------------+----------+
| id | purchasecount | poster | bucketid |
+----+---------------+---------------------+----------+
| 4 | 678 | NULL | NULL |
| 6 | 500 | NULL | NULL |
| 5 | 459 | swt | 11 |
| 1 | 456 | blah_blah_tie_break | 111 |
| 3 | 77 | qwqqe | 14 |
| 2 | 11 | z | 81 |
+----+---------------+---------------------+----------+
The significance of the cross join is merely to introduce and set starting values for 2 variables. That is all.
The results are the top ten books in descending order of purchasecount with the info from images if it exists (otherwise NULL) for the most upvoted image. The image selected honors tie-break rules picking the first one as mentioned above in the Visualization section with rownum.
Final Thoughts
I leave it to the OP to wedge in the appropriate where clause at the end as the sample data given had no useful book name to search on. That part is trivial. Oh, and do something about the schema for the large width of your primary keys. But that is off-topic at the moment.

Related

MySQL/MariaDB find one or more numbers in list, matching lottery numbers with past results

I have a MariaDB table with an archive of past lottery results, imagine EuroMillions or Powerball lotteries.
For example on EuroMillions numbers go from 1 to 50 and then the extra balls from 1 to 12, each result is 5 numbers form the main pool and 2 from the extra pool. So my historic results table could look like this:
Lottery Results table
(other columns like id, date, draw number, etc) | main_numbers | extra_numbers | (timestamp columns)
... | 1,2,3,4,5 | 1,2 | ...
... | 3,12,34,35,45 | 5,11 | ...
... | 4,15,34,39,45 | 10,11 | ...
... | 7,11,25,28,44 | 10,12 | ...
(you get the idea, I have thousands of records...)
So I could select main_numbers and get result "3,12,34,35,45" for that second example row. And for the extra_numbers I would get "5,11".
What I want is to given a set of numbers for main and extra to see if they match any of my results, finding any number of numbers (numbered lottery balls).
So for example if I SELECT to find main_numbers "5,9,22,34,45" with extra_numbers "2,11" I would get (from my extracted example) two records:
... | 3,12,34,35,45 | 5,11 | ...
... | 4,15,34,39,45 | 10,11 | ...
Matching two main numbers and one extra number, in this case finding lottery prizes in the results table. Makes sense?
I'm using MariaDB and I'm a bit lost on how to proceed, I tried WHERE IN, FIELD_IN_SET, etc.
Is there a way to perform a SELECT to find results in only one statement or do I have to pick all records and then iterate elsewhere, php for example?
My aim would be to have it in one statement, so I could just send the numbers and get the matching records... Possible?
I hope this makes sense.
Many thanks for your answers.
Consider the following.
For simplicity, let's say that a lottery comprises 3 main balls, and two bonus balls:
DROP TABLE IF EXISTS lottery_results;
CREATE TABLE lottery_results
(draw_id INT NOT NULL
,ball_no INT NOT NULL
,ball_val INT NOT NULL
,PRIMARY KEY(draw_id,ball_no)
);
INSERT INTO lottery_results VALUES
(1,1,22),
(1,2,35),
(1,3,62),
(1,4,27),
(1,5,17),
(2,1,18),
(2,2,33),
(2,3,49),
(2,4, 4),
(2,5,35);
And we want to find all results where 34, 35, or 36 were drawn as a main number...
SELECT draw_id
FROM lottery_results
WHERE ball_no <=3
AND ball_val IN(34,35,36);
+---------+
| draw_id |
+---------+
| 1 |
+---------+
Thanks Strawberry,
I found a solution if I have all numbers in distinct columns, but could I find if they are in the same column in CSV?
So if I put my CSV in distinct columns for numbers (n_1...n_5) and extra numbers for the stars in (s_1, s_2) I can seek matched in those multiple columns.
This is using multiple columns:
To find matches numbers 1,2,3,4,5 with stars 1,2...
In EuroMillions you get a prize with 2 or more numbers and any star (one or two).
SELECT
main_numbers, extra_numbers,
((n_1 IN (1,2,3,4,5)) +
(n_2 IN (1,2,3,4,5)) +
(n_3 IN (1,2,3,4,5)) +
(n_4 IN (1,2,3,4,5)) +
(n_5 IN (1,2,3,4,5))) AS matched_numbers,
((s_1 IN (1,2)) +
(s_2 IN (1,2))) AS matched_stars,
created_at
FROM `lottery_results_archive`
HAVING matched_numbers >= 3 OR matched_numbers = 2 AND matched_stars > 0
ORDER BY matched_numbers DESC, matched_stars DESC, created_at DESC
Makes sense?
Thanks.

Does MySQL have a way to "coalesce" as an aggregate function?

I'm attempting to take an existing application and re-architect the schema to support new customer requests and fix several outstanding issues (mostly around our current schema being heavily denormalized). In doing so, I've reached an interesting problem which at first glance seems to have a simple solution, but I can't seem to find the function I'm looking for.
The application is a media organization tool.
Our Old Schema:
Our old schema had separate models for "Groups", "Subgroups", and "Videos". A Group could have many Subgroups (one-to-many) and a Subgroup could have many Videos (one-to-many).
There were certain fields that were shared among Groups, Subgroups, and Videos. For instance, the Google Analytics ID to be used when the Video was embedded on a page. Whenever we displayed the embed page we would first look if the value was set on the Video. If not, we checked its Subgroup. If not, we checked its Group. The query looked roughly like so (I wish this were the real query, but unfortunately our application was written over many years by many junior developers, so the truth is much more painful):
SELECT
v.id,
COALESCE(v.google_analytics_id, sg.google_analytics_id, g.google_analytics_id) as google_analytics_id
FROM
Videos v
LEFT JOIN Subgroups sg ON sg.id = v.subgroup_id
LEFT JOIN Groups g ON g.id = sg.group_id
Pretty straight-forward. Now the issue we've run into is that customers want to be able to nest groups arbitrarily deep, and our schema clearly only allows for 2 levels (and, in fact, necessitates two levels - even if you only want one)
New Schema (First Pass):
As a first pass, I knew we'd want a basic tree structure for the Groups, so I came up with this:
CREATE TABLE Groups (
id INT PRIMARY KEY,
name VARCHAR(255),
parent_id INT,
ga_id VARCHAR(20)
)
We can then easily nest up to N levels deep with N joins like so:
SELECT
v.id,
COALESCE(v.ga_id, g1.ga_id, g2.ga_id, g3.ga_id, ...) as ga_id
FROM
Videos v
LEFT JOIN Groups g1 ON g1.id = v.group_id
LEFT JOIN Groups g2 ON g2.id = g1.parent_id
LEFT JOIN Groups g3 ON g3.id = g2.parent_id
...
There's obvious flaws with this approach: We don't know how many parents there will be so we don't know how many times we should JOIN, forcing us to implement a "max depth". Then even with a max depth, if a person only has a single level of groups we still perform multiple JOINs because our queries can't know how deep they need to go. MySQL offers recursive queries, but while looking into if that was the right option I found a smarter schema that produced the same results
New Schema (Take 2):
Looking into better ways to handle a tree structure, I learned about Adjacency Lists (my prior solution), Nested Sets, Materialized Paths, and Closure Tables. Other than Adjacency Lists (which depend on JOINs to grab the entire tree structure and so produces a single row with multiple columns per node on the tree), the other three solutions all return multiple rows for each node on the tree
I ended up going with a Closure Table solution like so:
CREATE TABLE Groups (
id INT PRIMARY KEY,
name VARCHAR(255),
ga_id VARCHAR(20)
)
CREATE TABLE Group_Closure (
ancestor_id INT,
descendant_id INT,
PRIMARY KEY (ancestor_id, descendant_id)
)
Now given a Video I can get all of its parents like so:
SELECT
v.id,
v.ga_id,
g.id,
g.ga_id
FROM
Videos v
JOIN Group_Closure gc ON v.group_id = gc.descendant
JOIN Groups g ON g.id = gc.ancestor;
This returns each group in the hierarchy as a separate row:
+------+---------+------+---------+
| v.id | v.ga_id | g.id | g.ga_id |
+------+---------+------+---------+
| 1 | abc123 | 2 | new_val |
| 1 | abc123 | 1 | default |
| 2 | NULL | 4 | xyz987 |
| 2 | NULL | 3 | NULL |
| 2 | NULL | 1 | default |
| 3 | NULL | 3 | NULL |
| 3 | NULL | 1 | default |
+------+---------+------+---------+
What I wish to do now is somehow achieve the same result I would have expected from using COALESCE on multiple self-joined Group tables: a single value for ga_id based on whichever node is "lowest" in the tree
Because I have multiple rows per Video, I suspect that this can be accomplished using GROUP BY and some kind of aggregate function:
SELECT
v.id,
COALESCE(v.ga_id, FIRST_NON_NULL(g.ga_id))
FROM
Videos v
JOIN Group_Closure gc ON v.group_id = gc.descendant
JOIN Groups g ON g.id = gc.ancestor
GROUP BY v.id, v.ga_id;
Note that because (ancestor, descendant) is my primary key, I believe the order of the group closure table can be guaranteed to always come back the same - meaning if I put the lowest node first, it will be the first row in the resulting query... If my understanding of this is incorrect, please let me know.
If you were to stick with an adjacency list, you could use a recursive CTE. This one traverses up from each video id value until it finds a non-NULL ga_id:
WITH RECURSIVE CTE AS (
SELECT id, ga_id, group_id
FROM videos
UNION ALL
SELECT CTE.id, COALESCE(CTE.ga_id, g.ga_id), g.parent_id
FROM `groups` g
JOIN CTE ON g.id = CTE.group_id AND CTE.ga_id IS NULL
)
SELECT id, ga_id
FROM CTE
WHERE ga_id IS NOT NULL
For my attempt to reconstruct your data from your question, this yields:
id ga_id
1 abc123
2 xyz987
3 default
Demo on dbfiddle

Remove Purge duplicate/multiplicate records from mariadb

Briefly: database imported from foreign source, so I cannot prevent duplicates, I can only prune and clean the database.
Foreign db changes daily, so, I want to automate the pruning process.
It resides on:
MariaDB v10.4.6 managed predominantly by phpMyadmin GUI v4.9.0.1 (both pretty much up to date as of this writing).
This is a radio browsing database.
It has multiple columns, but for me there are only few important:
StationID (it is unique entry number, thus db does not consider new entries as duplicates, all of them are unique because of this primary key)
There are no row numbers.
Name, url, home-page, country, etc
I do want to remove multiple url duplicated entries base on:
duplicate url has country to it, but some country values are NULL (=empty)
so I do want remove all duplicates except one containing country name, if there is one entry with it, if there is none, just one url, regardless of name (names are multilingual, so some duplicated urls have also various names, which I do not care for.
StationID (unique number, but not consecutive, also this is primary db key)
Name (variable, least important)
url (variable, but I do want to remove the duplicates)
country (variable, frequently NULL/empty, I want to eliminate those with empty entries as much as possible, if possible)
One url has to stay by any means (not to be deleted)
I have tried multitude of queries, some work for SELECT, but do NOT for DELETE, some hang my machine when executed. Here are some queries I tried (remember I use MariaDB, not oracle, or ms-sql)
SELECT * from `radio`.`Station`
WHERE (`radio`.`Station`.`Url`, `radio`.`Station`.`Name`) IN (
SELECT `radio`.`Station`.`Url`, `radio`.`Station`.`Name`
FROM `radio`.`Station`
GROUP BY `radio`.`Station`.`Url`, `radio`.`Station`.`Name`
HAVING COUNT(*) > 1)
This one should show all entries (not only one grouped), but this query hangs my machine
This query gets me as close as possible:
SELECT *
FROM `radio`.`Station`
WHERE `radio`.`Station`.`StationID` NOT IN (
SELECT MAX(`radio`.`Station`.`StationID`)
FROM `radio`.`Station`
GROUP BY `radio`.`Station`.`Url`,`radio`.`Station`.`Name`,`radio`.`Station`.`Country`)
However this query lists more entries:
SELECT *, COUNT(`radio`.`Station`.`Url`) FROM `radio`.`Station` GROUP BY `radio`.`Station`.`Name`,`radio`.`Station`.`Url` HAVING (COUNT(`radio`.`Station`.`Url`) > 1);
But all of these queries group them and display only one row.
I also tried UNION, INNER JOIN, but failed.
WITH cte AS..., but phpMyadmin does NOT like this query, and mariadb cli also did not like it.
I also tried something of this kind, published at oracle blog, which did not work, and I really had no clue what was what in this function:
select *
from (
select f.*,
count(*) over (
partition by `radio`.`Station`.`Url`, `radio`.`Station`.`Name`
) ct
from `radio`.`Station` f
)
where ct > 1
I did not know what f.* was, query did not like ct.
Given
drop table if exists radio;
create table radio
(stationid int,name varchar(3),country varchar(3),url varchar(3));
insert into radio values
(1,'aaa','uk','a/b'),
(2,'bbb','can','a/b'),
(3,'bbb',null,'a/b'),
(4,'bbb',null,'b/b'),
(5,'bbb',null,'b/b');
You could give the null countries a unique value (using coalesce), fortunately stationid is unique so:
select t.stationid,t.name,t.country,t.url
from radio t
join
(select url,max(coalesce(country,stationid)) cntry from radio t group by url) s
on s.url = t.url and s.cntry= coalesce(t.country,t.stationid);
Yields
+-----------+------+---------+------+
| stationid | name | country | url |
+-----------+------+---------+------+
| 1 | aaa | uk | a/b |
| 5 | bbb | NULL | b/b |
+-----------+------+---------+------+
2 rows in set (0.00 sec)
Translated to a delete
delete t from radio t
join
(select url,max(coalesce(country,stationid)) cntry from radio t group by url) s
on s.url = t.url and s.cntry <> coalesce(t.country,t.stationid);
MariaDB [sandbox]> select * from radio;
+-----------+------+---------+------+
| stationid | name | country | url |
+-----------+------+---------+------+
| 1 | aaa | uk | a/b |
| 5 | bbb | NULL | b/b |
+-----------+------+---------+------+
2 rows in set (0.00 sec)
Fix 2 problems at once:
Dup rows already in table
Dup rows can still be put in table
Do this fore each table:
CREATE TABLE new LIKE real;
ALTER TABLE new ADD UNIQUE(x,y); -- will prevent future dups
INSERT IGNORE INTO new -- IGNORE dups
SELECT * FROM real;
RENAME TABLE real TO old, new TO real;
DROP TABLE old;

MySql: Copy sql rows and keep some fields the same while updating others

I have a table with three columns, I want copy/paste all three columns within the same table, however, of the three columns, I want to update two columns with new data specific for that day while keeping one column the same. For the following table:
ticket_number | book_id | log_id
------------- | ------- | ------
1 | 1 | 120
12 | 2 | 120
23 | 3 | 120
I want to:
1) Copy all columns and paste into the same table
2) change the ticket_number column with new data for that day (e.g. 2, 13, 25) as well as the log_id column with the id for the current day (e.g. 121), while keeping book_id column the same.
I have tried with no avail:
INSERT INTO ticket (ticket_number, book_id, log_id) SELECT (2,13,24), (book_id), (121) FROM ticket;
This the schema for reference
Your SELECT query needs to return the rows that you want to insert.
UPDATE: You can use a separate table, which might be easier. Something like this:
CREATE TABLE id_map (
old_ticket_number NUMBER,
new_ticket_number NUMBER
);
You could insert the values into this table.
You can then use this query:
INSERT INTO ticket (ticket_number, book_id, log_id)
SELECT
m.new_ticket_number,
t.book_id,
121
FROM ticket t
INNER JOIN id_map m ON t.ticket_number = m.old_ticket_number;
Does this so what you're looking for?

How do I retrieve multiple random rows in MySQL without an id/auto_increment column?

My objective
I am trying to retrieve multiple random rows that contain only unique userid but for the type column to be random - type can only be 0 or 1. The table in question will contain less than 1,000 rows at any given time.
My table
CREATE TABLE tbl_message_queue (
userid bigint(20) NOT NULL,
messageid varchar(20) NOT NULL,
`type` int(1) NOT NULL,
PRIMARY KEY (userid,messageid,`type`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Sample data
userid | messageid | type
---------------------------------------------------
4353453 | 518423942 | 0
4353453 | 518423942 | 1
2342934 | 748475435 | 0
2342934 | 748475435 | 1
7657529 | 821516543 | 0
7657529 | 821516543 | 1
0823546 | 932843285 | 0
0823546 | 932843285 | 1
What to rule out
Using ORDER BY RAND() isn't feasible as over at least 18,000 of these types of queries are executed by applications at any given moment and are causing high load. Using SELECT DISTINCT or GROUP BY is (obviously) more efficient and will always pick unique userid but type will always equal to 0 with an acceptable load.
The common method is to create an id column but I'm looking for an alternative way only. The group primary key cannot change as it is required and deeply integrated into our application, however the structure of each column can be altered.
Thanks.
My understanding of your question is that for each userid you have two entries, but want to extract only one, at random.
To achieve this, you ought to generate a random value between 0 and 1 for each unique userid, and then JOIN this list with the starting list:
SELECT a.* FROM tbl_message_queue AS a
JOIN ( SELECT userid, FLOOR(2*RAND()) AS type
FROM tbl_message_queue GROUP BY userid ) AS b
ON ( a.userid = b.userid AND a.type = b.type );
But if an ORDER BY RAND() does not work for you, maybe we should compromise.
In the above sequence, any two userids will be uncorrelated -- i.e., the fact that user A gets type 0 tells you nothing about what user B will turn up with.
Depending on the use case, a less random (but "apparently random") sequence could be obtained with two queries:
SELECT #X := FLOOR(2*RAND()), #Y := POW(2,FLOOR(2+14*RAND()))-1;
SELECT * FROM tbl_message_queue WHERE (((userid % #Y) & 1) XOR type XOR #X);
This way, you can get what seems a random extraction. What really happens is that the userids are correlated, and you only have some couple dozens different extractions possible. But using only simple operators, and no JOINs, this query is very fast.