I got a list of IDs as comma seperated list of JSON values and some example data sets are like below [340596,340597,340595]
This list can be huge sometimes 50k IDs seperated by commas
The following query connects these IDs to a table primary key and fetch the records that is currently exists in the table
SELECT s.id,s.contactid, s.Quantity FROM
JSON_TABLE('[340596,340597,340595]', '$[*]' columns (Id int path '$')) AS sm
LEFT JOIN mastertable s ON s.Id = sm.id
The mastertable might contain these IDs or might be these records are erased from mastertable So the purpose of this query is to ensure the return result set contains only the active records
I have to apply one more filtering against this query and the filtering is based on another JSON int array and need to match it against the column ContactID
SELECT s.id,s.contactid, s.Quantity FROM
JSON_TABLE('[340596,340597,340595]', '$[*]' columns (Id int path '$')) AS sm
LEFT JOIN mastertable s ON s.Id = sm.id
WHERE s.ContactId IN (
SELECT cm.id FROM
JSON_TABLE('[12345,450597,640595]', '$[*]' columns (Id int path '$')) AS cm
)
However the Mysql IN perfomance is not better for large result sets . Can we replace this IN with some other better ways?
You can dump the ids inside IN clause in a temporary table and then join them with JSON_TABLE to get the result.
ALternatively you can use a CTE and join the same.
with temp as (
SELECT cm.id FROM
JSON_TABLE('[12345,450597,640595]', '$[*]' columns (Id int path '$')) AS cm
)
SELECT s.id,s.contactid, s.Quantity FROM
JSON_TABLE('[340596,340597,340595]', '$[*]' columns (Id int path '$')) AS sm
LEFT JOIN mastertable s ON s.Id = sm.id
INNER JOIN temp t ON s.ID = t.id;
Related
I am storing a list of integers as JSON array inside the column called ConvertedIds inside a table SelectionLogs
The type of column is MediumText and some of the example values of the column are [290766,319075,234525,325364,3472,34241,85643,11344556,88723,656378]
I am using following sql to generate the list of IDs from the column as rows
SELECT hm.id FROM SelectionLogs hlog,
JSON_TABLE(ConvertedIds, '$[*]' columns (Id int path '$')) AS hm
And then following query to extract further informations from other tables like
SELECT hm.id,hc.firstname ,hc.lastname ,hc.email FROM SelectionLogs hlog,
JSON_TABLE(ConvertedIds, '$[*]' columns (Id int path '$')) AS hm
LEFT JOIN contacts hc ON hc.Id = hm.id
Now i have to update this column based on the presence of a given value of IDs
For example if an ID exists in this column on any rows , i have to update the array after removing the ID
For example: [1,2,3,4,5,6,7] If ID : 3 exists , remove 3 and update column as [1,2,4,5,6,7]
I can use the following query to find the records from table SelectionLogs with given id present in column ConvertedIds
SELECT DISTINCT hlog.Id FROM SelectionLogs hlog,
JSON_TABLE(ConvertedIds, '$[*]' columns (Id int path '$')) AS hm
WHERE hm.id=427529
Now i have plans to iterate through each rows from my console program written in c#
foreach row in result
List<int> columnIds = read from column ConvertedIds as list of int
Remove the given int number from List
Update column ConvertedIds for given rowId refreshed List
Can i perform the updation via SQL itself ?
DEMO fiddle with some explanations.
-- source data
CREATE TABLE t1 (id INT, val JSON)
SELECT 1 id, '[1,2,3]' val UNION SELECT 2, '[3,4,5]' UNION SELECT 3, '[5,6,7]';
CREATE TABLE t2 (id INT) SELECT 1 id UNION SELECT 4;
-- UPDATE source table
UPDATE t1
JOIN ( SELECT t1.id, JSON_ARRAYAGG(jsontable.id) val
FROM t1
CROSS JOIN JSON_TABLE(t1.val,
'$[*]' COLUMNS (id INT PATH '$')) jsontable
LEFT JOIN t2 ON t2.id = jsontable.id
WHERE t2.id IS NULL
GROUP BY t1.id ) data_for_update USING (id)
SET t1.val = data_for_update.val;
I have a table with following structure
Table name: matches
That basically stores which product is matching which product. I need to process this table
And store in a groups table like below.
Table Name: groups
group_ID stores the MIN Product_ID of the Product_IDS that form a group. To give an example let's say
If A is matching B and B is Matching C then three rows should go to group table in format (A, A), (A, B), (A, C)
I have tried looking into co-related subqueries and CTE, but not getting this to implement.
I need to do this all in SQL.
Thanks for the help .
Try this:
;WITH CTE
AS
(
SELECT DISTINCT
M1.Product_ID Group_ID,
M1.Product_ID
FROM matches M1
LEFT JOIN matches M2
ON M1.Product_Id = M2.matching_Product_Id
WHERE M2.matching_Product_Id IS NULL
UNION ALL
SELECT
C.Group_ID,
M.matching_Product_Id
FROM CTE C
JOIN matches M
ON C.Product_ID = M.Product_ID
)
SELECT * FROM CTE ORDER BY Group_ID
You can use OPTION(MAXRECURSION n) to control recursion depth.
SQL FIDDLE DEMO
Something like this (not tested)
with match_groups as (
select product_id,
matching_product_id,
product_id as group_id
from matches
where product_id not in (select matching_product_id from matches)
union all
select m.product_id, m.matching_product_id, p.group_id
from matches m
join match_groups p on m.product_id = p.matching_product_id
)
select group_id, product_id
from match_groups
order by group_id;
Sample of the Recursive Level:
DECLARE #VALUE_CODE AS VARCHAR(5);
--SET #VALUE_CODE = 'A' -- Specify a level
WITH ViewValue AS
(
SELECT ValueCode
, ValueDesc
, PrecedingValueCode
FROM ValuesTable
WHERE PrecedingValueCode IS NULL
UNION ALL
SELECT A.ValueCode
, A.ValueDesc
, A.PrecedingValueCode
FROM ValuesTable A
INNER JOIN ViewValue V ON
V.ValueCode = A.PrecedingValueCode
)
SELECT ValueCode, ValueDesc, PrecedingValueCode
FROM ViewValue
--WHERE PrecedingValueCode = #VALUE_CODE -- Specific level
--WHERE PrecedingValueCode IS NULL -- Root
I have a table with a bunch of columns, but we only need to look at two of them. I'm trying to join another table on this table, but all we know about these two columns is that one will be null and the other won't:
client_id | note_id
The main table wants to join client_id (if not null) on clients.id OR note_id on notes.id if clients.id is null.
This will work for you. This is very basic query I wrote. Make changes if required.
SELECT * FROM YOUR_TABLE t
LEFT OUTER JOIN clients c ON t.client_id = c.id
LEFT OUTER JOIN notes n ON t.note_id = n.id
WHERE c.id IS NOT NULL OR n.id IS NOT NULL
Assuming there are 3 tables involved (the main table that contains client_id and note_id columns, clients table, and notes table), you can use a query such as this:
(select *
from mainTable inner join clients on mainTable.client_id = clients.id)
union
(select *
from mainTable inner join notes on mainTable.note_id = notes.id
where mainTable.client_id is NULL);
The above query contains 2 queries where each query will output rows where the joining column is not null. The results are then combined using union.
You can use coalesce in the join on clause. See demo here:
http://sqlfiddle.com/#!9/99911/2. If client id is null then use note id to join table1 and table2.
Select t1.client_id, t1.note_id,t2.client_id, t2.note_id
From table1 t1
Join table2 t2
on coalesce(t1.client_id, t1.note_id) =coalesce(t2.client_id, t2.note_id)
There are 4 columns in table A, id, name, create_time and content.
create table A
(
id int primary key,
name varchar(20),
create_time datetime,
content varchar(4000)
);
create table B like A;
I want to select max create_time records in the same name, and insert into another table B.
Execute sql as follow, but the time consumption is unacceptable.
insert into B
select A.*
from A,
(select name, max(create_time) create_time from B group by name) tmp
where A.name = tmp.name
and A.create_time = tmp.create_time;
A table has 1000W rows and 10GB, execute sql spend 200s.
Is there any way to do this job faster, or change which parameters in MySQL Server to run faster.
p:
table A can be any type, paration table or some else.
First be sure you have proper index on A (name, create_time) and B (name, create_time)
then try using explicit join and on condtion
insert into B
select A.*
from A
inner join (
select name, max(create_time) create_time
from B
group by name) tmp on ( A.name = tmp.name and A.create_time = tmp.create_time)
The query you need is:
INSERT INTO B
SELECT m.*
FROM A m # m from "max"
LEFT JOIN A l # l from "later"
ON m.name = l.name # the same name
AND m.create_time < l.create_time # "l" was created later than "m"
WHERE l.name IS NULL # there is no "later"
How it works:
It joins A aliased as m (from "max") against itself aliased as l (from "later" than "max"). The LEFT JOIN ensures that, in the absence of a WHERE clause, all the rows from m are present in the result set. Each row from m is combined with all rows from l that have the same name (m.name = l.name) and are created after the row from m (m.create_time < l.create_time). The WHERE condition keeps into the results set only the rows from m that do not have any match in l (there is no record with the same name and greater creation time).
Discussion
If there are more than one rows in A that have the same name and creation_time, the query returns all of them. In order to keep only one of them and additional condition is required.
Add:
OR (m.create_time = l.create_time AND m.id < l.id)
to the ON clause (right before WHERE). Adjust/replace the m.id < l.id part of the condition to suit your needs (this version favors the rows inserted earlier in the table).
Make sure the table A has indexes on the columns used by the query (name and create_time). Otherwise the performance improvement compared with your original query is not significant.
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Mysql: Perform of NOT EXISTS. Is it possible to improve permofance?
Is there a better/optimal way to do it. Should I use exists instead of join? Or two separate queries? And what about temporary tables, as I was reading about those but uncertain.
Getting members email from a group. Checking that they have not received a item yet.
SELECT m.email,g.id
FROM group g
LEFT JOIN members m
ON g.mid = m.id
AND g.gid='1'
WHERE NOT EXISTS
( SELECT id
FROM items AS i
WHERE i.mid=m.id
AND i.item_id='5'
)
Here's the same thing written as a JOIN:
SELECT m.email, g.id
From members m
JOIN group g ON g.mid = m.id AND g.gid = '1'
LEFT JOIN items i ON i.mid = m.id AND i.item_id = '5'
WHERE i.id IS NULL
Use the following compound indexes:
group (mid, gid)
items (mid, item_id)
I reversed the LEFT JOIN on members and group because it seems like you're returning members, not groups, and I changed the LEFT JOIN into an INNER JOIN since you only want members from that group.
I think this one might read better:
SELECT m.email, g.id
From members m
JOIN group g ON g.mid = m.id
LEFT JOIN items i ON i.mid = m.id AND i.item_id = 5
WHERE g.gid = 1
AND i.id IS NULL
You might be wondering if we can move the i.item_id = 5 part to the WHERE clause also. You can't because there are no rows where i.id IS NULL and i.item_id = 5. You must do the join first and then eliminate the NULL rows in the WHERE clause.
I don't believe a temporary table is necessary. We'd really only go that route if we can't get acceptable performance.
From your query, we gather your schema looks like this:
group (id INT PK, gid INT, mid INT)
items (id INT PK, item_id INT, mid INT)
members (id INT PK, email VARCHAR)
It looks like your group table is really a "membership" table, which resolves/implements a many-to-many relationship between a group and a person. (That is, a person can be a member of zero, one or more groups; a group can have zero, or or more persons as members.)
You are using a LEFT JOIN between group and members. This will return a row for group (returning group.id) when there are no matching members, with a NULL for members.email (which may be what you want). But if you only want to return email addresses, then this can be changed to an INNER JOIN.
The NOT EXISTS predicate can be replaced with an OUTER JOIN and a test for a NULL value returned from the JOINED table. If the group.gid and/or items.item_id columns are numeric datatype, then you can remove the quotes from around the integer literals in the predicates.
Here is an alternative which will return an equivalent resultset, and may perform better:
SELECT m.email
, g.id
FROM members m
JOIN group g ON g.mid = m.id AND g.gid = 1
LEFT
JOIN items i ON i.mid = m.id AND i.item_id = 5
WHERE i.id IS NULL
ADDENDUM:
TEST CASE (provided in comment on selected answer) demonstrates difference in result set between queries with the predicate items.item_id = 5 in the ON clause and in the WHERE clause. (Moving this predicate to the WHERE clause messes with the anti-join.)
CREATE TABLE `group` (`id` INT PRIMARY KEY, `gid` INT, `mid` INT);
CREATE TABLE `items` (`id` INT PRIMARY KEY, `item_id` INT, `mid` INT);
CREATE TABLE `members` (`id` INT PRIMARY KEY, `email` VARCHAR(40));
INSERT INTO `group` VALUES (1,1,1), (2,1,2);
INSERT INTO `items` VALUES (1,5,1);
INSERT INTO `members` VALUES (1,'one#m.com'),(2,'two#m.com');