I got a hash map in java like this
{(1,'2018-06-29 10:19:33'),(4,'2018-06-29 10:19:34'),(10,'2018-06-29 10:19:38'),....}
The length of map could go as high as 3000
and a mysql table
id name updated
1, firstProduct, 2018-06-29 09:19:33
2, secondproduct, 2014-06-29 10:19:33
4, fourthproduct, 2016-06-29 09:19:33
10, tenthproduct, 2018-06-29 06:13:32
.......
the key in the map is a id field in the table and the value is the updated timestamp field.
I would like to get all the products in the table whose timestamp is greater that value in the map.
like
select * from products where id = 1 and updated > '2018-06-29 10:19:33'
select * from products where id = 4 and updated > '2018-06-29 10:19:34'
...
But there could be as many as 3000 entries in the map.
I am thinking of passing the map values to mysql stored procedure. Inside the procedure a while loop will execute select statement for each map entry into a result set and finally return the result set back to java program.
Would like to know if this is possible and i feel there is a better solution for this but cant figure out.
I would dynamically generate a statement in Java.
Either:
SELECT p.id, p.name, p.update
FROM products p
WHERE 1=0
OR ( p.id = ? AND updated > ? )
OR ( p.id = ? AND updated > ? )
OR ( p.id = ? AND updated > ? )
(MySQL likely won't make effective use of an index for that query.)
Or, we could do :
SELECT p.id, p.name, p.update FROM products p WHERE p.id = ? AND updated > ?
UNION ALL
SELECT p.id, p.name, p.update FROM products p WHERE p.id = ? AND updated > ?
UNION ALL
SELECT p.id, p.name, p.update FROM products p WHERE p.id = ? AND updated > ?
With an appropriate index defined, each SELECT could make effective use of that. But with this approach, we are probably going to hit an upper limit on the number of table references in a single query. So this approach may need to be broken up into chunks.
Personally, I wouldn't hide the complexity in a database procedure.
It's not clear what benefit we would gain by implementing a database procedure.
Related
My problem is the following, I want set a list of ID in a variable, then use this variable in a subquery. The problem is that WorkBench (my GUI) return the following error : "subquery returning multiple rows". It seems to me that's what I want.
Please explain me where I am wrong.
This is my query :
set #listID := (select ID_VOIE as ID from voies
where ORIGINE = 'XXX'
group by CODE_INSEE, CODE_VOIE
having count(*) > 1);
select substring(v.CODE_INSEE,1,2), count(*) from voies v
where v.ID_VOIE in (#listID)
group by substring(vs.CODE_INSEE,1,2);
The thing is I'm blocked with the "group by", I want do a groupd by after a first group by, that's why I can't (or at least i didn't find a way) write the request with a single WHERE clause.
The thing is I know that I can put the whole request directly in my subquery instead of using variable but :
It can let me use this trick in another requests that needed this behaviour (DRY concept !)
I'm not sure but the subquery will be executed in each turn of my loop, and that will be very inefficient
So I seek 2 possible ways : a way that let me use a list in a variable in a subquery OR a way that let me use "group by" twice in a single query.
Thanks you in advance for your answers (oh and sorry for my english, this is not my maternal language).
Unless you need that variable for something else, you should be able to skip it entirely as follows:
SELECT
SUBSTRING(v.CODE_INSEE,1,2),
COUNT(*)
FROM
voies v
WHERE
v.ID_VOIE in
(SELECT
ID_VOIE as ID
FROM
voies
WHERE
ORIGINE = 'XXX'
GROUP BY
CODE_INSEE,
CODE_VOIE
HAVING COUNT(*) > 1)
GROUP BY
SUBSTRING(vs.CODE_INSEE,1,2);
As you say, the subquery will be executed for all rows. To avoid that, a variable would be best, but MySQL doesn't support table variables. Instead, you can use a temporary table:
IF EXISTS DROP TABLE myTempTable;
CREATE TEMPORARY TABLE myTempTable (ID_VOIE int); -- I don't know the datatype
INSERT INTO myTempTable (ID_VOIE)
SELECT DISTINCT -- using distinct so I can join instead of use IN.
ID_VOIE as ID from voies
WHERE
ORIGINE = 'XXX'
GROUP BY
CODE_INSEE, CODE_VOIE
HAVING COUNT(*) > 1
And now you can do this:
SELECT
SUBSTRING(v.CODE_INSEE,1,2), COUNT(*)
FROM
voies v
JOIN myTempTable tt ON
v.ID_VOIE = tt.ID_VOIE
GROUP BY SUBSTRING(vs.CODE_INSEE,1,2);
I'm not an expert in SQL, i have an sql statement :
SELECT * FROM articles WHERE article_id IN
(SELECT distinct(content_id) FROM contents_by_cats WHERE cat_id='$cat')
AND permission='true' AND date <= '$now_date_time' ORDER BY date DESC;
Table contents_by_cats has 11000 rows.
Table articles has 2700 rows.
Variables $now_date_time and $cat are php variables.
This query takes about 10 seconds to return the values (i think because it has nested SELECT statements) , and 10 seconds is a big amount of time.
How can i achieve this in another way ? (Views or JOIN) ?
I think JOIN will help me here but i don't know how to use it properly for the SQL statement that i mentioned.
Thanks in advance.
A JOIN is exactly what you are looking for. Try something like this:
SELECT DISTINCT articles.*
FROM articles
JOIN contents_by_cats ON articles.article_id = contents_by_cats.content_id
WHERE contents_by_cats.cat_id='$cat'
AND articles.permission='true'
AND articles.date <= '$now_date_time'
ORDER BY date DESC;
If your query is still not as fast as you would like then check that you have an index on articles.article_id and contents_by_cats.content_id and contents_by_cats.cat_id. Depending on the data you may want an index on articles.date as well.
Do note that if the $cat and $now_date_time values are coming from a user then you should really be preparing and binding the query rather than just dumping these values into the query.
This is the query we are starting with:
SELECT a.*
FROM articles a
WHERE article_id IN (SELECT distinct(content_id)
FROM contents_by_cats
WHERE cat_id ='$cat'
) AND
permission ='true' AND
date <= '$now_date_time'
ORDER BY date DESC;
Two things will help this query. The first is to rewrite it using exists rather than in and to simplify the subquery:
SELECT a.*
FROM articles a
WHERE EXISTS (SELECT 1
FROM contents_by_cats cbc
WHERE cbc.content_id = a.article_id and cat_id = '$cat'
) AND
permission ='true' AND
date <= '$now_date_time'
ORDER BY date DESC;
Second, you want indexes on both articles and contents_by_cats:
create index idx_articles_3 on articles(permission, date, article_id);
create index idx_contents_by_cats_2 on contents_by_cat(content_id, cat_id);
By the way, instead of $now_date_time, you can just use the now() function in MySQL.
Let's assume I have the following tables:
items table
item_id|view_count
item_views table
view_id|item_id|ip_address|last_view
What I would like to do is:
If last view of item with given item_id by given ip_address was 1+ hour ago I would like to increment view_count of item in items table. And as a result get the view count of item. How I will do it normally:
q = SELECT count(*) FROM item_views WHERE item_id='item_id' AND ip_address='some_ip' AND last_view < current_time-60*60
if(q==1) then q = UPDATE items SET view_count = view_count+1 WHERE item_id='item_id'
//and finally get view_count of item
q = SELECT view_count FROM items WHERE item_id='item_id'
Here I used 3 SQL queries. How can I merge it into one SQL query? And how can it affect the processing time? Will it be faster or slower than previous method?
I don't think your logic is correct for what you describe that you want. The query:
SELECT count(*)
FROM item_views
WHERE item_id='item_id' AND
ip_address='some_ip' AND
last_view < current_time-60*60
is counting the number of views longer ago than your time frame. I think you want:
last_view > current_time-60*60
and then have if q = 0 on the next line.
MySQL is pretty good with the performance of not exists, so the following should work well:
update items
set view_count = view_count+1
WHERE item_id='item_id' and
not exists (select 1
from item_views
where item_id='item_id' AND
ip_address='some_ip' AND
last_view > current_time-60*60
)
It will work much better with an index on item_views(item_id, ip_address, last_view) and an index on item(item_id).
In MySQL scripting, you could then write:
. . .
set view_count = (#q := view_count+1)
. . .
This would also give you the variable you are looking for.
update target
set target.view_count = target.view_count + 1
from items target
inner join (
select item_id
from item_views
where item_id = 'item_id'
and ip_address = 'some_ip'
and last_view < current_time - 60*60
) ref
on ref.item_id = target.item_id;
You can only combine the update statement with the condition using a join as in the above example; but you'll still need a separate select statement.
It may be slower on very large set and/or unindexed table.
I have a list of ids, and I want to query a mysql table for ids not present in the table.
e.g.
list_of_ids = [1,2,4]
mysql table
id
1
3
5
6
..
Query should return [2,4] because those are the ids not in the table
since we cant view ur code i can only work on asumption
Try this anyway
SELECT id FROM list_of_ids
WHERE id NOT IN (SELECT id
FROM table)
I hope this helps
There is a horrible text-based hack:
SELECT
substr(result,2,length(result)-2) AS notmatched
FROM (
SELECT
#set:=replace(#set,concat(',',id,','),',') AS result
FROM (
select #set:=concat(',',
'1,2,4' -- your list here
,',')
) AS setinit,
tablename --Your tablename here
) AS innerview
ORDER BY LENGTH(result)
LIMIT 1;
If you represent your ids as a derived table, then you can do this directly in SQL:
select list.val
from (select 1 as val union all
select 2 union all
select 4
) list left outer join
t
on t.id = list.val
where t.id is null;
SQL doesn't really have a "list" type, so your question is ambiguous. If you mean a comma separated string, then a text hack might work. If you mean a table, then something like this might work. If you are constructing the SQL statement, I would advise you to go down this route, because it should be more efficient.
I have the following query:
SELECT *
FROM products
INNER JOIN product_meta
ON products.id = product_meta.product_id
JOIN sales_rights
ON product_meta.product_id = sales_rights.product_id
WHERE ( products.categories REGEXP '[[:<:]]5[[:>:]]' )
AND ( active = '1' )
AND ( products.show_browse = 1 )
AND ( product_meta.software_platform_mac IS NOT NULL )
AND ( sales_rights.country_id = '240'
OR sales_rights.country_id = '223' )
GROUP BY products.id
ORDER BY products.avg_rating DESC
LIMIT 0, 18;
Running the query with the omission of the ORDER BY section and the query runs in ~90ms, with the ORDER BY section and the query takes ~8s.
I've browsed around SO and have found the reason for this could be that the sort is being executed before all the data is returned, and instead we should be running ORDER BY on the result set instead? (See this post: Slow query when using ORDER BY)
But I can't quite figure out the definitive way on how I do this?
I've browsed around SO and have found the reason for this could be
that the sort is being executed before all the data is returned, and
instead we should be running ORDER BY on the result set instead?
I find that hard to believe, but if that's indeed the issue, I think you'll need to do something like this. (Note where I put the parens.)
select * from
(
SELECT products.id, products.avg_rating
FROM products
INNER JOIN product_meta
ON products.id = product_meta.product_id
JOIN sales_rights
ON product_meta.product_id = sales_rights.product_id
WHERE ( products.categories REGEXP '[[:<:]]5[[:>:]]' )
AND ( active = '1' )
AND ( products.show_browse = 1 )
AND ( product_meta.software_platform_mac IS NOT NULL )
AND ( sales_rights.country_id = '240'
OR sales_rights.country_id = '223' )
GROUP BY products.id
) as X
ORDER BY avg_rating DESC
LIMIT 0, 18;
Also, edit your question and include a link to that advice. I think many of us would benefit from reading it.
Additional, possibly unrelated issues
Every column used in a WHERE clause should probably be indexed somehow. Multi-column indexes might perform better for this particular query.
The column products.categories seems to be storing multiple values that you filter with regular expressions. Storing multiple values in a single column is usually a bad idea.
MySQL's GROUP BY is indeterminate. A standard SQL statement using a GROUP BY might return fewer rows, and it might return them faster.
If you can, you may want to index your ID columns so that the query will run quicker. This is a DBA-level solution, rather than a SQL solution - tuning the database will help overall performance.
The issue in the instance of this query, was that by using GROUP BY and ORDER BY in a query, MySQL is unable to use the index if the GROUP BY and ORDER BY expressions are different.
Related Reading:
http://dev.mysql.com/doc/refman/5.0/en/order-by-optimization.html
http://mysqldba.blogspot.co.uk/2008/06/how-to-pick-indexes-for-order-by-and.html