I have about twenty rather small tables (the largest has about 2k rows, normaly about 100 rows, with from 4 up to 20 columns each) I try to join by
select ... from table1
left join table2 on table1.name = table2.t2name
left join table3 on table1.name = table3.othername
left join table4 on table2.t2name = table4.something
and so on
in theory it should return about 2k rows with maybe 80 columns, so I guess that the amount of data itself is not the problem.
But it runs out of memory. From reading several posts here I figured out that mysql internaly makes a big "all x all"-table first and reduces it later. How can I force it to excute the join after each join first, that it takes a lot less memory?
Just to make things clear, in your case the expected amount of data is not the problem.
What appears to be the problem is the fact that you are asking the system to compare A X B X C X D... rows (calculate what it means and you will get the picture).
The general idea described in one of my previous comments is to make you query look as follows:
SELECT * FROM (select ... from table1
where .....
) A
LEFT JOIN SELECT * FROM (select ... from table2
where .....
) B
ON A.name = B.t2name
LEFT JOIN SELECT * FROM (select ... from table3
where .....
) C
ON A.name = C.othername
LEFT JOIN SELECT * FROM (select ... from table4
where .....
) D
ON B.name = D.something
In this way, and assuming that this is applicable in the sense that you do have conditions to put in the where ..... clause of the inner selects, you will be reducing the number of records from each table that would need to be compared during the join.
Related
Here's the format of mysql code
select a,b,c
from table1
left join table2 on x=y
left join table3 on m=n
limit 100000, 10
I know know to optimize limit when I have a large offset. But I couldn't find the solution to optimize the one with multiple tables, is there any way to make my query faster?
First of all, offsets and limits are unpredictable unless you include ORDER BY clauses in your query. Without ORDER BY, your SQL server is allowed to return result rows in any order it chooses.
Second, Large offsets and small limits are a notorious query-performance antipattern. There's not much you can to do make the problem go away.
To get decent performance, it's helpful to rethink why you want to use this kind of access pattern, and then try to use WHERE filters on some indexed column value.
For example, let's say you're doing this kind of thing.
select a.user_id, b.user_email, c.user_account
from table1 a
left join table2 b on a.user_id = b.user_id
left join table3 c on b.account_id = c.account_id
limit whatever
Let's say you're paginating the query so you get fifty users at a time. Then you can start with a last_seen_user_id variable in your program, initialized to -1.
Your query looks like this:
select a.user_id, b.user_email, c.user_account
from (
select user_id
from table1
where user_id > ?last_seen_user_id?
order by user_id
limit 50
) u
join table1 a on u.user_id = a.user_id
left join table2 b on a.user_id = b.user_id
left join table3 c on b.account_id = c.account_id
order by a.user_id
Then, when you retrieve that result, set your last_seen_user_id to the value from the last row in the result.
Run the query again to get the next fifty users. If table1.user_id is a primary key or a unique index, this will be fast.
I have a sample query
SELECT COUNT(counts)
FROM
(SELECT COUNT(*) AS counts
FROM table AS table1
LEFT JOIN table_2 as table2 ON table2.id = table1.pr_id
WHERE table1.active = 1
GROUP BY table1.column1) as t
I got the result
20563
The result was correct, but it was so slow, I had to wait up to 15 seconds for 1 query
15 seconds is too big, is there any way I can improve the speed as quickly as possible?
Thanks everyone!
Inner counting makes no sense. So
SELECT COUNT(*) FROM (
SELECT DISTINCT table1.column1
FROM table AS table1
LEFT JOIN ....
WHERE ....
) as t
The index which matches joining condition and further WHERE / DISTINCT will improve.
PS. Check does LEFT makes sense, maybe INNER JOIN is enough.
I have an update that does this:
Update Table1 as T1
Inner Join Table2 as T2
On T1.X=T2.Y
Set T1.A=T2.B;
Table1 is around 10,000,000 records
Table2 is around 40,000 records
I have an index on both T1.X and T2.Y
Naturally this takes forever. Is there a way to reduce the time?
For instance, my understanding the join is not the sum of the table records but the product. Is there a way (if this is true) to step through the join 1000 Table1 records at at time?
Apparently the Join works on both full tables. SO I modified my query to this:
Update Table1 as T1
Inner Join (Select T2.Y,T2.B from Table2) as T2
On T1.X=T2.Y
Set T1.A=T2.B;
I haven't run the entire query again but when I limited it as a test to 'where T1.ID<100' it reduced from 18+ seconds to 1 second when I add the Select statement i.
UPDATE
I also tested pushing in a limited # of records at a time. Interstingly enough 10 records takes 1 second, 1000 records takes 6 seconds and 10000 take 250 seconds. So I am going to test nesting this in a while to step through 1000 records at a time.
It does not answer your question, but here is a way to use your logic to make it faster:
UPDATE table_A a
INNER JOIN (SELECT TRUNCATE(lat,3) AS lat, TRUNCATE(lon,3) AS lon, address FROM table_B) b
SET a.address = b.address
WHERE TRUNCATE(a.lat,3) = b.lat AND TRUNCATE(a.lon,3) = b.lon AND a.address IS NULL;
When tou need to make a work on your data to use it, instead of transform them in the 'Where', transform them when you take them aou of your table.
I need to calculate total available qty from the database, and for that I need to do joins with a couple of tables. I can not paste my whole query, but the following is the basic structure:
select sum(qty) as qty, field
from
(
(
select SUM(table1.qty) as qty , field
from
table1
left join table2 on table1.field = table2.field
left join table3 on table3.field = table2.field
where condition
group by fieldname
)
UNION ALL
(
selecy SUM(table1.qty) as qty,field
from
table1
left join table2 on table1.field = table2.field
left join table3 on table3.field = table2.field
where condition
group by fieldname
)
UNION ALL
(
select SUM(table1.qty) as qty, field
from
table1
left join table2 on table1.field = table2.field
left join table3 on table3.field = table2.field
where condition
group by fieldname
)
...
..
12 times
) as temp
LEFT JOIN another_main_table ON another_main_table.field = temp.field
I have taken care of indexes of each table, but there are some unions which are taking longer time than expected. There are around 45 tables used in this query and all are examined fully. Some tables have around 2.6 million records.
Can you please suggest me how I can get the result in 1/2 seconds? As of now I am getting the result in around one minute.
Since your given example one can not properly justify the proper solutions, but still if I roughly examine your query, you have used LEFT JOIN, So this will take a longer time compare to INNER JOIN.
So, Use INNER JOIN if your data permits
I have 3 tables which contain different types of data related to each other. the tables populate via an excel spreadsheet. I have:
table1 table2 table3
item_number item_number item_number
desc desc qty_sold
qty_instock vdf_cost upc
cost status
What I'm trying to do is use a join function to show all the data as they relate to each other, except the problem is that when I run
SELECT *
FROM table1 a
INNER JOIN table2 b
ON a.someColumn = b.otherColumn
INNER JOIN table3 c
ON b.anotherColumn = c.nextColumn
It just puts the tables side by side, If I run
SELECT *
FROM table1 a
INNER JOIN table2 b
USING(item_number)
It works but only joins the item number (i have no idea how to use multiple fields such as description which repeats), and for some reason I can only use the two tables when I try to add a third table (most likely being done completely wrong)
SELECT *
FROM table1 a
INNER JOIN table2 b
INNER JOIN table3 c
USING(item_number)
I just get a syntax error.
Thanks for all the help in advance
UPDATE:
I got it working
SELECT *
FROM master_list a
INNER JOIN bby_report ab USING (item_number, description)
INNER JOIN sales_report b USING (item_number)
Is there a way I can exclude the description from one of the tables and keep it from another one? Turns out the descriptions are not an exact match from one table to a another so it keeps returning zero results.
So to clarify keep description from table1 and leave out description from table2 while being able to JOIN the fields based on item_number
SELECT *
FROM master_list a
INNER JOIN bby_report ab USING (item_number, description)
INNER JOIN sales_report b USING (item_number)