subselect makes complex query really slow - mysql

Question

So I want to do the following for a project.
I have 3 tables. First two concern us now (the third is for your better understanding):
author {id, name}
authorship {id, id1, id2}
paper {id, title}
authorship connects author with paper and authorship.id1 refers to author.id, authorship.id2 refers to paper.id.
What I want to do is make a graph with a node for each author and edge that is determined by the amount of common papers between two authors.
w=1 - union_of_common_papers/intersection_of_common_papers
So what I have built (with some help from stackoverflow) an sql script that returns all couples of co-authors plus the amount of union and intersection of common papers. After that I will use the data with java. It's the following:
SELECT DISTINCT a1.name, a2.name, (
SELECT concat(count(a.id2), ',', count(DISTINCT a.id2))
FROM authorship a
WHERE a.id1=a1.id or a.id1=a2.id) as weight
FROM authorship au1
INNER JOIN authorship au2 ON au1.id2 = au2.id2 AND au1.id1 <> au2.id1
INNER JOIN author a1 ON au1.id1 = a1.id
INNER JOIN author a2 ON au2.id1 = a2.id;
this does my job and returns a list like:
+-----------------+---------------------+---------+
| name | name | weight |
+-----------------+---------------------+---------+
| Kurt | Michael | 161,157 |
| Kurt | Miron | 138,134 |
| Kurt | Manish | 19,18 |
| Roy | Gregory | 21,20 |
| Roy | Richard | 74,71 |
....
where in weight I can see 2 numbers a,b where b is intersection an b-a is the union of the common papers.
but this takes a lot of time.
And all the overhead is by this extra subselect
(SELECT concat(count(a.id2), ',', count(DISTINCT a.id2))
FROM authorship a
WHERE a.id1=a1.id or a.id1=a2.id) as weight
without this line all records (1M+) were returned in less than 2mins.
with this line 50 records need more than 1.5mins
I use mysql on linux through command line
Any ideas how I can optimize it?
author has ~130,000 records
authorship ~1,300,000 records
query should return ~1,200,000 records
This is what explain returns for this query. don't know how to use it.
+----+--------------------+-------+--------+---------------------+-----------+---------+--------------+---------+-----------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+-------+--------+---------------------+-----------+---------+--------------+---------+-----------------+
| 1 | PRIMARY | a1 | ALL | PRIMARY | NULL | NULL | NULL | 124768 | Using temporary |
| 1 | PRIMARY | au1 | ref | NewIndex1,NewIndex2 | NewIndex1 | 5 | dblp.a1.ID | 4 | Using where |
| 1 | PRIMARY | au2 | ref | NewIndex1,NewIndex2 | NewIndex2 | 5 | dblp.au1.id2 | 1 | Using where |
| 1 | PRIMARY | a2 | eq_ref | PRIMARY | PRIMARY | 4 | dblp.au2.id1 | 1 | |
| 2 | DEPENDENT SUBQUERY | a | ALL | NewIndex1 | NULL | NULL | NULL | 1268557 | Using where |
+----+--------------------+-------+--------+---------------------+-----------+---------+--------------+---------+-----------------+

Answer 1

You should be able to get your data directly from the joins in the outer query.
You can count the number of papers in common by counting the distinct id2 where that is the same for both authors.
You can count the total number of papers as the number of distinct papers for each author minus the ones in common (because otherwise, these would be counted twice):
SELECT a1.name, a2.name,
COUNT(distinct case when au1.id2 = au2.id2 then au1.id2 end) as CommonPapers,
COUNT(distinct au1.id2) + COUNT(distinct au2.id2) - COUNT(distinct case when au1.id2 = au2.id2 then au1.id2 end) as TotalPapers
FROM authorship au1 INNER JOIN
authorship au2
ON au1.id2 = au2.id2 AND au1.id1 <> au2.id1 INNER JOIN
author a1
ON au1.id1 = a1.id INNER JOIN
author a2
ON au2.id1 = a2.id
group by a1.name, a2.name;
In your data structure, id1 and id2 are lousy names. Have you considered something like idauthor and idpaper or something like that?
The above query counts the intersection correctly, but not the total, because of the initial inner join. one way around this is a full outer join, but that is not allowed in MySQL. We can do this with additional subqueries:
SELECT a1.name, a2.name,
COUNT(distinct case when au1.id2 = au2.id2 then au1.id2 end) as CommonPapers,
(ap1.NumPapers + ap2.NumPapers - COUNT(distinct case when au1.id2 = au2.id2 then au1.id2 end)
) as TotalPapers
FROM authorship au1 INNER JOIN
authorship au2
ON au1.id2 = au2.id2 AND au1.id1 <> au2.id1 INNER JOIN
author a1
ON au1.id1 = a1.id INNER JOIN
author a2
ON au2.id1 = a2.id inner join
(select au.id1, count(*) as numpapers
from authorship au
) ap1
on ap1.id1 = au1.id1 inner join
(select au.id1, count(*) as numpapers
from authorship au
) ap2
on ap2.id1 = au2.id1 inner join
group by a1.name, a2.name;