Perl with mysql, terribly slow, how to accelerate - mysql

Question

unit
id fir_name sec_name
author
id name unit_id
author_paper
id author_id paper_id
I want to unify authors['same author' means the names are the same and their units' fir_names are the same], and I have to change author_paper table at the same time.
Here is what i do:
$conn->do('create index author_name on author (name)');
my $sqr = $conn->prepare("select name from author group by name having count(*) > 1");
$sqr->execute();
while(my #row = $sqr->fetchrow_array()) {
my $dup_name = $row[0];
$dup_name = formatHtml($dup_name);
my $sqr2 = $conn->prepare("select id, unit_id from author where name = '$dup_name'");
$sqr2->execute();
my %fir_name_hash = ();
while(my #row2 = $sqr2->fetchrow_array()) {
my $author_id = $row2[0];
my $unit_id = $row2[1];
my $fir_name = getFirNameInUnit($conn, $unit_id);
if (not exists $fir_name_hash{$fir_name}) {
$fir_name_hash{$fir_name} = []; #anonymous arr reference
}
$x = $fir_name_hash{$fir_name};
push #$x, $author_id;
}
while(my ($fir_name, $author_id_arr) = each(%fir_name_hash)) {
my $count = scalar #$author_id_arr;
if ($count == 1) {next;}
my $author_id = $author_id_arr->[0];
for ($i = 1; $i < $count; $i++) {
#print "$author_id_arr->[$i] => $author_id\n";
unifyAuthorAndAuthorPaperTable($conn, $author_id, $author_id_arr->[$i]); #just delete in author table, and update in author_paper table
}
}
}
select count(*) from author; #240,000
select count(distinct(name)) from author; #7,7000
It is terribly slow!!I've runned it for 5hours, it just removed about 4,0000 dup names.
How to make it run faster.I am eager for your advice

Answer 1

You should not prepare the second sql statement within the loop and you can make real use of the preparation when you use the ? placeholder:
$conn->do('create index author_name on author (name)');
my $sqr = $conn->prepare('select name from author group by name having count(*) > 1');
# ? is the placeholder and the database driver knows if its an integer or a string and
# quotes the input if needed.
my $sqr2 = $conn->prepare('select id, unit_id from author where name = ?');
$sqr->execute();
while(my #row = $sqr->fetchrow_array()) {
my $dup_name = $row[0];
$dup_name = formatHtml($dup_name);
# Now you can reuse the prepared handle with different input
$sqr2->execute( $dup_name );
my %fir_name_hash = ();
while(my #row2 = $sqr2->fetchrow_array()) {
my $author_id = $row2[0];
my $unit_id = $row2[1];
my $fir_name = getFirNameInUnit($conn, $unit_id);
if (not exists $fir_name_hash{$fir_name}) {
$fir_name_hash{$fir_name} = []; #anonymous arr reference
}
$x = $fir_name_hash{$fir_name};
push #$x, $author_id;
}
while(my ($fir_name, $author_id_arr) = each(%fir_name_hash)) {
my $count = scalar #$author_id_arr;
if ($count == 1) {next;}
my $author_id = $author_id_arr->[0];
for ($i = 1; $i < $count; $i++) {
#print "$author_id_arr->[$i] => $author_id\n";
unifyAuthorAndAuthorPaperTable($conn, $author_id, $author_id_arr->[$i]); #just delete in author table, and update in author_paper table
}
}
}
This should speed up things as well.

Answer 2

The moment I see a query and a loop I think that you have a latency problem: you query to get a set of values and then iterate over the set to do something else. That's a LOT of latency if it means a network round trip to the database for each row in the set.
It'd be better if you could do it in a single query using an UPDATE and a sub-select OR if you could batch those requests and perform all of them in one round trip.
You'll get an additional speed up if you use indexes wisely. Every column in a WHERE clause should have an index. Every foreign key should have an index.
I'd run EXPLAIN PLAN on your queries and see if there are any TABLE SCAN going on. If there is, you've got to index properly.
I wonder if a properly designed JOIN would come to your rescue?
240,000 rows in one table and 77,000 in another isn't that large a database.