Perl search lookup efficiency - mysql

Question

I have a batch of urls that I have to search through the database for a match or rather if the url contains the url in the database.
An example of a url is
http://www.foodandnuts.com/login.html
The database has a table filled with urls
Currently my script created a array at the start that has all the urls in my database
my $results = $dbh->selectall_hashref('SELECT * FROM urltable;', 'url');
foreach my $j (keys %$results) {
push(#urldb, $j);
}
It will then go through the array to see if the url contains the url from the database
foreach(#urldb){
if($searchedurl=~ /$_/){
#do things here
}}
The problem is that is extremely slow as the array has more than 10000 urls so each searched url has to go through that array. Is there any way to make this faster?

Answer 1

The question can be answered differently depending on which of 3 kinds of URL matches you wish:
Exact full matches only (string equality). E.g. if DB url is "google.com", then search string "http://google.com" will NOT match, nor will "google.com/q=a".
In this case, drop using regexps, and either simply do SELECT * FROM urls WHERE url="$search", or do a hash lookup as Andreas' answer details.
Both search URL and URLs in DB are valid URLs (e.g. start with http://) and therefore MUST match starting with beginning of string, but the search URL can contain a DB URL+suffix to match. E.g. if DB URL is "http://google.com", then search strings "http://google.com" AND "http://google.com/q=a" match.
In this case, either do a start-anchored RegEx, or start-anchored "LIKE" DB match - see details in the next part of the answer.
Any substring match. E.g. if DB URL is "google", then any URL containing "google" string matches anywhere.
In this case, either do word-lookup table, or even smarter substring lookups algoritrhms; or do a batched regex matches using "|" to join multiple DB urls. See details in the last part of the answer.
This part of the answer assumes your URLs in DB can be substrings of search URL but they all start with "http", meaning they always match at the beginning of the string; but are not exact matches.
Solution 1 for start-anchored match (Perl):
Fix your RegExes to be anchored at the beginning: if($searchedurl=~ /^$_/){
Solution 2 for start-anchored match (DB):
Index your URL table by URL field, and do (Sybase syntax)
$query = qq[SELECT * FROM urls WHERE url LIKE "$searchurl\%"];
This will do a very efficient DB search for start-anchored substrings.
NOTE: the tradeoff between doing matches in DB vs Perl is:
If you have 1 DB and 100s of clients, you don't want to overload the DB doing string matching. Distribute the CPU load onto clients.
If you only have 1-2 clients, DB may be better as you will transfer less data from disk IO in DB (index on the table will help) and over network.
This part of the answer assumes your URLs in DB can be full substrings of search URL, not necessarily exact or even anchored matches.
Solution 1 for random substring match (Perl):
One purely Perl way you can make this faster is by combining your search strings into batches:
Split off first N elements from #urldb, in a loop
my $N = 10;
my $start = 0;
my $end = $N - 1;
while ($start < #urldb ) {
search_with($searchedurl, #urldb[$start..$end]); # see next bullet
$start += $N;
$end += $N;
$end = #urldb if $end > #urldb;
}
For each of length-N arrays, join the elements with "|" and create a regex
sub search_with {
my $searchedurl = shift;
my $regex_string = join("|", #_);
if ($searchedurl =~ /($regex_string)/) {
# Do stuff, $1 will contain what matched.
}
}
Solution 2 for random substring match (DB):
Another more algorithmic way to do it is to build an "word lookup" table (aka index, but I'd rather not use the term index to avoid confusion with database indices).
Split off each URL into words.
In the DB, add a unique ID to URL table
In the DB, add an "word lookup" table mapping (1-to-N) URL ID to every individual word (1 per row) in that URL
Use the "word lookup" table to narrow down the list of URLs to query out.
You can use a database index on "word lookup" table to make that search VERY fast.
You will of course need to split search URL into words as well.
Further speed up/narrow down by separately indexing domain name words from paths.
NOTE: using a simple "WHERE" clause in-database to search your URL table is a VERY bad idea if the URLS can be substrings that don't match on the first character - this way, you can't use and index and will simply scan the table.
NOTE2: For even more efficient substring matching against arrays of strings, there are more advanced algorithms based on graphs of substrings.
NOTE3: Tradeoff between doing matching in Perl and DB is same as in the first half of the answer.

Answer 2

#DVK is right about the fact that it is usually more efficient if you can anchor the match at the beginning. That way you can use a standard btree index to search against (MySQL doesn't have PostgreSQL's richer range of index types afaik).
I'd disagree with him/her about where to do the matching. It almost always makes sense to do this in the database itself. That's what a database is for.
The most efficient way is probably something like:
Create a TEMPORARY TABLE to hold your target urls
Bulk insert your targets to that temporary table
Create an index on them (assuming indexes will help here)
Join from your main url table to your targets using a LIKE match.
Even if you can't use indexes, the database should really be quicker than your perl. You're reading the entire table, packaging up the raw data into the transport protocol, transferring it, parsing that into perl values, assembling a hash and then checking it. Assuming your list of target urls is much smaller than the full list in the database you'll win just by not transferring so much data.

Answer 3

Note: OP asked for a solution where the search string should contain the url. I've changed my solution to try to normalize the urls so that hash matches are exact lookups after
getting comments of this.
This code is not tested, it should serve as some form of pseudo code that might work
Create a hash instead of an array. Hashes are ordered and better suited as lookups.
my $results = $dbh->selectall_hashref('SELECT * FROM urltable;', 'url');
my %urldb = map { normalize($_) => 1 } keys %$results;
sub normalize {
my $url = shift;
$url =~ s|http://||; # strip away http:// if present
$url =~ s|www\.||; # strip away www if present
$url =~ s|/.*||; # strip away anything after and including /
return $url;
}
Then you would search with
if (exists($urldb{normalize($searchedurl)})) {
#do things here
}