Different result for Haversine formulas - mysql

I am using mysql to count the proximity and for that i have created one procedure named distance which is as follows but that procedure is not working properly but the sql statement is working so what is the difference over here as both are i guess Haversine formulas but not giving me the proper result. i really don't know wht i am missing in formula one.
Data structure of my table is as follows
for formula one
id varchar(100)
userid varchar(100)
username varchar(100)
currLoc point
radius int(10)
for formula two
id varchar(30)
userid varchar(30)
username varchar(40)
lat float(10,6)
lan float(10,6)
radius varchar(100)
Formula One: reference
sql statement to execute distance function
SELECT userid, username, distance(userstatus.currLoc,
GeomFromText('POINT(23.039574 72.56602)')) AS cdist
FROM userstatus HAVING cdist <= 0.6 ORDER BY cdist LIMIT 10
RETURN 6371 * 2 *
ASIN( SQRT(POWER(SIN(RADIANS(ABS(X(a)) - ABS(X(b)))), 2) +
COS(RADIANS(ABS(X(a)))) * COS(RADIANS(ABS(X(b)))) *
POWER(SIN(RADIANS(Y(a) - Y(b))), 2)));
Formula two: reference
SELECT *,(((acos(sin((23.039574*pi()/180)) *
sin((lat *pi()/180))+cos((23.039574*pi()/180)) *
cos((lat *pi()/180)) * cos(((72.56602- lon)*pi()/180))))*
180/pi())*60*1.1515*1.609344) as distance
FROM status HAVING distance <= 0.6
here 0.6 is a radius in kilometers

One version of the expression is using ABS(X(a)) etc and the other is not. The one using ABS is suspect. You can't afford to ignore the sign on the angles. You'll get different results in some areas of the world (near the equator or the prime meridian, for example, or near the poles).
Your constants are also different.
60*1.1515*1.609344
vs
6371 * 2
One expression involves SQRT, the other does not.
One expression involves ASIN and the other uses ACOS.
There is essentially nothing in common between the two...
See the discussion at Wikipedia 'Haversine Formula', and in particular the references to numerical stability when the distance between the points is small.
You could also improve the chances of people helping you by making the formulae you're using semi-readable, by splitting them over lines.
For example:
RETURN 6371 * 2 *
ASIN( SQRT(POWER(SIN(RADIANS(ABS(X(a)) - ABS(X(b)))), 2) +
COS(RADIANS(ABS(X(a)))) * COS(RADIANS(ABS(X(b)))) *
POWER(SIN(RADIANS(Y(a) - Y(b))), 2)));
And:
(((acos(sin((23.039574*pi()/180)) * sin((lat *pi()/180)) +
cos((23.039574*pi()/180)) * cos((lat *pi()/180)) *
cos(((72.56602-lan)*pi()/180))
)
) * 180/pi()) * 60 * 1.1515 * 1.609344)
The latter references 'lan'; is that meant to be 'lon'? In the second example, you appear to have encoded one of the two positions as 23.039574°N and 72.56602°W, and lat and lan come from the table in the SQL query.

Related

distance calculation between two tables of lat/lon

I have the following two tables
cities
id,lat,lon
mountains
id,latitude,longitude
SELECT cities.id,
(SELECT id FROM mountains
WHERE SQRT(POW(69.1 * ( latitude - cities.lat ) , 2 ) +
POW( 69.1 * (cities.lon - longitude ) *
COS( latitude / 57.3 ) , 2 ) )<20 LIMIT 1) as mountain_id
FROM cities
(Query took 0.5060 seconds.)
I've removed some parts of the query (e.g. order by, where) for the complexity's sake. However it doesn't affect the execution time really.
The EXPLAIN below.
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY cities ALL NULL NULL NULL NULL 478379
2 DEPENDENT SUBQUERY mountains ALL NULL NULL NULL NULL 15645 Using where
Using the SELECT itself is not my problem but when I try to use the given result... e.g.
id mountain_id
588437 NULL
588993 4269
589014 4201
589021 4213
589036 4952
589052 7625
589113 9235
589125 NULL
589176 1184
589210 4317
...to UPDATE a table everything gets awfully slow. I tried pretty much everything that I know of. I do know that a dependent sub-query isn't optimal but I don't know how to get rid of it.
Is there any way to improve my query. Maybe changing it into a JOIN?
The 2 tables itself have nothing really in common except latitude and longitude which are different and are only brought into relation when using calculations.
Spatial distance search (km,miles) in MariaDB seems not to be available yet.
The trick to making this sort of operation fast is to avoid doing all that computation on every possible pair of lat/lon points. To do that you should incorporate a bounding-box operation.
Let's start by using a JOIN. In pseudocode, you want something like this, but it doesn't matter if you catch a few extra pairs, as long as they are further apart than the others.
SELECT c.city_id, m.mountain_id
FROM cities c
JOIN mountains m ON distance_in_miles(c, m) < 20
So we need to figure out how to make that ON clause fast -- make it use indexes rather than rambling around all the cities and mountains (with apologies to Woody Guthrie).
Let's try this for the ON clause. It searches within square bounding boxes of +/- 20 miles for nearby pairs.
SELECT c.city_id, m.mountain_id
FROM cities c
JOIN mountains m
ON m.lat BETWEEN c.lat - (20.0 / 69.0)
AND c.lat + (20.0 / 69.0)
AND m.lon BETWEEN c.lon - (20.0 / (69.0 * COS(RADIANS(c.lat))))
AND c.lon + (20.0 / (69.0 * COS(RADIANS(c.lat))))
In this query, 20.0 is the comparison limit radius, and 69.0 is the constant defining statute miles per degree of latitude.
Then, put compound indexes on (lat, lon, id) on both tables, and your JOIN operation will be able to use index range scans to make the query more efficient.
Finally, you can augment that query with these sorts of clauses, in pseudocode
ORDER BY dist_in_miles (c,m) ASC
LIMIT 1
Here you actually need to use a distance formula. The cartesian-distance formula in your question is an approximation that works tolerably well unless you're near the pole. You may want to use a great circle formula instead. Those are called spherical cosine law, haversine, or Vincenty formulas.

How can I speed up this MySQL query that finds the closest locations to a given latitude/longitude?

I have a zip code table in my database which is used in conjunction with a business table to find businesses matching certain criteria that is closest to a specified zip code. The first thing I do is grab just the latitude and longitude since it's used in a couple places on the page. I use:
$zipResult = mysql_fetch_array(mysql_query("SELECT latitude,longitude FROM zipCodes WHERE zipCode='".mysql_real_escape_string($_SESSION['zip'])."' Limit 1"));
$latitude = $zipResult['latitude'];
$longitude = $zipResult['longitude'];
$radius = 100;
$lon1 = $longitude - $radius / abs(cos(deg2rad($latitude))*69);
$lon2 = $longitude + $radius / abs(cos(deg2rad($latitude))*69);
$lat1 = $latitude - ($radius/69);
$lat2 = $latitude + ($radius/69);
From there, I generate the query:
$query2 = "Select * From (SELECT business.*,zipCodes.longitude,zipCodes.latitude,
(3956 * 2 * ASIN ( SQRT (POWER(SIN((zipCodes.latitude - $latitude)*pi()/180 / 2),2) + COS(zipCodes.latitude* pi()/180) * COS($latitude *pi()/180) * POWER(SIN((zipCodes.longitude - $longitude) *pi()/180 / 2), 2) ) )) as distance FROM business INNER JOIN zipCodes ON (business.listZip = zipCodes.zipCode)
Where business.active = 1
And (3958*3.1415926*sqrt((zipCodes.latitude-$latitude)*(zipCodes.latitude-$latitude) + cos(zipCodes.latitude/57.29578)*cos($latitude/57.29578)*(zipCodes.longitude-$longitude)*(zipCodes.longitude-$longitude))/180) <= '$radius'
And zipCodes.longitude between $lon1 and $lon2 and zipCodes.latitude between $lat1 and $lat2
GROUP BY business.id ORDER BY distance) As temp Group By category_id ORDER BY distance LIMIT 18";
Which turns out something like:
Select *
From (SELECT business.*,zipCodes.longitude,zipCodes.latitude, (3956 * 2 * ASIN ( SQRT (POWER(SIN((zipCodes.latitude - 39.056784)*pi()/180 / 2),2) + COS(zipCodes.latitude* pi()/180) * COS(39.056784 *pi()/180) * POWER(SIN((zipCodes.longitude - -84.343573) *pi()/180 / 2), 2) ) )) as distance
FROM business
INNER JOIN zipCodes ON (business.listZip = zipCodes.zipCode)
Where business.active = 1
And (3958*3.1415926*sqrt((zipCodes.latitude-39.056784)*(zipCodes.latitude-39.056784) + cos(zipCodes.latitude/57.29578)*cos(39.056784/57.29578)*(zipCodes.longitude--84.343573)*(zipCodes.longitude--84.343573))/180) <= '100'
And zipCodes.longitude between -86.2099407074 and -82.4772052926
and zipCodes.latitude between 37.6075086377 and 40.5060593623
GROUP BY business.id
ORDER BY distance) As temp
Group By category_id
ORDER BY distance
LIMIT 18
The code runs and executes just fine, but it takes just over a second to complete (usually around 1.1 seconds). However, I've been told that in some browsers the page loads slowly. I have tested this is multiple browsers and multiple versions of those browsers without ever seeing an issue. However, I figure if I can get the execution time down it will help either way. The problem is I do not know what else I can do to cut down on the execution time. The zip code table already came with preset indexes which I assume are good (and contains the columns I'm using in my queries). I've added indexes to the business table as well, though I'm not too knowledgeable about them. But I've made sure to include the fields used in the Where clause at least, and maybe a couple more.
If I need to add my indexes to this question just let me know. If you see something in the query itself I can improve also please let me know.
Thanks,
James
EDIT
Table structure for the business table:
CREATE TABLE IF NOT EXISTS `business` (
`id` smallint(6) unsigned NOT NULL AUTO_INCREMENT,
`active` tinyint(3) unsigned NOT NULL,
`featured` enum('yes','no') NOT NULL DEFAULT 'yes',
`topFeatured` tinyint(1) unsigned NOT NULL DEFAULT '0',
`category_id` smallint(5) NOT NULL DEFAULT '0',
`listZip` varchar(12) NOT NULL,
`name` tinytext NOT NULL,
`address` tinytext NOT NULL,
`city` varchar(128) NOT NULL,
`state` varchar(32) NOT NULL DEFAULT '',
`zip` varchar(12) NOT NULL,
`phone` tinytext NOT NULL,
`alt_phone` tinytext NOT NULL,
`website` tinytext NOT NULL,
`logo` tinytext NOT NULL,
`index_logo` tinytext NOT NULL,
`large_image` tinytext NOT NULL,
`description` text NOT NULL,
`views` int(5) unsigned NOT NULL,
PRIMARY KEY (`id`),
KEY `featured` (`featured`,`topFeatured`,`category_id`,`listZip`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=3085 ;
SQL Fiddle
http://sqlfiddle.com/#!2/2e26ff/1
EDIT 2014-03-26 09:09
I've updated my query, but the shorter query actually takes about .2 seconds longer to execute every time.
Select * From (
SELECT Distinct business.id, business.name, business.large_image, business.logo, business.address, business.city, business.state, business.zip, business.phone, business.alt_phone, business.website, business.description, zipCodes.longitude, zipCodes.latitude, (3956 * 2 * ASIN ( SQRT (POWER(SIN((zipCodes.latitude - 39.056784)*pi()/180 / 2),2) + COS(zipCodes.latitude* pi()/180) * COS(39.056784 *pi()/180) * POWER(SIN((zipCodes.longitude - -84.343573) *pi()/180 / 2), 2) ) )) as distance
FROM business
INNER JOIN zipCodes ON (business.listZip = zipCodes.zipCode)
Where business.active = 1
And zipCodes.longitude between -86.2099407074 and -82.4772052926
And zipCodes.latitude between 37.6075086377 and 40.5060593623
GROUP BY business.category_id
HAVING distance <= '50'
ORDER BY distance
) As temp LIMIT 18
There is already an index on the zip code, latitude, and longitude fields in the zip codes database, both all in one index, and each with their own index. That's just how the table came when purchased.
I had updated the listZip data type to match the zip code table's zip data type yesterday.
I did take out the GROUP BY business.id and replace it with DISTINCT, but left the GROUP BY business.category_id because I only want one business per category.
Also, I started getting the 0.2 second execution difference as soon as I changed the query to use the HAVING clause instead of the math formula in the WHERE clause. I did try using WHERE distance <= 50 in the outer-query, but that didn't speed anything up either. Also using 50 miles instead of 100 miles doesn't seem to effect this particular query either.
Thanks for all of the suggestions so far though.
Put indexes on zipCodes.longitude and zipCodes.latitude. That should help a lot.
See here for more information. http://www.plumislandmedia.net/mysql/haversine-mysql-nearest-loc/
Edit you need an index in the zipCodes table on longitude alone or starting with longitude. It looks to me like you should try a composite index on
(longitude, latitude, zipCode)
for best results.
Make the data types of zipCodes.zipCode and business.listingZip the same, so the join will be more efficient. If those data types are different, MySQL will typecast one to the other as it does the join, and so the join will be inefficient. Make sure business.listingZip has an index.
You are misusing GROUP BY. (Did you maybe mean SELECT DISTINCT?) It makes no sense unless you also use an aggregate function like MAX() In a similar vein, see if you can get rid of the * in SELECT business.*, and instead give a list of the columns you need.
100 miles is a very wide search radius. Narrow it a bit to speed things up.
You're computing the great circle distance twice. You surely can recast the query to do it once.

Slow SQL Query by Limit/Order dynamic field (coordinates from X point)

I'm trying to make a SQL query on a database of 7 million records, the database "geonames" have the "latitude" and "longitude" in decimal(10.7) indexed both, the problem is that the query is too slow:
SELECT SQL_NO_CACHE DISTINCT
geonameid,
name,
(6367.41 * SQRT(2 * (1-Cos(RADIANS(latitude)) * Cos(0.704231626533) * (Sin(RADIANS(longitude))*Sin(-0.0669560660943) + Cos(RADIANS(longitude)) * Cos(-0.0669560660943)) - Sin(RADIANS(latitude)) * Sin(0.704231626533)))) AS Distance
FROM geoNames
WHERE (6367.41 * SQRT(2 * (1 - Cos(RADIANS(latitude)) * Cos(0.704231626533) * (Sin(RADIANS(longitude)) * Sin(-0.0669560660943) + cos(RADIANS(longitude)) * Cos(-0.0669560660943)) - Sin(RADIANS(latitude)) * Sin(0.704231626533))) <= '10')
ORDER BY Distance
The problem is sort by the "Distance" field, which when created dynamically take long to seep into the condition "WHERE", if I remove the condition of the "WHERE ... <= 10" takes only 0.34 seconds, but the result is 7 million records and to transfer data from MySQL to PHP takes almost 120 seconds.
Can you think of any way to make the query to not lose performance by limiting the Distance field, given that the query will very often change the values?
This kind of query cannot use an index but must compute whether the lat/lon of each row falls within the specified distance. Therefore, it is typical that some form of preprocessing is used to limit the scan to a subset of rows. You could create tables corresponding to distance "bands" (2, 5, 8, 10, 20 miles/km -- whatever makes sense for your application requirements) and then populate these bands and keep them up to date. If you want only those medical providers, say, or hotels, or whatever, within 10 miles of a given location, there's no need to worry about the ones that are hundreds or thousands of miles away. With ad hoc queries you could inner join on the "within 10 miles" band, say, and thereby exclude from the comparison scan all rows where the computed distance > 10. When the location varies, the "elegant" way to handle this is to implement an RTREE, but you can define your encompassing region in any arbitrary way you like if you have access to additional data -- e.g. by using zipcodes or counties or states.
There are two things you can do:
Make sure the datatypes are the same on both sides of a comparison: ie compare with 10 (a number), not '10' (a char type) - it will make less work for the DB
In cases like this, I create a view, which means the calculation to be made just once, even if you refer to it more than once in the query
If these two points are incorporated into you code, you get:
CREATE VIEW geoNamesDistance AS
SELECT SQL_NO_CACHE DISTINCT
geonameid,
name,
(6367.41 * SQRT(2 * (1-Cos(RADIANS(latitude)) * Cos(0.704231626533) * (Sin(RADIANS(longitude))*Sin(-0.0669560660943) + Cos(RADIANS(longitude)) * Cos(-0.0669560660943)) - Sin(RADIANS(latitude)) * Sin(0.704231626533)))) AS Distance
FROM geoNames;
SELECT * FROM geoNamesDistance
WHERE Distance <= 10
ORDER BY Distance;
I came up with:
select * from retailer
where latitude is not null and longitude is not null
and pow(2*(latitude - ?), 2) + pow(longitude - ?, 2) < your_magic_distance_value
With this fast & easy flat-Earth code, Los Angeles is closer to Honolulu than San Fransisco, but i doubt customers will consider that when going that far to shop.

MySQL query performance tuning

I'm trying to run a query over a table with around a 1 million records.
The structure of the table with its indexes is:
CREATE TABLE `table` (
`Id` int(11) NOT NULL,
`Name` varchar(510) DEFAULT NULL,
`Latitude` float NOT NULL DEFAULT '0',
`Longitude` float NOT NULL DEFAULT '0',
PRIMARY KEY (`Latitude`,`Longitude`,`Id`),
KEY `IX_Latitude_Longitude` (`Latitude`,`Longitude`),
KEY `IX_Latitude` (`Latitude`),
KEY `IX_Longitude` (`Longitude`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
I'm running the following query:
SELECT m.Id, m.Name, sqrt(69.1 * (m.Latitude - :latitude) * 69.1 * (m.Latitude - :latitude) +
53.0 * (m.longitude - :longitude) * 53.0 * (m.longitude - :longitude)) as Distance,
m.Latitude as Latitude, m.Longitude as Longitude
FROM table m
WHERE sqrt(69.1 * (m.Latitude - :latitude) * 69.1 * (m.Latitude - :latitude)
+ 53.0 * (m.longitude - :longitude) * 53.0 * (m.longitude - :longitude)) < :radius
ORDER BY sqrt(69.1 * (m.Latitude - :latitude) * 69.1 * (m.Latitude - :latitude) +
53.0 * (m.longitude - :longitude) * 53.0 * (m.longitude - :longitude)) desc
LIMIT 0, 100
That suppose to return all the records in a specific radius (distance calculation information: http://www.meridianworlddata.com/Distance-Calculation.asp)
BUT the query takes a lot of time...
Here is the explain plan that I get:
id|select_type |table|type|possible_keys|key |key_len|ref |rows |Extra
1 |SIMPLE |m |ALL |{null} |{null}|{null} |{null}|1264001|Using where; Using filesort
What am I doing wrong?
Which index do I need to add in order to cause the query to use it instead of table scan?
Do I need to change the table structure?
You're using functions inside your WHERE clause, so it will always result in a table scan. The database has no way to index against the result of a function. I think your best option is to come up with some way to limit the results before trying to evaluate the distance algorithm.
For instance, for a given location, you can know the minimum and maximum possible latitude that can fall within your set distance, so filter by that first. A degree of latitude is ~69 miles, so if your search radius is 50 miles, it would never be possible for anything more than 0.725 degrees of latitude away to fall within 50 miles of your location. Since this is just a numeric comparison WHERE m.latitude > (:latitude - 0.725) AND m.latitude < (:latitude + 0.725), not a call to a function, the database will be able to use your indexes to evaluate it.
Longitude is more complicated, since the distance for each degree varies depending on how far north/south the location happens to be, but depending on how much work you want to put into it, you could do the same with longitude as well.

Need Help understanding the HAVING clause as it relates to COUNT

I am starting to learn sql and I am currently getting hung up on the 'having' clause when I use it in conjunction with COUNT. I have done a lot of research and my general understanding is that unlike where it waits to apply until any functions in the query have run.
This lead me to find the rather neat little function that allows me to find the closest locations to a given zip code by latitude and longitude. I started playing with this and used it to tie together three tables. One has a list of zip codes and their latitude longitudes while the other has a list of events and their zip codes and the last has a list of any special requirements for each event. The three event tables are tied to each other through an index. The latitude/longitude table is tied in via the zip code.
So for example:
SELECT `EVENT_CAT`,
`NAME`,
(3959 * acos(cos(radians(44.643418)) * cos(radians(`Latitude`)) * cos(radians(`Longitude`) - radians(-73.121685) ) + sin( radians(44.643418)) * sin(radians( `Latitude`)))) AS distance
FROM DATA_ZipCodes
JOIN `EVENT_POST_General` ON ZIP_CODE = ZipCode
JOIN `DATA_EVENTCategories` ON EVENT_CAT = DATA_EVENTCategories.ID
JOIN `EVENT_POST_Filtering` ON EVENT_POST_General.EVENT_ID = EVENT_POST_Filtering.EVENT_ID
WHERE `REQUIRE_TICKET` = '0'
HAVING distance < 10
ORDER BY distance
This works great and returns:
EVENT_CAT NAME DISTANCE
-------------------------------
1 CONCERT 1
1 CONCERT 1
1 CONCERT 1
2 GAMES 1
2 GAMES 2
3 DANCE 4
4 DINNER 4
5 MOVIES 4
The catch is I also want to be able to just query a count of how many of each category of events I have.
To this end I tried just encorporating a COUNT and GROUP BY
SELECT COUNT(`EVENT_CAT`),
`NAME`,
(3959 * acos(cos(radians(44.643418)) * cos(radians(`Latitude`)) * cos(radians(`Longitude`) - radians(-73.121685) ) + sin( radians(44.643418)) * sin(radians( `Latitude`)))) AS distance
FROM DATA_ZipCodes
JOIN `EVENT_POST_General` ON ZIP_CODE = ZipCode
JOIN `DATA_EVENTCategories` ON EVENT_CAT = DATA_EVENTCategories.ID
JOIN `EVENT_POST_Filtering` ON EVENT_POST_General.EVENT_ID = EVENT_POST_Filtering.EVENT_ID
WHERE `REQUIRE_TICKET` = '0'
GROUP BY `EVENT_CAT`
HAVING distance < 10
ORDER BY distance
When I do this however the response does not error but it also does not return anything. I am very baffled as to how to best do this. I tried moving the group by but that just caused errors.
*EDITED TO CORRECT THE GROUP_BY TO GROUP BY as that was a type when I was pasting it in here and is not in the original code :)
Your query a bit shorted:
SELECT COUNT(`EVENT_CAT`),`NAME`, ... as `distance`
FROM ...
WHERE `REQUIRE_TICKET` = '0'
GROUP_BY `EVENT_CAT`
HAVING distance < 10
ORDER BY distance
The group by clause (I think it should not have a _ there, too) groups the result by different event_cat results, unifying the rest of the results. But:
your query result does not even contain such a column.
in each group you try to count how much different values of event_cat there are. There can only be one, if you would group by this.
additionally, the other columns of the result are not depending only on the group-by column ... this does not work.
The having clause is used to filter the resulting groups after grouping. Since your distance is not unique in each group, this is not a good way to do it - put it instead in the WHERE clause, if you want to count how many events in 10 km distance are in each category. The problem is that you can't refer in the WHERE clause to fields only defined in the SELECT clause, so we need to put the formula into the WHERE clause.
Then count the different name values (or something else unique), not the event_cat.
SELECT `EVENT_CAT`, COUNT(`NAME`)
FROM ...
WHERE `REQUIRE_TICKET` = '0' AND ... < 10
GROUP_BY `EVENT_CAT`
Ordering by distance also is not possible, since the distance is not unique by category. Maybe ordering by count, or minimal distance, or such?
Use:
SELECT dzc.name,
COUNT(dzc.name),
(3959 * acos(cos(radians(44.643418)) * cos(radians(`Latitude`)) * cos(radians(`Longitude`) - radians(-73.121685) ) + sin( radians(44.643418)) * sin(radians( `Latitude`)))) AS distance
FROM DATA_ZipCodes dzc
JOIN EVENT_POST_General epg ON epg.zip_code = dzc.zipcode
JOIN DATA_EVENTCategories dec ON dec.id = dzc.event_cat
JOIN EVENT_POST_Filtering epf ON epf.event_id = epg.event_id
WHERE REQUIRE_TICKET = '0'
GROUP BY dzc.name
HAVING distance < 10
ORDER BY distance
you had a typo - there's no underscore in "GROUP BY"
MySQL supports hidden columns in the GROUP BY, but few other databases do
You only need backticks in MySQL queries if you are escaping reserved/key words
use table aliases -- makes the query more readable, and will give you more specific errors in the event a table gets changed