I am trying to find our nearby branches that can serve a list of our customers (company_id) locations. The two tables (customers 'locations' and our 'branches') both have indexed lat / lng fields. This MySQL query works using the haversine formula and returns all branches within 25 miles of each location.
select locations.id,locations.street, locations.city, locations.state,branches.id, branchname,branches.city,branches.state,
(3956 * acos(cos(radians(locations.lat))
* cos(radians(branches.lat))
* cos(radians(branches.lng)
- radians(locations.lng))
+ sin(radians(locations.lat))
* sin(radians(branches.lat)))) as branchdistance
from locations
left join branches on
(3956 * acos(cos(radians(locations.lat))
* cos(radians(branches.lat))
* cos(radians(branches.lng)
- radians(locations.lng))
+ sin(radians(locations.lat))
* sin(radians(branches.lat)))) < 25
where locations.company_id = 388
order by locations.id, branchdistance
However I want to limit the number of branches (left join) returned to a max of 5.
Any help would be greatly appreciated.
Thanks
Well finally after a lot of hair pulling I found the answer:
select
a.id,
b.id,
st_distance_sphere(a.position, b.position) * 0.00062137119 dist
from addresses a
inner join branches b
on b.id = (
select b1.id
from branches b1
where st_distance_sphere(a.position, b1.position) * 0.00062137119 < 25
order by st_distance_sphere(a.position, b1.position)
limit 1
)
By way of explanation we are trying to allocate thousands of leads (addresses) to the closest of many hundreds of branches each of which have a service radius of 25 miles.
Each table (addresses & branches) have lat, lng and a geospatial position column.
I modified the original query to use MySQL 5.7 geospatial functions (st_distance_sphere) but the Haversine formula would still work.
Hope this is of value to someone.
Related
I have the following two tables
cities
id,lat,lon
mountains
id,latitude,longitude
SELECT cities.id,
(SELECT id FROM mountains
WHERE SQRT(POW(69.1 * ( latitude - cities.lat ) , 2 ) +
POW( 69.1 * (cities.lon - longitude ) *
COS( latitude / 57.3 ) , 2 ) )<20 LIMIT 1) as mountain_id
FROM cities
(Query took 0.5060 seconds.)
I've removed some parts of the query (e.g. order by, where) for the complexity's sake. However it doesn't affect the execution time really.
The EXPLAIN below.
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY cities ALL NULL NULL NULL NULL 478379
2 DEPENDENT SUBQUERY mountains ALL NULL NULL NULL NULL 15645 Using where
Using the SELECT itself is not my problem but when I try to use the given result... e.g.
id mountain_id
588437 NULL
588993 4269
589014 4201
589021 4213
589036 4952
589052 7625
589113 9235
589125 NULL
589176 1184
589210 4317
...to UPDATE a table everything gets awfully slow. I tried pretty much everything that I know of. I do know that a dependent sub-query isn't optimal but I don't know how to get rid of it.
Is there any way to improve my query. Maybe changing it into a JOIN?
The 2 tables itself have nothing really in common except latitude and longitude which are different and are only brought into relation when using calculations.
Spatial distance search (km,miles) in MariaDB seems not to be available yet.
The trick to making this sort of operation fast is to avoid doing all that computation on every possible pair of lat/lon points. To do that you should incorporate a bounding-box operation.
Let's start by using a JOIN. In pseudocode, you want something like this, but it doesn't matter if you catch a few extra pairs, as long as they are further apart than the others.
SELECT c.city_id, m.mountain_id
FROM cities c
JOIN mountains m ON distance_in_miles(c, m) < 20
So we need to figure out how to make that ON clause fast -- make it use indexes rather than rambling around all the cities and mountains (with apologies to Woody Guthrie).
Let's try this for the ON clause. It searches within square bounding boxes of +/- 20 miles for nearby pairs.
SELECT c.city_id, m.mountain_id
FROM cities c
JOIN mountains m
ON m.lat BETWEEN c.lat - (20.0 / 69.0)
AND c.lat + (20.0 / 69.0)
AND m.lon BETWEEN c.lon - (20.0 / (69.0 * COS(RADIANS(c.lat))))
AND c.lon + (20.0 / (69.0 * COS(RADIANS(c.lat))))
In this query, 20.0 is the comparison limit radius, and 69.0 is the constant defining statute miles per degree of latitude.
Then, put compound indexes on (lat, lon, id) on both tables, and your JOIN operation will be able to use index range scans to make the query more efficient.
Finally, you can augment that query with these sorts of clauses, in pseudocode
ORDER BY dist_in_miles (c,m) ASC
LIMIT 1
Here you actually need to use a distance formula. The cartesian-distance formula in your question is an approximation that works tolerably well unless you're near the pole. You may want to use a great circle formula instead. Those are called spherical cosine law, haversine, or Vincenty formulas.
I am trying to repeat the following query for all rows. Basically I am trying to map the closest city (based on the latitude and longitude) to the places latitude and longitude. I have a table places which contains the places that need to be mapped, and a table CityTable with the places to be matched to. I have the following query which works for a single row:
SELECT p.placeID, p.State, p.City, p.County, p.name,
SQRT(POW((69.1 * (p.lat - z.Latitude)), 2 )
+ POW((53 * (p.lng - z.Loungitude)), 2)) AS distance,
p.lat,p.lng,z.Latitude,z.Loungitude,z.City
FROM places p,CityTable z
WHERE p.placeID = 1
ORDER BY distance ASC
LIMIT 1;
This works for a single location. Obviously I would need to remove the WHERE constraints to apply it to the entire table.The problem that I am encountering is that it seems to want to make a copy to compare to every other element in the table. For example, if there are 100 rows in p and 100 rows in z, then the resulting table seems to be 10,000 rows. I need the table to be of size count(*) for p. Any ideas? Also, are there any more efficient ways to do this if my table p contains over a million rows? Thanks.
You can find the nearest city to a place using:
SELECT p.placeID, p.State, p.City, p.County, p.name,
(select z.City
from CityTable z
order by SQRT(POW((69.1 * (p.lat - z.Latitude)), 2 ) + POW((53 * (p.lng - z.Loungitude)), 2))
limit 1
) as City,
p.lat, p.lng
FROM places p
ORDER BY distance ASC;
(If you want additional city information, join the city table back in on City.)
This doesn't solve the problem of having to do the Cartesian product. It does, however, frame it in a different way. If you know that a city is within five degrees longitude/latitude of any place, then you can make the subquery more efficient:
(select z.City
from CityTable z
where z.lat >= p.lat + 5 and z.lat <= p.lat - 5 and
z.long <= p.long + 5 and z.long <= p.lat - 5
order by SQRT(POW((69.1 * (p.lat - z.Latitude)), 2 ) + POW((53 * (p.lng - z.Loungitude)), 2))
limit 1
) as City,
p.lat, p.lng;
This query will use an index on lat. It might even use an index on lat, long.
If this isn't sufficient, then you might consider another way of reducing the search space, by looking only at neighboring states (in the US) or countries.
Finally, you may want to consider the geospatial extensions to MySQL if you are often dealing with this type of data.
I'm trying to make a SQL query on a database of 7 million records, the database "geonames" have the "latitude" and "longitude" in decimal(10.7) indexed both, the problem is that the query is too slow:
SELECT SQL_NO_CACHE DISTINCT
geonameid,
name,
(6367.41 * SQRT(2 * (1-Cos(RADIANS(latitude)) * Cos(0.704231626533) * (Sin(RADIANS(longitude))*Sin(-0.0669560660943) + Cos(RADIANS(longitude)) * Cos(-0.0669560660943)) - Sin(RADIANS(latitude)) * Sin(0.704231626533)))) AS Distance
FROM geoNames
WHERE (6367.41 * SQRT(2 * (1 - Cos(RADIANS(latitude)) * Cos(0.704231626533) * (Sin(RADIANS(longitude)) * Sin(-0.0669560660943) + cos(RADIANS(longitude)) * Cos(-0.0669560660943)) - Sin(RADIANS(latitude)) * Sin(0.704231626533))) <= '10')
ORDER BY Distance
The problem is sort by the "Distance" field, which when created dynamically take long to seep into the condition "WHERE", if I remove the condition of the "WHERE ... <= 10" takes only 0.34 seconds, but the result is 7 million records and to transfer data from MySQL to PHP takes almost 120 seconds.
Can you think of any way to make the query to not lose performance by limiting the Distance field, given that the query will very often change the values?
This kind of query cannot use an index but must compute whether the lat/lon of each row falls within the specified distance. Therefore, it is typical that some form of preprocessing is used to limit the scan to a subset of rows. You could create tables corresponding to distance "bands" (2, 5, 8, 10, 20 miles/km -- whatever makes sense for your application requirements) and then populate these bands and keep them up to date. If you want only those medical providers, say, or hotels, or whatever, within 10 miles of a given location, there's no need to worry about the ones that are hundreds or thousands of miles away. With ad hoc queries you could inner join on the "within 10 miles" band, say, and thereby exclude from the comparison scan all rows where the computed distance > 10. When the location varies, the "elegant" way to handle this is to implement an RTREE, but you can define your encompassing region in any arbitrary way you like if you have access to additional data -- e.g. by using zipcodes or counties or states.
There are two things you can do:
Make sure the datatypes are the same on both sides of a comparison: ie compare with 10 (a number), not '10' (a char type) - it will make less work for the DB
In cases like this, I create a view, which means the calculation to be made just once, even if you refer to it more than once in the query
If these two points are incorporated into you code, you get:
CREATE VIEW geoNamesDistance AS
SELECT SQL_NO_CACHE DISTINCT
geonameid,
name,
(6367.41 * SQRT(2 * (1-Cos(RADIANS(latitude)) * Cos(0.704231626533) * (Sin(RADIANS(longitude))*Sin(-0.0669560660943) + Cos(RADIANS(longitude)) * Cos(-0.0669560660943)) - Sin(RADIANS(latitude)) * Sin(0.704231626533)))) AS Distance
FROM geoNames;
SELECT * FROM geoNamesDistance
WHERE Distance <= 10
ORDER BY Distance;
I came up with:
select * from retailer
where latitude is not null and longitude is not null
and pow(2*(latitude - ?), 2) + pow(longitude - ?, 2) < your_magic_distance_value
With this fast & easy flat-Earth code, Los Angeles is closer to Honolulu than San Fransisco, but i doubt customers will consider that when going that far to shop.
I am starting to learn sql and I am currently getting hung up on the 'having' clause when I use it in conjunction with COUNT. I have done a lot of research and my general understanding is that unlike where it waits to apply until any functions in the query have run.
This lead me to find the rather neat little function that allows me to find the closest locations to a given zip code by latitude and longitude. I started playing with this and used it to tie together three tables. One has a list of zip codes and their latitude longitudes while the other has a list of events and their zip codes and the last has a list of any special requirements for each event. The three event tables are tied to each other through an index. The latitude/longitude table is tied in via the zip code.
So for example:
SELECT `EVENT_CAT`,
`NAME`,
(3959 * acos(cos(radians(44.643418)) * cos(radians(`Latitude`)) * cos(radians(`Longitude`) - radians(-73.121685) ) + sin( radians(44.643418)) * sin(radians( `Latitude`)))) AS distance
FROM DATA_ZipCodes
JOIN `EVENT_POST_General` ON ZIP_CODE = ZipCode
JOIN `DATA_EVENTCategories` ON EVENT_CAT = DATA_EVENTCategories.ID
JOIN `EVENT_POST_Filtering` ON EVENT_POST_General.EVENT_ID = EVENT_POST_Filtering.EVENT_ID
WHERE `REQUIRE_TICKET` = '0'
HAVING distance < 10
ORDER BY distance
This works great and returns:
EVENT_CAT NAME DISTANCE
-------------------------------
1 CONCERT 1
1 CONCERT 1
1 CONCERT 1
2 GAMES 1
2 GAMES 2
3 DANCE 4
4 DINNER 4
5 MOVIES 4
The catch is I also want to be able to just query a count of how many of each category of events I have.
To this end I tried just encorporating a COUNT and GROUP BY
SELECT COUNT(`EVENT_CAT`),
`NAME`,
(3959 * acos(cos(radians(44.643418)) * cos(radians(`Latitude`)) * cos(radians(`Longitude`) - radians(-73.121685) ) + sin( radians(44.643418)) * sin(radians( `Latitude`)))) AS distance
FROM DATA_ZipCodes
JOIN `EVENT_POST_General` ON ZIP_CODE = ZipCode
JOIN `DATA_EVENTCategories` ON EVENT_CAT = DATA_EVENTCategories.ID
JOIN `EVENT_POST_Filtering` ON EVENT_POST_General.EVENT_ID = EVENT_POST_Filtering.EVENT_ID
WHERE `REQUIRE_TICKET` = '0'
GROUP BY `EVENT_CAT`
HAVING distance < 10
ORDER BY distance
When I do this however the response does not error but it also does not return anything. I am very baffled as to how to best do this. I tried moving the group by but that just caused errors.
*EDITED TO CORRECT THE GROUP_BY TO GROUP BY as that was a type when I was pasting it in here and is not in the original code :)
Your query a bit shorted:
SELECT COUNT(`EVENT_CAT`),`NAME`, ... as `distance`
FROM ...
WHERE `REQUIRE_TICKET` = '0'
GROUP_BY `EVENT_CAT`
HAVING distance < 10
ORDER BY distance
The group by clause (I think it should not have a _ there, too) groups the result by different event_cat results, unifying the rest of the results. But:
your query result does not even contain such a column.
in each group you try to count how much different values of event_cat there are. There can only be one, if you would group by this.
additionally, the other columns of the result are not depending only on the group-by column ... this does not work.
The having clause is used to filter the resulting groups after grouping. Since your distance is not unique in each group, this is not a good way to do it - put it instead in the WHERE clause, if you want to count how many events in 10 km distance are in each category. The problem is that you can't refer in the WHERE clause to fields only defined in the SELECT clause, so we need to put the formula into the WHERE clause.
Then count the different name values (or something else unique), not the event_cat.
SELECT `EVENT_CAT`, COUNT(`NAME`)
FROM ...
WHERE `REQUIRE_TICKET` = '0' AND ... < 10
GROUP_BY `EVENT_CAT`
Ordering by distance also is not possible, since the distance is not unique by category. Maybe ordering by count, or minimal distance, or such?
Use:
SELECT dzc.name,
COUNT(dzc.name),
(3959 * acos(cos(radians(44.643418)) * cos(radians(`Latitude`)) * cos(radians(`Longitude`) - radians(-73.121685) ) + sin( radians(44.643418)) * sin(radians( `Latitude`)))) AS distance
FROM DATA_ZipCodes dzc
JOIN EVENT_POST_General epg ON epg.zip_code = dzc.zipcode
JOIN DATA_EVENTCategories dec ON dec.id = dzc.event_cat
JOIN EVENT_POST_Filtering epf ON epf.event_id = epg.event_id
WHERE REQUIRE_TICKET = '0'
GROUP BY dzc.name
HAVING distance < 10
ORDER BY distance
you had a typo - there's no underscore in "GROUP BY"
MySQL supports hidden columns in the GROUP BY, but few other databases do
You only need backticks in MySQL queries if you are escaping reserved/key words
use table aliases -- makes the query more readable, and will give you more specific errors in the event a table gets changed
I have one table with two location IDs (pointa and pointb) and how far apart they are. Multiple pairs can be the same distance apart.
Example data:
pointa pointb distance
1 2 250
3 4 250
4 5 250
.....
6 8 500
10 12 500
13 17 500
I want to select one pair from each distance class (one from 250, one from 500 and so on) and join those to another table which contain attributes for that location.
If I were to write this as an algorithm, it would go:
Select one pair at random from each distance class from table distance
join with table data based on distance.pointa=data.location.
Then do the same for pointb such that pointb=data.location
So after the join, I end up with:
pointa pointb data_a data_b
1 2 234.5 440.2
Does anyone have ideas on how I can achieve this?
For now I am doing this using PHP (looking up attributes for a, then b, and then updating a new table. Clearly this is inefficient and I want to learn a better way to do this directly in MySQL.
tia.
getting random rows in SQL can be less than efficient. I'd try something like:
SELECT distance, GROUP_CONCAT(pointa) a, GROUP_CONCAT(pointb) b FROM mytable
GROUP BY distance;
Then in PHP,
while($distance_points= $result->fetch_assoc()){
$distance=$distance_points['distance'];
$as=explode($distance_points['a']);
$random_a=array_rand($as,1);
$bs=explode($distance_points['b']);
$random_b=$bs[array_search($as,$random_a)]; //to get the pair assuming points don't repeat
echo "$distance $random_a $random_b";
}
ETA: Since you have too much data for GROUP_CONCAT, a multiquery solution might work better. Something like:
SELECT DISTINCT distance FROM mytable ORDER BY distance; //get an array with the possible distances
foreach($distances as $d)
$q="SELECT distance, a, b FROM mytable WHERE distance=$d ORDER BY RAND() LIMIT 1";
//run query, retrieve single result and output or store
}
Since you want to get extra info about your locations a and b, you could add that join to $q. Something like:
SELECT distance, a, li1.gps, b, li2.gp2
FROM mytable m
JOIN location_info li1 ON (m.a=li1.id)
JOIN location_info li2 ON (m.b=li2.id)
WHERE distance=$d
ORDER BY RAND() LIMIT 1