mysql large table with geo-locations - find intersections - mysql

Question

I have a large table ( > 20 millions rows ) with this structure
[ Id, IdUser (int), Latitude(double), Longitude (double), EventDateTime (datetime) ]
and I need to find all the moments where users have been in the same area( within 500 meters ).
What is the best solution for this?

Answer 1

First, so we don't have to write insanely complex SQL queries full of transcendental functions, let's define a stored function distance(lat1, lon1, lat2, lon2) to get ourselves a distance between two pairs of points.
DELIMITER $$
DROP FUNCTION IF EXISTS distance$$
CREATE FUNCTION distance(
lat1 FLOAT, lon1 FLOAT,
lat2 FLOAT, lon2 FLOAT
) RETURNS FLOAT
NO SQL DETERMINISTIC
COMMENT 'Returns the distance in metres on the Earth
between two known points of latitude and longitude'
BEGIN
RETURN 111045 * DEGREES(ACOS(
COS(RADIANS(lat1)) *
COS(RADIANS(lat2)) *
COS(RADIANS(lon2) - RADIANS(lon1)) +
SIN(RADIANS(lat1)) * SIN(RADIANS(lat2))
));
END$$
DELIMITER ;
Now we need to compare pairs of items in your table to find coincidences. Let's say we want one-minute resolution on the time comparison. This query will do the trick, but take a while.
SELECT DISTINCT a.IdUser, b.IdUser,
DATE_FORMAT (a.EventDateTime, '%Y-%m-%d %H:%i:00) AS EventDateTime
FROM table a
JOIN table b
ON a.IdUser < b.IdUser /* compare different users */
AND a.EventDateTime >= b.EventDateTime - INTERVAL 1 HOUR
AND a.EventDateTime <= b.EventDateTime + INTERVAL 1 HOUR
AND distance(a.Latitude, a.Longitude, b.Latitude, b.Longitude) <= 500.0
This will work, giving a list of pairs of users and the hours in which they were near one another. But it won't be very fast.
You'll to experiment with indexes. Probably an index on (EventDateTime, IdUser) will help. You probably should experiment with this query by adding a time restriction like this...
WHERE a.EventDateTime >= CURDATE - INTERVAL 2 DAY
AND a.EventDateTime < CURDATE - INTERVAL 1 DAY
so you don't take hours to run the query.
Now, let's try to do an optimization pass over the self-join, in an attempt to cut down the use of the distance function, and to use indexes better. In order to do this, we need to know that there are ~11045m per degree of (north-south) latitude, so that 500m is 500/111045 degrees.
This query will generate pairs of observations that are within 500m north-to-south of each other, then use a WHERE clause to further eliminate points that are still too far apart. That will reduce the use of the distance function.
SELECT a.IdUser, b.IdUser,
DATE_FORMAT (a.EventDateTime, '%Y-%m-%d %H:%i:00) AS EventDateTime
FROM table a
JOIN table b
ON a.IdUser < b.IdUser /* compare different users */
AND a.EventDateTime >= b.EventDateTime - INTERVAL 1 HOUR
AND a.EventDateTime <= b.EventDateTime + INTERVAL 1 HOUR
AND a.Latitude >= b.Latitude - (500.0/111045.0)
AND a.Latitude <= b.Latitude + (500.0/111045.0)
WHERE distance(a.Latitude, a.Longitude, b.Latitude, b.Longitude) <= 500.0
It is worth trying a compound covering index on (IdUser, EventDateTime, Latitude, Longitude) to try to optimize this query.