I'm attempting to get my head around geospatial indexes in mysql and I'm having trouble getting simple queries to work and I think it's down to my lack of understanding. I'm basically trying to get a number of locations within a 10 mile radius of a lat/lon I pass in.
The table structure is very simple
Points:
name (varchar)
location (Point)
at present there's only one row
name - "point1" location:POINT(31.5 42.2)
Now I'm trying the following sql that I've lifted from here https://www.percona.com/blog/2013/10/21/using-the-new-mysql-spatial-functions-5-6-for-geo-enabled-applications/
set #lat= 31.5;
set #lon = 42.2;
set #dist = 10;
set #rlon1 = #lon-#dist/abs(cos(radians(#lat))*69);
set #rlon2 = #lon+#dist/abs(cos(radians(#lat))*69);
set #rlat1 = #lat-(#dist/69);
set #rlat2 = #lat+(#dist/69);
select astext(location), name from Points
where st_contains(location, envelope(linestring(point(#rlon1, #rlat1), point(#rlon2, #rlat2))))
order by st_distance(point(#lon, #lat), location) limit 10;
but it returns an empty result set. Anyone got any ideas or pointers?
thanks
I managed to get this working
I was assuming a 'Point' object was lat,lon but it's not as far as I can tell it's lon,lat. So in theory the problem was I had stored my initial 'Point' coordinate values the wrong way round in the table even though I thought the standard way was lat/lon
Related
I am looking for a solution for finding wether a LAT,LNG Point is contained inside any Polygon in my MySQL table.
For some reason that extends past my SQL knowledge, my queries using ST_Intersects returning 'Function not found'. So i have stried in it's place:
SELECT `area_title` FROM `service_areas` WHERE MBRIntersects(`area_poly`, GEOMFROMTEXT('POINT(40.775032, -73.970778)'));
My table for testing is fairly simply and stores a
1. area_poly GEOMETRY stores POLYGON((-71.740723 41.360319,-71.685791 42.106374,-71.71875 42.407235,-71.905518 42.771211,-72.070312 43.036776,-72.432861 43.157109,-72.718506 43.397065,-73.190918 43.55651,-73.619385 43.580391,-74.32251 43.572432,-75.201416 43.277205,-75.717773 43.004647,-75.926514 42.795401,-76.135254 42.528796,-76.256103 42.138968,-76.289062 41.869561,-76.234131 41.442726,-76.190185 40.955011,-75.992432 40.472024,-75.849609 40.153687,-75.629883 39.783213,-75.311279 39.529467,-74.94873 39.368279,-74.520264 39.257778,-74.256592 39.478606,-71.5979 40.971604,-71.740723 41.360319))
And
2. area_title VARCHAR(20) stores service area 1
I am trying to pass a Point as POINT(40.775032, -73.970778) as shown above.
Expected result would be to return any area_title's that the point is within it's polygon; However, my query returns 0 Rows.
I know there are a ton of questions/answers on slack and the web, but i have not found a solution trying most of the methods. With my lack of SQL knowledge i'm not sure if i'm storing the Poly correctly or not using the correct function to find it.
Any help, even pointing in the right direction is appreciated.
A simple solution for anyone who is running into the same issue:
SELECT ea.name, ea.area_poly from service_areas as ea WHERE contains(ea.area_poly, GeomFromText(AsText(point(#lat, #lng)), 4326))
My polygon ea.area_poly is saved as Polygon((lat1 lng1,lat2 lng2, etc..)) as a geometry type in SQL.
I have a database with pretty static objects e.g buildings with x and y coorinates for a game in which I will be sending http requests to my server to get all objects around some given x and y coordinates.
Currently I am using this simple sql on the server which then returns the data in JSON.
SELECT OBJECTS.id \"id\", POINTS.x, POINTS.y
FROM OBJECT, OBJECTPOINTS, POINTS
WHERE OBJECTPOINTS.OID = OBJECTS.ID AND OBJECTPOINTS.PID = POINTS.ID AND
ABS(POINTS.x -\"+x+") < 0.01 AND ABS(POINTS.y - "+y+") < 0.01;"
Each object is represented by points which will be used to draw on the client.
I am currently achieving ~ 5 seconds respond time for around 1.5M Points and 200k objects.
To me this is fairly reasonable however the problem I have is that the db is blocked with each request. Here are 10 requests sent at the same time
:
36 seconds for map data with 10 clients requesting at the same time is way too much.
So my question is what would be a better way to handle the request rather then comparing distance in sql?
Would it be deliberatly faster to hold all of those objects in memory and iterate them on the server?
I have also thought of abstracting all the data in to some kind of grid and then first checking in which grid the request coords are to then run the same query as above on the db with only the objects in that certain square. Is there some clever solution I might be overlooking in the sql maybe?
Your query cannot be utilized by an index, because you are using a function on your column data in the where-clause of your query (ABS(POINTS.x ...)). If you rewrite your query to compare the raw value of your columns against another value you can add an index to your table and your query no longer needs to scan your full table to answer the query.
Rewrite your where clause to something like this to replace the ABS() function.
(POINTS.x < (x + 0.01) AND POINTS.x > (x - 0.01))
Then add an index to your table like:
alter table POINTS add index position(x, y);
Check the changing of the scanned rows of both queries with and without the index by adding the explain keyword infront of your query.
I have an issue and don't really know where to start :
I have user that choose geolocalize data (eg : San Franciso, 50 miles)
I need to perform search on this (eg : San Jose) and return user that have the city in their perimeter.
I use Sphinx (but change can be an option) and MySQL, I tried to make an multi attribute with all the cities while indexing : it crash my server while indexing.
So I make a cron that store User -> City links in database. It works fine for Sphinx, but my cron is a bit too greedy (and the table grow very fast, an user can have nearly 1000 cities in his perimeter).
Do you know a good solution for this?
Thanks
This is possible through Sphinx. You need to have latitude and longitude of each of the cities in database and need to add as attribute in Sphinx.
Sphinx uses a magic function #geodist for calculation of points of interest (POI) within range.
Need to change Sphinx config file as below:
source geolocation
{
sql_query = select city_id, radians(longitude) as longitude,
radians(latitude) as latitude from cities
sql_attr_float = longitude
sql_attr_float = latitude
}
Note that the sql_query uses the radians function to convert degrees to radians. sql_attr_float tells Sphinx that the longitude/latitude are floats. THis is needed for the magic #geodist function. The sql_query_info is handy if you debugging on the command line.
Below is PHP code to execute Sphinx Search to retrieve Geo Location search
$_longitude = $_GET['longitude'];
$_latitude = $_GET['latitude'];
$_radius = $_GET['radius'];
$search = new SphinxClient();
$search->SetServer("localhost", 3312);
$search->SetMatchMode(SPH_MATCH_ALL);
$search->SetArrayResult(true);
$search->SetLimits(0, 100);
$search->SetGeoAnchor('latitude', 'longitude', (float) deg2rad($_latitude),
(float) deg2rad($_longitude));
$circle = (float) $_radius * 1.61;
$search->SetFilterFloatRange('#geodist', 0.0, $circle);
$result = $search->Query('', 'geolocation');
I had a project similar to yours last year and got the same problem. I had to switch to SphinxQL instead of the php API and first worked much faster and you have more options as API give you.
All-
I thing i've finally out grown MySQL for one of my solutions. Right now I have 70 million rows that simply store the x,y,z of objects in 3D space. Unfortunately I don't know how else to optimize my database to handle the inserts/queries anymore. I need to query based on distance (get objects within distance).
Does anyone have a suggestions on a good replacement? I don't know if I should be looking at something like hbase or non-relational databases, as I may run into a similar problem. I generally insert about 100 rows per minute, and my query looks like:
// get objects within 500 yards
SELECT DISTINCT `object_positions`.`entry` FROM `object_positions` WHERE `object_positions`.`type` = 3 AND `object_positions`.`continent` = '$p->continent' AND SQRT(POW((`object_positions`.`x` - $p->x), 2) + POW((`object_positions`.`y` - $p->y), 2) + POW((`object_positions`.`z` - $p->z), 2)) < 500;
Nothing crazy complicated, but I think the math involved is what is causing MySQL to explode and I'm wondering if I should be looking at a cloud based database solution? It could easily have to handle 10-100 queries per second.
It's not MySQL that's giving you trouble, it's the need to apply indexing to your problem. You have a problem that no amount of NoSQL or cloud computing is going to solve by magic.
Here's your query simplified just a bit for clarity.
SELECT DISTINCT entry
FROM object_positions
WHERE type = 3
AND continent = '$p->continent'
AND DIST(x,$p->x, y, $p->y, z,$p-z) < 500
DIST() is shorthand for your Cartesian distance function.
You need to put separate indexes on x, y, and z in your table, then you need to do this:
SELECT DISTINCT entry
FROM object_positions
WHERE type = 3
AND continent = '$p->continent'
AND x BETWEEN ($p->x - 500) AND ($p->x + 500)
AND y BETWEEN ($p->y - 500) AND ($p->y + 500)
AND z BETWEEN ($p->z - 500) AND ($p->z + 500)
AND DIST(x,$p->x, y, $p->y, z,$p-z) < 500
The three BETWEEN clauses of the WHERE statement will allow indexes to be used to avoid a full table scan of your table for each query. They'll select all your points in a 1000x1000x1000 cube surrounding your candidate point. Then the DIST computation will toss out the ones that are outside the radius you want. You'll get the same batch of points but much more efficiently.
You don't have to actually create a DIST function; the formula you have in your question is fine.
You do have an index on (type, continent), don't you? If not you need it too.
I need to organize a Mysql Database of 7,000,000 records. It needs to be queryable by a Lat/Long square, example: 22.54x -134.74x. It also needs to be broken down by month and name. There are about 700 possible names.
I have no clue how to manage all of this data so that someone within that lat/long square can quickly query only the names that exist inside that month and lat/long square.
Do I need 10,000+ tables for every lat/long possibility? Then from there broken down into months and names? My mind is about to explode.
Thanks for your help in advance!
MySQL offers spatial extensions that are specifically designed to efficiently deal with lat/long type problems.
If you use these extensions, a given row would contain the relevant latitude and longitude of a place, and the query would describe the geometric boundary that you are interested in.
Your query can look something along the lines of
SELECT name, AsText(location) FROM Points
WHERE X(location) > 0 AND X(location) < 1 AND
Y(location) > 38 AND Y(location) < 39
This would select places whose location (location here is a spatial column in the database) has a longitude between 0 and 1, and a latitude between 38 and 39.
For a modern database 7,000,000 rows are not much. If you use an index you will improve the access time significantly. Splitting the tables is not necessary. I have made experiments with mySQL and about 1 million locations and was not using the spatial extension and was happy with the access times (been still below 1 second).
You would do something like:
CREATE INDEX my_idx ON my_pos_table (month,name,lng,lat);
I suggest that you give it a try. There is nothing better than empirical information. Also observe how you access the data. Maybe you do not always access by month and name, so you can add more indicies, which will be only a disadvantage when inserting a row.
CREATE INDEX my_idx_2 ON my_pos_table (name,lng,lat);
CREATE INDEX my_idx_3 ON my_pos_table (lng,lat);
Hope it helps.
You can find a usefull post here:
http://blog.jcole.us/2007/11/24/on-efficiently-geo-referencing-ips-with-maxmind-geoip-and-mysql-gis/
I have used this post as a "starting point" for a project over 150milions of record -myisam- and it works fine
BTW, in general the best way is to use the GEOMETRY and SPATIAL INDEX -only for myisam- like as follow:
ALTER TABLE <table> ADD geom_point GEOMETRY NOT NULL ;
ALTER TABLE <table> ADD SPATIAL INDEX geom_point (geom_point) ;
UPDATE <table> SET geom_point = POINT(latitude, longitude);
Now you can find all "names" into a "square" using a query like this:
SELECT * from <table> WHERE
MBRContains(GeomFromText('LineString(<lata> <lona>, <latb> <lonb>)'), geom_point)
Or by distance like:
$longitude = 8.449997;
$latitude = 45.550003;
$distance = 50; # km
$point1 = $latitude + $distance / ( 111.1 / cos($latitude));
$point2 = $longitude + $distance / 111.1;
$point3 = $latitude - $distance / ( 111.1 / cos($latitude));
$point4 = $longitude - $distance / 111.1;
SELECT * from <table> WHERE
MBRContains(GeomFromText('LineString(<$point1> <$point2>, <$point3> <$point4>)'), geom_point)
Try on your test table ;)