I have an issue and don't really know where to start :
I have user that choose geolocalize data (eg : San Franciso, 50 miles)
I need to perform search on this (eg : San Jose) and return user that have the city in their perimeter.
I use Sphinx (but change can be an option) and MySQL, I tried to make an multi attribute with all the cities while indexing : it crash my server while indexing.
So I make a cron that store User -> City links in database. It works fine for Sphinx, but my cron is a bit too greedy (and the table grow very fast, an user can have nearly 1000 cities in his perimeter).
Do you know a good solution for this?
Thanks
This is possible through Sphinx. You need to have latitude and longitude of each of the cities in database and need to add as attribute in Sphinx.
Sphinx uses a magic function #geodist for calculation of points of interest (POI) within range.
Need to change Sphinx config file as below:
source geolocation
{
sql_query = select city_id, radians(longitude) as longitude,
radians(latitude) as latitude from cities
sql_attr_float = longitude
sql_attr_float = latitude
}
Note that the sql_query uses the radians function to convert degrees to radians. sql_attr_float tells Sphinx that the longitude/latitude are floats. THis is needed for the magic #geodist function. The sql_query_info is handy if you debugging on the command line.
Below is PHP code to execute Sphinx Search to retrieve Geo Location search
$_longitude = $_GET['longitude'];
$_latitude = $_GET['latitude'];
$_radius = $_GET['radius'];
$search = new SphinxClient();
$search->SetServer("localhost", 3312);
$search->SetMatchMode(SPH_MATCH_ALL);
$search->SetArrayResult(true);
$search->SetLimits(0, 100);
$search->SetGeoAnchor('latitude', 'longitude', (float) deg2rad($_latitude),
(float) deg2rad($_longitude));
$circle = (float) $_radius * 1.61;
$search->SetFilterFloatRange('#geodist', 0.0, $circle);
$result = $search->Query('', 'geolocation');
I had a project similar to yours last year and got the same problem. I had to switch to SphinxQL instead of the php API and first worked much faster and you have more options as API give you.
Related
I am looking for a solution for finding wether a LAT,LNG Point is contained inside any Polygon in my MySQL table.
For some reason that extends past my SQL knowledge, my queries using ST_Intersects returning 'Function not found'. So i have stried in it's place:
SELECT `area_title` FROM `service_areas` WHERE MBRIntersects(`area_poly`, GEOMFROMTEXT('POINT(40.775032, -73.970778)'));
My table for testing is fairly simply and stores a
1. area_poly GEOMETRY stores POLYGON((-71.740723 41.360319,-71.685791 42.106374,-71.71875 42.407235,-71.905518 42.771211,-72.070312 43.036776,-72.432861 43.157109,-72.718506 43.397065,-73.190918 43.55651,-73.619385 43.580391,-74.32251 43.572432,-75.201416 43.277205,-75.717773 43.004647,-75.926514 42.795401,-76.135254 42.528796,-76.256103 42.138968,-76.289062 41.869561,-76.234131 41.442726,-76.190185 40.955011,-75.992432 40.472024,-75.849609 40.153687,-75.629883 39.783213,-75.311279 39.529467,-74.94873 39.368279,-74.520264 39.257778,-74.256592 39.478606,-71.5979 40.971604,-71.740723 41.360319))
And
2. area_title VARCHAR(20) stores service area 1
I am trying to pass a Point as POINT(40.775032, -73.970778) as shown above.
Expected result would be to return any area_title's that the point is within it's polygon; However, my query returns 0 Rows.
I know there are a ton of questions/answers on slack and the web, but i have not found a solution trying most of the methods. With my lack of SQL knowledge i'm not sure if i'm storing the Poly correctly or not using the correct function to find it.
Any help, even pointing in the right direction is appreciated.
A simple solution for anyone who is running into the same issue:
SELECT ea.name, ea.area_poly from service_areas as ea WHERE contains(ea.area_poly, GeomFromText(AsText(point(#lat, #lng)), 4326))
My polygon ea.area_poly is saved as Polygon((lat1 lng1,lat2 lng2, etc..)) as a geometry type in SQL.
I have a data dump (csv) of place names from https://www.ordnancesurvey.co.uk/business-and-government/help-and-support/products/os-open-names.html
I need to import this into mysql however the geometry co-ordinates use BNG (OSGB36). Does Mysql have any functions to convert these co-ordinates to wgs84 lat/long or is there any other sql method to achieve this?
another option is perhaps loading it into postgis - does postgis have any funtion to transform BNG to lat/long? Perhaps I could do that and then export the data once transformed and load it into mysql?
In postgis I could do something as follows (not tested)
select AddGeometryColumn('locations', 'the_geom', 27700, 'POINT', 2);
-- X and Y are the BNG co-ordinates
UPDATE locations SET the_geom = ST_GeomFromText('POINT(' || x || ' ' || y || ')', 27700 );
alter table locations add column lat real;
alter table locations add column long real;
update locations set long=st_x(st_transform(the_geom,4326)),
lat=st_y(st_transform(the_geom,4326));
Is it possible to do these type of function in mysql - basically what are the equivalent functions in mysql. I cant seem to figure the syntax out? The following doesnt work in mysql:
update locations set long=ST_X(ST_Transform(the_geom,4326)),
lat=ST_Y(ST_Transform(the_geom,4326));
I get error function ST_Transform does not exist. I'm using mysql 5.7
* UPDATE *
Sample data:
NAME1 LOCAL_TYPE GEOMETRY_X GEOMETRY_Y DISTRICT_BOROUGH REGION COUNTRY
Southport Town 333510 417225 Sefton North West England
The answer is, of all places, here
Answered by Hartmut Holzgraefe in this comment.
So far the SRID property is just a dummy in MySQL, it is stored as
part of a geometries meta data but all actual calculations ignore it
and calculations are done assuming Euclidean (planar) geometry.
So ST_Transform would not really do anything at this point anyway.
I think the same is still true for MariaDB, at least the knowledge [sic]
base page for the SRID() function still says so:
This WorkLog discusses the progress of implementing ST_Transform.
MySQL 8.0 does seem to have it implemented: https://dev.mysql.com/doc/refman/8.0/en/spatial-operator-functions.html#function_st-transform
So, the solution may require upgrading to MySQL 8.0.
The changelog for the very recent 8.0.13 says:
----- 2018-10-22 8.0.13 General Availability -- -- -----
MySQL now implements the
ST_Transform()
spatial function for use in converting geometry values from one
spatial reference system (SRS) to another. Currently, it supports
conversion between geographic SRSs. For details, see Spatial Operator
Functions.
I'm attempting to get my head around geospatial indexes in mysql and I'm having trouble getting simple queries to work and I think it's down to my lack of understanding. I'm basically trying to get a number of locations within a 10 mile radius of a lat/lon I pass in.
The table structure is very simple
Points:
name (varchar)
location (Point)
at present there's only one row
name - "point1" location:POINT(31.5 42.2)
Now I'm trying the following sql that I've lifted from here https://www.percona.com/blog/2013/10/21/using-the-new-mysql-spatial-functions-5-6-for-geo-enabled-applications/
set #lat= 31.5;
set #lon = 42.2;
set #dist = 10;
set #rlon1 = #lon-#dist/abs(cos(radians(#lat))*69);
set #rlon2 = #lon+#dist/abs(cos(radians(#lat))*69);
set #rlat1 = #lat-(#dist/69);
set #rlat2 = #lat+(#dist/69);
select astext(location), name from Points
where st_contains(location, envelope(linestring(point(#rlon1, #rlat1), point(#rlon2, #rlat2))))
order by st_distance(point(#lon, #lat), location) limit 10;
but it returns an empty result set. Anyone got any ideas or pointers?
thanks
I managed to get this working
I was assuming a 'Point' object was lat,lon but it's not as far as I can tell it's lon,lat. So in theory the problem was I had stored my initial 'Point' coordinate values the wrong way round in the table even though I thought the standard way was lat/lon
I need to organize a Mysql Database of 7,000,000 records. It needs to be queryable by a Lat/Long square, example: 22.54x -134.74x. It also needs to be broken down by month and name. There are about 700 possible names.
I have no clue how to manage all of this data so that someone within that lat/long square can quickly query only the names that exist inside that month and lat/long square.
Do I need 10,000+ tables for every lat/long possibility? Then from there broken down into months and names? My mind is about to explode.
Thanks for your help in advance!
MySQL offers spatial extensions that are specifically designed to efficiently deal with lat/long type problems.
If you use these extensions, a given row would contain the relevant latitude and longitude of a place, and the query would describe the geometric boundary that you are interested in.
Your query can look something along the lines of
SELECT name, AsText(location) FROM Points
WHERE X(location) > 0 AND X(location) < 1 AND
Y(location) > 38 AND Y(location) < 39
This would select places whose location (location here is a spatial column in the database) has a longitude between 0 and 1, and a latitude between 38 and 39.
For a modern database 7,000,000 rows are not much. If you use an index you will improve the access time significantly. Splitting the tables is not necessary. I have made experiments with mySQL and about 1 million locations and was not using the spatial extension and was happy with the access times (been still below 1 second).
You would do something like:
CREATE INDEX my_idx ON my_pos_table (month,name,lng,lat);
I suggest that you give it a try. There is nothing better than empirical information. Also observe how you access the data. Maybe you do not always access by month and name, so you can add more indicies, which will be only a disadvantage when inserting a row.
CREATE INDEX my_idx_2 ON my_pos_table (name,lng,lat);
CREATE INDEX my_idx_3 ON my_pos_table (lng,lat);
Hope it helps.
You can find a usefull post here:
http://blog.jcole.us/2007/11/24/on-efficiently-geo-referencing-ips-with-maxmind-geoip-and-mysql-gis/
I have used this post as a "starting point" for a project over 150milions of record -myisam- and it works fine
BTW, in general the best way is to use the GEOMETRY and SPATIAL INDEX -only for myisam- like as follow:
ALTER TABLE <table> ADD geom_point GEOMETRY NOT NULL ;
ALTER TABLE <table> ADD SPATIAL INDEX geom_point (geom_point) ;
UPDATE <table> SET geom_point = POINT(latitude, longitude);
Now you can find all "names" into a "square" using a query like this:
SELECT * from <table> WHERE
MBRContains(GeomFromText('LineString(<lata> <lona>, <latb> <lonb>)'), geom_point)
Or by distance like:
$longitude = 8.449997;
$latitude = 45.550003;
$distance = 50; # km
$point1 = $latitude + $distance / ( 111.1 / cos($latitude));
$point2 = $longitude + $distance / 111.1;
$point3 = $latitude - $distance / ( 111.1 / cos($latitude));
$point4 = $longitude - $distance / 111.1;
SELECT * from <table> WHERE
MBRContains(GeomFromText('LineString(<$point1> <$point2>, <$point3> <$point4>)'), geom_point)
Try on your test table ;)
I'm working on a project where I have places with latitude and longitude coordinates stored in my database. I use Google Maps to plot these places as markers on a map. I don't want to plot any "invisible" markers (markers outside the current viewport/bounding box) to the map. Therefore I follow Googles advice regarding viewport marker management.
I have a working solution where i use AJAX to query an ASP.NET web service whenever the viewport of my map has changed. This web service calls a stored procedure in my MSSQL database to get all places that have coordinates that are inside the current viewport/bounding box. The stored procedure looks like this:
ALTER PROCEDURE dbo.GetPlacesInsideBoundingBox
(
#swLat VarChar(11), /* south-west point latitude */
#swLng VarChar(11), /* south-west point longitude */
#neLat VarChar(11), /* north-east point latitude */
#neLng VarChar(11) /* north-east point longitude */
)
AS
/* SET NOCOUNT ON */
SELECT * FROM dbo.Place WHERE
place_lat >= CONVERT(Decimal(11,8), #swLat)
AND place_lat <= CONVERT(Decimal(11,8), #neLat)
AND place_lng >= CONVERT(Decimal(11,8), #swLng)
AND place_lng <= CONVERT(Decimal(11,8), #neLng)
RETURN
The parameters that are sent to the stored procedure are simply the latitude and longitude of the south-west and the north-east corners of the viewport/bounding box. Then I compare these points with the longitude and latitude values stored in my database to see which places are inside the current viewport/bounding box.
This works fine! But I read that you can run into problems if you have a negative value for the longitude (west of the Prime Meridian) and that the simple solution was to add 360 to the longitude if it was negative.
I have two questions:
How do I alter my stored procedure (above) to take into account negative
longitudes?
Are there any other modifications I should consider making to this
stored procedure to make it foolproof?
If you are wondering about the conversion from VarChar to Decimal, I found it much easier to work with simple strings on the client side (javascript) and then convert them to decimal numbers in my stored procedure when I needed to do the calculations.
Thanks in advance!
1) Longitude normaly should be between -180 to 180 so just check if the coordinates from google fulfil this condition. Probably yes. If not, just normalize them to be within this interval by adding or subtracting 360. Normalize them before you call the stored procedure.
2) Normally #swLng <= #neLng, but you must also handle the case #swLng > #neLng. So inside your stored procedure, your longitude condition would look like:
AND
(
(CONVERT(Decimal(11,8), #swLng) <= CONVERT(Decimal(11,8), #neLng)
AND place_lng >= CONVERT(Decimal(11,8), #swLng)
AND place_lng <= CONVERT(Decimal(11,8), #neLng)
)
OR
(CONVERT(Decimal(11,8), #swLng) > CONVERT(Decimal(11,8), #neLng)
AND (place_lng >= CONVERT(Decimal(11,8), #swLng)
OR place_lng <= CONVERT(Decimal(11,8), #neLng))
)
)
P.S.: Note that you should probably use GIS like PostGIS for this to be really efficient :) I was about to solve this problem but I still postponed it "until I have a GIS". You inspired me and maybe I'll go for this solution too.
P.S.: I'm not sure how your database engine optimizes but I guess for increased efficiency it might help if you define latitude and longitude as indexes - so that it can solve the condition faster. But I'm not sure at all.
Actually I found a better solution. SQL Server has built-in data types for working with spatial data. So I added a new value to my table and called it place_geo with the data type geography. Then I populated it with geography points (created from the latitudes and longitudes I already had in the database). When I had the points stored as geography points in my database, the stored procedure becomes much simpler in its syntax and I also think it will execute much faster.
Here is my stored procedure:
ALTER PROCEDURE dbo.GetPlaceClosestToMe
(
#my_lat Decimal(11,8),
#my_lng Decimal(11,8)
)
AS
DECLARE #my_point geography;
SET #my_point = geography::Point(#my_lat, #my_lng , 4326); /* SRID 4326 */
SET NOCOUNT ON
SELECT TOP 1
place_id,
place_name,
#my_point.STDistance(place_geo)/1000 [Distance in km]
FROM Places
ORDER BY #my_point.STDistance(place_geo)
So what does it do?
First I get a longitude and a latitude as input parameters in decimal format. I declare a new variable (#my_point) that has the data type "geography". I then assign a new geography point to the variable I just created using the longitude and latitude input parameters.
Then I ask for the point in my database table (place_geo) that is closest to #my_point. (STDistance returns the shortest line between two geography types.) The division by 1000 is to get a distance expressed in kilometers.
If you want it to return several results, just alter the select: SELECT TOP 5 or SELECT TOP 10.
At the moment I don't have enough test data to test if this is faster than using the old method, but perhaps someone else have done some tests? I would guess that using the geography data type is much faster than the old method.
With my limited test data, the old stored procedure and this new stored procedure return the same results.
Hope you found this follow-up useful!