Migrating a MariaDB table with geometric data to MySql, some data cannot be inserted, because they are not well-formed, even if it's not a issue for MariaDB.
This request works on MariaDB (10.2).
CREATE TABLE IF NOT EXISTS geo (
id INT AUTO_INCREMENT NOT NULL,
value GEOMETRY NOT NULL,
SPATIAL INDEX idx_value (value),
PRIMARY KEY(id)
) ENGINE = InnoDB;
INSERT INTO geo (value) SELECT ST_GeomFromText('LINESTRING(1 2)');
Not on MySql (5.7.20), where the error is:
3037 - Invalid GIS data provided to function st_geometryfromtext.
There are three functions to identify such geometries in MySql: ST_IsSimple(), ST_IsValid(), and ST_Validate() but they don't work with badly formatted geometries:
SELECT ST_IsSimple(ST_GeomFromText('LINESTRING(1 2)'));
SELECT ST_IsValid(ST_GeomFromText('LINESTRING(1 2)'));
SELECT ST_AsText(ST_Validate(ST_GeomFromText('LINESTRING(1 1)')));
3055 - Geometry byte string must be little endian.
This example comes from https://dev.mysql.com/doc/refman/5.7/en/spatial-convenience-functions.html, but it doesn't work. So it's strange (the doc was not updated for 5.7). More details about validity on mysql: https://dev.mysql.com/doc/refman/5.7/en/geometry-well-formedness-validity.html (mysql accepts any syntactically well-formed input, but not the geometrically invalid).
Similar issues here:
MySQL spatial geometry validate wkt, where the answer is : the functions provided by MySQL to test validity of geometries requires well formed geometry as input...
MySQL 5.7: Invalid GIS data, where there is the idea to use a stored function and to create an exception handler, so a bit complicated.
A bug report asking for the same issue, but without response: https://bugs.mysql.com/bug.php?id=76595
https://github.com/creof/doctrine2-spatial/issues/155 (doctrine php orm), that says that the result is changing between versions of mysql.
But none of them answer to the issue: how to identify badly formatted geometries on mysql 5.7?
LineString needs at least two Points. Perhaps 5.6 was negligent in pointing that out.
mysql> SELECT hex(ST_GeomFromText('LINESTRING(1 1)'));
ERROR 3037 (22023): Invalid GIS data provided to function st_geometryfromtext.
mysql> SELECT hex(ST_GeomFromText('LINESTRING(1 1, 2 3)'));
+--------------------------------------------------------------------------------------------+
| hex(ST_GeomFromText('LINESTRING(1 1, 2 3)')) |
+--------------------------------------------------------------------------------------------+
| 00000000010200000002000000000000000000F03F000000000000F03F00000000000000400000000000000840 |
+--------------------------------------------------------------------------------------------+
mysql> SELECT ##version;
+-----------+
| ##version |
+-----------+
| 5.7.15 |
+-----------+
Related
I am no MySQL expert so no doubt I have overlooked something simple.
I have a MySQL DB (V 5.7.26) and if I run the following query (as an example) on one of my tables I get a result of 39 which is what I expect:
SELECT count(*)
FROM `entities`
WHERE
ST_CONTAINS(
ST_GeomFromText('POLYGON((-0.4120 51.6009, -0.4120 51.3467, 0.1533 51.3467, 0.1533 51.6009, -0.4120 51.6009))', 4326),
gloc
) = 1;
I created the same DB in MySQL V 8.0.23 and did so by exporting table definitions and recreating them by running the queries in the new DB, so as far as I can tell tables definitions etc are identical.
I imported the same data into the new DB and I know the data is correct, for example if I pull a record by another criteria(selecting by record id for example) I can display the record on a map (leaflet) and it is in the correct location.
However if I run the same query as above in my new DB it returns a result of 0.
If I leave out the SRID of 4326 then I get an error (which I would expect) of
/* SQL Error (3033): Binary geometry function st_contains given two geometries of different srids: 0 and 4326, which should have been identical. */
/* Affected rows: 0 Found rows: 0 Warnings: 0 Duration for 0 of 1 query: 0.000 sec. */
So I don't think its an SRID issue.
gloc is defined as type 'geometry', NULL values are allowed though none exist. However if I disallow NULL values and then add a spatial index to gloc column the result is exactly the same.
I would check lat:lon order. MySql 8 follows lat:lon order for 4326 as defined by CRS. Assuming your data is in England, rather than Indian ocean, you should swap coordinate order. Also check other data for same issue.
See also
https://mysqlserverteam.com/axis-order-in-spatial-reference-systems/
I have a data dump (csv) of place names from https://www.ordnancesurvey.co.uk/business-and-government/help-and-support/products/os-open-names.html
I need to import this into mysql however the geometry co-ordinates use BNG (OSGB36). Does Mysql have any functions to convert these co-ordinates to wgs84 lat/long or is there any other sql method to achieve this?
another option is perhaps loading it into postgis - does postgis have any funtion to transform BNG to lat/long? Perhaps I could do that and then export the data once transformed and load it into mysql?
In postgis I could do something as follows (not tested)
select AddGeometryColumn('locations', 'the_geom', 27700, 'POINT', 2);
-- X and Y are the BNG co-ordinates
UPDATE locations SET the_geom = ST_GeomFromText('POINT(' || x || ' ' || y || ')', 27700 );
alter table locations add column lat real;
alter table locations add column long real;
update locations set long=st_x(st_transform(the_geom,4326)),
lat=st_y(st_transform(the_geom,4326));
Is it possible to do these type of function in mysql - basically what are the equivalent functions in mysql. I cant seem to figure the syntax out? The following doesnt work in mysql:
update locations set long=ST_X(ST_Transform(the_geom,4326)),
lat=ST_Y(ST_Transform(the_geom,4326));
I get error function ST_Transform does not exist. I'm using mysql 5.7
* UPDATE *
Sample data:
NAME1 LOCAL_TYPE GEOMETRY_X GEOMETRY_Y DISTRICT_BOROUGH REGION COUNTRY
Southport Town 333510 417225 Sefton North West England
The answer is, of all places, here
Answered by Hartmut Holzgraefe in this comment.
So far the SRID property is just a dummy in MySQL, it is stored as
part of a geometries meta data but all actual calculations ignore it
and calculations are done assuming Euclidean (planar) geometry.
So ST_Transform would not really do anything at this point anyway.
I think the same is still true for MariaDB, at least the knowledge [sic]
base page for the SRID() function still says so:
This WorkLog discusses the progress of implementing ST_Transform.
MySQL 8.0 does seem to have it implemented: https://dev.mysql.com/doc/refman/8.0/en/spatial-operator-functions.html#function_st-transform
So, the solution may require upgrading to MySQL 8.0.
The changelog for the very recent 8.0.13 says:
----- 2018-10-22 8.0.13 General Availability -- -- -----
MySQL now implements the
ST_Transform()
spatial function for use in converting geometry values from one
spatial reference system (SRS) to another. Currently, it supports
conversion between geographic SRSs. For details, see Spatial Operator
Functions.
We're running MySQL 5.5.47 on a number of Debian servers. On some of them, we're seeing the following strange behavior:
mysql> set #TKEY:='ARDARD:fae590c4.ffa2.11e5.a318.0cc47a39aeb4-1460351116';
mysql> select replace(substring_index(substring_index(#TKEY,':',-1),'-',1), '.','-') as guid;
+--------------------------------------+
| guid |
+--------------------------------------+
| fae5a2.1--0cc47a 9ae47a 9aeb4a 9aeb4 |
+--------------------------------------+
This is supposed to extract the middle part of #TKEY (between the : and -) and replace all the periods with hyphens. Where are the spaces coming from? Other parts of the result seem to be jumbled up: 9aeb4 has been duplicated, a2.1 has been shifted left.
This doesn't happen if I assign the substring_index to an intermediate variable.
mysql> set #temp = substring_index(substring_index(#TKEY,':',-1),'-',1);
mysql> select replace(#temp, '.', '-') as guid;
+--------------------------------------+
| guid |
+--------------------------------------+
| fae590c4-ffa2-11e5-a318-0cc47a39aeb4 |
+--------------------------------------+
This only happens on our production servers. I can't reproduce it on our development server or sqlfiddle. I compared all the server variables, and there are no differences that look like they should affect the behavior of string functions (there were initially some differences in character set and collation variables, but I changed the dev server to match the production server and still couldn't replicate the error.
On another production server running MySQL 5.5.41 I get a slightly different wrong result:
mysql> select replace(substring_index(substring_index(#TKEY,':',-1),'-',1), '.','-') as guid;
+--------------------------------------+
| guid |
+--------------------------------------+
| fae590c4-ffa2-11e5-a318-0cc47a 9aeb4 |
+--------------------------------------+
This is correct except that there's a space in place of of the last 3.
Can anyone explain this? Is it just a MySQL bug? I couldn't find anything at bugs.mysql.com.
This appears to be a bug that was fixed in MySQL 5.6.5. There's a somewhat similar bug report regarding LOWER(SUBSTRING_INDEX(...)). It was closed with the comment:
Noted in 5.6.5 changelog.
The result of SUBSTRING_INDEX() could be missing characters when used
as an argument to conversion functions such as LOWER().
I suspect the underlying cause is pointer misuse resulting in buffer overflow and undefined behavior. Hopefully I haven't corrupted any long-lived memory in the server.
Rails Active Record save uses datetime in insert query although the datatype is time in mysql and time is set in model before saving
mysql schema:
rtime time DEFAULT NULL
ActiveRecord model: Abc
abc = Abc.new {'rtime'=> '18:23 PM'}
abc.save!
corresponding mysql query generated by active record:
insert into abces (rtime) values('2000-01-01 18:23:00');
Later in mysql only the time is stored and date is sliced off, and a warning is also generated.
+-----------------------+
| rtime |
+-----------------------+
| 18:23:00 |
+-----------------------+`
Why is the date appended with time while mysql insertion?
Rails version: 3.2.16
Looks like ActiveRecord often falls back to a DateTime object when assigning a Time in Rails 3+: https://github.com/rails/rails/blob/3-2-stable/activerecord/lib/active_record/attribute_assignment.rb#L130
There's some documentation on the conversion method here: http://apidock.com/rails/Time/time_with_datetime_fallback/class. Also, there's a comment that hints at this on line 180 of the same file: # we instantiate Time object and convert it back to a date thus using Time's logic in handling invalid dates
Thus it seems Rails 3 often treats Time objects as though they were DateTimes, which explains the odd insert query you're seeing. The behavior seems to be changed in Rails 4 so upgrading may resolve the issue.
In any case the date values seem to simply get truncated by MySQL hence the warning, but the stored time seems correct to me.
Finally, do you need a timezone for this time?
I'm storing geo coordinates in a MySQL table using a spatial index. The problem is that queries where I want to know if a tuple/location is within a bounding box (closed polygon described by 5 coordinates; first an last coordinate are the same) return an empty results. Here's a minimum example that is not working for me as expected:
CREATE TABLE osTest(
loc Point NOT NULL
) ENGINE=MyISAM;
CREATE SPATIAL INDEX locIndex ON osTest(loc);
-- New York Central Park, The Great Lawn
INSERT INTO osTest (loc) VALUES (POINT(40.781343, -73.966598));
If I execute the following table scan query:
SELECT AsText(loc), X(loc), Y(loc) FROM osTest;
all seems to work fine -- the result looks like this:
+-----------------------------+-----------+------------+
| AsText(loc) | X(loc) | Y(loc) |
+-----------------------------+-----------+------------+
| POINT(40.781343 -73.966598) | 40.781343 | -73.966598 |
+-----------------------------+-----------+------------+
However, when I try to execute a query that is using the spatial index, the result is always empty. The polygon the in following polygon describes the whole Central Park:
SELECT AsText(loc), X(loc), Y(loc) FROM osTest WHERE WITHIN(loc, GeomFromText('POLYGON(40.800716 -73.958358, 40.796858 -73.949120, 40.764216 -73.973153, 40.768108 -73.981929, 40.800716 -73.958358)') );
I've also tried INTERSECTS, OVERLAPS -- same empty result set. What am I missing here?
I am not an expert on mysql GIS, but in postgis the points have to be Latitude,Longitude. My belief is that it is the same in mysql 5.6 GIS data types as well.
According to that theory; in your query, y and x should be switched
I found the issue. I had an error in my SQL query. It doesn't result in a syntax error, but obviously it doesn't work correctly. More specifically, I had to add additional brackets. I had to change:
GeomFromText('POLYGON(40.800716 -73.958358, ...)')
to
GeomFromText('POLYGON((40.800716 -73.958358, ...))')
I simply overlooked it when I searched for examples. And that I didn't get any syntax errors or something didn't help either.