Spatial index on generated (virtual/stored) column? - mysql

I have a generated (storedAs) column in my Laravel database that takes two decimal columns lat and lng and returns a point geospatial type. I'm using a generated column so as to avoid a scenario where the decimal lat/lng columns somehow get updated and the spatial latlng column doesn't (or vice versa).
I'd like to also index this column so as to perform speedy lookups, but running into a slight hitch.
I'm using the Laravel migration builder to generate the table schema. Although the source columns are not nullable and default to 0, it won't run if I also add a spatial index, citing all parts must be not null. I'm assuming that "all parts" refers to just the latlng column.
The migration below uses the spatialIndex() method from Grimzy's Laravel MySQL Spatial extension, but I don't think that's the issue, as the standard Laravel Blueprint class also returns the same result.
public function up()
{
Schema::table('locations', function (Blueprint $table) {
$table->decimal('lat', 8, 6)->default(0);
$table->decimal('lng', 9, 6)->default(0);
$table->point('latlng')->storedAs('POINT(lat, lng)');
$table->spatialIndex('latlng');
});
}
Upon migration:
Illuminate\Database\QueryException : SQLSTATE[42000]: Syntax error or access violation: 1252 All parts of a SPATIAL index must be NOT NULL (SQL: alter table `locations` add spatial `locations_latlng_spatial`(`latlng`))
My only guess is that MySQL doesn't support spatial indexes on generated columns. Or am I doing something wrong?
UPDATE: seems I'm using MariaDB and not MySQL, which doesn't currently support them.

Related

MySQL 8: Create Collections via DDL

I’d like to be able to create MySQL Document Store Collections via simple SQL DDL statements rather than using the X-Protocol clients.
Is there any way to do so?
Edit: I’ll try and clarify the question.
Collections are tables using JSON datatypes and functions. That much is clear.
I would like know how I can create a Collection without using the X-Protocol calls and make sure that the aforementioned collection is picked up as an actual Collection.
Judging from MySQL workbench, collection tables have a _id blob PK with an expression, a doc JSON column and a few other elements I do not recall at the moment (might be indexes, etc).
I have no means to tell via the Workbench whatever additional schema/metadata information is required for a table to be considered a Document Store Collection, or if the mere presence of an _id and doc columns are enough.
I hope this clears things up.
All "x-api" instructions are directly mapped to sql syntax. When you e.g. run db.createCollection('my_collection'), MySQL will literally just execute
CREATE TABLE `my_collection` (
`doc` json DEFAULT NULL,
`_id` varbinary(32) GENERATED ALWAYS AS
(json_unquote(json_extract(`doc`,_utf8mb4'$._id'))) STORED NOT NULL,
`_json_schema` json GENERATED ALWAYS AS (_utf8mb4'{"type":"object"}') VIRTUAL,
PRIMARY KEY (`_id`),
CONSTRAINT `$val_strict` CHECK (json_schema_valid(`_json_schema`,`doc`))
NOT ENFORCED
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
You can run the corresponding sql statements yourself if you follow that format.
The doc and _id (with their type and the given expression) are required, the _json_schema is optional, the check too (and only added since MySQL 8.0.17). Since MySQL 8, no additional columns are allowed, except generated columns that use JSON_EXTRACT on doc and which are supposed to be used in an index, see below (although they don't actually have to be used in an index).
Any table that looks like that - doc and _id with their correct type/expression and no other columns except an optional _json_schema and generated JSON_EXTRACT(doc,-columns - will be found with getCollections().
To add an index, the corresponding syntax for
my_collection.createIndex("age", {fields: [{field: "$.age", type: "int"}]})
would be
ALTER TABLE `test`.`my_collection` ADD COLUMN `$ix_i_somename` int
GENERATED ALWAYS AS (JSON_EXTRACT(doc, '$.age')) VIRTUAL,
ADD INDEX `age` (`$ix_i_somename`)
Obviously,
db.dropCollection('my_collection')
simply translates to
DROP TABLE `my_collection`
Similarly, all CRUD operations on documents have a corresponding sql DML syntax (that will actually be executed when you use them via x-api).

Getting all values of an index in mysql

I'm working on a heatmap application which retrieves longitude and latitude points from a mysql database. Since the table I'm using is innodb I cannot use spatial index. So instead I divided the region where the points are located into a 1000*1000 block grid and assign each point an index based on the block it falls into using this formula:
set blockindex=FLOOR((lon-minlon)/londifference*1000)*1000+FLOOR(lat-minlat)/latdifference*1000);
Is there a syntax in mysql to get all the values in this index that's been assigned to a point?
MariaDB 5.2 (etc) has "Virtual Columns" , which can be indexed.
5.7.6 will have "functional indexes" by using "Functional Expressions" .

Table structure for Geo Spatial Data

What is the suggested way to structure a table in a MYSQL Database containing Geo Spatial data. As a POC i am working to insert the data as in the image as below into a database table.
This below is the snapshot of the table creation of Phpmyadmin table creation
looking for suggestions to create table for geospatial data as in screenshots.
EDIT: I am working on XAMPP V1.8.3 Windows8 running MYSQL version 5.6.16.
Created Table geomduplicate and columns and inserted data as in screenshot with the below sql
CREATE TABLE geomduplicate1(
zip INTEGER(3) NOT NULL PRIMARY KEY,
latitude NUMERIC(9,6),
longitude NUMERIC(10,6),
city VARCHAR(10),
state VARCHAR(2),
county VARCHAR(9)
);
REPLACE INTO geomduplicate1(zip,latitude,longitude,city,state,county) VALUES
(501,40.922326,-72.637078,'Holtsville','NY','Suffolk');
REPLACE INTO geomduplicate1(zip,latitude,longitude,city,state,county) VALUES
(544,40.922326,-72.637078,'Holtsville','NY','Suffolk');
REPLACE INTO geomduplicate1(zip,latitude,longitude,city,state,county) VALUES
(601,18.165273,-66.722583,'Adjuntas','PR','Adjuntas');
REPLACE INTO geomduplicate1(zip,latitude,longitude,city,state,county) VALUES
(602,18.393103,-67.180953,'Aguada','PR','Aguada');
REPLACE INTO geomduplicate1(zip,latitude,longitude,city,state,county) VALUES
(603,18.455913,-67.14578,'Aguadilla','PR','Aguadilla');
REPLACE INTO geomduplicate1(zip,latitude,longitude,city,state,county) VALUES
(604,18.49352,-67.135883,'Aguadilla','PR','Aguadilla');
REPLACE INTO geomduplicate1(zip,latitude,longitude,city,state,county) VALUES
(605,18.465162,-67.141486,'Aguadilla','PR','Aguadilla');
REPLACE INTO geomduplicate1(zip,latitude,longitude,city,state,county) VALUES
(606,18.172947,-66.944111,'Maricao','PR','Maricao');
REPLACE INTO geomduplicate1(zip,latitude,longitude,city,state,county) VALUES
(610,18.288685,-67.139696,'Anasco','PR','Anasco');
and the data was successfully inserted in mysql database.
Store it is as a geometry data type. MySQL supports Geometry (generic), as well as Point, Linestring and Polygon data types, see creating spatial data types. A single longitude or latitude value can not be a geometry on its own, as you have it in your screen shot.
If you go the route of using geometry types, it gives you two advantages over having separate latitude and longitude fields: you can add a spatial index and you will be able to use some of MySQL's spatial operator functions such as ST_Buffer, ST_Intersects, ST_Distance to do further analysis. Spatial indexes are based on R-trees and will perform far better than two B-tree indexes on non spatial columns, latitude and longitude -- and this performance difference will grow as your table size grows.
You can still get the latitude and longitude values back by using the X and Y point functions so you will not lose anything by storing your data as a Point.
If you already have your data in two separate lat/lon columns, and you want to go the geometry/point datatype route, you can use the Point function to create the Point datatype:
alter table mytable add column pt POINT;
update mytable set pt=Point(longitude, latitude);
alter table mytable modify pt POINT NOT NULL;
create spatial index ix_spatial_mytable_pt ON mytable(pt);
Note that the Point function was only introduced in MySQL 5.1.x (it isn't too well documented, so I'm not sure of exact version), and before that you had to use concat with the GeomFromText function, see Moving lat/lon text columns into a 'point' type column for some more information on this, although note that Quassnoi's answer has lon and lat the wrong way round -- it is Point(lon, lat) not the other way, though this is a very common mistake.
NOTE: Until recently, you could only index a spatial column if using the MyISAM engine.
EDIT: In the upcoming release, MySQL 5.7.5, InnoDB will finally support indexes on spatial data types (and not just store spatial types without an index, which is considerably less useful). This means you can have foreign keys, ACID guarantees, spatial indexes all in one engine, which has been a long time in coming.
Since version 5.6 MySQL's innodb engine supports spatial data.
Refer the following links -
http://dev.mysql.com/doc/refman/5.6/en/creating-spatial-columns.html
https://dev.mysql.com/doc/refman/5.5/en/gis-data-formats.html
https://dev.mysql.com/doc/refman/5.5/en/populating-spatial-columns.html

Symfony doctrine::build task cannot create table named order

I have a database with a table named order.
When i run php symfony doctrine:build --all, i got the folowing error:
SQLSTATE[42000]: Syntax error or
access violation: 1064 You have an
error in your SQL syntax; check the
manual that corresponds to your MySQL
server version for the right syntax to
use near 'order (id BIGINT
AUTO_INCREMENT, status VARCHAR(255),
colissimonumber VARCHAR(25' at line 1.
Failing Query: "CREATE TABLE order (id
BIGINT AUTO_INCREMENT, status
VARCHAR(255), colissimonumber
VARCHAR(255) NOT NULL, created_at
DATETIME NOT NULL, updated_at DATETIME
NOT NULL, PRIMARY KEY(id)) ENGINE =
MyISAM".
The problem is clearly that order has no backquotes arount it (if i run manually the query in phpmyadmin with backquotes, it works)
How do i tell doctrine to add backquotes around table and field names? Any workaround exept renaming my table ?
I run symfony 1.4.9 with doctrine 1.2
You can turn on Doctrine_Core::ATTR_QUOTE_IDENTIFIER in your doctrine configuration mthod on projectConfiguration which will quote tables and col names but its not recommended:
Just because you CAN use delimited
identifiers, it doesn't mean you
SHOULD use them. In general, they end
up causing way more problems than they
solve. Anyway, it may be necessary
when you have a reserved word as a
field name (in this case, we suggest
you to change it, if you can).
http://www.doctrine-project.org/projects/orm/1.2/docs/manual/configuration/en#identifier-quoting
You probably want your model to be named Order, but this doesn't mean that the corresponding RDBMS table must be named the same.
Order:
tableName: project_order
columns: ...
Got similar error when upgrading from MySQL 5.7 to MySQL 8.0. RANK() is a function added in MySQL 8.0 whereas one of the table in our database has rank as column name.
My Setup:
- Symfony 1.5
- Doctrine 1.2
- PHP 7.4.24
- MySQL 8.0.27
- Ubuntu 20.04.1
Here are three possible solutions.
Solution 1: Add quote_identifier: true in config/databases.yml file. Also, clear cache after change in configuration with php symfony cc or php symfony cache:clear.
all:
doctrine:
param:
attributes:
quote_identifier: true
Solution 2: Turn on Doctrine_Core::ATTR_QUOTE_IDENTIFIER in config/ProjectConfiguration.class.php file on configureDoctrine() method.
$conn = Doctrine_Manager::getInstance()->getCurrentConnection();
$conn->setAttribute(Doctrine_Core::ATTR_QUOTE_IDENTIFIER, true);
Solution 3: Turn on Doctrine_Core::ATTR_QUOTE_IDENTIFIER on specific table(s) which are potentially breaking the system.
$table = Doctrine_Core::getTable('table_name');
$table->setAttribute(Doctrine_Core::ATTR_QUOTE_IDENTIFIER, true);
Note: From Doctrine 1 docs
Just because you CAN use delimited identifiers, it doesn't mean you
SHOULD use them. In general, they end up causing way more problems
than they solve. Anyway, it may be necessary when you have a reserved
word as a field name (in this case, we suggest you to change it, if
you can).

Applying an Index to a Blob/Longtext field

I am trying to create an index on a particular text string using Hibernate annotations. What I am currently doing is:
#Index(name="guid_index")
#Column(length=1400)
private String myGUID;
However, since the length of the column is 1400, this maps to SQL using the longtext type. hbm2ddl is able to create the table without mishap, but when it tries to create the index, I get:
Unsuccessful: create index guid_index on table_name (myguid);
BLOB/TEXT column 'myguid' used in key specification without a key length
How can I create an index on this column using Hibernate?
I initially thought that HHH-4389 was what you're looking for but your case seems to be a different issue: Hibernate is simply not generating the expected DDL statement. I would open a new Jira issue.
You have to specify a length of index.
Maximum length depends on storage engine, but usually it's not more than 1000 bytes,
if charset is utf-8, you have to divide maximum lengh by 3.
create index guid_index on table_name (myguid (1000) );
It's good enough for WHERE field LIKE 'prefix%' and WHERE field = 'text' conditions.