I'm using Symfony 3.3.10 and MySQL 5.7.18, and trying to use a fulltext index like:
/**
* #ORM\Table(name="article", indexes={#ORM\Index(name="body_idx", columns={"body"}, flags={"fulltext"})})
* #ORM\Entity(repositoryClass="AppBundle\Repository\ArticleRepository")
*/
class Article
{
With this entity definition, bin/console doctrine:schema:update --dump-sql gives
CREATE FULLTEXT INDEX body_idx ON article (body);
but I want to use WITH PARSER like
CREATE FULLTEXT INDEX body_idx ON article (body) WITH PARSER mecab;
Question: How can I achieve this? My intuitive is any of the following would be the strategy, but I've not come to the answer: 1) configuring ORM\Index's option to add an extra string to the query or 2) somehow using raw SQL to build the query.
Related
I've recently picked up Slick and Scala, and realised my database tables created by Slick where using MYISAM, seems this is the default engine on my MySQL config and Slick create table statements do not specify engine. That's all good but I wanted to change to InnoDB without changing my default table engine in MySQL. Slick documentation has nothing on the matter. Is there anyway to change table engine with Slick (before tables creation) without modying default engine in MySQL my.ini ?
You can define the engine or otherwise extend the query by manipulating the statements:
val statements: Iterator[String] = myTableQuery.schema.createStatements
val action: SqlAction[Int, NoStream, Effect] =
sqlu"#${createStatement.mkString(" ")} engine=InnoDB"
db.run(createAction)
Take care to use the # verbatim interpolation prefix for the sqlu interpolator - without it, the statements will be escaped, leading to a syntax error.
see related docs: http://slick.lightbend.com/doc/3.2.3/schemas.html#data-definition-language
EDIT:
I have noticed that the approach does not work if the DDL generates multiple statements, like additional alter table.... In this case, there is an even less elegant solution:
def createTable(ddl: profile.DDL, db: Database)(implicit ec: ExecutionContext): Future[Int] = {
val mapped = ddl.createStatements.toList map (
s => if (s.startsWith("create table")) s"$s engine=InnoDB" else s
)
mapped.foldLeft(Future.successful(1)) {
case (acc: Future[Int], s: String) =>
acc.flatMap(_ => db.run(sqlu"#$s"))
}
}
You cannot use traverse or map to create the futures, because alter table depends on the table being already present. Use flatMap for sequential execution.
I know that Doctrine 2 doesn't support FULLTEXT indexes. I'm actually using a result set mapping and native queries to FULLTEXT search innodb tables (MySQL 5.6). But I still need to mark one or more entity fields as part of the index.
Is there any way to add the index using annotations? It seems that #Index annotation doesn't specify the type of...
According to DDC-3014 in the issue tracker of Doctrine, the possibility to specify full-text indices using annotations was implemented on April 14 and will be available in release 2.5. If you don't like to wait, you could try to use the unstable, development version or to backport the commit implementing the feature.
Here is a usage example:
/**
* #Entity
* #Table(indexes={#Index(columns={"content"}, flags={"fulltext"})})
*/
class Comment
{
/**
* #Column(type="text")
*/
private $content;
...
}
Here is an example how to make a fulltext index with the yaml mapping driver.
Doctrine\Tests\ORM\Mapping\Comment:
type: entity
fields:
content:
type: text
indexes:
xy_fulltext:
columns: ["name", "content", "keywords"]
flags: fulltext
In PostgreSQL 9.3, there are multiple ways to build an expression, which points to a json field's nested property:
data->'foo'->>'bar'
data#>>'{foo,bar}'
json_extract_path_text(data, 'foo', 'bar')
Therefore PostgreSQL only use these indexes, if the query's expression is an exact match with the index's expression.
CREATE TABLE json_test_index1(data json);
CREATE TABLE json_test_index2(data json);
CREATE TABLE json_test_index3(data json);
CREATE INDEX ON json_test_index1((data->'foo'->>'bar'));
CREATE INDEX ON json_test_index2((data#>>'{foo,bar}'));
CREATE INDEX ON json_test_index3((json_extract_path_text(data, 'foo', 'bar')));
-- these queries use an index, while all other combinations not:
EXPLAIN SELECT * FROM json_test_index1 WHERE data->'foo'->>'bar' = 'baz';
EXPLAIN SELECT * FROM json_test_index2 WHERE data#>>'{foo,bar}' = 'baz';
EXPLAIN SELECT * FROM json_test_index3 WHERE json_extract_path_text(data, 'foo', 'bar') = 'baz';
My questions are:
Is this behaviour intended? I thought the query optimizer should (at least) use the index with the #>> operator, when the query contains the appropriate call of json_extract_path_text() -- and vice versa.
If I want to use more of these expressions in my application (not just one, f.ex. stick to the -> & ->> operators), what indexes should I build? (I hope, not all of them.)
Are there any chance, that some future Postgres versions' optimizers will understand the equivalence of these expressions?
EDIT:
When i create an additional operator for that:
CREATE OPERATOR ===> (
PROCEDURE = json_extract_path_text,
LEFTARG = json,
RIGHTARG = text[]
);
This query (table from the previous example) still not uses its index:
EXPLAIN SELECT * FROM json_test_index3 WHERE data ===> '{foo,bar}' = 'baz';
Bonus question:
While Postgres expands the operators into function calls (behind the scenes), why this still not using its index?
You must use GIN index for JSON and JSONB datatype.
You can use operator parameters for your planned query
Examples:
CREATE INDEX idx_tbl_example ON tbl_example USING GIN(your_jsonb_field);
If you are planning only use #> operator, you can use with jsonb_path_ops parameter
CREATE INDEX idx_tbl_example ON tbl_example USING GIN(your_jsonb_field jsonb_path_ops);
Other choices is documented on postgresql site
I think you can use this:
CREATE INDEX idx_tbl_example ON tbl_example USING GIN(your_jsonb_field json_extract_path_text);
I am trying to build a very basic fulltext search app in php. I have found an example here Mysql fulltext search so I used it to build my own.
This is my table/data (field types are text)
name site result
RRR1 COS COSMETICS P R15-500 000847719903 20110607 094742.VER RRR1 P
RRR3 BIST MIDDLEWARE P R22-200 029051946829 20110607 101331.VER RRR3 P
RRR2 PRE PREHEAT F R16-500 000897546214 20110607 085111.VER RRR2 F
RRR1 COS COSMETICS P R16-300 000899331425 20110607 091337.VER RRR1 P
This is my index
ALTER TABLE automation_search_test ADD FULLTEXT search_idx (name ,site)
This is my query
SELECT * FROM automation_search_test WHERE MATCH (name,site) AGAINST ('RRR1' with query expansion);
I have however a few problems:
If I remove with query expansion, the query returns an empty set
If I add the RESULT column to my fulltext index, the query returns an
empty set after I change my SQL to this
SELECT * FROM automation_search_test WHERE MATCH (name,site,result)
AGAINST ('RRR1' with query expansion);
Is my index incorrectly setup? Is there an issue with the data?
I welcome any input you have. Thanks.
Not enough data was my problem. Mysql documentation clearly explains it. Thread closed.
I need to implement full text search for my Django application, running MySQL as backend.
Let's say I've got a model as follows:
class MyItem(models.Model):
title = models.CharField()
short_description = models.TextField()
description = models.TextField()
I would like to have results first for search term occurences in title, then in short_description and at the end in description field. I'll be happier if I don't have to use additional modules/applications for this task.
The previously highest rated answer is deprecated. As of Django 1.10 there is no more search field lookup for MySQL databases (see the search section in the 1.10 documentation).
The release notes for 1.10 also propose a solution to this, by defining a custom lookup:
###__search query lookup
The search lookup, which supports MySQL only and is extremely limited in
features, is deprecated. Replace it with a custom lookup:
from django.db import models
class Search(models.Lookup):
lookup_name = 'search'
def as_mysql(self, compiler, connection):
lhs, lhs_params = self.process_lhs(compiler, connection)
rhs, rhs_params = self.process_rhs(compiler, connection)
params = lhs_params + rhs_params
return 'MATCH (%s) AGAINST (%s IN BOOLEAN MODE)' % (lhs, rhs), params
models.CharField.register_lookup(Search)
models.TextField.register_lookup(Search)
You can use full text search in django
MyItem.objects.filter(title__search="some search text")
One thing is - you can't define a fulltext index from a Django model, you need to do in directly in a database (using PHPMyAdmin or SQL query)
See Django documentation for field lookup called search
I don't know if it helps now but I've created a new Python Library for Django which supports MySQLs' and MariaDBs' native full-text search engine over one or multiple columns:
You can have a look at it on the GitHub repository
There's also a description how to install it, use it and how to create the FULLTEXT INDEX stuff via Django migrations (Django 1.7+).
If you've configured the indexes and set the SearchManager for your model you should be able to run something like:
Mymodel.objects.search('Something')
Mymodel.objects.search('Somet*')
Mymodel.objects.search('+Something -Awesome')
Just wanted to update this topic because I didn't find an acceptable solution so far and it might help someone out there as well :)
Cheers
Domi
If you are looking for a beefy solution I recommend http://haystacksearch.org/
It is very well thought out.
Django provides full-text functionality for PostgreSQL's only.
From django docs regarding full-text search:
Example:
Entry.objects.filter(headline__search="+Django -jazz Python")
SQL equivalent:
SELECT ... WHERE MATCH(tablename, headline) AGAINST (+Django -jazz Python IN BOOLEAN MODE);
Note this is only available in MySQL and requires direct manipulation of the database to add the full-text index. By default Django uses BOOLEAN MODE for full text searches. See the MySQL documentation for additional details.
Now to the direct manipulation of the database. In MySQL you can create full-text index by following these steps (source article):
Open command prompt, and enter mysql -u root -p. On prompt enter the root password.
Enter use your_db_name to switch to your django database.
Enter CREATE FULLTEXT INDEX index_name ON table_name (column_names).
That's it! FTS indexing in enabled in your django database. Now you can use the django's rich QuerySet API to do full-text searches.
Edit: The above quote no longer exists in the django version >1.9.