Django MySQL full text search - mysql

I need to implement full text search for my Django application, running MySQL as backend.
Let's say I've got a model as follows:
class MyItem(models.Model):
title = models.CharField()
short_description = models.TextField()
description = models.TextField()
I would like to have results first for search term occurences in title, then in short_description and at the end in description field. I'll be happier if I don't have to use additional modules/applications for this task.

The previously highest rated answer is deprecated. As of Django 1.10 there is no more search field lookup for MySQL databases (see the search section in the 1.10 documentation).
The release notes for 1.10 also propose a solution to this, by defining a custom lookup:
###__search query lookup
The search lookup, which supports MySQL only and is extremely limited in
features, is deprecated. Replace it with a custom lookup:
from django.db import models
class Search(models.Lookup):
lookup_name = 'search'
def as_mysql(self, compiler, connection):
lhs, lhs_params = self.process_lhs(compiler, connection)
rhs, rhs_params = self.process_rhs(compiler, connection)
params = lhs_params + rhs_params
return 'MATCH (%s) AGAINST (%s IN BOOLEAN MODE)' % (lhs, rhs), params
models.CharField.register_lookup(Search)
models.TextField.register_lookup(Search)

You can use full text search in django
MyItem.objects.filter(title__search="some search text")
One thing is - you can't define a fulltext index from a Django model, you need to do in directly in a database (using PHPMyAdmin or SQL query)
See Django documentation for field lookup called search

I don't know if it helps now but I've created a new Python Library for Django which supports MySQLs' and MariaDBs' native full-text search engine over one or multiple columns:
You can have a look at it on the GitHub repository
There's also a description how to install it, use it and how to create the FULLTEXT INDEX stuff via Django migrations (Django 1.7+).
If you've configured the indexes and set the SearchManager for your model you should be able to run something like:
Mymodel.objects.search('Something')
Mymodel.objects.search('Somet*')
Mymodel.objects.search('+Something -Awesome')
Just wanted to update this topic because I didn't find an acceptable solution so far and it might help someone out there as well :)
Cheers
Domi

If you are looking for a beefy solution I recommend http://haystacksearch.org/
It is very well thought out.

Django provides full-text functionality for PostgreSQL's only.

From django docs regarding full-text search:
Example:
Entry.objects.filter(headline__search="+Django -jazz Python")
SQL equivalent:
SELECT ... WHERE MATCH(tablename, headline) AGAINST (+Django -jazz Python IN BOOLEAN MODE);
Note this is only available in MySQL and requires direct manipulation of the database to add the full-text index. By default Django uses BOOLEAN MODE for full text searches. See the MySQL documentation for additional details.
Now to the direct manipulation of the database. In MySQL you can create full-text index by following these steps (source article):
Open command prompt, and enter mysql -u root -p. On prompt enter the root password.
Enter use your_db_name to switch to your django database.
Enter CREATE FULLTEXT INDEX index_name ON table_name (column_names).
That's it! FTS indexing in enabled in your django database. Now you can use the django's rich QuerySet API to do full-text searches.
Edit: The above quote no longer exists in the django version >1.9.

Related

Prisma Unsupported("point") MySql Approach

So I have my location column using Point data type, I'm using Apollo Server and Prisma, and when I use "npx prisma db pull" generates this data type because is not currently supported on Prisma (generated script)
so I say "Ok, I'm using string and I manage how to insert this data type" so I changed to this script, surprise! didn't work enter image description here, try to find any approach to handling MySql Point data type in Prisma but no info at soever, I really appreciate any ideas
You cannot convert it to String and use it as it isn't supported yet. You need to leave it as unsupported and you can only add data via raw queries.
For now, only adding data is supported. You cannot query for it using PrismaClient.
We can query data using Prisma Client, via raw queries as SELECT id, ST_AsText(geom) as geom from training_data where geom has dataType geometry for using Unsupported("geometry").

Python3, MySQL, and SqlAlchemy -- does SqlAlchemy always require a DBAPI?

I am in the process of migrating databases from sqlite to mysql. Now that I've migrated the data to mysql, I'm not able to use my sqlalchemy code (in Python3) to access it in the new mysql db. I was under the impression that sqlalchemy syntax was database agnostic (i.e. the same syntax would work for accessing sqlite and mysql), but this appears not to be the case. So my question is: Is it absolutely required to use a DBAPI in addition to Sqlalchemy to read the data? Do I have to edit all of my sqlalchemy code to now read mysql?
The documentation says: The MySQL dialect uses mysql-python as the default DBAPI. There are many MySQL DBAPIs available, including MySQL-connector-python and OurSQL, which I think means that I DO need a DBAPI.
My old code with sqlite successfully worked like this with sqlite:
engine = create_engine('sqlite:///pmids_info.db')
def connection():
conn = engine.connect()
return conn
def load_tables():
metadata = MetaData(bind=engine) #init metadata. will be empty
metadata.reflect(engine) #retrieve db info for metadata (tables, columns, types)
inputPapers = Table('inputPapers', metadata)
return inputPapers
inputPapers = load_tables()
def db_inputPapers_retrieval(user_input):
result = engine.execute("select title, author, journal, pubdate, url from inputPapers where pmid = :0", [user_input])
for row in result:
title = row['title']
author = row['author']
journal = row['journal']
pubdate = row['pubdate']
url = row['url']
apa = str(author+' ('+pubdate+'). '+title+'. '+journal+'. Retrieved from '+url)
return apa
This worked fine and dandy. So then I tried to update it to work with the mysql db like this:
engine = create_engine('mysql://snarkshark#localhost/pmids_info')
At first when I tried to run my sample code like this, it complained because I didn't have MySqlDB. Some googling around informed me that MySqlDB does NOT work for Python 3. So then I tried pip installing pymysql and changing my engine statement to
engine = create_engine('mysql+pymysql://snarkshark#localhost/pmids_info')
which also ends up giving me various syntax errors when I try to adjust things.
So what I want to know, is if there is any way I can get my current syntax to work with mysql? Since the syntax is from sqlalchemy, I thought it would work perfectly for the exact same data in mysql that was previously in sqlite. Will I have to go through and update ALL of my db functions to use the syntax of the DBAPI?
This will sound like a dumb answer, but you'll need to change all the places where you're using database-specific behavior. SQLAlchemy does not guarantee that anything you do with it is portable across all backends. It leaks some abstractions on purpose to allow you to do things that are only available on certain backends. What you're doing is like using Python because it's cross-platform, then doing a bunch of os.fork()s everywhere, and then being surprised that it doesn't work on Windows.
For your specific case, at a minimum, you need to wrap all your raw SQL in text() so that you're not affected by the supported paramstyle of the DBAPI. However, there are still subtle differences between different dialects of SQL, so you'll need to use the SQLAlchemy SQL expression language instead of raw SQL if you want portability. After all that, you'll still need to be careful not to use backend-specific features in the SQL expression language.

'Relation does not exist' error after transferring to PostgreSQL

I have transfered my project from MySQL to PostgreSQL and tried to drop the column as result of previous issue, because after I removed the problematic column from models.py and saved. error didn't even disappear. Integer error transferring from MySQL to PostgreSQL
Tried both with and without quotes.
ALTER TABLE "UserProfile" DROP COLUMN how_many_new_notifications;
Or:
ALTER TABLE UserProfile DROP COLUMN how_many_new_notifications;
Getting the following:
ERROR: relation "UserProfile" does not exist
Here's a model, if helps:
class UserProfile(models.Model):
user = models.OneToOneField(User)
how_many_new_notifications = models.IntegerField(null=True,default=0)
User.profile = property(lambda u: UserProfile.objects.get_or_create(user=u)[0])
I supposed it might have something to do with mixed-case but I have found no solution through all similar questions.
Yes, Postgresql is a case aware database but django is smart enough to know that. It converts all field and it generally converts the model name to a lower case table name. However the real problem here is that your model name will be prefixed by the app name. generally django table names are like:
<appname>_<modelname>
You can find out what exactly it is by:
from myapp.models import UserProfile
print (UserProfile._meta.db_table)
Obviously this needs to be typed into the django shell, which is invoked by ./manage.py shell the result of this print statement is what you should use in your query.
Client: DataGrip
Database engine: PostgreSQL
For me this worked opening a new console, because apparently from the IDE cache it was not recognizing the table I had created.
Steps to operate with the tables of a database:
Database (Left side panel of the IDE) >
Double Click on PostgreSQL - #localhost >
Double Click on the name of the database >
Right click on public schema >
New > Console
GL

Drupal : How can I know if the db is mysql or postgres

I have a complicated query and since I need that my module work on both mysql and postgres, I need to write two version of it.
Unfortunately, I don't know how I can check if the db I use is mysql or postgres, to know which query use. Do you know if a function can return this value?
As #kordirko says, one option is to query the server version: SELECT version(); will work on both MySQL and PostgreSQL, though not most other database engines.
Parsing version strings is always a bit fragile though, and MySQL returns just a version number like 5.5.32 wheras PostgreSQL returns something like PostgreSQL 9.4devel on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.7.2 20121109 (Red Hat 4.7.2-8), 64-bit. What do you do if you're connecting to a PostgreSQL-compatible database like EnterpriseDB Postgres Plus, or a MySQL-compatible database?
It's much safer to use the Drupal function for the purpose, DatabaseConnection::databaseType. This avoids a query round-trip to the DB, will work on databases that won't understand/accept SELECT version(), and will avoid the need to parse version strings.
You'll find this bug report useful; it suggests that the correct usage is Database::getConnection()->databaseType().
(I've never even used Drupal, I just searched for this).
As long as the abstract DatabaseConnection class extends PDO class, you can invoking pdo methods in order to know the current database driver.
For instance:
$conn = Database::getConnection();
print $conn->getAttribute($conn::ATTR_DRIVER_NAME); #returns mysql, pgsql...
There is a second way to do it using DatabaseConnection::driver():
print $conn->driver();
or DatabaseConnection::databaseType();
print $conn->databaseType();
Note that DatabaseConnection::driver() and DatabaseConnection::databaseType() are similar functions but not equals!
The return value from DatabaseConnection::driver() method depends on the implementation and other factors.
in the Drupal Database API page:
database.inc abstract public DatabaseConnection::driver()
This is not necessarily the same as the type of the database itself. For instance, there could be two MySQL drivers, mysql and mysql_mock. This function would return different values for each, but both would return "mysql" for databaseType().
In the most cases you just gonna want to use only
$conn->getAttribute($conn::ATTR_DRIVER_NAME)
or $conn->databaseType()
If you want get more specific properties, you should take advantage the PHP ReflectionClass features:
$conn = Database::getConnection();
$ref = new ReflectionClass($conn);
#ref->getProperties, ref->getConstants $ref->isAbstract...
Reference:
PDO::getAttribute
PDO::ATTR_DRIVER_NAME
Drupal Database API
Drupal Base Database API class

NHibernate Full Text Search

What is the best way trying to get a text search function using nhibernate? I have read about NHibernate.Search, but cannot find the library anywhere.
I downloaded the latest NHibernate source code(2.1.2) and compiled it, but i still cannot find NHibernate.Search.
Does anyone have any suggestions? Or any other methods to do text search?
EDIT: Using MySQL database, incase it makes any difference.
NHiberante.Search a separate dll to make Nhiberante and Lucene work together. You have to download and reference it if you want to use it. You might like to read some introduction about Lucene to understand how Nhiberante.Search works.
One of the places where you can get the dll is hornget
If you want to the MySql specific full text search options, there won't be any help from Nhibernate for you to use them.
You can use Expression.Sql but i think it's good idea using mysql stored procedure
Your MySql Stored Procedure :
CREATE PROCEDURE `GetProductsByText`(IN `queryText` VARCHAR(100) CHARSET utf8)
SELECT *
FROM Products
WHERE MATCH(Title, Description) AGAINST (queryText)
Your nhibernate mapping xml file :
<sql-query name="GetProductsByText">
<return class="Product"/>
call `GetProductsByText`( :queryText )
</sql-query>
your c# nhibernate query:
public IList<Product> FindByText (string text)
{
var session = SessionFactory.GetCurrentSession ();
IQuery query = session.GetNamedQuery ("GetProductsByText");
return query.SetString ("queryText", text).List<Product> ();
}