I consider using Sphinx search in one of my projects so I have a few questions related to it.
When using SphinxSE and RT index, every UPDATE or INSERT in the SphinxSE table will update the index, right? No need to call indexer or anything?
Can I search on both tags (user entered keywords for a document) and the content and give more relevance to the tag matches? And if it's possible how do I implement the tag search (now I have them in separate tables like an inverted index)
For the fillter attributes is it better to stick duplicates of them in the SphinxSE table or fillter using mysql from the regular documents table I have?
Thanks in advance!
OK, I finally understand how things work with the sphinx thing.
You cannot INSERT or UPDATE directly the SphinxSE table. Instead you use INSERT/REPLACE while connected to SphinxQL (directly to sphinx daemon).
With 1.10 you can add multiple FullText searchable fields. I added title, tags and content. And the query to give more weight to the title, then tags and then content looks like this:
SELECT SQL_NO_CACHE * FROM sphinx_docs WHERE query = 'a lot of keywords;weights=3,2,1;';
I use the SQL_NO_CACHE to tell mysql not to cache the result of this, because on next calls I can't get the number of rows returned from sphinx (SHOW STATUS LIKE 'sphinx_total_found')
It's better to let sphinx do all the sorting, filltering and use mysql only to JOIN the table you need more info from.
In addition I have to say that I tried many times to add the sphinxse plugin to mysql without success (endless make waiting hours) so I switched to MariaDB 5.2.4 which includes the SphinxSE storage engine.
Related
SQL Server: Index columns used in like?
I've tried using the query method in the link above with Postgres (0.3ms improvement), it seems to only work with MySQL (10x faster).
MYSQL
User Load (0.4ms) SELECT * FROM users WHERE reverse_name LIKE REVERSE('%Anderson PhD')
User Load (5.8ms) SELECT * FROM users WHERE name LIKE ('%Anderson Phd')
POSTGRES
User Load (2.1ms) SELECT * FROM users WHERE reverse_name LIKE REVERSE('%Scot Monahan')
User Load (2.5ms) SELECT * FROM users WHERE name LIKE '%Scot Monahan'
Did some googling but couldn't quite understand as I'm quite new to DBs. Could anyone explain why this is happening?
To support a prefix match in Postgres for character type columns you either need an index with a suitable operator class: text_pattern_ops for text etc. (Unless you work with the "C" locale ...)
Or you use a trigram index to support any pattern - then you don't need the "reverse" trick any more.
See:
PostgreSQL LIKE query performance variations
You'll see massive performance improvement for tables of none-trivial size, as soon as an index can be used for the query. Have a look at the query plan with EXPLAIN.
I've a web app developed by java. Currently I'm in a part of my app that I need to use MySql like in order to search for a string in mysql table contain 100000+ rows. When I had my research I found that MySql like doesn't use indexes but if you have the wildcard at the end of your string example: hello% but I need %hello% which like doesn't use index in these kinds of wildcards. And I also read on the internet that there are other technologies such as postgresql which can give you the ability of using indexes for searching string.
My question is Just because of like do I need to change MySql DB with all it's other features to postgresql DB, Do we have any alternative way on MySql To search for a string that uses indexes?, Do I Install them both and use each for it's own use ( If there is no other way );
All replies are much appreciated.
Do we have any alternative way on MySql To search for a string
Have you looked into MySQL Full-Text Search which uses fulltext index; provided you are using either InnoDB or MyISAM engine
I have a question about integrating Sphinx with MariaDB 10.0. The db comes packaged with the SphinxSE storage engine, but I'm having trouble finding detailed information about how to set this up. Consider the following scenario:
I have a book-selling app using a database with the following tables: "Book", "Listing", and "School". A "listing" contains a relationship to a single book and a single school. A "book" contains several columns, most importantly "title" and "author". I would like to perform full-text searches on the "Book" table using Sphinx, and a typical query might go something like this (in pseudo-SQL): "Find all books whose title contains the string 'Harry P' for which a current listing exists at Harvard University".
According to MariaDB's article: "you need to create a special ENGINE=SPHINX 'search table', and then SELECT from it with full text query..." - so does this mean that Sphinx cannot access/index my existing MariaDB tables at all? Must I recreate my "Book" table as a SphinxSE table? If this is the case, how do I specify relationships between this table and my other non-Sphinx tables (Listing, School, etc)? The above article gives a very simple JOIN example, but I'm still confused. If all "books" are stored in a Sphinx table, can MariaDB access this table for different types of queries (where a full-text search is not needed)?
As a related question, what would be a good way to enable Sphinx to index both the Title and the Author of the books? The app will always specify which field is being queried.
Any help would be greatly appreciated, because hours of searching and reading has still left me mostly in the dark on this topic.
The SphinxSE engine is NOT a real engine. All it does is serve a 'proxy' to a underlying Sphinx daemon. It runs the query against sphinx engine, then presents the 'results' as a FAKE table, so that mysql can then join it back with the original table(s).
Basically a convenience method to get sphinx results and the apprirate data in one query (with mysql doing the hard work of doing the 'JOIN')
... so before "worrying" about sphinxSE. You need to figure out a normal Sphinx index, that you can run queries against. ie create an index to be able to run that query. Normally the sphinx query will just give you a list of ids.
So play with getting sphinx setup independantly (without SphinxSE or MariaDB) - create an index, and run some queries against it. Get that working first.
Only once you have the index working (and you understand the concepts) - come back and work out how to set it up with SphinxSE.
There are plenty of articles on getting sphinx working
https://www.google.com/search?q=sphinx+search+getting+started
I am currently trying to figure out how to set up a search database on localhost which is set up with sphinx which I have no experience with, nor did I create this crazy database and scripting I have to work with. I successfully have install sphinx on the computer and it connects well. Now my question is do I need to set up the same index fields as it was set up in phpMyAdmin?
Thanks
No.
You setup sphinx to index the column(s) you need to be able to search. They dont need to match mysql indexes at all.
But if you recreating a search that used to use the mysql full-text indexes, then you would create the sphinx index on the same column(s) plus any additional columns you need to be able to filter, group or sort by.
Does anyone know of a way to embed comments in MySQL statements? When I search for mysql and comments I get only ways to put comments in tables, etc
The idea, if I implement this the way my boss wants it, is to prepend the user id to the statement apparently so that when MySQL is analyzed later (via the binary log) we know who did what.
Example:
SELECT id
FROM customer
WHERE handle='JBH'
Would now show up as:
-- user:jwilkie
SELECT id
FROM customer
WHERE handle='JBH'
(or similar)
EDIT FOR CLARITY: The reason for this is that we have perl modules that are interfacing with MySQL and we are retrieving the user id by reading $ENV{USER} (which in this case is "jwilkie"). It is a situation where we have one MySQL user defined but multiple people running the perl mod.
Does anyone have experience with this? Many many thanks! Jane
Normally, comments are stripped before the SQL statement is recorded in the binary log. However, a nasty workaround is to pretend that ypur comment contains syntax for some future version of MySQL - eg. 9.99.99:
/*!99999 user:jwilkie */ insert into tbl values (yyy);
These comments will then be passed through into the binary log.
If you have control over the SQL queries being generated, then you should be able to embed comments in them programatically in your query builder.
Select queries don't go in the binary log, but the comments may make it into the slow query log, general query log, etc.
Here's a blog post from Percona that touches on the subject a bit, specifically in the context of mk-query-digest. It might help:
http://www.mysqlperformanceblog.com/2010/07/05/mk-query-digest-query-comments-and-the-query-cache/