i have such long text that over bytes limit of index.
The text doesn't have any space. it's combination of things like dbname and table_name.
I will query only check exactly same
select * from table where column='text'
or
if i use full text search it may like this.
select * from table where Match (column) Against(text)
i don't know which one is best between two options below.
i will use text_example = '{dbName}/{tableName}' for explain but in real, it's combination of at least 5types of string.
option1. use full text search
option2. use index
To use index, i have to split text_example to 5types. And then create index each types.
create table info (
dbName text,
tableName text,
index dbIdx (dbName),
index tableIdx (tableName)
);
insert query will often.
In this case, which one is best aspects of storage?
how about performance?
or is there any good way to improve select query with such long text?
FYI, i use mysql 8.0
Punctuation, such as /, separates "words" for the purpose of FULLTEXT.
"dbname/tablename" is not very long text -- that is limited to perhaps 129 characters. (So, I don't understand where "over bytes limit of index" is coming from.)
col = 'text' checks the whole column; MATCH..AGAINST checks "words" in the column. That is, they do different things.
Please clarify what the real text is like and what the real query is like. Then we can advise on FULLEXT versus BTREE.
Related
I tried searching in different terms & got some answers too but they were not matching to my requirements. like This Link
I am using a sql statement something like below to fetch matching results from MySQL table.
SELECT statements... WHERE keyword_title_field REGEXP 'abc|axy|91store';
My questions is:
What data type (e.g. varchar, text etc) should i choose for keyword_title_field field in MySQL table to fetch results quickly without putting much load on table/server.
My current data type is Text due to unknown character length supply by user. Is this best suited or should i change?
Though it's not mandatory but any reference reading along with answer would be great for my understanding.
Here are some things to consider:
When you use any field in conditions (like REGEXP, LIKE or even '=') it is importand that you put an INDEX on the field. This will make MySQL not search every record 1 by 1, but find it via its INDEX instead. So make sure to look into that -> https://www.tutorialspoint.com/mysql/mysql-indexes.htm
The less characters allowed in your field, the smaller the INDEX is. You however have variable lengths to consider, so a TEXT is fine. If you know the maximum length and it's less than 256 characters, use a VARCHAR. Just make sure to index the field.
Note that REGEXP is relatively slow. LIKE '%term%' would be prefered, but that of course depends on your needs. If it's just 'abc' OR 'axy' OR '91store', you could consider this query: SELECT statements... WHERE keyword_title_field IN ('abc', 'axy', '91store');
I have a table and one of the column is TEXT type. I need to search the table for the rows with the text similar to the given string.
As the string can be pretty long (let's say 10000 bytes) I decided that it will be enough to compare only first 20 bytes of the string. To do this search faster I created a key:
KEY `description` (`description`(20))
So what I want to do now is one of the following query:
SELECT * FROM `table` WHERE STRCMP(SUBSTRING(`description`,0,20),'string_to_compare') = 0
or
SELECT * FROM `table` WHERE `description` LIKE 'string_to_compare%')
Note that I put only one percentage sign at the end of string_to_compare for saying to DB that I want to compare only first bytes.
I hope that MySQL brains will do the best to use key and not to do any extra moves.
Questions:
Is there any difference which query is better? I'm personally prefer the second as it looks clearer and hopefully will be better understand by the DB engine (MyISAM).
Is that correct the MySQL MyISAM will make an efficient code for these queries?
How do I put '%' in the PDO's prepare statement? SELECT * FROM table
WHERE description LIKE ":text%"?
Yes, there's a difference. When the WHERE condition calls a function on the column value, indexes can't be used. I don't think it will realize that your SUBSTRING() call happens to match the indexed part of the text and use that. On the other hand, LIKE is specifically coded to recognize the cases where it can use an index. Also, if you want to compare two strings for equality, you should use =, not STRCMP(), e.g.
WHERE SUBSTRING(`description`,1,20) = 'string_to_compare'
I believe it will make an efficient query for the LIKE version.
The placeholder can't be in quotes for it to work. Use CONCAT() to combine it: WHERE description LIKE CONCAT(:text, '%'). Or you can put the % at the end of the PHP variable that you bind to the placeholder, and use WHERE description LIKE :text.
I have strings like the following in my VARCHAR InnoDB table column:
"This is a {{aaaa->bbb->cccc}} and that is a {{dddd}}!"
Now, I'd like to search for e.g. {{xxx->yyy->zzz}}. Brackets are part of the string. Sometimes searched together with another colum, but which only contains an ordinary id and hence don't need to be considered (I guess).
I know I can use LIKE or REGEXP. But these (already tried) ways are too slow. Can I introduce a fulltext index? Or should I add another helping table? Should I replace the special characters {, }, -, > to get words for the fulltext search? Or what else could I do?
The search works with some ten-thousand rows and I assume that I often get about one hundred hits.
This link should give you all the info you need regarding FULLTEXT indexes in MySQL.
MySQL dev site
The section that you will want to pay particular attention to is:
"Full-text searching is performed using MATCH() ... AGAINST syntax. MATCH() takes a comma-separated list that names the columns to be searched. AGAINST takes a string to search for, and an optional modifier that indicates what type of search to perform. The search string must be a string value that is constant during query evaluation. This rules out, for example, a table column because that can differ for each row."
So in short, to answer your question you should see an improvement in query execution times by implementing a full text index on wide VARCHAR columns. Providing you are using a compatible storage engine ( InnoDB or MyISAM)
Also here is an example of how you can query the full text index and also an additional ID field as hinted in your question:
SELECT *
FROM table
WHERE MATCH (fieldlist) AGAINST ('search text here')
AND ( field2= '1234');
I have a one-word column with about 10000 rows. Searching for a certain word is ok but should be faster - right now I'm using
SELECT * FROM words WHERE word='hello'
This query takes 0.0004 secs to be exec.
I've created a FULLTEXT on the word column, but the time for the query to exec is exactly the same.
What would you do?
SELECT * FROM words WHERE MATCH (word) AGAINST ('hello')
For more info: http://dev.mysql.com/doc/refman/5.1/en/fulltext-search.html
There is no gain in a FULLTEXT index if it's only one world. That is used to index parts of larger texts so that they can be searched.
I assume you already have a normal index on word. You could experiment with setting the length of the index but it might not improve.
I'd like to make my own custom full text search and I am not sure what is the best way to make index table.
Ok, I take text field and extract all the words that are longer than 3 to the index table. But what do I have to store about them? word, ID of the table where I am searching? Anything else? Frequency of the word?
And support question: How do I split the text field to words, is there any mysql function or should I do this using server side language?
UPDATE: To make things clear: I don't need full text search just a wordlist of the words that are in all records of my text field, so I could search for the endings with LIKE 'word%'
If you are only going to implement what MySQL calls boolean mode (no relevance counting), you should implement the following basic functionality:
A wordbreaker, an algorithm that splits the strings into words. This is trivial in English but can be a problem for some Asian languages which do not use spaces between words.
Optionally, a stemmer, an algorithm which reduces words to their basic forms, so that went and gone both become go.
Optionally, a spellchecker, an algorithm which corrects the common spelling errors.
Optionally, a thesaurus, which reduces the synonyms to their common form.
A result of all this is that you have a string like this:
a fast oburn vixen jmups over an indolent canine
split into the basic forms of the words with the synonyms replaced and errors corrected:
quick
brown
fox
jump
over
lazy
dog
Then you just create a composite index on (word, rowid), where word is the basic form and rowid is the PRIMARY KEY of the record indexed.
To query for, say, '+quick +fox', you should search your index for these words and find an intersection on rowid. The intersecting rowid will contain both words.
If you are going to take relevance into account, you should additionally maintain a per-word statistics in a separate index over the whole corpus.
I should warn you that this is not a simple task. Just take a look at Sphinx source code.
Don't do it
Unless you know what you are doing forget about rolling your own full-text-search.
Let MySQL do the heavy lifting.
Use MyISAM for the table your want to search on
Put a FULLTEXT index on the text-fields you want to index.
Then do
SELECT *, MATCH(field1, field2) AGAINST 'text to search'
IN NATURAL LANGUAGE MODE WITH QUERY EXPANSION AS relevancy
FROM table1
WHERE MATCH(field1, field2) AGAINST 'text to search'
IN NATURAL LANGUAGE MODE WITH QUERY EXPANSION
ORDER BY relevancy
See: http://dev.mysql.com/doc/refman/5.5/en/fulltext-search.html#function_match
OP indicated that he want to search for word endings.
Where I live (Holland) we do poems in December, so I do a lot of word ending searching to find words that ryhme.
Here's my trick.
Add a new field to your table named visa_versa: varchar indexed
UPDATE mytable SET mytable.visa_versa = REVERSE(mytable.myword);
Now you can do an indexed search on word endings with
SET #ending = 'end';
SELECT myword FROM mytable where visa_versa LIKE REVERSE(CONCAT('%',#ending));