Completely new to mongo here.
I have following fields in on of my mysql tables:
id (BIGINT), text (LONGTEXT) #this would contain a long description
I am hoping to change my project from Mysql to MongoDB, but before I do that there is very crucial query that needs to be resolved.
My current query looks for various terms in the description and returns all ids, e.g.
select id from <table> where instr(<table>.text, 'value') or instr(<table>.text, 'value2)
Is it possible for this to be recreated in Mongo? if so how? right now using either the $or or $in seems that I need to have those specific values in some kind of an array in my document.
MongoDB does not natively support full text search at the moment.
You could use regular expressions but it would be slow (due to not using indexes unless they are rooted).
Query would be like:
db.collection.find({ $or: [{description: /value1/}, {description: /value2/}] })
You could do some preprocessing to insert each word into a searchable array of keywords but if the text is really long you probably don't want to go this route.
Related
I have a table dictionary which contains a list of words Like:
ID|word
---------
1|hello
2|google
3|similar
...
so i want if somebody writes a text like
"helo iam looking for simlar engines for gogle".
Now I want to check every word if it exists in the database, if not it should
get me the similar word for the word. For example: helo = hello, simlar = similar, gogle = google.
Well, i want to fix the spelling errors. In my database i have a full dictionary of all english words. I coudn't find any mysql function which helps me. LIKE isn't helpfull in my situation.
you can use soundex() function for comparing phonetically
your query should be something like:
select * from table where soundex(word) like soundex('helo');
and this will return you the hello row
There is a function that does roughly want you want, but it's intensive and will slow queries down. You might be able to use in your circumstances, I have used it before. It's called Levenshtein. You can get it here How to add levenshtein function in mysql?
What you want to do is called a fuzzy search. You could use the SOUNDEX function in MySQL, documented here:
http://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_soundex
You query would look like:
SELECT * FROM dictionary where SOUNDEX(word) = SOUNDEX(:yourSearchTerm)
... where your search term is bound to the :yourSearchTerm parameter value.
A next step would be to try implementing and making use of a Levenshtein function in MySQL. One is described here:
http://www.artfulsoftware.com/infotree/qrytip.php?id=552
The Levenshtein distance between two strings is the minimum number of
operations needed to transform one string into the other, where an
operation may be insertion, deletion or substitution of one character.
You might also consider looking into databases that are aimed at full text searching, such as Elastic Search, which provides this natively:
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-fuzzy-query.html
I have an JSON string stored in the database and I need to SQL COUNT based on the WHERE condition that is in the JSON string. I need it to work on the MYSQL 5.5.
The only solution that I found and could work is to use the REGEXP function in the SQL query.
Here is my JSON string stored in the custom_data column:
{"language_display":["1","2","3"],"quantity":1500,"meta_display:":["1","2","3"]}
https://regex101.com/r/G8gfzj/1
I now need to create a SQL sentence:
SELECT COUNT(..) WHERE custom_data REGEXP '[HELP_HERE]'
The condition that I look for is that the language_display has to be either 1, 2 or 3... or whatever value I will define when I create the SQL sentence.
So far I came here with the REGEX expression, but it does not work:
(?:\"language_display\":\[(?:"1")\])
Where 1 is replaced with the value that I look for. I could in general look also for "1" (with quotes), but it will also be found in the meta_display array, that will have different values.
I am not good with REGEX! Any suggestions?
I used the following regex to get matches on your test string
\"language_display\":\[(:?\"[0-9]\"\,)*?\"3\"(:?\,\"[0-9]\")*?\]
https://regex101.com/ is a free online regex tester, it seems to work great. Start small and work big.
Sorry it doesn't work for you. It must be failing on the non greedy '*?' perhaps try without the '?'
Have a look at how to serialize this data, with an eye to serializing the language display fields.
How to store a list in a column of a database table
Even if you were to get your idea working it will be slow as fvck. Better off to process through each row once and generate something more easily searched via sql. Even a field containing the comma separated list would be better.
I have a table dictionary which contains a list of words Like:
ID|word
---------
1|hello
2|google
3|similar
...
so i want if somebody writes a text like
"helo iam looking for simlar engines for gogle".
Now I want to check every word if it exists in the database, if not it should
get me the similar word for the word. For example: helo = hello, simlar = similar, gogle = google.
Well, i want to fix the spelling errors. In my database i have a full dictionary of all english words. I coudn't find any mysql function which helps me. LIKE isn't helpfull in my situation.
you can use soundex() function for comparing phonetically
your query should be something like:
select * from table where soundex(word) like soundex('helo');
and this will return you the hello row
There is a function that does roughly want you want, but it's intensive and will slow queries down. You might be able to use in your circumstances, I have used it before. It's called Levenshtein. You can get it here How to add levenshtein function in mysql?
What you want to do is called a fuzzy search. You could use the SOUNDEX function in MySQL, documented here:
http://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_soundex
You query would look like:
SELECT * FROM dictionary where SOUNDEX(word) = SOUNDEX(:yourSearchTerm)
... where your search term is bound to the :yourSearchTerm parameter value.
A next step would be to try implementing and making use of a Levenshtein function in MySQL. One is described here:
http://www.artfulsoftware.com/infotree/qrytip.php?id=552
The Levenshtein distance between two strings is the minimum number of
operations needed to transform one string into the other, where an
operation may be insertion, deletion or substitution of one character.
You might also consider looking into databases that are aimed at full text searching, such as Elastic Search, which provides this natively:
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-fuzzy-query.html
Situation:
I have user model. attribute "meta_data" in db represents "text" type field.
In model it seriazized by custom class. ( serialize :meta_data, CustomJsonSerializer.new )
It means, when I have an instance of user, I can work with meta_data like with Hash.
User.first.meta_data['username']
Problem:
I need to write a search function, which will search users by given string. I can do it by manual building search query in rails ex. User.where("email LIKE '%#{string}%'")...
But what about meta_data ? Should I search in this field by LIKE statement too? If I will do so, it will decrease relevance of found record.
For example:
I have 2 users. One of them has username "patrick", another one is "sergio"
meta data in db will look like this:
1) {username: patrick}
2) {username: sergio}
I want to find sergio , I enter a search string "ser" => but I have 2 results, instead of one. This meta_data string "{uSERname: Patrick}" also has "ser", so it makes this record irrelevant.
Do you have any idea how to solve it?
That's really the problem with serialized data. In theory, the serialization could be an algorithm that is very unsearchable. It could do a Hoffman encoding, or other compression, and store the serialization in binary. You are relying on the assumption that the serialization uses JSON and your string will still be findable as a sub-string in the serialization.
Then the problem you are having is another issue. Other data in the serialization can mess up your results.
In general, if you serialize data, you are making a choice to not be searchable.
So a solution would be to add an additional field that you populate in a way that you control. Have a values field and store a pipe (|) delimited value that you can search. So if the data is {firstname: "Patrick", lastname: "Stern"}, your meta_values field might be "Patrick|Stern".
Also, don't use the where method with a string with #{} expansion of input values. The makes it vulnerable to SQL attacks. Instead use:
where("meta_values is like :pattern", pattern: "%#{string}%")
I know that may not look very different, but ActiveRecord will go through a sanitizing this way. If someone has a semi-colon in string, then ActiveRecord will escape the semi-colon in the search condition.
This question already has an answer here:
Closed 11 years ago.
Possible Duplicate:
Simulating regex capture groups in mysql
Good day,
I have many rows of data stored in a MySQL table. A typical value could look something like this:
::image-gallery::
::gallery-entry::images/01.jpg::/gallery-entry::
::/image-gallery::
Is there a way - by means of a regular expression that I can a) extract the term image gallery from the first line (it could be any phrase, not just image-gallery) and then extract the center line as two separate values like this:
gallery-entry and then images/01.jpg
There could be many lines of ::gallery-entry:: values, and they could be called anything as well. A more complete example would be:
::image-gallery::
::title::MY GALLERY::/title::
::date::2011-05-20::/date::
::gallery-entry::images/01.jpg::/gallery-entry::
::/image-gallery::
In essence I want this information: The content type (image-gallery) in the above case, first line and last line. Then I need the title as a key value style pair, so title as the key and MY GALLERY as the value. Then, subsequently, I would need all the rows of fields thereafter (gallery-entry) as key value pairs too.
This is for a migration script where data from an old system will be migrated over to a new system with different syntax.
If MySQL select statements would not work, would it be easier to parse the results with a PHP script for data extraction?
Any and all help is always appreciated.
Kind regards,
Simon
Try this regex:
::image-gallery::\s+::title::(.*?)::/title::.*?::gallery-entry::(.*?)::/gallery-entry::\s+::/image-gallery::
Use single-line mode (/pattern/s) so the .*? chews up newlines.
Your key-value pairs will be:
title: $1 (matching group 1)
gallery-entry: $2 (matching group 2)
From simulating-regex-capture-groups-in-mysql there does not seem to be a way to easily capture groups with a regex in mysql. The reason is that MySQL does not natively support capture groups in a regex. If you want that functionality you can use a server side extension like lib_mysqludf_preg to add that capability to MySQL.
The easiest way is to extract the whole column with SQL and then do the text matching in another language (such as php).
In my tests kenbritton's regex didn't work, but building off of it the following regex worked on your test data:
::image-gallery::\s+::title::(.*?)::\/title::\s+(?:.*\s+)*::gallery-entry::(.*?)::\/gallery-entry::\s+::\/image-gallery::