How can I recognize when a user has missed a space when entering a search term? For example, if the user enters "usbcable", I want to search for "usb cable". I'm doing a REGEX search in MySQL to match full words.
I have a table with every term used in a search, so I know that "usb" and "cable" are valid terms. Is there a way to construct a WHERE clause that will give me all the rows where the term matches part of the string?
Something like this:
SELECT st.term
FROM SearchTerms st
WHERE 'usbcable' LIKE '%' + st.term + '%'
Or any other ideas?
Text Segmentation is a part of Natural Language Processing, and is what you're looking for in this specific example. It's used in search engines and spell checkers, so you might have some luck with example source code looking at open source spell checkers and search engines.
Spell checking might be the correct paradigm to consider anyway, as you first need to know whether it's a legitimate word or not before trying to pry it apart.
-Adam
Posted in the comments, but I thought it important to bring up as an answer:
Does that query not work? – Simon Buchan
Followed by:
Well, I should've tested it before I
posted. That query does not work, but
with a CONCAT it does, like so: WHERE
'usbcable' LIKE Concat('%', st.term,
'%'). I think this is the simplest,
and most relevant (specific to my
site), way to go. – arnie0674
Certainly far easier than text segmentation for this application...
-Adam
Related
I am searching in my Eclipse project for id="* *" in order to find something like below in all files of my project
id="Text1 Text2"
Since space between attribute value needs to be removed I am searching like this to find out all the instances in my project.
But it searches and brings up lots of results since * represent any string.
For example:
id="xyzImg" onLoad="test();" SRC="abc.png"
It taking the complete element instead of desired results.
Please suggest me is there any way to get my desired result or is there any online tool that serve the same purpose.
I am going to write it here as an answer instead. Try with *|*. By asterisk I meant the wildcard symbol.
Lets say an internet user searches for "trouble with gmail".
How can I return entries with "problem|problems|issues|issue|trouble|troubles with gmail|googlemail|google mail"?
I don't like to manually add these linkings between different keywords so the links between "issue <> problem <> trouble" and "gmail <> googlemail <> google mail" are completly unknown. They should be found in an automated process.
Approach to solve the problem
I provide a synonyms/thesaurus plattform like thesaurus.com, synonym.com, etc. or use an synomys database/api and use this user generated input for my queries on a third website.
But this won't cover all synonyms like the "gmail"-example.
Which other options do I have? Maybe something based on the given data and logged search phrases of the past?
You have to think of it ignoring the language.
When you show a baby the same thing using two words, he understand that those words are synonym. He might not have understood perfectly, but he will learn when this is repeated.
You type "problem with gmail".
Two choices:
Your search give results: you click on one item.
The system identify that this item was already clicked before when searching for "google mail bug". That's a match, and we will call it a "relative search".
Your search give poor results:
We will search in our history for a matching search:
We propose : "do you mean trouble with yahoo mail? yes/no". You click no, that's a "no match". And we might propose others suggestions like a list of known "relative search" or a list of might be related playing with both full text search in our history and levenshtein distance.
When a term is sufficiently scored to be considered as a "synonym", you can consider it is. Algorithm might be wrong, but in fact it depends on what you really expect.
If i search "sending a message is difficult with google", and "gmail issue", nothing is synonym, but search are relatively the same. This is more important to me than true synonyms.
And if you really want to get the synonym, i would do it in a second phase comparing words inside "relative searches" and would include a manual check.
I think google algorithm use synonym mainly to highlight search terms in page result, but not to do an actual search where they use the relative search terms, except in known situations, as the result for "gmail" and "google mail" are not the same.
But if you identify 10 relative searches for "gmail" which all contains "google mail", that will be a good start point to guess they are synonyms.
This is a bit long for a comment.
What you are looking for is called a "thesaurus" or "synonyms" list in the world of text searching. Apparently, there is a proposal for such functionality in MySQL. It is not yet implemented. (Here is a related question on Stack Overflow, although the link in the question doesn't seem to work.)
The work-around would be to modify queries before sending them to the database. That is, parse the query into words, then look up all the synonyms for those words, and reconstruct the query. This works better for the natural language searches than the boolean searches (which require more careful reconstruction).
Pseudo-code for getting the final word list with synonyms would be something like:
select #finalwords = concat_ws(' ', group_concat(synonyms separator ' ') )
from synonyms s
where find_in_set(s.baseword, #words) > 0;
Seems to me that you have two problems on your hands:
Lemmatisation, which breaks words down into their lemma, sometimes called the headword or root word. This is more difficult than Stemming, as it doesn't just chop suffixes off of words, but tries to find a true root, e.g. "are" => "be". This is something that is often done programatically, although it appears to be a complex task. Here is an online example of text being lemmatized: http://lemmatise.ijs.si/Services
Searching for synonymous lemmas. This is a very complex problem. One approach to this that I have heard of is modifying the lemmatisation engine to return more than one lemma for a given set of words, i.e. "problems" => "problem" and "issue", thereby allowing a more flexible set of results. However, this means that the synonymous lemmas must be provided to the lemmatisation engine from elsewhere. I truly have no idea how you would build a list of synonyms programatically.
So, you may consider a strategy whereby you lemmatise the text to be searched for, then pass each lemma out to your synonym finder (however that works) to get a final list of lemmas to perform your search with.
I think you have bitten off a very large problem for yourself.
If the system in question is a publicly accessible website, one 'out there' option is to ensure all content can be crawled by Google and then use a Google search on your own site, which should give you the synonym capability 'for free'. There would obviously be some vagaries in the results though and lag in getting match results for newly created content, depending upon how regularly the crawlers hit the site. Probably not suitable in your use case, but for some people, this may be sufficient.
Seeing your revised question, what about using a public API?
http://www.programmableweb.com/category/reference/apis?category=20066&keyword=synonym
I am currently working on a form and am stuck on a keyword/tag input field (think youtube.. or even stackoverflow). Anyway, I thought it was pretty logical to use ',' to separate the tags... which would allow users to use combinations of words as tags using ' '. However my boss wants it separated with just ' '. Which worries me as I think we will end up with millions of 'The' tags. Personally I like the tag system used here on stackoverflow... but its not up to me.
So far my only idea was to have a list of common words that would automatically be removed.... problem being is its an international site, so not much good making a list of English words.
Any ideas?
As the user is typing in the keywords show a preview of the matched keywords right below. This might help clarify to the user that each word separated by spaces is a distinct keyword and that duplicates are eliminated.
Enter keywords: [[[ the house the boat the cat the coat ]]]
Keyword preview: [the] [house] [boat] [cat] [coat]
*** Warning: Duplicate keywords found. To combine multiple words use a "-"
If you really can't convince your boss otherwise, I can only think of your dictionary idea coupled with scheduled maintenance. So, if you notice a word repeatedly being used, such as an equivalent of 'the', then you can blacklist it and remove it from current tags. Also, you could make it clear that words such as 'the' are not required and other words should be hyphenated (as is on SO).
When I'm programming, I often find myself writing functions that -should- (to be proper english) contain apostrophes (too bad C started everyone thinking that an apostrophe was an appropriate delimiter). For example: get_user's_group() -> get_users_group() . What do you guys do with that forced-bad-english ambiguous english? Just ignore the apostrophe? Create a different phrasing?
In that case, I would do get_group_for_user().
So, yes, I would "create a different phrasing" :)
Either that, or user.get_group().
getGroupForUser()
or
getGroupByUser()
My original answer of Ignore it, move on! is incomplete. You should ignore the fact you can't use ' in your method/function names. But you should continue to look at the naming of them to better explain what they do. I think this is a worthwhile pursuit in programming.
Picking on JavaScript, you could if you wanted to use apostrophes:
const user = {
"get_user's_group": () => console.log("Naming things! Am I right?!")
}
user["get_user's_group"]()
But don't do that 😬
Taking it further, you could if you wanted to, use a transpiler to take your grammatically correct name and transform it into something you never see.
Again with JavaScript as an example, maybe you could write a babel transform.
But don't do that 😛
As others have said, if there is context available from an object, that's a nice option:
user.get_group()
Failing that, the context of the surrounding code should be enough to make this your choice:
get_users_group()
How about getGroupByUser?
Either get_user_ApostropheShouldBeHereButLanguageWillNotLetMe_s_group or just ignore it because it really doesn't matter.
I ignore the apostraphe getGroupyUser and group_from_user are both perfectly understandable. Worrying about having correct grammer in your function names is a waste of time and distracts from the correct goal of having clear and understandable user names.
the point of proper english in function naming is a bit extreme ...
i mean why is the apostrophe bothering you but the _ instead of a space is not ?
Depending on the programming language you may be able to use Unicode variable names, this SO thread lists a few.
With Unicode identifiers you could use one of the unicode apostrophes to give the proper english language formatting to your variable name. Though this only speculative. And it would be hard to maintain. Actually, now that I think about it, it sounds downright evil.
Two points: First, don't use a name that would otherwise require an apostrophe if you can avoid it. Second, you are right in being concerned about ambiguity. For example, you could have:
getUsersGroup: gets the group of a list of users. If you are using an object-oriented language, this could have more information than just a group ID string. You could also have something like createUsersGroup, which would create a group object from a list of users passed in.
getGroupOfUser: takes in some sort of user object; returns the name of the group of the user
getGroupByUserId: takes in the user's name or a unique ID associated with that user; returns the name of the group of the user
The best way to delineate the difference between all of these is to just use standard method comments that explain the method names. This would depend on what language you are working with and what style of method comments your organization conventionally uses.
Normally I just drop the apostrophe, but do back-ticks work? (get_user`s_group)
getGroupOfUser? getUserGroup?
It's a programming language, not literature...
It would be getBackgroundColour in proper English (rather than getBackgroundColor)
Personally I'd write get_user_group() rather than get_group_for_user() since it feels like it reads better to me. Of course, I use a programming language where apostrophes are allowed in names:
proc get_user's_group {id} {#...}
Although, some of the more prolific non-English-native European users use it as a word separator:
proc user'group {id} {#...}
to each his own I guess..
Typically languages have keywords that you are unable to use directly with the exact same spelling and case for naming things (variables,functions,classes ...) in your program. Yet sometimes a keyword is the only natural choice for naming something. What is your system for avoiding/getting around this clash in your chosen technology?
I just avoid the name, usually. Either find a different name or change it slightly - e.g. clazz instead of class in C# or Java. In C# you can use the # prefix, but it's horrible:
int #int = 5; // Ick!
There is nothing intrinsically all-encompassing about a keyword, in that it should stop you from being able to name your variables. Since all names are just generalized instances of some type to one degree or another, you can always go up or down in the abstraction to find another useful name.
For example, if your writing a system that tracks students and you want an object to represent their study in a specific field, i.e. they've taken a "class" in something, if you can't use the term directly, or the plural "classes", or an alternative like "studies", you might find a more "instanced" variation: studentClass, currentClass, etc. or a higher perspective: "courses", "courseClass" or a specfic type attribute: dailyClass, nightClass, etc.
Lots of options, you should just prefer the simplest and most obvious one, that's all.
I always like to listen to the users talk, because the scope of their language helps define the scope of the problem, often if you listen long enough you'll find they have many multiple terms for the same underlying things (with only subtle differences). They usually have the answer ...
Paul.
My system is don't use keywords period!
If I have a function/variable/class and it only seems logical to name it with a keyword, I'll use a descriptive word in front of the keyword.
(adjectiveNoun) format. ie: personName instead of Name where "Name" is a keyword.
I just use a more descriptive name. For instance, 'id' becomes identifier, 'string' becomes 'descriptionString,' and so on.
In Python I usually use proper namespacing on my modules to avoid name clashes.
import re
re.compile()
instead of:
from re import *
compile()
Sometimes, when I can't avoid keyword name clashes I simply drop the last letter off the name of my variable.
for fil in files:
pass
As stated before either change class to clazz in Java/C#, or use some underscore as a prefix, for example
int _int = 0;
There should be no reason to use keywords as variable names. Either use a more detailed word or use a thesaraus. Capitalizing certain letters of the word to make it not exactly like the keyword is not going to help much to someone inheriting your code later.
Happy those with a language without ANY keywords...
But joke apart, I think in the seldom situations where "Yet sometimes a keyword is the only natural choice for naming something." you can get along by prefixing it with "my", "do", "_" or similar.
I honestly can't really think of many such instances where the keyword alone makes a good name ("int", "for" and "if" are definitely bad anyway). The only few in the C-language family which might make sense are "continue" (make it "doContinue"), "break" (how about "breakWhenEOFIsreached" or similar ?) and the already mentioned "class" (how about "classOfThingy" ?).
In other words: make the names more reasonable.
And always remember: code is WRITTEN only once, but usualy READ very often.
Typically I follow Hungarian Notation. So if, for whatever reason, I wanted to use 'End' as a variable of type integer I would declare it as 'iEnd'. A string would be 'strEnd', etc. This usually gives me some room as far as variables go.
If I'm working on a particular personal project that other people will only ever look at to see what I did, for example, when making an add-on to a game using the UnrealEngine, I might use my initials somewhere in the name. 'DS_iEnd' perhaps.
I write my own [vim] syntax highlighters for each language, and I give all keywords an obvious colour so that I notice them when I'm coding. Languages like PHP and Perl use $ for variables, making it a non-issue.
Developing in Ruby on Rails I sometime look up this list of reserved words.
In 15 years of programming, I've rarely had this problem.
One place I can immediately think of, is perhaps a css class, and in that case, I'd use a more descriptive name. So instead of 'class', I might use 'targetClass' or something similar.
In python the generally accepted method is to append an '_'
class -> class_
or -> or_
and -> and_
you can see this exemplified in the operator module.
I switched to a language which doesn't restrict identifier names at all.
First of all, most code conventions prevent such a thing from happening.
If not, I usually add a descriptive prose prefix or suffix:
the_class or theClass infix_or (prefix_or(class_param, in_class) , a_class) or_postfix
A practice, that is usually in keeping with every code style advice you can find ("long names don't kill", "Longer variable names don't take up more space in memory, I promise.")
Generally, if you think the keyword is the best description, a slightly worse one would be better.
Note that, by the very premise of your question you introduce ambiguity, which is bad for the reader, be it a compiler or human. Even if it is a custom to use class, clazz or klass and even if that custom is not so custom that it is a custom: it takes a word word, precisely descriptive as word may be, and distorts it, effectively shooting w0rd's precision in the "wrd". Somebody used to another w_Rd convention or language might have a few harsh wordz for your wolds.
Most of us have more to say about things than "Flower", "House" or "Car", so there's usually more to say about typeNames, decoratees, class_params, BaseClasses and typeReferences.
This is where my personal code obfuscation tolerance ends:
Never(!!!) rely on scoping or arcane syntax rules to prevent name clashes with "key words". (Don't know any compiler that would allow that, but, these days, you never know...).
Try that and someone will w**d you in the wörd so __rd, Word will look like TeX to you!
My system in Java is to capitalize the second letter of the word, so for example:
int dEfault;
boolean tRansient;
Class cLass;