I am currently improving the search functionality of my cms so that users can search for entries by copying and pasting text from a web page and finding it in the database.
The query is simple. It takes the search term and does a LIKE '% text here %' query.
The problem is, I'm not getting many results and have figured out why.
In the CMS itself, a lot of the text that has been entered from MS Word seems to be double spaced. Such as
"Hello my name is James"
However on the front end website it renders properly, with single spaces, like:
"Hello my name is James"
This means my query is never picking up the database entry based on what is shown on the web page.
Any suggestions? Do I tackle the double spaces in the CMS (seems risky to me with so much HTML in there!), or can I adjust my query to cope with it?
if it is only double spaces that are creating the issue, then just
replace(columnToSearch,' ',' ')
when searching, or as #ManMohan suggests, before inserting the data into your table in the first place
Take a look at MySQL REPLACE
REPLACE(str,from_str,to_str)
Returns the string str with all occurrences of the string from_str replaced by the string to_str.
REPLACE() performs a case-sensitive match when searching for from_str.
Related
In the pages I've checked, they all return the same thing, but the Mediawiki documentation says there are differences.
I'm not worried about the differences, but which one is actually stored in the page table?
Neither of them. The internal representation ("DB key form") is title without namespace (it's stored separately as a number in page_namespace), spaces replaced with underscores. The code is here. Thus it's neither {{PAGENAME}} which is human-readable title, nor {{PAGENAMEE}} which is {{#urlencode:{{PAGENAME}}}} with special case for spaces -> underscores.
Got it. I saved the page "Texas A & M" and in the page table it shows as "Texas_A_&_M".
According to Mediawiki's Manual:PAGENAMEE_encoding page (I can't post more than two links), PAGENAME is the only one that will convert an ampersand to & while the others convert it to %26.
The following is still not correct!
I thought it was PAGENAME, but PAGENAME actually doesn't replace the spaces with underscores.
Instead, I found here and here that you can access the string that is stored in the Page table by using this:
$dbk = $title->getDBkey();
That snippet is pulled straight from Mediawiki code.
It doesn't appear there is a Magic Word associated with this key.
I can't find where the page_title in the database comes from, but it looks like it's simply the page name with the spaces, quotes, and ampersand replaced. Maybe it's database dependent. I'm using MySQL.
I am trying to isolate an email address from a block of free field text (column name is TEXT).
There are many different variations of preceding and succeeding characters in the free text field, i.e.:
email me! john#smith.com
e:john#smith.com m:555-555-5555
john#smith.com--personal email
I've tried variations of INSTR() and SUBSTRING_INDEX() to first isolate the "#" (probably the one reliable constant in finding an email...) and extracting the characters to the left (up until a space or non-qualifying character like "-" or ":") and doing the same thing with the text following the #.
However - everything I've tried so far hasn't filtered out the noise to the level I need.
Obviously 100% accuracy isn't possible but would someone mind taking a crack at how I can structure my select statement?
There is no easy solution to do this within MySQL. However you can do this easily after you have retrieved it using regular expressions.
Here would be a an example of how to use it in your case: Regex example
If you want it to select all e-mail addresses from one string: Regex Example
You can use regex to extract the ones where it does contain an e-mail in MySQL but it still doesn't extract the group from the string. This has to be done outside MySQL
SELECT * FROM table
WHERE column RLIKE '\w*#\w*.\w*'
RLIKE is only for matching it, you can use REGEXP in the SELECT but it only returns 1 or 0 on whether it has found a match or not :s
If you do want to extract it in MySQL maybe this other stackoverflow post helps you out. But it seems like a lot of work instead of doing it outside MySQL
Now in MySQL 5 and 8 you can use REGEXP_SUBSTR to isolate just the email from a block of free text.
SELECT *, REGEXP_SUBSTR(`TEXT`, '([a-zA-Z0-9._%+\-]+)#([a-zA-Z0-9.-]+)\.([a-zA-Z]{2,4})') AS Emails FROM `mytable`;
If you want to get just the records with emails and remove duplicates ...
SELECT DISTINCT REGEXP_SUBSTR(`TEXT`, '([a-zA-Z0-9._%+\-]+)#([a-zA-Z0-9.-]+)\.([a-zA-Z]{2,4})') AS Emails FROM `mytable` WHERE `TEXT` REGEXP '([a-zA-Z0-9._%+\-]+)#([a-zA-Z0-9.-]+)\.([a-zA-Z]{2,4})';
I have a CSV file, which has two columns and 4500 rows. In one column, I have several phrases that are surrounded in quotation marks. I need to delete all the text that comes before and after the quotations marks.
For example:
How would you say "Hello, my Friend" when speaking outside?
should become "Hello, my Friend"
I also have several rows that have the word NULL in the second column. I need these rows deleted in full.
What's the best way of doing something like this? I have been looking at regular expressions, but I'm not sure if they are flexible enough to do what I want to do, or how you would use them on a CSV file (I need the table structure to remain).
EDIT:
1) At the moment I am just using Apple Numbers, but I know that wont don't it, so I am happy to any suggestions. It must support Kanji characters.
2) I have removed all the NULL rows, so that is no longer needed (I simply added a column of numbers, sorted the table so all the NULLs were together, deleted them and the sorted back by the column of numbers).
Find a text editor that supports regular expression search and replace.
Something like this would match ,NULL in the second column: ^.*,NULL.*$. Replace it with "DELETEMEDELETEME" to mark the line, or as an empty string or find a way to have it match on `\n' or '\r' to catch the line break and remove the entire line completely.
Stripping out parts of the quoted string might work like this:
^(.*,){n}(.*)(\".\")(.*)(,.*)$ replaced with \1\3\5 where n is the number of columns preceding the one you want to edit. Repeat (.*,) if that's not available. It will depend on the regex flavor of your tool.
Somewhere along the way, between all the imports and exports I have done, a lot of the text on a blog I run is full of weird accented A characters.
When I export the data using mysqldump and load it into a text editor with the intention of using search-and-replace to clear out the bad characters, searching just matches every "a" character.
Does anyone know any way I can successfully hunt down these characters and get rid of them, either directly in MySQL or by using mysqldump and then reimporting the content?
This is an encoding problem; the  is a non-breaking space (HTML entity ) in Unicode being displayed in Latin1.
You might try something like this... first we check to make sure the matching is working:
SELECT * FROM some_table WHERE some_field LIKE BINARY '%Â%'
This should return any rows in some_table where some_field has a bad character. Assuming that works properly and you find the rows you're looking for, try this:
UPDATE some_table SET some_field = REPLACE( some_field, BINARY 'Â', '' )
And that should remove those characters (based on the page you linked, you don't really want an nbsp there as you would end up with three spaces in a row between sentences etc, you should only have one).
If it doesn't work then you'll need to look at the encoding and collation being used.
EDIT: Just added BINARY to the strings; this should hopefully make it work regardless of encoding.
The accepted answer did not work for me.
From here http://nicj.net/mysql-converting-an-incorrect-latin1-column-to-utf8/ I have found that the binary code for  character is c2a0 (by converting the column to VARBINARY and looking what it turns to).
Then here http://www.oneminuteinfo.com/2013/11/mysql-replace-non-ascii-characters.html found the actual solution to remove (replace) it:
update entry set english_translation = unhex(replace(hex(english_translation),'C2A0','20')) where entry_id = 4008;
The query above replaces it to a space, then a normal trim can be applied or simply replace to '' instead.
I have had this problem and it is annoying, but solvable. As well as  you may find you have a whole load of characters showing up in your data like these:
“
This is connected to encoding changes in the database, but so long as you do not have any of these characters in your database that you want to keep (e.g. if you are actually using a Euro symbol) then you can strip them out with a few MySQL commands as previously suggested.
In my case I had this problem with a Wordpress database that I had inherited, and I found a useful set of pre-formed queries that work for Wordpress here http://digwp.com/2011/07/clean-up-weird-characters-in-database/
It's also worth noting that one of the causes of the problem in the first place is opening a database in a text editor which might change the encoding in some way. So if you can possibly manipulate the database using MySQL only and not a text editor this will reduce the risk of causing further trouble.
I have a database of phrases that users will search for from their own input. I want them to find the phrase regardless of what punctuation they use. For example if the phrase, "Hey, how are you?" is in the row, I want all of the following searches to return it:
"Hey! How are you?!"
"Hey how are you?"
"Hey :) How are you?"
Right now, I have the columns 'phrase' and 'phrase_search'. Phrase search is a stripped down version of phrase so our example would be 'hey-how-are-you'.
Is there anyway to achieve this without storing the phrase twice?
Thank you!
-Nicky
What you've done is probably the most time-efficient way of doing it. Yes, it requires double the space, but is that an issue?
If it is an issue, a possible solution would be to convert your search string to use wildcards (eg. %Hey%how%are%you%) and then filter the SQL results in your code by applying the same stripping function to the database input and the search string and comparing them. The rationale behind this is that there should be relatively few matches with non-punctuation characters in-between the words, so you're still getting MySQL to do the "heavy lifting" while your PHP/Perl/Python/whatever code can do a more fine-grained check on a relatively small number of rows.
(This assumes that you have some code calling this, rather than a user typing the SQL query from the command line, of course.)