I have this column in my MySQL database. It is of type TINYTEXT. I fetch this column with a PHP script and displays it in an HTML. I find that all of my text has a trailing character in it. It shows up as a question mark with a diamond behind it in the browser.
I opened MySQL workbench and selected the column into the editor. I opened the value in the editor. The words seemed normal, except that they all have a trailing space behind it. I also took a look at the binary tab and the last group of letter/numbers is always 'a0'.
What could this be? I tried to trim the title, replace '\r' with '', replace '\n' with '', but nothing seem to work. How do I get rid of this? I also tried replacing '\xa0', though I don't know what this does (one of the solutions google turned up). But it does not work.
Any help would be appreciated.
Related
I have a SQL query that returns a value similar to "7 KI" for a description column, but when I export my data to a .csv/text file using SQLCMD, the resulting data is "7KI" (no space inbetween 7 and K). I believe that this causes a formatting issue when I open this csv file in Excel later on...(I'm wondering if "7K" is getting read as 7,000 because the data is splitting into a new column as-if it contained a comma there?)...
This is so weird to me. I am using the Quotename function to return the Unicode version of this description field in the first place - do you think that's why I am experiencing this issue? I used the Quotename function to fix issues with this field a few days ago. To me, it's odd that the export is stripping the space inbetween 7 and K in the first place.
Has anyone ever seen/fixed this before? It really looks like there is a space included in the SQL data, and I copied-and-pasted the value into a online Unicode converter to confirm there is a space. Why would a random space be getting stripped in the SQLCMD export? Plenty of my other text fields are not having this issue, even though they have spaces too (although not followed by "K"...) so maybe the issue is with CMD somehow reading the value...?
I am migrating an excel sheet (csv) to mysql, however when I do an insert, some fields end up with empty spaces at the end, and I cant get rid of them for some reason. So I assume there is a wierd character at the end, since not even this:
UPDATE FOO set FIELD2 = TRIM(Replace(Replace(Replace(FIELD2,'\t',''),'\n',''),'\r',''));
Gets rid of it completely, I still have a whitespace at the end and I dont know how to get rid of it. I have over 2000 entries, so doing it manually is not an option. I am using Laravel with the revision package and it doesnt work because it thinks that those spaces at the end are changes and it creates a bunch of duplicates. Thank you for your help.
If you think there are weird characters in the original csv, you could open it in a text processor capable of doing regex replaces, and then replace all non ascii characters with nothing.
Your regex would look like this:
[^\u0000-\u007F]+
then after removing any possible strange characters, re-import the data into the database.
Unfortunately, I don't think regex replaces are possible in sql, so you'll need to re-import.
I am going crazy. Please help.
I have a form with a text input. This is inserted into My SQL text column. Sometime a user will enter some unknown character that breaks the insert.
For example one that I just found is Topkapı, which is a town in Turkey. You will notice the last character ı. On insert, this causes a database error:
Error Executing Database Query. Incorrect string value: '/xC4/xB1
and...' for column 'country_description' at row 1
Is there a simple method to either remove these characters or escape them? I am using cfqueryparam and tried HTMLEditFormat, CFSavecontent etc to no avail.
EncodeForHTML() does not fix this particular issue if you are actually inserting HTML from TinyMCE for example.
What fixed this was changing the Collation to utf8mb4. You can do this in Workbench by expanding the header. It's collapsed by default.
Backup your table.
Go to "Alter Table".
Click the arrows on the top right of the windows
Select utf8mb4 from the Collation dropdown.
Click "Apply"
Here are your options in my opinion:
If you're using ColdFusion 10 or above, try using EncodeForHTML()
Validate your UI to accept only US and UK English characters, numbers etc.
Change the column data type in MySQL to VARCHAR(n) CHARSET utf8.
Hope this helps.
I am using JSP servlets and have a mysql database. I have an input field "Introduction". The error is when a user copy pastes a para from word then the character "(double quotes) is entered as ? in my table but only when the character is copied from a word or some other source. Also, if a user copies two paragraph's with spaces in between then a buggy character enters my sql table and the JS which is trying to load the introduction in my jsp page fails. i have also attached the screenshot for this. Please help me how can i resolve this.
MicroSoft, in its infinite wisdom, decided to have non-standard double quotes -- a left version and a right version. But that should be fixable, since those quotes do exist somewhere in the huge world of utf8 characters.
However, the data from your 'copy' was probably not copied in utf8 encoding. Since is is unclear how that is being done, we can't give you complete details on fixing it.
The "best" plan is to establish "utf8" at all stages of data/client/server/database/table/column/etc.
The quick-and-dirty fix is to replace the funny quotes with ascii quotes.
Somewhere along the way, between all the imports and exports I have done, a lot of the text on a blog I run is full of weird accented A characters.
When I export the data using mysqldump and load it into a text editor with the intention of using search-and-replace to clear out the bad characters, searching just matches every "a" character.
Does anyone know any way I can successfully hunt down these characters and get rid of them, either directly in MySQL or by using mysqldump and then reimporting the content?
This is an encoding problem; the  is a non-breaking space (HTML entity ) in Unicode being displayed in Latin1.
You might try something like this... first we check to make sure the matching is working:
SELECT * FROM some_table WHERE some_field LIKE BINARY '%Â%'
This should return any rows in some_table where some_field has a bad character. Assuming that works properly and you find the rows you're looking for, try this:
UPDATE some_table SET some_field = REPLACE( some_field, BINARY 'Â', '' )
And that should remove those characters (based on the page you linked, you don't really want an nbsp there as you would end up with three spaces in a row between sentences etc, you should only have one).
If it doesn't work then you'll need to look at the encoding and collation being used.
EDIT: Just added BINARY to the strings; this should hopefully make it work regardless of encoding.
The accepted answer did not work for me.
From here http://nicj.net/mysql-converting-an-incorrect-latin1-column-to-utf8/ I have found that the binary code for  character is c2a0 (by converting the column to VARBINARY and looking what it turns to).
Then here http://www.oneminuteinfo.com/2013/11/mysql-replace-non-ascii-characters.html found the actual solution to remove (replace) it:
update entry set english_translation = unhex(replace(hex(english_translation),'C2A0','20')) where entry_id = 4008;
The query above replaces it to a space, then a normal trim can be applied or simply replace to '' instead.
I have had this problem and it is annoying, but solvable. As well as  you may find you have a whole load of characters showing up in your data like these:
“
This is connected to encoding changes in the database, but so long as you do not have any of these characters in your database that you want to keep (e.g. if you are actually using a Euro symbol) then you can strip them out with a few MySQL commands as previously suggested.
In my case I had this problem with a Wordpress database that I had inherited, and I found a useful set of pre-formed queries that work for Wordpress here http://digwp.com/2011/07/clean-up-weird-characters-in-database/
It's also worth noting that one of the causes of the problem in the first place is opening a database in a text editor which might change the encoding in some way. So if you can possibly manipulate the database using MySQL only and not a text editor this will reduce the risk of causing further trouble.