I am migrating an excel sheet (csv) to mysql, however when I do an insert, some fields end up with empty spaces at the end, and I cant get rid of them for some reason. So I assume there is a wierd character at the end, since not even this:
UPDATE FOO set FIELD2 = TRIM(Replace(Replace(Replace(FIELD2,'\t',''),'\n',''),'\r',''));
Gets rid of it completely, I still have a whitespace at the end and I dont know how to get rid of it. I have over 2000 entries, so doing it manually is not an option. I am using Laravel with the revision package and it doesnt work because it thinks that those spaces at the end are changes and it creates a bunch of duplicates. Thank you for your help.
If you think there are weird characters in the original csv, you could open it in a text processor capable of doing regex replaces, and then replace all non ascii characters with nothing.
Your regex would look like this:
[^\u0000-\u007F]+
then after removing any possible strange characters, re-import the data into the database.
Unfortunately, I don't think regex replaces are possible in sql, so you'll need to re-import.
Related
I'm encountering the character %C2%92 in a bunch of MySQL records and looking to replace it with a single quote.
Something like this:
update
tablename
set field = replace(field, UNHEX('92'), "'")
where field RLIKE UNHEX('92')
Any ideas on how to actually accomplish this?
What comes right before/after the C292? Although that looks like a UTF-8 encoding, I don't recognize it.
You really should go back to where the data came from and avoid "url-encoding" the string. (cf PHP's urlencode()). If you are using PHP, then urldecode() will undo the mess: Fetch the row, field = urlencode(field);, then UPDATE the row in the database.
I have a SQL query that returns a value similar to "7 KI" for a description column, but when I export my data to a .csv/text file using SQLCMD, the resulting data is "7KI" (no space inbetween 7 and K). I believe that this causes a formatting issue when I open this csv file in Excel later on...(I'm wondering if "7K" is getting read as 7,000 because the data is splitting into a new column as-if it contained a comma there?)...
This is so weird to me. I am using the Quotename function to return the Unicode version of this description field in the first place - do you think that's why I am experiencing this issue? I used the Quotename function to fix issues with this field a few days ago. To me, it's odd that the export is stripping the space inbetween 7 and K in the first place.
Has anyone ever seen/fixed this before? It really looks like there is a space included in the SQL data, and I copied-and-pasted the value into a online Unicode converter to confirm there is a space. Why would a random space be getting stripped in the SQLCMD export? Plenty of my other text fields are not having this issue, even though they have spaces too (although not followed by "K"...) so maybe the issue is with CMD somehow reading the value...?
I converted a large amount of data from mssql to mysql and a few of them came in with the 'other' double quotes ... the sharper ones.
I would like to do a query in phpmyadmin for all the entries that have that symbol because its breaking my query (coming back as null) but cannot figure out how to write it ...
SELECT * FROM table where id LIKE '% %' <-- i dont actually know what the ascii symbol is for that one and this html ascii convention doesnt work anyway.
If you find one of them, can't you just copy it and paste it into your LIKE clause? That will prevent you from having to figure out the ASCII code for it.
I need something like this:
Attivato;Nome;Categorie;Prezzo tasse escluse;Descrizione;Immagini
1;"Bracciale rock";11,12,13;130;"This is a long description.";http://s20.postimg.org/r08w8i4i5/perle.jpg,http://s20.postimg.org/tmjtbp6bx/bracciale.jpg
But if I open it with neooffice calc (or anyway in some spreadsheet program) it then export like this, at the best:
Attivato;Nome;Categorie;Prezzo tasse escluse;Descrizione;Immagini
1;"Bracciale rock";"11,12,13";130;"This is a long description.";"http://s20.postimg.org/r08w8i4i5/perle.jpg,http://s20.postimg.org/tmjtbp6bx/bracciale.jpg"
It won't retain things like 11,12,13 without converting them to strings
How can I fix this?
I tried really everything but no way... Tried any kind of import/export options, different programs, etc... I cannot do it.
I finally found a couple of ways.
(1) In neooffice/openoffice:
left empty the text separator field
check the 'detect special numbers' option
(2) As another alternative, I used google docs. That's good also to use for clients who usually use exel. Google docs/drive saves csv files in utf-8 encoding by default.
Also, for the problem of "string conversion" of multiple values like 5,7,9,6, I found that if you use semicolons (;) instead of commas, that works (I mean, it doesn't add "" when you save as csv AND doesn't read them as dates or other wrong data types). And in prestashop you can set the field and test separators accordingly.
Hope it helps other people.
Somewhere along the way, between all the imports and exports I have done, a lot of the text on a blog I run is full of weird accented A characters.
When I export the data using mysqldump and load it into a text editor with the intention of using search-and-replace to clear out the bad characters, searching just matches every "a" character.
Does anyone know any way I can successfully hunt down these characters and get rid of them, either directly in MySQL or by using mysqldump and then reimporting the content?
This is an encoding problem; the  is a non-breaking space (HTML entity ) in Unicode being displayed in Latin1.
You might try something like this... first we check to make sure the matching is working:
SELECT * FROM some_table WHERE some_field LIKE BINARY '%Â%'
This should return any rows in some_table where some_field has a bad character. Assuming that works properly and you find the rows you're looking for, try this:
UPDATE some_table SET some_field = REPLACE( some_field, BINARY 'Â', '' )
And that should remove those characters (based on the page you linked, you don't really want an nbsp there as you would end up with three spaces in a row between sentences etc, you should only have one).
If it doesn't work then you'll need to look at the encoding and collation being used.
EDIT: Just added BINARY to the strings; this should hopefully make it work regardless of encoding.
The accepted answer did not work for me.
From here http://nicj.net/mysql-converting-an-incorrect-latin1-column-to-utf8/ I have found that the binary code for  character is c2a0 (by converting the column to VARBINARY and looking what it turns to).
Then here http://www.oneminuteinfo.com/2013/11/mysql-replace-non-ascii-characters.html found the actual solution to remove (replace) it:
update entry set english_translation = unhex(replace(hex(english_translation),'C2A0','20')) where entry_id = 4008;
The query above replaces it to a space, then a normal trim can be applied or simply replace to '' instead.
I have had this problem and it is annoying, but solvable. As well as  you may find you have a whole load of characters showing up in your data like these:
“
This is connected to encoding changes in the database, but so long as you do not have any of these characters in your database that you want to keep (e.g. if you are actually using a Euro symbol) then you can strip them out with a few MySQL commands as previously suggested.
In my case I had this problem with a Wordpress database that I had inherited, and I found a useful set of pre-formed queries that work for Wordpress here http://digwp.com/2011/07/clean-up-weird-characters-in-database/
It's also worth noting that one of the causes of the problem in the first place is opening a database in a text editor which might change the encoding in some way. So if you can possibly manipulate the database using MySQL only and not a text editor this will reduce the risk of causing further trouble.