I'm having issues with the encoding of the the '£' character. (Issue: When POSTing '£' from a form field and doing an insert, nothing is inserted in the MySQL table). I've checked everything wrt to UTF-8 support on my PHP code headers, server, collation/char set on MySQL etc.
I'm using MAMP as my dev environment (PHP 5.3.5).
Everything works fine on my production server (commercial host) (PHP 5.2.6) so I've ruled out any issues with my code
However, I think I have tracked down the culprit: When comparing both environments, this line is missing from my dev server:
_ENV["HTTP_ACCEPT_CHARSET"] ISO-8859-1,utf-8;q=0.7,*;q=0.3
However, there is nothing in php.ini I can see to change it. Any ideas, or am I barking up the wrong tree?
Cheers
Roland
I'd write a simple test to check out where things are going wrong.
echo() out the value from $_POST in PHP and verify whether the browser is sending the data correctly and that it's being parsed into PHP correctly. WHen you do this test make sure that the browser has correctly detected the character set.
If that works then you're likely to have mis-configured something with the database. If you have both the table collation and the connection encoding as "UTF-8" then you should have no problems saving the data into MySQL (if both SET NAMES and the table collation are the same no translation will go on so it'll be stored correctly in the table).
You didn't mention the MySQL connection anywhere in your question (just "etc"). Just in case there's anything you've missed have a good look over this article How to Avoid Character Encoding Problems in PHP
Related
I switched an self developed Web-Application based on Perl/CGI and MySQL from one server to another and since then, special characters from database content like german umlauts are shown as black diamond with question mark. Everything else, even text with umlauts which is coming from the script is ok. After lot of research I have still no clue what might cause the issue.
The original server is openSuSE 13.1 with Perl 5.18.1 and MySQL 5.6.25 while the new server is Debian 8 with Perl 5.20.2 and MySQL 5.5.44. I transfered all files with zipped tar files and configured MySQL and Apache identically, i.e. setting utf-8 and so on. After dumping and restoring sql data I was able to verify that data in new MySQL is fine.
I tried many things so far, playing around with use Encode and use utf8 in Perl, setting DBI with utf8 but nothing helps. Feel quite lost now, so any hint is appreciated. Let me know if more info is necessary.
Kind regards,
Uwe
The post suggested by bytepusher helped me. When I use use utf8; AND use open ':std', ':encoding(UTF-8)'; everything works fine. However, I am still wondering why it did work without these statements on the old machine...
I have an old site where visitors can add their comments. Until now it always worked well, it doesn't have many visitors (it's for a niche audience). It was built in classic ASP and it's using MySQL (now 5.6). It's running on IIS 8.5 and it connects to the DB without DSN.
Whenever someone adds emoji characters to their posts it'll make the IIS service go into some kind of loop using more than 60% of the CPU and never stops.
I do not want to filter these characters out, I think they fit in well with the site's premise, however I did not foresee this issue. When I first set up MySQL I used UTF-16 to make sure my users could write in any language, and I never had issues until now. There are messages in what looks like Japanese and Korean, and I only figured out it was an issue with Emojis when a user told me what he was doing when the site crashed on him.
All the site's pages/files are saved in Unicode and for all of them the charset is set as "utf-8".
The database's collation is utf16_unicode_ci and so are the tables'.
I can insert Emojis into the tables directly from command line or via HeidiSQL, however....
The server is sending the Emojis as question marks (?).
Here's my connection string:
Driver={MySQL ODBC 5.3 Unicode
Driver};Server=...;User=******;Password=******;Option=3;charset=utf16;
Use CHARACTER SET utf8mb4 end-to-end in MySQL.
I set up a web project on a web server with apparently wrong encoding. But somehow (I really don't know why), we figured how we had to deal with it and the encoding worked for us.
But now we moved the data to a new server with a correctly set up UTF8 database and surprise, the encoding is wrong.
How can we "correct" the data, is there any best practice?
Example
old server: http://www.orat.io/stmt/200
new server: http://www.firefile.net/stmt/200
Thanks a lot!
This actually happens in practice - I've seen it sometimes :-) What usually happens:
database is in utf-8
input/output is in encoding1 - say it is iso-8859-2 (i.e. the meta-charset in HTML)
but, because the previous admin didn't know how to properly set the database connection encoding (this is the charset which is specified using set names SQL command), the database connection is configured as if the input/output is in different encoding2 - say iso-8859-1 - default on many linux systems.
To get the correct data out of the database, you must access it in the same erroneous way as you did before - e.g. for the example above, set iso-8859-1 encoding but in fact get the iso-8859-2. Everything "works" until someone starts to access the database correctly.
You have not provided enough detail about what is encoding1/encoding2 in your case and you probably don't know it. So, either look at the old setup or try to figure out by trial and error.
The easiest way to re-code your database is probably to do mysqldump in encoding2, then claim it's encoding1 (just put the set names encoding1 at the beginning of sql file) and import the database into an empty one by source-ing the sql file. All database fields of course stay in utf-8 all the time.
Be careful and use PHPMyAdmin as an independent tool to see the "real" state of the database. :-) Good luck with this rebus.
In a web application I'm working on all the content seems fine, except for the content which seems to be retrieved from the database. Some special characters are used, and they break - so it's deffinatly a character encoding error somewhere.
When I manually in the browser try to switch from iso-8859-1 to utf-8 the database-content looks fine, but the static is messed up. And vice versa. So I suspect that the static content is iso-8859-1, and the content from the database is utf-8.
I've looked around for some configuration files which states the charset, but when I try changing it, nothing happen.
Will converting my database content to iso-8859-1 help maybe, so it correlates with the static content? In that case, how? I've tried changing the schema and collation but the effect was seamless.
Edit: My apologies. This is an MySQL database.
Maybe it is an issue with your client library that communicates to MySQL.
Before you try to retrieve data from the database, execute this query:
SET NAMES 'utf8'
reference:
http://forums.mysql.com/read.php?103,46870,47245#msg-47245
Hope this helps.
I recently deployed my application. For development I used SQLite and everything was right so far. I have a controller which uses Nokogiri to populate data into my database.
The problem is on production I'm using MySQL instead of SQLite and now my script is populating the data with the wrong encoding.
For instance, it writes "Aragón" instead of "Aragón". The MySQL is using utf8 for both the database and every table.
Nokogiri is probably returning things correctly. I suspect you have a mismatch in the character set of the content you are parsing with Nokogiri, and the database.
Your data being parsed might be ISO-8859-1 or WIN-1252, which are the most common on the internet. You'll need to look in the data to see what it is declared as. Also look at the source for the word "Aragón" and see whether it has embedded upper-bit characters, or entity-encoded characters. By looking at the value for the accented characters you can also get an idea when encoding the characters are.
Odds are good they're not UTF8, so when Nokogiri passes them to your code that writes to the database they will be wrong.
To fix the problem you'll need to either tell Nokogiri what the encoding is, or convert the text to UTF-8 before storing it.
You've got the encoding wrong somewhere in your stack. I bet it's set wrong in MySQL.
Take a look at this: I need help fixing Broken UTF8 encoding