I am developing an app in which I am using MySQL database. The database contains certain characters which can not be encoded to the client side & I found those values null.
Like, a string containing a special character is represented as null at the client side.
I found that the default charset for the db was latin1, I changed it to utf-8, including all tables and individual columns of those tables. Also in my pdo_construct I have mentioned the charset to be utf-8,
$db = new PDO('mysql:dbname=$dbname;host=$dbhost;charset=utf8',$dbname,$dbhost);
I also configured the response headers to use utf-8 charset. But the characters are still not encoded, I am still getting null string in case where the special character is present.
I tried changing the my.ini file configuration by setting the default charset, it gives me error in my connection file at PDO construct.
Its urgent for me to fix this! Can someone help?
Related
My MySQL database contains some Chinese symbols and such (non-ASCII symbols). When I view them in PHPMyAdmin, they look garbled. However, if I display them on my website with PHP using the regular mysqli API, it looks fine so I assume the data is uploaded/stored properly in the database, so maybe the server connection collation is incorrect.
My PHP code for opening the database connection is:
function openConnection(): mysqli
{
$databaseHost = "localhost";
$databaseUser = "root";
$databasePassword = '';
$databaseName = "my-database-name";
$connection = new mysqli($databaseHost, $databaseUser,
$databasePassword, $databaseName);
if ($connection->connect_error) {
die("Connection failed: " . $connection->connect_error);
}
return $connection;
}
My PHPMyAdmin server connection collation is the default utf8mb4_unicode_ci which seems to be reasonable as well. My tables are also created with the default utf8mb4_general_ci. Shouldn't that work fine for any input users might make?
Calling $connection->get_charset() in PHP also returns the correct charset:
If I export the database data in MyPHPAdmin, the export is also garbled in Notepad++, I made sure to view it with UTF-8 encoding. If I import the garbled export again, the database will show the data as garbled once more and on the website the data now also shows as garbled. In this case, an actually corrupted export happened.
How can I solve this encoding problem? Clearly PHP can handle UTF-8 properly, my Apache web server is also serving UTF-8 and my database is configured seemingly correctly as well but there is an issue with PHPMyAdmin or the database/database table collation.
It looks like the issue was entirely elsewhere since I'm supplying data to PHP with C++ code. The C++ code uses the nlohmann JSON libary to build the data submitted to the PHP script. The issue was my inability to specifically encode std::strings to UTF-8 like described here when putting data into a C++ JSON object. With that said, everything is now working as expected.
⚈ If using mysqli, do $mysqli_obj->set_charset('utf8mb4');
⚈ If using PDO do somethin like $db = new PDO('dblib:host=host;dbname=db;charset=utf8mb4', $user, $pwd);
⚈ Alternatively, execute SET NAMES utf8mb4
Any of these will say that the bytes in the client are UTF-8 encoded. Conversion, if necessary, will occur between the client and the database if the column definition is something other than utf8mb4.
More notes on PHP: http://mysql.rjweb.org/doc.php/charcoll#php
If you have specific garbling, see Trouble with UTF-8 characters; what I see is not what I stored
If you suspect the data being fed from PHP to Notepad, dump a few Chinese characters in hex and shown to us. I would expect every 4th character to be hex F0 or every 3rd to be between E3 and EA. (These are the first byte for 4-char and 3-char UTF-8 encoding of Chinese characters.)
Does Notepad properly handle UTF-8, or does it need a setting?
If you are in the "cmd" in Windows, you may need chcp 65001; see http://mysql.rjweb.org/doc.php/charcoll#entering_accents_in_cmd That way, more non-English characters will display correctly.
Trying to quickly convert a latin1 mysql DB to utf8, I tried the following:
Dump the DB
run iconv -f latin1 -t utf8 on the resulting file
import into a fresh DB with UTF8 default encoding
This mostly works except... some letters get converted wrong (an example: uppercase accented 'U' becomes some garbled sequence starting with a question mark). Some conversion is taking place (od an a query result shows a two byte sequence where the latin1 byte was) and te latin1 version is alright. While I have so far been unsystematic in isolating the problem (late night; under deadline; etc.) the weirdness of the issue kills me: why would it fail on some letters and not all? Client connection? Column charset? Why I am not getting any diagnostics? I'm stymied.
Sure, I can work on isolating the issue and its details, but thought that maybe somebody ran into this already and can recognize it by this (admittedly rather poor) description.
Cheers
The data may have been stored as latin1 but it's possible that what ever client you used to dump the data has already exported it as UTF-8.
Open the dump file in a decent text editor (Notepad++, TextWrangler, Atom) and check which encoding allows all characters to be displayed properly.
Then when it comes to import the data back in, ensure your client is set to use UTF-8 on the import.
Don't use iconv, it only muddies the works.
Assuming that a table is declared to be latin1 and correctly contains latin1 bytes, but you would like to change it to utf8, do this to the table:
ALTER TABLE tbl CONVERT TO CHARACTER SET utf8;
It is also possible to do it with a dump and reload; it involves some changes to the arguments. Sorry I don't have the details.
I'm using phpMyAdmin version 4.4.4 with MySQL 5.6 (charset is set to UTF-8 Unicode). The table in question has the collation set to utf8-general-ci and all fields are also set to utf8-general-ci collation as well. My php.ini file has default_charset = "UTF-8".
Despite all the UTF-8 settings for all three applications, unicode characters appear garbled when viewing a table within phpMyAdmin. So, instead of seeing ...
Søren
... in phpMyAdmin I see ...
Søren
Even though it displays garbled in phpMyAdmin, it displays correctly on the website. The only problem is with phpMyAdmin.
If I attempt to Insert a new record using phpMyAdmin and enter Søren in a text field, it displays like this within phpMyAdmin...
Søren
Which looks correct there, but, on the web page, it displays like this...
S�ren
The ø character is replaced with a question mark inside a black diamond instead of displaying the proper unicode character on the website.
What the heck is going on? How do I make phpMyAdmin display and insert the unicode characters properly into the table without mangling them? Thanks!
My php.ini file has default_charset = "UTF-8".
That only affects the charset used for some PHP built-in functions like htmlentities.
MySQL uses its own charset to decode stuff you send it. This can be set using $mysqli->set_charset('utf8') for mysqli, or mysql_set_charset('utf8') for the deprecated mysql module, or using charset=utf8 in the connection string in PDO.
I started working on a legacy mysql database whose collation: latin1-default but tables are utf-8-default. Even though tables are mentioned with utf-8 (universal standard encoding) it doesn't render Swedish characters. It seems application related to this database encoding is ISO-8859-1. So, I would like to convert this database and data in it to ISO-8859-1 encoding. I tried with this command
iconv -f UTF-8 -t ISO-8859-1 webtest_backu_01.sql > converted-file.sql
it gives error: illegal input sequence at position
any help is appreciated. thanks.
Please take a look at this link: http://dev.mysql.com/doc/refman/5.0/en/charset-conversion.html
You can use the alter table command to make this conversion per-table if it is possible. I used this before successfully.
Example from the link:
ALTER TABLE t MODIFY col1 CHAR(50) CHARACTER SET utf8;
Also an important detail... Conversion may be lossy if the column contains characters that are not in both character sets... but I don't think ISO-8859-1 to UTF-8.
Give this a try for one of the tables and see if it works.
I have an MYSQL Database in utf-8 format, but the Characters inside the Database are ISO-8859-1 (ISO-8859-1 Strings are stored in utf-8). I've tried with recode, but it only converted e.g. ü to ü). Does anybody out there has an solution??
If you tried to store ISO-8859-1 characters in the a database which is set to UTF-8 you just managed to corrupt your "special characters" -- as MySQL would retrieve the bytes from the database and try to assemble them as UTF-8 rather than ISO-8859-1. The only way to read the data correctly is to use a script which does something like:
ResultSet rs = ...
byte[] b = rs.getBytes( COLUMN_NAME );
String s = new String( b, "ISO-8859-1" );
This would ensure you get the bytes (which came from a ISO-8859-1 string from what you said) and then you can assemble them back to ISO-8859-1 string.
The other problem as well -- what do you use to "view" the strings in the database -- is it not the case that your console doesn't have the right charset to display those characters rather than the characters being stored wrongly?
NOTE: Updated the above after the last comment
I just went through this. The biggest part of my solution was exporting the database to .csv and Find / Replace the characters in question. The character at issue may look like a space, but copy it directly from the cell as your Find parameter.
Once this is done - and missing this is what took me all morning:
Save the file as CSV ( MS-DOS )
Excellent post on the issue
Source of MS-DOS idea