I am inserting some non-ascii(specifically asian characters into mysql table column with charset utf8, but after insertion, if I retrieve it again, it shows up as ????. I checked the db, table and column charset, they are all utf8. what's wrong?
CREATE TABLE `test_utf` (
`test_id` bigint(20) NOT NULL auto_increment,
`raw_text` longtext,
PRIMARY KEY (`test_id`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8;
insert into test_utf (raw_text) values('黄剑');
Problem solved!
I need to add characterEncoding=UTF-8 to whatever mysql client I'm using,
for example, when I use jdbc I need to specify "jdbc:mysql://localhost/dbname?characterEncoding=UTF-8" in connection url.
Related
I have two words ('বাঁধা' and 'বাধা') to be inserted in a mysql (8.0.12 - MySQL Community Server - GPL) table. The word 'বাঁধা' is inserted correctly. But when inserting 'বাধা', mysql produces an error:
INSERT INTO lc6_words(jp_word, jp_fcharascii) VALUES('বাঁধা', 2476);
/*Query OK*/
INSERT INTO lc6_words(jp_word, jp_fcharascii) VALUES('বাধা', 2476);
/*#1062 - Duplicate entry 'বাধা' for key 'jp_word'*/
The table structure:
CREATE TABLE IF NOT EXISTS `lc6_words` (
`jp_wkey` BIGINT NOT NULL AUTO_INCREMENT,
`jp_word` varchar(255) NOT NULL,
`jp_fcharascii` int UNSIGNED NOT NULL,
`jp_word_occ` BIGINT UNSIGNED NOT NULL DEFAULT 1,
UNIQUE(`jp_word`),
PRIMARY KEY (`jp_wkey`)
) ENGINE=MyISAM DEFAULT CHARSET=UTF8MB4 COLLATE=utf8mb4_bin;
Relevant queries and their output:
SELECT jp_wkey FROM lc6_words WHERE BINARY jp_word='বাঁধা';
/* 1 */
SELECT jp_wkey FROM lc6_words WHERE BINARY jp_word='বাধা';
/* Empty */
Thanks for reading this far. And some more too if you share your thoughts :).
There seems to be problem in collation. After running the command below, all worked perfectly:
ALTER TABLE lc6_words MODIFY jp_word VARCHAR(191) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;
Note: The VARCHAR size changed from 255 to 191.
When trying to insert a unicode emoji character (😎) to a MYSQL table, the insert fails due to the error;
Incorrect string value: '\\xF0\\x9F\\x98\\x8E\\xF0\\x9F...' for column 'Title' at row 1
From what I've red about this issue, it's apparently caused by the tables default character set, and possible the columns default character set, being set incorrectly. This post suggests to use utf8mb4, which I've tried, but the insert is still failing.
Here's my table configuration;
CREATE TABLE `TestTable` (
`Id` int(11) NOT NULL AUTO_INCREMENT,
`InsertDate` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
`Title` text,
`Description` text,
`Info` varchar(250) CHARACTER SET utf8 DEFAULT NULL,
PRIMARY KEY (`Id`),
KEY `xId_TestTablePK` (`Id`)
) ENGINE=InnoDB AUTO_INCREMENT=2191 DEFAULT CHARSET=utf8mb4;
Note that the Title and Text columns dont have an explicitly stated character set. Initially I had no default table character set, and had these two columns were setup with DEFAULT CHARSET=utf8mb4. However, when I altered the table's default charset to the same, they were removed (presumably because the columns inherit the type from the table?)
Can anyone please help me understand how I can store these unicode values in my table?
Its worth noting that I'm on Windows, trying to perform this insert on the MYSQL Workbench. I have also tried using C# to insert into the database, specifying the character set with CHARSET=utf8mb4, however this returned the same error.
EDIT
To try and insert this data, I am executing the following;
INSERT INTO TestTable (Title) SELECT '😎😎';
Edit
Not sure if this is relevant or not, but my database is also set up with the same default character set;
CREATE DATABASE `TestDB` /*!40100 DEFAULT CHARACTER SET utf8mb4 */;
The connection needs to establish that the client is talking utf8mb4, not just utf8. This involves changing the parameters used at connection time. Or executing SET NAMES utf8mb4 just after connecting.
I have a problem when trying to execute this line in MySQL (Workbench):
INSERT INTO classification (`Type`, `Subtype`) VALUES ("тип", "подтип");
I have tried to set different charsets for table classification : cp1251, utf-8, utf8mb4, cp1251_bin.
This is a table with all charsets in my database that I have found, maybe it will help you:
UPD. I have found a solution. However, I had to change my table, so now the table risk is an edited table classification. The result of SHOW CREATE TABLE risk is:
'CREATE TABLE `risk` (
`IdRisk` int(11) NOT NULL AUTO_INCREMENT,
`IdSubtype` int(11) DEFAULT NULL,
`Content` varchar(4000) CHARACTER SET utf8 DEFAULT NULL,
PRIMARY KEY (`IdRisk`),
KEY `FK_subtype_risk_idx` (`IdSubtype`),
CONSTRAINT `FK_subtype_risk` FOREIGN KEY (`IdSubtype`) REFERENCES `subtype` (`IdSubtype`) ON DELETE SET NULL ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=48 DEFAULT CHARSET=latin1'
Can't find the solution to this issue. I'm hope that someone knows a solution to it.
Thank You!
The CHARACTER SET for the table is the default for columns in the table. Please provide SHOW CREATE TABLE so we can verify what the columns are set to.
What is the encoding of the bytes in the client? cp1251 is different than utf8; utf8mb4 == utf8 for Russian.
In what way are things bad? Based on the symptom, see this for specific tips on what else might be set incorrectly.
Perhaps it was your change to NVARCHAR that forced CHARACTER SET utf8 on the columns?
I have a MySQL Database (myDB; ~2GB in size) with 4 Tables (tab1, tab2, tab3, tab4). Currently, the data that is stored in the tables was added using the charset ISO-8859-1 (i.e. Latin-1).
I'd like to convert the data in all tables to UTF-8 and use UTF-8 as default charset of the tables/database/columns.
On https://blogs.harvard.edu/djcp/2010/01/convert-mysql-database-from-latin1-to-utf8-the-right-way/ I found an interesting approach:
mysqldump myDB | sed -i 's/CHARSET=latin1/CHARSET=utf8/g' | iconv -f latin1 -t utf8 | mysql myDB2
I haven't tried it yet, but are there any caveats?
Is there a way to do it directly in the MySQL shell?
[EDIT:]
Result of SHOW CREATE TABLE messages; after running ALTER TABLE messages CONVERT TO CHARACTER SET utf8mb4;
CREATE TABLE `messages` (
`number` int(11) NOT NULL AUTO_INCREMENT,
`status` enum('0','1','2') NOT NULL DEFAULT '1',
`user` varchar(30) NOT NULL DEFAULT '',
`comment` varchar(250) NOT NULL DEFAULT '',
`text` mediumtext NOT NULL,
`date` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`number`),
KEY `index_user_status_date` (`user`,`status`,`date`)
) ENGINE=InnoDB AUTO_INCREMENT=3285217 DEFAULT CHARSET=utf8mb4
It is possible to convert the tables. But then you need to convert the application, too.
ALTER TABLE tab1 CONVERT TO utf8mb4;
etc.
To check, do SHOW CREATE TABLE tab1; it should show you CHARACTER SET utf8mb4.
Note: There are 3 things going on:
Convert the encoding of the data in any VARCHAR and TEXT columns.
Change the CHARACTER SET for such columns.
Change the DEFAULT CHARACTER SET for the table -- this comes into play if you add any new columns without specifying a charset.
The application...
When you connect from a client to MySQL, you need to tell it, in a app-specific way or via SET NAMES, the encoding of the bytes in the client. This does not have to be the same as the column declarations; conversion will occur during INSERT and SELECT, if necessary.
I recommend you take a backup and/or test a copy of one of the tables. Be sure to go all the way through -- insert, select, display, etc.
Am developing a tamil website using ASP.NET MVC and MYSQL.
While updating the values (tamil language text) from ASP.NET MVC website to Database all my values as storing as format like this ??????????????
When I directly run the insert query into my database I am able to insert the tamil text into the database.
Asp.NET MVC : I have the below code
<meta charset="utf-8" />
MySQL - Create table syntax :
CREATE TABLE IF NOT EXISTS `Login` (
`ID` int(11) NOT NULL AUTO_INCREMENT,
`username` text COLLATE utf8_unicode_ci,
`password` mediumtext COLLATE utf8_unicode_ci,
PRIMARY KEY (`ID`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=6 ;
Note: I have been using the Entity Framework to connect with the MySQL database.
Is any thing missing from my side?
From Comment
Solution Found - just add ";Charset=utf8" to connection string.
Here is the working solution:
<add name="photostorageEntities"
connectionString="metadata=res://*/Models.Photos.csdl|
res://*/Models.Photos.ssdl|res://*/Models.Photos.msl;
provider=MySql.Data.MySqlClient;
provider connection string="server=ServerIP;
User Id=UID;password=PASS;
Persist Security Info=True;database=photostorage; Charset=utf8""
providerName="System.Data.EntityClient" />
Thanks everyone! :)
I also faced the exact same problem.
Add this connection property to the jdbc driver where you are defining the database connection.
<property name="connectionProperties" value="characterEncoding=UTF-8;characterSetResults=UTF-8"/>
I hope this will solve your problem.
Use following query to set the default character set to already created table
ALTER TABLE Login CONVERT TO CHARACTER SET utf8;
You are using collate in multiple places. The following will be enough
CREATE TABLE IF NOT EXISTS `Login` (
`ID` int(11) NOT NULL AUTO_INCREMENT,
`username` text,
`password` mediumtext,
PRIMARY KEY (`ID`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=6 ;