I'm having an issue with emojis not being transferred correctly to my staging site. Some emojis are replaced by a question mark, some don't. It's mainly affecting the blog posts. I'm not sure what is causing this and was hoping someone could point me in the right direction to solving this problem! Could it be something with the character encoding in the database?
For example:
These work: ☀️ ☔️❤️✨
These get replaced by a question mark: 🇸🇪💛🎉😍😘🌸😎
and show this Error:
1366 Incorrect string value: '\xF0\x9F\x98\x83\xF0\x9F...' for column 'text' at row 1
MY SQL Code:
CREATE TABLE `posts` (
`id` int(11) UNSIGNED NOT NULL,
`user_id` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '0',
`text` longtext COLLATE utf8mb4_unicode_ci NOT NULL,
`status` varchar(50) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '0',
`timestamp` varchar(50) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '0'
) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
Have run and tried this SQL Codes:
ALTER TABLE posts CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_bin
ALTER DATABASE test CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci;
The table definition is correct.
Where are you trying to insert from? Make sure the connection is using utf8mb4-charset. From command line use the --default-character-set=utf8mb4 option.
See db-fiddle
Related
I don't know a lot of mysql and have an error in my sql script. Currently running mysql 8.0.24. Does anyone know what might be the problem?
Error:
https://prnt.sc/226xk5x
Sql:
-- Dumping structure for table gtav_rp2._vehicle
CREATE TABLE IF NOT EXISTS `_vehicle` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`cid` int(11) unsigned NOT NULL,
`vin` varchar(50) NOT NULL DEFAULT '',
`type` varchar(50) NOT NULL DEFAULT '',
`size` int(11) NOT NULL,
`plate` varchar(50) NOT NULL DEFAULT '',
`model` varchar(50) NOT NULL DEFAULT '',
`name` varchar(50) DEFAULT NULL,
`garage` varchar(59) DEFAULT NULL,
`state` varchar(50) DEFAULT NULL,
`appearance` longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL CHECK (json_valid('appearance')),
`mods` longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL CHECK (json_valid('mods')),
`data` longtext DEFAULT NULL,
`damage` longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL CHECK (json_valid('damage')),
`degredation` longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL CHECK (json_valid('degredation')),
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
-- Dumping data for table gtav_rp2._vehicle: ~0 rows (approximately)
/*!40000 ALTER TABLE `_vehicle` DISABLE KEYS */;
/*!40000 ALTER TABLE `_vehicle` ENABLE KEYS */;
/*!40101 SET SQL_MODE=IFNULL(#OLD_SQL_MODE, '') */;
/*!40014 SET FOREIGN_KEY_CHECKS=IF(#OLD_FOREIGN_KEY_CHECKS IS NULL, 1, #OLD_FOREIGN_KEY_CHECKS) */;
/*!40101 SET CHARACTER
_SET_CLIENT=#OLD_CHARACTER_SET_CLIENT */;
Your CHECK constraint does not reference the column. Or any column, actually. By using single-quotes, you're using a string literal, not a column name.
`appearance` longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL
CHECK (json_valid('appearance')),
I assume this is meant to be:
`appearance` longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL
CHECK (json_valid(`appearance`)),
Use the right type of quotes for SQL identifiers, not string literals.
In addition, this CHECK constraint would be unnecessary if you used the JSON data type instead of LONGTEXT. The JSON data type already enforces that the content of the column must be valid JSON format.
MySQL's JSON data type already uses utf8mb4 character set and utf8mb4_bin collation.
So your column definition could be simply as follows:
`appearance` JSON
I am trying to add Turkish names on my table but then when displayed it gives me ? instead of any of them. Any help what I am missing here? This is my table:
CREATE TABLE IF NOT EXISTS `offerings` (
`dep` varchar(5) CHARACTER SET utf16 COLLATE utf16_turkish_ci DEFAULT NULL,
`grade` varchar(4) CHARACTER SET utf16 COLLATE utf16_turkish_ci DEFAULT NULL,
`section` varchar(3) CHARACTER SET utf16 COLLATE utf16_turkish_ci DEFAULT NULL,
`name` varchar(100) CHARACTER SET utf16 COLLATE utf16_turkish_ci DEFAULT NULL,
`teacher` varchar(50) CHARACTER SET utf16 COLLATE utf16_turkish_ci DEFAULT NULL,
`quota` varchar(2) CHARACTER SET utf16 COLLATE utf16_turkish_ci DEFAULT NULL,
`lec1` varchar(35) CHARACTER SET utf16 COLLATE utf16_turkish_ci DEFAULT NULL,
`lec2` varchar(35) CHARACTER SET utf16 COLLATE utf16_turkish_ci DEFAULT NULL,
`lec3` varchar(35) DEFAULT NULL,
`lec4` varchar(35) DEFAULT NULL,
`lec5` varchar(35) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf16;
As suggested from the answer I choose here is the solution to the problem for whoever googles this topic. Special thanks to all who contributed in the solution of my problem.
CREATE TABLE IF NOT EXISTS `offerings` (
`dep` varchar(5) NOT NULL,
`grade` varchar(4) COLLATE utf8_unicode_ci NOT NULL,
`section` varchar(3) COLLATE utf8_unicode_ci NOT NULL,
`name` varchar(100) COLLATE utf8_unicode_ci NOT NULL,
`teacher` varchar(50) COLLATE utf8_unicode_ci NOT NULL,
`quota` varchar(2) COLLATE utf8_unicode_ci,
`lec1` varchar(35) DEFAULT NULL,
`lec2` varchar(35) DEFAULT NULL,
`lec3` varchar(35) DEFAULT NULL,
`lec4` varchar(35) DEFAULT NULL,
`lec5` varchar(35) DEFAULT NULL
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
(Beginnings of an answer...)
Please don't use utf16; there is virtually no reason for such in a MySQL table.
So, assuming you switch to utf8, let's see if we can get rid of the ? problems.
utf8 needs to be established in about 4 places.
The column(s) in the database -- Use SHOW CREATE TABLE to verify that they are explicitly set to utf8, or defaulted from the table definition. (It is not enough to change the database default.)
The connection between the client and the server. See SET NAMES utf8.
The bytes you have. (This is probably the case.)
If you are displaying the text in a web page, check the <meta> tag.
What probably happened:
you had utf8-encoded data (good)
SET NAMES latin1 was in effect (default, but wrong)
the column was declared CHARACTER SET latin1 (default, but wrong)
Since the CHARACTER SET disagrees with what you have shown, the problem is possibly more complex. Please provide
SELECT col, HEX(col) FROM tbl WHERE ...
for some simple cell with Turkish characters. With this, I may be able to figure out what happened.
Also, Reference notes on encodings.
VARCHARs are character strings, while NVARCHARS are Unicode character strings. NVARCHARS require more bits per character to store, but have a greater range. Try updating your data types. This should fix your problem.
EDIT This answer is wrong. The OP clearly asked for a MySQL solution, but the above applies only to SQL Server.
I already tried This solution which says
ALTER TABLE title
CHARACTER SET utf8
COLLATE utf8_unicode_ci;
Ok here are some screen shots which might help you.
Update
here's what happens when i insert Japanese characters.
Update 2
Show create table gives this
CREATE TABLE `productInfo` (
`pID` int(11) NOT NULL AUTO_INCREMENT,
`pOperation` varchar(40) CHARACTER SET latin1 DEFAULT NULL,
`year` year(4) DEFAULT NULL,
`season` varchar(10) CHARACTER SET latin1 DEFAULT NULL,
`pName` varchar(40) CHARACTER SET latin1 DEFAULT NULL,
`category` varchar(40) CHARACTER SET latin1 DEFAULT NULL,
`margin1` text CHARACTER SET latin1,
`margin2` text CHARACTER SET latin1,
PRIMARY KEY (`pID`)
) ENGINE=InnoDB AUTO_INCREMENT=12 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
just see that
DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
But now see that the query
SELECT character_set_name, collation_name
FROM information_schema.columns
WHERE table_schema = 'trac_data'
AND table_name = 'productInfo'
AND column_name = 'pOperation';
gives
character_set_name collation_name
'latin1' 'latin1_swedish_ci'
Thats weird !
Update 3
SELECT hex(pOperation),pOperation FROM trac_data.productInfo;
gave 3F3F3F3F3F which is hex code for actual '?' and not any japanese character so that means no japanese characters are being stored
You have a mix of charsets in your table structure. The table itself uses utf8, but the column in question uses latin 1. You have it defined that way. As long as you have an own charset for your column you can change the table's or the schema's column a thousand times. It won't have any effect on your column. So, instead change the column's charset to either default (to use that of the table) or make it using utf8 explicitely.
When you alter the column's charset existing data will be converted (if possible). Your wrong input however stays wrong, so you have to fill the data again.
Ok i found the cause
CREATE TABLE `productInfo` (
`pID` int(11) NOT NULL AUTO_INCREMENT,
`pOperation` varchar(40) CHARACTER SET latin1 DEFAULT NULL,
`year` year(4) DEFAULT NULL,
`season` varchar(10) CHARACTER SET latin1 DEFAULT NULL,
`pName` varchar(40) CHARACTER SET latin1 DEFAULT NULL,
`category` varchar(40) CHARACTER SET latin1 DEFAULT NULL,
`margin1` text CHARACTER SET latin1,
`margin2` text CHARACTER SET latin1,
PRIMARY KEY (`pID`)
) ENGINE=InnoDB AUTO_INCREMENT=12 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
I noticed how in front of each column SET latin1 was present.
So I Just changed to sjis and problem solved.
You have to set the database collation to UTF-8, not only the table collation :
Here is the SQL script result :
I'm working on a website in which we save Japanese characters in a text field.
The table looks like this:
| contactlogs | CREATE TABLE `contactlogs` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`contact_email` varchar(128) CHARACTER SET latin1 NOT NULL,
`name` varchar(128) CHARACTER SET latin1 DEFAULT NULL,
`company_name` varchar(128) CHARACTER SET latin1 DEFAULT NULL,
`email` varchar(128) CHARACTER SET latin1 DEFAULT NULL,
`telephone` varchar(30) CHARACTER SET latin1 DEFAULT NULL,
`fax` varchar(30) CHARACTER SET latin1 DEFAULT NULL,
`subject` text COLLATE utf8_unicode_ci NOT NULL,
`message` text COLLATE utf8_unicode_ci NOT NULL,
`created` datetime DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=4 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci |
When I retrieve the data that was saved, the message field is often cut off, with a garbage character at the end (I assume because it might not have saved all the data of the last character.
The website itself is in cakephp. The data is being saved by just doing a $this->model->save($data) (Haven't changed anything about the way it's saving, the model itself is empty). There's no special database settings. Just set host, login, persistent => false, driver => mysql, database, prefix.
Your DB and tables are probably under latin1 default collation, so without 'encoding' => 'utf8' in your database config cake is gonna be talking to the DB in latin1. The transcoding of UTF to latin1 and back leaves space for data to be lost. This is not really cake's problem.
So go on and change the connection to UTF and all default collations in the DB too. Newly saved data will be displayed fine, for old ones it will depend, I won't go into details cause it's out of context.
Im having a strange problem with some data that is saved in mysql.
im saving a certain value as text in my db but for some reason some of the text is transformed to this "ＡＴＥＬＯＰＨＯＢＩＡ" kind of symbols.
the code:
$data = mysql_real_escape_string($_GET['data']);
$id = $_GET["id"];
mysql_query("INSERT INTO table (ID, name, pos,data) VALUES ('$id', '$regionname','$regionps[0]','$data')");
table structure:
CREATE TABLE IF NOT EXISTS `sim_scanner` (
`ID` varchar(64) COLLATE latin1_general_ci NOT NULL,
`simname` varchar(36) COLLATE latin1_general_ci NOT NULL,
`simpos` varchar(25) COLLATE latin1_general_ci NOT NULL,
`time` varchar(25) COLLATE latin1_general_ci NOT NULL,
`data` text COLLATE latin1_general_ci NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_general_ci;
the data that is transforming itself to that is a value like this , <151.25,255.54,1254.12>(vector), as the data is generated by a script i don't know the exact value its basicly a characters position once it moves the data is another vector
thank you for your time
your mysql character encoding is different to the data character encoding that's why ...
for example try use the utf-8_general_ci in DB character set..
UTF-8 definition for the characters they support, usually UTF8 supports most of the characters