View Japanese Characters in MySQL Workbench - mysql

I already tried This solution which says
ALTER TABLE title
CHARACTER SET utf8
COLLATE utf8_unicode_ci;
Ok here are some screen shots which might help you.
Update
here's what happens when i insert Japanese characters.
Update 2
Show create table gives this
CREATE TABLE `productInfo` (
`pID` int(11) NOT NULL AUTO_INCREMENT,
`pOperation` varchar(40) CHARACTER SET latin1 DEFAULT NULL,
`year` year(4) DEFAULT NULL,
`season` varchar(10) CHARACTER SET latin1 DEFAULT NULL,
`pName` varchar(40) CHARACTER SET latin1 DEFAULT NULL,
`category` varchar(40) CHARACTER SET latin1 DEFAULT NULL,
`margin1` text CHARACTER SET latin1,
`margin2` text CHARACTER SET latin1,
PRIMARY KEY (`pID`)
) ENGINE=InnoDB AUTO_INCREMENT=12 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
just see that
DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
But now see that the query
SELECT character_set_name, collation_name
FROM information_schema.columns
WHERE table_schema = 'trac_data'
AND table_name = 'productInfo'
AND column_name = 'pOperation';
gives
character_set_name collation_name
'latin1' 'latin1_swedish_ci'
Thats weird !
Update 3
SELECT hex(pOperation),pOperation FROM trac_data.productInfo;
gave 3F3F3F3F3F which is hex code for actual '?' and not any japanese character so that means no japanese characters are being stored

You have a mix of charsets in your table structure. The table itself uses utf8, but the column in question uses latin 1. You have it defined that way. As long as you have an own charset for your column you can change the table's or the schema's column a thousand times. It won't have any effect on your column. So, instead change the column's charset to either default (to use that of the table) or make it using utf8 explicitely.
When you alter the column's charset existing data will be converted (if possible). Your wrong input however stays wrong, so you have to fill the data again.

Ok i found the cause
CREATE TABLE `productInfo` (
`pID` int(11) NOT NULL AUTO_INCREMENT,
`pOperation` varchar(40) CHARACTER SET latin1 DEFAULT NULL,
`year` year(4) DEFAULT NULL,
`season` varchar(10) CHARACTER SET latin1 DEFAULT NULL,
`pName` varchar(40) CHARACTER SET latin1 DEFAULT NULL,
`category` varchar(40) CHARACTER SET latin1 DEFAULT NULL,
`margin1` text CHARACTER SET latin1,
`margin2` text CHARACTER SET latin1,
PRIMARY KEY (`pID`)
) ENGINE=InnoDB AUTO_INCREMENT=12 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
I noticed how in front of each column SET latin1 was present.
So I Just changed to sjis and problem solved.

You have to set the database collation to UTF-8, not only the table collation :
Here is the SQL script result :

Related

Sql constraint error references other column

I don't know a lot of mysql and have an error in my sql script. Currently running mysql 8.0.24. Does anyone know what might be the problem?
Error:
https://prnt.sc/226xk5x
Sql:
-- Dumping structure for table gtav_rp2._vehicle
CREATE TABLE IF NOT EXISTS `_vehicle` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`cid` int(11) unsigned NOT NULL,
`vin` varchar(50) NOT NULL DEFAULT '',
`type` varchar(50) NOT NULL DEFAULT '',
`size` int(11) NOT NULL,
`plate` varchar(50) NOT NULL DEFAULT '',
`model` varchar(50) NOT NULL DEFAULT '',
`name` varchar(50) DEFAULT NULL,
`garage` varchar(59) DEFAULT NULL,
`state` varchar(50) DEFAULT NULL,
`appearance` longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL CHECK (json_valid('appearance')),
`mods` longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL CHECK (json_valid('mods')),
`data` longtext DEFAULT NULL,
`damage` longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL CHECK (json_valid('damage')),
`degredation` longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL CHECK (json_valid('degredation')),
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
-- Dumping data for table gtav_rp2._vehicle: ~0 rows (approximately)
/*!40000 ALTER TABLE `_vehicle` DISABLE KEYS */;
/*!40000 ALTER TABLE `_vehicle` ENABLE KEYS */;
/*!40101 SET SQL_MODE=IFNULL(#OLD_SQL_MODE, '') */;
/*!40014 SET FOREIGN_KEY_CHECKS=IF(#OLD_FOREIGN_KEY_CHECKS IS NULL, 1, #OLD_FOREIGN_KEY_CHECKS) */;
/*!40101 SET CHARACTER
_SET_CLIENT=#OLD_CHARACTER_SET_CLIENT */;
Your CHECK constraint does not reference the column. Or any column, actually. By using single-quotes, you're using a string literal, not a column name.
`appearance` longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL
CHECK (json_valid('appearance')),
I assume this is meant to be:
`appearance` longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL
CHECK (json_valid(`appearance`)),
Use the right type of quotes for SQL identifiers, not string literals.
In addition, this CHECK constraint would be unnecessary if you used the JSON data type instead of LONGTEXT. The JSON data type already enforces that the content of the column must be valid JSON format.
MySQL's JSON data type already uses utf8mb4 character set and utf8mb4_bin collation.
So your column definition could be simply as follows:
`appearance` JSON

Emojis replaced by question marks when insert in Database

I'm having an issue with emojis not being transferred correctly to my staging site. Some emojis are replaced by a question mark, some don't. It's mainly affecting the blog posts. I'm not sure what is causing this and was hoping someone could point me in the right direction to solving this problem! Could it be something with the character encoding in the database?
For example:
These work: ☀️ ☔️❤️✨
These get replaced by a question mark: 🇸🇪💛🎉😍😘🌸😎
and show this Error:
1366 Incorrect string value: '\xF0\x9F\x98\x83\xF0\x9F...' for column 'text' at row 1
MY SQL Code:
CREATE TABLE `posts` (
`id` int(11) UNSIGNED NOT NULL,
`user_id` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '0',
`text` longtext COLLATE utf8mb4_unicode_ci NOT NULL,
`status` varchar(50) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '0',
`timestamp` varchar(50) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '0'
) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
Have run and tried this SQL Codes:
ALTER TABLE posts CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_bin
ALTER DATABASE test CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci;
The table definition is correct.
Where are you trying to insert from? Make sure the connection is using utf8mb4-charset. From command line use the --default-character-set=utf8mb4 option.
See db-fiddle

Utf8 characters cut off when saving text field in mysql

I'm working on a website in which we save Japanese characters in a text field.
The table looks like this:
| contactlogs | CREATE TABLE `contactlogs` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`contact_email` varchar(128) CHARACTER SET latin1 NOT NULL,
`name` varchar(128) CHARACTER SET latin1 DEFAULT NULL,
`company_name` varchar(128) CHARACTER SET latin1 DEFAULT NULL,
`email` varchar(128) CHARACTER SET latin1 DEFAULT NULL,
`telephone` varchar(30) CHARACTER SET latin1 DEFAULT NULL,
`fax` varchar(30) CHARACTER SET latin1 DEFAULT NULL,
`subject` text COLLATE utf8_unicode_ci NOT NULL,
`message` text COLLATE utf8_unicode_ci NOT NULL,
`created` datetime DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=4 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci |
When I retrieve the data that was saved, the message field is often cut off, with a garbage character at the end (I assume because it might not have saved all the data of the last character.
The website itself is in cakephp. The data is being saved by just doing a $this->model->save($data) (Haven't changed anything about the way it's saving, the model itself is empty). There's no special database settings. Just set host, login, persistent => false, driver => mysql, database, prefix.
Your DB and tables are probably under latin1 default collation, so without 'encoding' => 'utf8' in your database config cake is gonna be talking to the DB in latin1. The transcoding of UTF to latin1 and back leaves space for data to be lost. This is not really cake's problem.
So go on and change the connection to UTF and all default collations in the DB too. Newly saved data will be displayed fine, for old ones it will depend, I won't go into details cause it's out of context.

Is there any way to make a UNIQUE index case insensitive in Mysql 5.1.x ?

If so - What must change in this table ?
CREATE TABLE contestants
(
idContestants int(10) unsigned NOT NULL AUTO_INCREMENT,
idEvent int(10) unsigned NOT NULL,
ContestantName varchar(50) DEFAULT NULL,
PRIMARY KEY (idContestants),
UNIQUE KEY Index_UniqueName (idEvent,ContestantName),
)
ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=latin1;
If you mean case sensitive then:
ALTER TABLE `contestants` CHANGE `ContestantName` `ContestantName` VARCHAR( 50 )
CHARACTER SET latin1 COLLATE latin1_bin NULL DEFAULT NULL
If you mean case insensitive then:
ALTER TABLE `contestants` CHANGE `ContestantName` `ContestantName` VARCHAR( 50 )
CHARACTER SET latin1 COLLATE latin1_general_ci NULL DEFAULT NULL
For table level do (for case insensitive):
ALTER TABLE `contestants` DEFAULT CHARACTER SET latin1 COLLATE latin1_general_ci
Note that table level only affects new columns.
For database level do (for case insensitive):
ALTER DATABASE `database_name` CHARACTER SET latin1 COLLATE latin1_general_ci
Note that database level only affect new tables.
Yes, use a case-insensitive collation on the columns involved.
MySQL Manual :: Column Character Set and Collation
This worked for me in Mysql 5.5
ALTER TABLE `contestants` MODIFY
`ContestantName` VARCHAR(50)
CHARACTER SET latin1
COLLATE latin1_bin;

Mysql german accents not-sensitive search in full-text searches

Let`s have a example hotels table:
CREATE TABLE `hotels` (
`HotelNo` varchar(4) character set latin1 NOT NULL default '0000',
`Hotel` varchar(80) character set latin1 NOT NULL default '',
`City` varchar(100) character set latin1 default NULL,
`CityFR` varchar(100) character set latin1 default NULL,
`Region` varchar(50) character set latin1 default NULL,
`RegionFR` varchar(100) character set latin1 default NULL,
`Country` varchar(50) character set latin1 default NULL,
`CountryFR` varchar(50) character set latin1 default NULL,
`HotelText` text character set latin1,
`HotelTextFR` text character set latin1,
`tagsforsearch` text character set latin1,
`tagsforsearchFR` text character set latin1,
PRIMARY KEY (`HotelNo`),
FULLTEXT KEY `fulltextHotelSearch` (`HotelNo`,`Hotel`,`City`,`CityFR`,`Region`,`RegionFR`,`Country`,`CountryFR`,`HotelText`,`HotelTextFR`,`tagsforsearch`,`tagsforsearchFR`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_german1_ci;
In this table for example we have only one hotel with Region name = "Graubünden" (please note umlaut ü character)
And now I want to achieve same search match for phrases:
'graubunden' and
'graubünden'
This is simple with use of MySql built in
collations in regular searches as follows:
SELECT *
FROM `hotels`
WHERE `Region` LIKE CONVERT(_utf8 '%graubunden%' USING latin1)
COLLATE latin1_german1_ci
This works fine for 'graubunden' and 'graubünden' and
as a result I receive proper result, but problem is
when we make MySQL full text search
Whats wrong with this SQL statement?:
SELECT
*
FROM
hotels
WHERE
MATCH (`HotelNo`,`Hotel`,`Address`,`City`,`CityFR`,`Region`,`RegionFR`,`Country`,`CountryFR`, `HotelText`, `HotelTextFR`, `tagsforsearch`, `tagsforsearchFR`)
AGAINST( CONVERT('+graubunden' USING latin1) COLLATE latin1_german1_ci IN BOOLEAN MODE)
ORDER BY Country ASC, Region ASC, City ASC
This doesn`t return any result.
Any ideas where the dog is buried ?
When you define individual CHARACTER SETS for your columns, you override the collation you set default on table level.
Each of your columns has default latin1 collation (which is latin1_swedish_ci). You can see it by running SHOW CREATE TABLE.
In FULLTEXT queries, indexed columns have COERCIBILITY of 0, that is all fulltext queries are converted to the collation used in the index, not vice versa.
You need to remove CHARACTER SET definitions from your columns or explicitly set all columns to latin1_german_ci:
CREATE TABLE `hotels` (
`HotelNo` varchar(4) NOT NULL default '0000',
`Hotel` varchar(80) NOT NULL default '',
`City` varchar(100) default NULL,
`CityFR` varchar(100) default NULL,
`Region` varchar(50) default NULL,
`RegionFR` varchar(100) default NULL,
`Country` varchar(50) default NULL,
`CountryFR` varchar(50) default NULL,
`HotelText` text,
`HotelTextFR` text,
`tagsforsearch` text,
`tagsforsearchFR` text,
PRIMARY KEY (`HotelNo`),
FULLTEXT KEY `fulltextHotelSearch` (`HotelNo`,`Hotel`,`City`,`CityFR`,`Region`,`RegionFR`,`Country`,`CountryFR`,`HotelText`,`HotelTextFR`,`tagsforsearch`,`tagsforsearchFR`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_german1_ci;
INSERT
INTO hotels (hotelText, HotelTextFR, tagsforsearch, tagsforsearchFR)
VALUES ('text', 'text', 'graubünden', 'tags');
SELECT *
FROM hotels
WHERE MATCH (`HotelNo`,`Hotel`,`City`,`CityFR`,`Region`,`RegionFR`,`Country`,`CountryFR`, `HotelText`, `HotelTextFR`, `tagsforsearch`, `tagsforsearchFR`)
AGAINST (CONVERT('+graubunden' USING latin1) COLLATE latin1_german1_ci IN BOOLEAN MODE)
ORDER BY
Country ASC, Region ASC, City ASC;