When I change the character set of a table from utf8 to latin1, the columns still keep the old setting.
For e.g. in the example below, name column is still utf8 even if the table is converted to latin1
mysql> create table mytest (id int, name varchar(255)) engine=myisam default charset=utf8;
Query OK, 0 rows affected (0.02 sec)
mysql> alter table mytest charset=latin1;
Query OK, 0 rows affected (0.02 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> show create table mytest\G
*************************** 1. row ***************************
Table: mytest
Create Table: CREATE TABLE `mytest` (
`id` int(11) DEFAULT NULL,
`name` varchar(255) CHARACTER SET utf8 DEFAULT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1
1 row in set (0.00 sec)
This must be to preseve the data integrity. But is there any way to change everything to latin1. I am not much worried about data. I am ok if there are a few junk characters due to conversion.
Related
I am using an aurora rds (mysql) in a cluster and cannot insert emoji's. The column which I am trying to insert into has the collation:
utf8mb4 - utf8mb4_unicode_ci
I have tried inserting using client and also from mysql workbench writing the query but in both cases I just see ???? in the field.
I have updated the table default character set:
utf8mb4
and default collation:
utf8mb4_unicode_ci
But still getting ??? instead of emoji
Edit 1:
I've tried to edit the parameter group of the cluster and set all character set values to utf8mb4 and all collation values to utf8mb4_unicode_ci but still not working.
Your cluster parameter group should have the following options set:
character_set_client: utf8mb4
character_set_connection: utf8mb4
character_set_database: utf8mb4
character_set_server: utf8mb4
collation_connection: utf8mb4_unicode_ci
collation_server: utf8mb4_unicode_ci
Rebooting your instances after updating this might be required. When you connect to the cluster you want to set the correct collation for your connection, like this:
SET NAMES utf8mb4 COLLATE utf8mb4_unicode_ci;
If you run SHOW VARIABLES LIKE "%collation%" after this you should see three variables that all have the correct collation (utf8mb4_unicode_ci).
You also need to convert your tables and columns to the correct charset and collation, this has been answered before on the DBA Stack Exchange: How to easily convert utf8 tables to utf8mb4 in MySQL 5.5
Changing the default character set of a table doesn't alter any of the existing columns. You can run SHOW CREATE TABLE MyTable and see the difference.
For example, we can create a table with an old-fashioned charset:
mysql> create table MyTable (string1 varchar(100) ) default character set = latin1;
mysql> show create table MyTable\G
*************************** 1. row ***************************
Table: MyTable
Create Table: CREATE TABLE `MyTable` (
`string1` varchar(100) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1
1 row in set (0.00 sec)
Next, we change the table's default, but we see that does not change the column. The existing column automatically gets an option showing us what it actually uses for a charset, since it's now different from the table default:
mysql> alter table MyTable default charset = utf8mb4;
Query OK, 0 rows affected (0.04 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> show create table MyTable\G
*************************** 1. row ***************************
Table: MyTable
Create Table: CREATE TABLE `MyTable` (
`string1` varchar(100) CHARACTER SET latin1 DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
1 row in set (0.00 sec)
The default is used for columns added to the table subsequently:
mysql> alter table MyTable add column string2 varchar(100);
Query OK, 0 rows affected (0.03 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> show create table MyTable\G
*************************** 1. row ***************************
Table: MyTable
Create Table: CREATE TABLE `MyTable` (
`string1` varchar(100) CHARACTER SET latin1 DEFAULT NULL,
`string2` varchar(100) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
To convert the existing column, use CONVERT TO CHARACTER SET. This rewrites the table, converting the existing data of each column if needed:
mysql> alter table MyTable convert to character set utf8mb4;
Query OK, 0 rows affected (0.09 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> show create table MyTable\G
*************************** 1. row ***************************
Table: MyTable
Create Table: CREATE TABLE `MyTable` (
`string1` varchar(100) DEFAULT NULL,
`string2` varchar(100) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
Now both columns say nothing about their respective charsets, because they match the table's default.
I wrote a detailed blog post (a few years back) explaining how to support emoji and Aurora MySQL in Ruby on Rails here:
https://josephecombs.com/2018/05/06/how-to-support-emojis-with-rails-elasticbeanstalk-and-amazon-aurora
I know external links are frowned upon here, but this could help others in a similar situation. The guide is too long to paste as an answer.
create temporary table if not exists tmp engine=memory
SELECT id, CONCAT(TRIM(lastName),TRIM(firstName),TRIM(zip)) AS identify
FROM customers
GROUP BY identify;
While running the procedure I get the following error message:
The used table type doesn't support BLOB/TEXT columns
I already saw this thread but it didn't help me.
The types on the columns are the following:
lastName -> VARCHAR(255)
firstName -> VARCHAR(255)
zip -> VARCHAR(10)
when I exclude the zip from the procedure it works as It should so I guess there is a problem with the length of the varchar?
Does anyone knew a solution without changing the varchar length of zip from 10 to 255?
Incidence is presented by the value of the constant CONVERT_IF_BIGGER_TO_BLOB:
/**
CHAR and VARCHAR fields longer than this number of characters are converted
to BLOB.
Non-character fields longer than this number of bytes are converted to BLOB.
Comparisons should be '>' or '<='.
*/
#define CONVERT_IF_BIGGER_TO_BLOB 512 /* Used for CREATE ... SELECT */
mysql-server/sql/sql_const.h::52
and
16.3 The MEMORY Storage Engine
...
Support for variable-length data types (including BLOB and TEXT) not supported by MEMORY.
Example:
mysql> DROP TEMPORARY TABLE IF EXISTS `tmp`;
Query OK, 0 rows affected (0.00 sec)
mysql> CREATE TEMPORARY TABLE IF NOT EXISTS `tmp` ENGINE=MEMORY
-> SELECT SPACE(512) `tmp_col`;
Query OK, 1 row affected (0.00 sec)
Records: 1 Duplicates: 0 Warnings: 0
mysql> DROP TEMPORARY TABLE IF EXISTS `tmp`;
Query OK, 0 rows affected (0.00 sec)
mysql> CREATE TEMPORARY TABLE IF NOT EXISTS `tmp` ENGINE=MEMORY
-> SELECT SPACE(513) `tmp_col`;
ERROR 1163 (42000): The used table type doesn't support BLOB/TEXT columns
Try:
mysql> DROP TABLE IF EXISTS `tmp`, `customers`;
Query OK, 0 rows affected (0.00 sec)
mysql> CREATE TABLE IF NOT EXISTS `customers` (
-> `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY,
-> `lastName` VARCHAR(255) NOT NULL,
-> `firstName` VARCHAR(255) NOT NULL,
-> `zip` VARCHAR(10) NOT NULL
-> );
Query OK, 0 rows affected (0.00 sec)
mysql> CREATE TEMPORARY TABLE IF NOT EXISTS `tmp` (
-> `id` BIGINT UNSIGNED NOT NULL PRIMARY KEY,
-> `identify` VARCHAR(520) NOT NULL
-> ) ENGINE=MEMORY;
Query OK, 0 rows affected (0.00 sec)
mysql> INSERT INTO `tmp`
-> SELECT `id`, CONCAT(TRIM(`lastName`),
-> TRIM(`firstName`),
-> TRIM(`zip`)) `identify`
-> FROM `customers`
-> GROUP BY `id`, `identify`;
Query OK, 0 rows affected (0.00 sec)
Records: 0 Duplicates: 0 Warnings: 0
My experience with ENGINE=MEMORY was that the charset is also relevant.
varchar(65000) with charset latin1 worked; charset UTF8 did not
ENGINE=MEMORY DEFAULT CHARSET=latin1
I have a table as
mysql> show create table tbl_name\G
************* 1. row *************
Table: tbl_name
Create Table: CREATE TABLE `tbl_name` (
`name` char(15) CHARACTER SET latin1 DEFAULT NULL,
`id` int(5) NOT NULL AUTO_INCREMENT,
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=5 DEFAULT CHARSET=latin7 COLLATE=latin7_estonian_cs
1 row in set (0.00 sec)
The DATA In Table is as
mysql> select * from tbl_name;
name id
manaf 1
manaf 2
MANAF 3
MAnaf 4
4 rows in set (0.00 sec)
Now i want the Case Sensitivity in all the records
mysql> select distinct(name) from tbl_name;
It should return the 3 rows in result but it is returing just 1.why..??
As I have set the table collation as latin7_estonian_cs
try this
SELECT DISTINCT BINARY value FROM tableName
Reference
I have a 4 GB table and the default charset was set to utf8 though I am saving only latin1 characters.
I changed it to latin1 using alter table statement on a test machine.
The index file log_details.MYI was reduced by 5% while there was no difference noted in the data file, log_details.MYD
I have a few questions:
1) Should I alter the table on production? is it worth it?
2) Will it improve the select speed?
3) I guess I can have longer indexes once I change the default charset to latin1. Any
other advantage?
I have also noted that after changing the default charset using alter table statement, the varchar columns types were changed automatically.
Item_ID varchar(32) character set utf8 How do I avoid this?
mysql> create table char_test( id int, Item_ID varchar(32) ) default charset = utf8;
Query OK, 0 rows affected (0.02 sec)
mysql> insert into char_test values (1, 'abc');
Query OK, 1 row affected (0.00 sec)
mysql> show create table char_test\G
*************************** 1. row ***************************
Table: char_test
Create Table: CREATE TABLE `char_test` (
`id` int(11) default NULL,
`Item_ID` varchar(32) default NULL
) ENGINE=MyISAM DEFAULT CHARSET=utf8
1 row in set (0.00 sec)
mysql> alter table char_test default charset = latin1;
Query OK, 1 row affected (0.03 sec)
Records: 1 Duplicates: 0 Warnings: 0
mysql> show create table char_test\G
*************************** 1. row ***************************
Table: char_test
Create Table: CREATE TABLE `char_test` (
`id` int(11) default NULL,
`Item_ID` varchar(32) character set utf8 default NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1
1 row in set (0.00 sec)
Performance is probably the last thing you should care about. What character set is you client-side app using? What natural language is the info written on? Those are the questions you must make.
If you stick to Latin1 you won't be able to store Japanese characters but also some common chars like the € symbol. On the other side, using UTF-8 in the database may be useless (or plain wrong) if your application cannot handle multi-byte input.
How to create a table in memory which is identical to "regular" table? I want a copy without indexes and constraints. Pure SQL (no external tools).
This works (however indexes are created):
create table t_copy (like t_original)
This doesn't:
create table t_copy (like t_original) engine=memory
CREATE TABLE t_copy ENGINE=MEMORY SELECT * FROM t_original;
I actually tried it, it works !!!
mysql> show create table queue\G
*************************** 1. row ***************************
Table: queue
Create Table: CREATE TABLE `queue` (
`ndx` int(11) NOT NULL AUTO_INCREMENT,
`folderid` int(11) NOT NULL,
PRIMARY KEY (`ndx`)
) ENGINE=MyISAM AUTO_INCREMENT=2 DEFAULT CHARSET=latin1
1 row in set (0.02 sec)
mysql> create table queue_memory engine=MEMORY as select * from queue;
Query OK, 0 rows affected (0.05 sec)<BR>
Records: 0 Duplicates: 0 Warnings: 0<BR>
lwdba#localhost (DB test) :: show create table queue_memory\G
*************************** 1. row ***************************
Table: queue_memory
Create Table: CREATE TABLE `queue_memory` (
`ndx` int(11) NOT NULL DEFAULT '0',
`folderid` int(11) NOT NULL
) ENGINE=MEMORY DEFAULT CHARSET=latin1
1 row in set (0.00 sec)
Give it a try !!!
This should get you an empty table without any indexes.
create table t_copy
select *
from t_original
where 0
To create an in-memory table with the same structure AND indexes as the first table, try:
create table t_copy like t_original;
alter table t_copy engine=mem;