Is there a way the status command be called on another connection. Something like:
status 231073
where the number is the ID from SHOW PROCESSLIST. In particular I'm interested in connection/client character sets.
EDIT:
By status I mean the command you call from the MySQL shell (not show status)
You could use the following to capture info of your connections, say, at the start of your app or at anytime. For instance max_allowed_packet could change during the client run lifecycle. So, in short, have in as part of your app startup.
Then you have collected info on your client connections, such as character set settings.
Schema:
DROP TABLE IF EXISTS connectionSnapshot;
CREATE TABLE connectionSnapshot
( -- based off of (more or less) mysql 5.6 show create table information_schema.PROCESSLIST;
-- 3 columns at least
-- TODO: compare to same structure on mysql 5.7
id INT AUTO_INCREMENT PRIMARY KEY,
connId BIGINT UNSIGNED NOT NULL,
user VARCHAR(16) NOT NULL,
host VARCHAR(64) NOT NULL,
login VARCHAR(100) NOT NULL,
-- the followed just dreamt up:
character_set_client VARCHAR(100) NOT NULL,
character_set_connection VARCHAR(100) NOT NULL,
character_set_results VARCHAR(100) NOT NULL,
collation_connection VARCHAR(100) NOT NULL,
lc_time_names VARCHAR(100) NOT NULL,
max_allowed_packet BIGINT NOT NULL,
time_zone VARCHAR(100) NOT NULL,
theWhen DATETIME NOT NULL
);
Stored Proc:
DROP PROCEDURE IF EXISTS uspConnectionSnapshotMe;
DELIMITER $$
CREATE PROCEDURE uspConnectionSnapshotMe()
BEGIN
DECLARE lconnId INT;
DECLARE luser VARCHAR(16);
DECLARE lhost VARCHAR(64);
SELECT connection_id() INTO lconnID;
SELECT USER,HOST into luser,lhost FROM information_schema.PROCESSLIST WHERE ID=lconnId;
INSERT connectionSnapshot (connId,user,host,login,character_set_client,character_set_connection,character_set_results,
collation_connection,lc_time_names,max_allowed_packet,time_zone,theWhen) VALUES
(lconnId,luser,lhost,current_user(),##session.character_set_client,##session.character_set_connection,
##session.character_set_results,##session.collation_connection,
##session.lc_time_names,##session.max_allowed_packet,##session.time_zone,now());
END$$
DELIMITER ;
-- ****************************************************************************************
Test:
call uspConnectionSnapshotMe();
Results:
select * from connectionSnapshot;
+----+--------+------+-----------------+----------------+----------------------+--------------------------+-----------------------+----------------------+---------------+--------------------+-----------+---------------------+
| id | connId | user | host | login | character_set_client | character_set_connection | character_set_results | collation_connection | lc_time_names | max_allowed_packet | time_zone | theWhen |
+----+--------+------+-----------------+----------------+----------------------+--------------------------+-----------------------+----------------------+---------------+--------------------+-----------+---------------------+
| 1 | 3825 | root | localhost:4660 | root#localhost | utf8 | utf8 | utf8 | utf8_general_ci | en_US | 4194304 | SYSTEM | 2016-09-08 02:40:18 |
| 2 | 37007 | root | localhost:52071 | root#localhost | utf8 | utf8 | cp850 | utf8_general_ci | en_US | 4194304 | SYSTEM | 2016-09-08 02:44:17 |
+----+--------+------+-----------------+----------------+----------------------+--------------------------+-----------------------+----------------------+---------------+--------------------+-----------+---------------------+
Related
I'm trying to create a local Docker image that represents a development version of my production database. I've started a MariaDB container with the same version as production (10.1) and loaded a schema based on a mysqldump of production.
However when I start inserting data, I get weird hex values in every column.
The production data looks like:
select * from projects where p_project = 'Water';
+-----------+----------------+------------+----------+-------------+---------+----------+----------+---------+--------------------+
| p_project | p_timestamp | p_wikipage | p_parent | p_shortname | p_count | p_qcount | p_icount | p_scope | p_upload_timestamp |
+-----------+----------------+------------+----------+-------------+---------+----------+----------+---------+--------------------+
| Water | 20200305045828 | NULL | NULL | NULL | 841 | 644 | 554 | 0 | 20190813001026 |
+-----------+----------------+------------+----------+-------------+---------+----------+----------+---------+--------------------+
While the dev database looks like:
select * from projects where p_project = 'Water';
+----------------------+--------------------------------+------------------------+--------------------+--------------------------+---------+----------+----------+---------+----------------------------------------+
| p_project | p_timestamp | p_wikipage | p_parent | p_shortname | p_count | p_qcount | p_icount | p_scope | p_upload_timestamp |
+----------------------+--------------------------------+------------------------+--------------------+--------------------------+---------+----------+----------+---------+----------------------------------------+
| 0x5761746572 | 0x3230323030333035303532333538 | 0x | 0x | 0x | NULL | NULL | NULL | 0 | 0x |
+----------------------+--------------------------------+------------------------+--------------------+--------------------------+---------+----------+----------+---------+----------------------------------------+
The production database is defined as so:
show create database enwp10;
enwp10 | CREATE DATABASE `enwp10` /*!40100 DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci */ |
And dev is:
show create database enwp10_dev;
| enwp10_dev | CREATE DATABASE `enwp10_dev` /*!40100 DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci */ |
Here's the table definition:
show create table projects;
----------------------------------------------------------------------+
| projects | CREATE TABLE `projects` (
`p_project` varbinary(63) NOT NULL,
`p_timestamp` binary(14) NOT NULL,
`p_wikipage` varbinary(255) DEFAULT NULL,
`p_parent` varbinary(63) DEFAULT NULL,
`p_shortname` varbinary(255) DEFAULT NULL,
`p_count` int(10) unsigned DEFAULT '0',
`p_qcount` int(10) unsigned DEFAULT '0',
`p_icount` int(10) unsigned DEFAULT '0',
`p_scope` int(10) unsigned NOT NULL DEFAULT '0',
`p_upload_timestamp` binary(14) DEFAULT NULL,
PRIMARY KEY (`p_project`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 |
+----------
The interesting part is that the select works in both cases, so maybe it's my mysql client display settings that are off?
My kwargs to my pymysql connect call include:
'charset': None,
'use_unicode': False,
Which has always worked perfectly fine in production. Additionally, I have set my global server charset to 'utf8mb4' and server collation to 'utf8mb4_unicode_ci' for the development database.
All of the tables in the development database are defined with CHARSET=latin1, but that's also true for the production tables.
Any ideas what is going on here? Thanks!
The problem was that I was using the mysql client binary that was installed by mysql, and the protocol must be slightly different with this old (10.1) version of MariaDB. See comment on question.
The solution was to install MariaDB for MacOS, at the proper version, and use /usr/local/mariadb/server/bin/mariadb to connect to my docker database.
I have the following table on MySQL. I am using 5.6.32. The table contains about ~40 million records. I am only sharing columns which I feel are necessary to understand the issue.
Table Structure
create table `random` (
`id` bigint(20) not null auto_increment,
`some_id` bigint(20) not null,
`latitude` decimal(20,14) default null,
`longitude` decimal(20,14) default null,
`new_column` varchar(255) collate utf8_unicode_ci default null,
primary key (`id`)
) engine=innodb auto_increment=40878872 default charset=utf8 collate=utf8_unicode_ci;
So, I added a new column in this table called new_column varchar(255). But, when I do length(new_column), there are entries which have more than 255 characters.
The actual value being inserted:
random*GS02,355234054262743,GPS:356728;A;N33.614073;E77.063096;0;0;230118,STT:400;0,ADC:0���&�������������r�������r�������r������*GS02,39233054663793,GPS:173158;A;N33.614057;E77.0263201;0;0;210118,STT:200;0,ADC:0;24.7;1;29.9;2;4.2;3;0.0
On the MySQL Master (say, machine #1, my application was able to insert this value in new_column in the table without an issue. I have a MySQL slave (say, machine #2) using native MySQL replication and it was also able to replicate this record easily. But then I have another slave replicating from machine #2 which is using tungsten replicator. Whenever there is a string which is more than 255 characters, tungsten throws the following error and replication breaks
pendingError : Event application failed: seqno=2395306016 fragno=0 message=java.sql.SQLDataException: Data too long for column 'new_column' at row 1
pendingExceptionMessage: java.sql.SQLDataException: Data too long for column 'new_column' at row 1
EDIT:
Variables on Master and both Slaves are set to
+--------------------------+----------------------------+
| Variable_name | Value |
+--------------------------+----------------------------+
| character_set_client | latin1 |
| character_set_connection | latin1 |
| character_set_database | utf8 |
| character_set_filesystem | binary |
| character_set_results | latin1 |
| character_set_server | utf8 |
| character_set_system | utf8 |
| character_sets_dir | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
Collation while creating the table is set to utf8_unicode_ci on all instances.
Why is it that MySQL native replication is allowing more characters to be written to the column? And why is it that Tungsten replicator is preventing it?
I'm using MariaDb server (Ver 15.1 Distrib 10.2.7-MariaDB).
When I execute
CREATE TABLE `my_table` (
`id` INT NOT NULL,
`name` NVARCHAR(64) NULL,
PRIMARY KEY (`id`)
);
Describe output:
MariaDB [db]> describe my_table;
+-------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------+-------------+------+-----+---------+-------+
| id | int(11) | NO | PRI | NULL | |
| name | varchar(64) | YES | | NULL | |
+-------+-------------+------+-----+---------+-------+
2 rows in set (0.00 sec)
Why there is no error, and "name" column datatype is varchar (not nvarchar)?
db schema details:
Default collation: utf8_general_ci
Default characterset: utf8
NVARCHAR is a synonym for VARCHAR in MySQL/MariaDB. But you need to add the CHARACTER SET utf8mb4 to be sure that you get full UTF-8 support.
What you show as the default for that database is only the subset, called 'utf8'. It will not handle Emoji or some of Chinese.
Can I use UTF-8 names in column name on data base? Like example here:
$zapytaj = mysql_query("SELECT * FROM users WHERE `użytkownicy` = '$nazwaużytkownika' ");
This give me error:
Unknown column 'użytkownicy' in 'where clause'
Can someone explain why this is not working?
mysql> SHOW VARIABLES LIKE 'character%';
+--------------------------+-------------
| Variable_name | Value
+--------------------------+-------------
| character_set_client | utf8mb4
| character_set_connection | utf8mb4
| character_set_database | utf8mb4
| character_set_filesystem | binary
| character_set_results | utf8mb4
| character_set_server | latin1
| character_set_system | utf8
mysql> SELECT COLUMN_NAME, HEX(COLUMN_NAME)
FROM information_schema.columns WHERE table_name = "so31349641";
+--------------+--------------------------+
| COLUMN_NAME | HEX(COLUMN_NAME) |
+--------------+--------------------------+
| id | 6964 |
| użytkownicy | 75C5BC79746B6F776E696379 | -- Note the C5BC for ż
| hasło | 686173C5826F | -- and the C582 for ł
+--------------+--------------------------+
If I delete `` from użytkownicy I see this error:
You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '�ytkownicy = 'xxx'' at line 1
Maybe PHP file are don't have UTF8 coding? How to check this file in PHPStorm?
!SOLUTION!
If You have this error just change mysql to PDO that should fix Your problem.
To answer your stated question, column names are utf8:
mysql> SHOW CREATE TABLE information_schema.columns\G
*************************** 1. row ***************************
Table: COLUMNS
Create Table: CREATE TEMPORARY TABLE `COLUMNS` (
`TABLE_CATALOG` varchar(512) NOT NULL DEFAULT '',
`TABLE_SCHEMA` varchar(64) NOT NULL DEFAULT '',
`TABLE_NAME` varchar(64) NOT NULL DEFAULT '',
`COLUMN_NAME` varchar(64) NOT NULL DEFAULT '', -- NOTE --
`ORDINAL_POSITION` bigint(21) unsigned NOT NULL DEFAULT '0',
`COLUMN_DEFAULT` longtext,
`IS_NULLABLE` varchar(3) NOT NULL DEFAULT '',
`DATA_TYPE` varchar(64) NOT NULL DEFAULT '',
`CHARACTER_MAXIMUM_LENGTH` bigint(21) unsigned DEFAULT NULL,
`CHARACTER_OCTET_LENGTH` bigint(21) unsigned DEFAULT NULL,
`NUMERIC_PRECISION` bigint(21) unsigned DEFAULT NULL,
`NUMERIC_SCALE` bigint(21) unsigned DEFAULT NULL,
`DATETIME_PRECISION` bigint(21) unsigned DEFAULT NULL,
`CHARACTER_SET_NAME` varchar(32) DEFAULT NULL,
`COLLATION_NAME` varchar(32) DEFAULT NULL,
`COLUMN_TYPE` longtext NOT NULL,
`COLUMN_KEY` varchar(3) NOT NULL DEFAULT '',
`EXTRA` varchar(30) NOT NULL DEFAULT '',
`PRIVILEGES` varchar(80) NOT NULL DEFAULT '',
`COLUMN_COMMENT` varchar(1024) NOT NULL DEFAULT ''
) ENGINE=MyISAM DEFAULT CHARSET=utf8
To get to the root of the implied question ("Why does the query fail"), let's see
SHOW VARIABLES LIKE 'character%';
Edit
Well, something un-obvious going on. This works for me:
mysql> create table so31349641 (
id int(11) NOT NULL AUTO_INCREMENT,
użytkownicy varchar(24) NOT NULL,
hasło varchar(24) NOT NULL, PRIMARY KEY (id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
mysql> INSERT INTO so31349641 VALUES (1, 'a', 'b');
mysql> SELECT * FROM so31349641 WHERE użytkownicy = 'a';
+----+--------------+--------+
| id | użytkownicy | hasło |
+----+--------------+--------+
| 1 | a | b |
+----+--------------+--------+
This seems ordinary:
mysql> SHOW VARIABLES LIKE 'character%';
+--------------------------+-------------
| Variable_name | Value
+--------------------------+-------------
| character_set_client | utf8
| character_set_connection | utf8
| character_set_database | latin1
| character_set_filesystem | binary
| character_set_results | utf8
| character_set_server | latin1
| character_set_system | utf8
Looking in the IS:
mysql> SELECT COLUMN_NAME, HEX(COLUMN_NAME)
FROM information_schema.columns WHERE table_name = "so31349641";
+--------------+--------------------------+
| COLUMN_NAME | HEX(COLUMN_NAME) |
+--------------+--------------------------+
| id | 6964 |
| użytkownicy | 75C5BC79746B6F776E696379 | -- Note the C5BC
| hasło | 686173C5826F | -- and the C582 for ł
+--------------+--------------------------+
That is as I would expect it.
My char% values are different than yours, but I think we are both "OK" for this situation.
Try a SELECT on the information_schema similar to what I did.
Next, what is your client? PHP? Something else? Perhaps the encoding is incorrect in the client.
(Rather than trying to use HTML tags in a Comment, Edit your original question with the added info.)
It looks like UTF-8 in SQL is not default, but tables/databases can be changed to be so.
Some potentially helpful links:
The mySQL documentation on charsets:
https://dev.mysql.com/doc/refman/5.0/en/charset.html
A SO question on determining the charset:
determining the character set of a table / database?
On changing the charset: http://makandracards.com/makandra/2529-show-and-change-mysql-default-character-set
Hope this helps.
I'm storing Unicode strings in a MySQL database with a Django web application. I can store Unicode data fine, but when querying, I find that é and e are treated as if they were the same character:
In [1]: User.objects.filter(last_name='Jildén')
Out[1]: [<User: Anders Jildén>]
In [2]: User.objects.filter(last_name='Jilden')
Out[2]: [<User: Anders Jildén>]
This is also the case when using the MySQL shell directly:
mysql> select last_name from auth_user where last_name = 'Jildén';
+-----------+
| last_name |
+-----------+
| Jildén |
+-----------+
1 row in set (0.00 sec)
mysql> select last_name from auth_user where last_name = 'Jilden';
+-----------+
| last_name |
+-----------+
| Jildén |
+-----------+
1 row in set (0.01 sec)
Here are the database charset settings:
mysql> SHOW variables LIKE '%character_set%';
+--------------------------+------------------------------------------------------+
| Variable_name | Value |
+--------------------------+------------------------------------------------------+
| character_set_client | latin1 |
| character_set_connection | latin1 |
| character_set_database | utf8 |
| character_set_filesystem | binary |
| character_set_results | latin1 |
| character_set_server | latin1 |
| character_set_system | utf8 |
| character_sets_dir | /usr/local/Cellar/mysql/5.1.54/share/mysql/charsets/ |
+--------------------------+------------------------------------------------------+
here's the table schema:
CREATE TABLE `auth_user` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`username` varchar(30) CHARACTER SET utf8 NOT NULL,
`first_name` varchar(30) CHARACTER SET utf8 NOT NULL,
`last_name` varchar(30) CHARACTER SET utf8 NOT NULL,
`email` varchar(200) CHARACTER SET utf8 NOT NULL,
`password` varchar(128) CHARACTER SET utf8 NOT NULL,
`is_staff` tinyint(1) NOT NULL,
`is_active` tinyint(1) NOT NULL,
`is_superuser` tinyint(1) NOT NULL,
`last_login` datetime NOT NULL,
`date_joined` datetime NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `username` (`username`)
) ENGINE=InnoDB AUTO_INCREMENT=7952 DEFAULT CHARSET=utf8 COLLATE=utf8_bin
and here are the options I'm passing via Django's DATABASES setting:
DATABASES = {
'default': {
# ...
'OPTIONS': {
'charset': 'utf8',
'init_command': 'SET storage_engine=INNODB;',
},
},
}
Note that I have tried setting the table collation to utf8_bin, with no effect:
mysql> alter table auth_user collate utf8_bin;
mysql> select last_name from auth_user where last_name = 'Jilden';
+-----------+
| last_name |
+-----------+
| Jildén |
+-----------+
1 row in set (0.00 sec)
How can I get MySQL to treat these as different characters?
You were nearly there when you changed the table collation, but not quite. In MySQL, each column in a table has its own character set and collation. The table has its own character set and collation, but this does not override the column collations; it only determines what the collation will be for new columns that are added for which you don't specify the collation. So you haven't changed the collation of the column that you're interested in.
ALTER TABLE tablename MODIFY columnname
varchar(???) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL
You need to set a collation that treats diacritics as significant. Try using utf8_bin
I think it would be important to know the charset of the table and the field you are querying.
The answer to your question could be found here
http://dev.mysql.com/doc/refman/5.0/en/charset-unicode-sets.html
Maybe the field you are querying has the utf8_general_ci charset.
To obtain what you want you should set the charset of that field as utf8_unicode_ci
Remember that, as the manual says, queries on utf8_unicode_ci charset fields are slower than queries on utf8_general_ci fields