I have a simple 'people' table:
mysql> desc people;
+-------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------+--------------+------+-----+---------+-------+
| id | int | NO | PRI | NULL | |
| name | varchar(255) | YES | | NULL | |
+-------+--------------+------+-----+---------+-------+
2 rows in set (0.01 sec)
When I do show create table people; command, it shows me whole command that created my table:
mysql> show create table people;
+--------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+--------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| people | CREATE TABLE `people` (
`id` int NOT NULL,
`name` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci |
+--------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
In this command, I see a sentence ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci.
Question is: What does it mean? Especially, what does ENGINE=InnoDB mean?
InnoDB is a storage engine in MySQL. As explained in the documentation:
Storage engines are MySQL components that handle the SQL operations for different table types. InnoDB is the default and most general-purpose storage engine, and Oracle recommends using it for tables except for specialized use cases. (The CREATE TABLE statement in MySQL 8.0 creates InnoDB tables by default.)
MySQL Server uses a pluggable storage engine architecture that enables storage engines to be loaded into and unloaded from a running MySQL server.
As of the current state of art, MyISAM, that was the only available engine in the early 5.x days, is being abandonned and InnoDB is the de facto standard: it offers most typical functionalities that can be find in other high-profile databases, such as transactions and row-level locking.
InnoDB it's a storage engin that allow Foriegn Key ,Rollback , commit and ACID Compliant...
CHARSET = is a set of characters that are legal in a string.
uft8mb4 means that each character is stored as a maximum of 4 bytes in the UTF-8 encoding scheme.
ai refers accent insensitivity
ci refers to case insensitivity.
COLLATE = set of rules used to compare characters
Related
I have a MySQL 8.0.19 running in a Docker container and using the InnoDB engine. I have noticed that table integer field lengths are getting ignored.
The issue occurs with integer datatypes regardless if running a CREATE or ALTER query
CREATE TABLE `test` (
`id` int DEFAULT NULL,
`text_field` varchar(20) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`decimal_field` decimal(6,2) DEFAULT NULL,
`int_field` int DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
The lengths are showing as 0 in my MySQL client (Navicat), but the same occurs if checking in the console with SHOW FULL COLUMNS FROMtest;
mysql> SHOW FULL COLUMNS FROM `test`;
+---------------+--------------+--------------------+------+-----+---------+-------+---------------------------------+---------+
| Field | Type | Collation | Null | Key | Default | Extra | Privileges | Comment |
+---------------+--------------+--------------------+------+-----+---------+-------+---------------------------------+---------+
| id | int | NULL | YES | | NULL | | select,insert,update,references | |
| text_field | varchar(20) | utf8mb4_unicode_ci | YES | | NULL | | select,insert,update,references | |
| decimal_field | decimal(6,2) | NULL | YES | | NULL | | select,insert,update,references | |
| int_field | int | NULL | YES | | NULL | | select,insert,update,references | |
+---------------+--------------+--------------------+------+-----+---------+-------+---------------------------------+---------+
The Type column should be showing int(11) for the two integer fields, but it isn't.
Is this related to something in my MySQL settings, and if so, which variable would have to be changed?
This is a change documented in the MySQL 8.0.19 release notes:
Display width specification for integer data types was deprecated in
MySQL 8.0.17, and now statements that include data type definitions in
their output no longer show the display width for integer types, with
these exceptions:
The type is TINYINT(1). MySQL Connectors make the assumption that
TINYINT(1) columns originated as BOOLEAN columns; this exception
enables them to continue to make that assumption.
The type includes the ZEROFILL attribute.
This change applies to tables, views, and stored routines, and affects
the output from SHOW CREATE and DESCRIBE statements, and from
INFORMATION_SCHEMA tables.
For DESCRIBE statements and INFORMATION_SCHEMA queries, output is
unaffected for objects created in previous MySQL 8.0 versions because
information already stored in the data dictionary remains unchanged.
This exception does not apply for upgrades from MySQL 5.7 to 8.0, for
which all data dictionary information is re-created such that data
type definitions do not include display width. (Bug #30556657, Bug #97680)
The "length" of an integer column doesn't mean anything. A column of int(11) is the same as int(2) or int(40). They are all a fixed-size, 32-bit integer data type. They support the same minimum and maximum value.
The "length" of integer columns has been a confusing feature of MySQL for years. It's only a hint that affects the display width, not the storage or the range of values. Practically, it only matters when you use the ZEROFILL option.
mysql> create table t ( i1 int(6) zerofill, i2 int(12) zerofill );
Query OK, 0 rows affected (0.02 sec)
mysql> insert into t set i1 = 123, i2 = 123;
Query OK, 1 row affected (0.00 sec)
mysql> select * from t;
+--------+--------------+
| i1 | i2 |
+--------+--------------+
| 000123 | 000000000123 |
+--------+--------------+
1 row in set (0.00 sec)
So it's a good thing that the misleading integer "length" is now deprecated and removed. It has caused confusion for many years.
I can confirm that having upgraded AWS RDS to MySQL 8.0.19 that you can now sync using Navicat correctly.
However, PLEASE BE AWARE!!
When updating the id column, if auto_increment is set, Navicat removes auto_increment to change the length and then re-applies it at the end. This causes the auto_increment column to reassign the ids is sequencial order!
ALTER TABLE `database`.`table` MODIFY COLUMN `id` mediumint(0) NOT NULL FIRST;
...
...
ALTER TABLE `database`.`table` MODIFY COLUMN `id` mediumint(0) NOT NULL AUTO_INCREMENT;
If you are using table relationships and do not have the foreign keys setup properly, this will break your database!
Also, if you have id numbers of zero or below in your auto_increment column, this will cause the following error:
Result: 1062 - ALTER TABLE causes auto_increment resequencing,
resulting in duplicate entry '1' for key 'table.PRIMARY'
To avoid the above, you will need to manually change each tables id length to 0 and then save the changes before attempting to use the Navicat sync feature. When saving the changes using Navicat this will automatically change any other int column lengths to 0.
Please ensure you throughly test your changes on a copy of the database before trying to apply to any production databases.
We have the following table with about 200 million records:
CREATE TABLE IF NOT EXISTS `history` (
`airline` char(2) NOT NULL,
`org` char(3) NOT NULL,
`dst` char(3) NOT NULL,
`departat` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`arriveat` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`validon` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`price` int(11) NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=utf8
/*!50500 PARTITION BY RANGE COLUMNS(org)
(PARTITION p0 VALUES LESS THAN ('AHI') ENGINE = MyISAM,
PARTITION p1 VALUES LESS THAN ('ARO') ENGINE = MyISAM,
...
PARTITION p39 VALUES LESS THAN ('WMA') ENGINE = MyISAM,
PARTITION p40 VALUES LESS THAN (MAXVALUE) ENGINE = MyISAM) */;
--
-- Indexes for table `history`
--
ALTER TABLE `history`
ADD KEY `tail` (`org`,`dst`,`departat`);
We're doing bulk inserts of some VALUES frequently, usually up to 1000 records in simple INSERT queries, without any decoration such as ON DUPLICATE KEY (the index is not unique anyway).
Sometimes when I go to server status in phpMyAdmin, a see a bunch of INSERT statements waiting for each other, sometimes for up to 300-400 seconds. Nothing else seems to be going on the server at the particular time. We got 32 GB and otherwise excellent performance.
How to troubleshoot this issue? Thanks for help.
Probably first step is to do couple of test runs with profiling on.
Usually you'd do something like:
SET LOCAL PROFILING=ON;
-- run your INSERT, like:
INSERT INTO yourtable (id) VALUES (1),(2),(3);
SHOW PROFILES;
+----------+------------+------------------------------------------------+
| Query_ID | Duration | Query |
+----------+------------+------------------------------------------------+
| 1012 | 6.25220000 | INSERT INTO yourtable (id) VALUES (1),(2),(3); |
+----------+------------+------------------------------------------------+
This tells you very basic information, like duration of the query (6.25 sec in this case). To get the actual details you need to pull up the profile for said query:
SHOW PROFILE FOR QUERY 1025;
+------------------------------+----------+
| Status | Duration |
+------------------------------+----------+
| starting | 0.004356 |
| checking permissions | 0.000015 |
| Opening tables | 6.202999 |
| System lock | 0.000017 |
| init | 0.000342 |
| update | 0.023951 |
| Waiting for query cache lock | 0.000008 |
| update | 0.000007 |
| end | 0.000011 |
| query end | 0.019984 |
| closing tables | 0.000019 |
| freeing items | 0.000304 |
| logging slow query | 0.000006 |
| cleaning up | 0.000181 |
+------------------------------+----------+
You may notice that 'Opening tables' took very long. In this example query execution was delayed by locking the table (LOCK TABLES) by another process to delay the execution. Further information about the states is available in the manual.
set default 0 for timestamp fields and try
eg:
departat timestamp NOT NULL DEFAULT 0,
arriveat timestamp NOT NULL DEFAULT 0,
Timestamp will store a value like integer (mean timestamp of the passing time), it will not keep record like datetime.
In your case you have set the default as datetime format in timestap field type
There are several things you can do to optimize bulk inserts.
One of the things is setting off these variables if you are sure your data doesn't contain duplicates (don't forget to set them to 1 after the upload is complete):
SET AUTOCOMMIT = 0; SET FOREIGN_KEY_CHECKS = 0; SET UNIQUE_CHECKS = 0;
Also you need to check if no other users are accessing the table. You can also try using Innodb since it's said is better than MyISAM handling bulk inserts with data already on it.
Also you can check for fragmentation on your tables, sometimes the overhead the OS gives when assigning free space on fragmented drives is the cause of the delay.
MariaDB 5.3 introduced dynamic columns. From my understanding the next version of mysql should have similar features mariadb has?
I am currently running mysql 5.5.9 and I wanted to mess around with dynamic columns per row.
So I read up on the mysql website, in order to get this working:
innodb_file_format should be set to Barracuda.
Done.
--------------
show variables like "%innodb_file%"
--------------
+--------------------------+-----------+
| Variable_name | Value |
+--------------------------+-----------+
| innodb_file_format | Barracuda |
| innodb_file_format_check | ON |
| innodb_file_format_max | Barracuda |
| innodb_file_per_table | ON |
+--------------------------+-----------+
4 rows in set (0.00 sec)
I then go ahead and create my table for testing
CREATE TABLE `test` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`dyn` blob,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 ROW_FORMAT=DYNAMIC
I then try to insert
insert into test(`dyn`) VALUES (COLUMN_CREATE(1, "black", 3, "Linux"))
I get the following error:
FUNCTION db.COLUMN_CREATE does not exist
So my question is does mysql not offer these functions? Should I switch to mariadb for testing?
Based on the documentation, it seems that MySQL does not offer this feature.
The dynamic row format actually offers a different method for storing rows that may be more efficient in some cases, not a way to have a variable table schema:
When innodb_file_format is set to Barracuda and a table is created
with ROW_FORMAT=DYNAMIC or ROW_FORMAT=COMPRESSED, long column values
are stored fully off-page, and the clustered index record contains
only a 20-byte pointer to the overflow page.
If you need such a flexible schema, check out the entity-attribute-value pattern.
I got myself into a MySQL design scalability issue. Any help would be greatly appreciated.
The requirements:
Storing users' SOCIAL_GRAPH and USER_INFO about each user in their social graph. Many concurrent reads and writes per second occur. Dirty reads acceptable.
Current design:
We have 2 (relevant) tables. Both InnoDB for row locking, instead of table locking.
USER_SOCIAL_GRAPH table that maps a logged in (user_id) to another (related_user_id). PRIMARY key composite user_id and related_user_id.
USER_INFO table with information about each related user. PRIMARY key is (related_user_id).
Note 1: No relationships defined.
Note 2: Each table is now about 1GB in size, with 8 million and 2 million records, respectively.
Simplified table SQL creates:
CREATE TABLE `user_social_graph` (
`user_id` int(10) unsigned NOT NULL,
`related_user_id` int(11) NOT NULL,
PRIMARY KEY (`user_id`,`related_user_id`),
KEY `user_idx` (`user_id`)
) ENGINE=InnoDB;
CREATE TABLE `user_info` (
`related_user_id` int(10) unsigned NOT NULL,
`screen_name` varchar(20) CHARACTER SET latin1 DEFAULT NULL,
[... and many other non-indexed fields irrelevant]
`last_updated` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`related_user_id`),
KEY `last_updated_idx` (`last_updated`)
) ENGINE=InnoDB;
MY.CFG values set:
innodb_buffer_pool_size = 256M
key_buffer_size = 320M
Note 3: Memory available 1GB, these 2 tables are 2GBs, other innoDB tables 3GB.
Problem:
The following example SQL statement, which needs to access all records found, takes 15 seconds to execute (!!) and num_results = 220,000:
SELECT SQL_NO_CACHE COUNT(u.related_user_id)
FROM user_info u LEFT JOIN user_socialgraph u2 ON u.related_user_id = u2.related_user_id
WHERE u2.user_id = '1'
AND u.related_user_id = u2.related_user_id
AND (NOT (u.related_user_id IS NULL));
For a user_id with a count of 30,000, it takes about 3 seconds (!).
EXPLAIN EXTENDED for the 220,000 count user. It uses indices:
+----+-------------+-------+--------+------------------------+----------+---------+--------------------+--------+----------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+--------+------------------------+----------+---------+--------------------+--------+----------+--------------------------+
| 1 | SIMPLE | u2 | ref | user_user_idx,user_idx | user_idx | 4 | const | 157320 | 100.00 | Using where |
| 1 | SIMPLE | u | eq_ref | PRIMARY | PRIMARY | 4 | u2.related_user_id | 1 | 100.00 | Using where; Using index |
+----+-------------+-------+--------+------------------------+----------+---------+--------------------+--------+----------+--------------------------+
How do we speed these up without setting innodb_buffer_pool_size to 5GB?
Thank you!
The user_social_graph table is not indexed correctly !!!
You have ths:
CREATE TABLE user_social_graph
(user_id int(10) unsigned NOT NULL,
related_user_id int(11) NOT NULL,
PRIMARY KEY (user_id,related_user_id),
KEY user_idx (user_id))
ENGINE=InnoDB;
The second index is redundant since the first column is user_id. You are attempting to join the related_user_id column over to the user_info table. That column needed to be indexed.
Change user_social_graphs as follows:
CREATE TABLE user_social_graph
(user_id int(10) unsigned NOT NULL,
related_user_id int(11) NOT NULL,
PRIMARY KEY (user_id,related_user_id),
UNIQUE KEY related_user_idx (related_user_id,user_id))
ENGINE=InnoDB;
This should change the EXPLAIN PLAN. Keep in mind that the index order matters depending the the way you query the columns.
Give it a Try !!!
What is the MySQL version? Its manual contains important information for speeding up statements and code in general;
Change your paradigm to a data warehouse capable to manage till terabyte table. Migrate your legacy MySQL data base with free tool or application to the new paradigm. This is an example: http://www.infobright.org/Downloads/What-is-ICE/ many others (free and commercial).
PostgreSQL is not commercial and there a lot of tools to migrate MySQL to it!
I'm looking at this Mysql question link:
And I can't repeat the behavior described in the answer to that question.
I tried creating tables with CHAR and VARCHAR column of various lengths and it doesn't matter what length - SHOW CREATE TABLE always return the data type that I've originally defined.
So - no CHAR->VARCHAR switching is going on.
Is answer to the question below only partially correct (I'm talking only about items 1 and 2)?
Q:
When you create a table, and then run SHOW CREATE TABLE on it, you occasionally get different results than what you typed in. What does MySQL modify in your newly created tables?
A (supposedly):
VARCHARs with length less than 4 become CHARs
CHARs with length more than 3 become VARCHARs.
NOT NULL gets added to the columns declared as PRIMARY KEYs
Default values such as NULL are specified for each column
There is a page in the MySQL's manual that answers some of your questions : 12.1.14.2. Silent Column Specification Changes.
Quoting some portions that correspond to items you posted in your question :
For item 3 :
Columns that are part of a PRIMARY
KEY are made NOT NULL even if not
declared that way.
About the size of varchar columns (not exactly one of your items, though) :
If strict SQL mode is not enabled, a
VARCHAR column with a length
specification greater than 65535 is
converted to TEXT, and a VARBINARY
column with a length specification
greater than 65535 is converted to
BLOB. Otherwise, an error occurs in
either of these cases.
And that page ends with the following sentence :
To see whether MySQL used a data type
other than the one you specified,
issue a DESCRIBE or SHOW CREATE
TABLE statement after creating or
altering the table.
So I'm guessing you might expect some additional differences, that are not listed.
Doing a quick test, here's a create table statement :
create table test_2 (
id int primary key,
blah_vc varchar(2),
blah_c char(5)
) engine=InnoDb;
And the table that's created gives :
mysql> desc test_2;
+---------+------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------+------------+------+-----+---------+-------+
| id | int(11) | NO | PRI | NULL | |
| blah_vc | varchar(2) | YES | | NULL | |
| blah_c | char(5) | YES | | NULL | |
+---------+------------+------+-----+---------+-------+
3 rows in set (0.00 sec)
mysql> show create table test_2;
+--------+--------------------------------------------+
| Table | Create Table |
+--------+--------------------------------------------+
| test_2 | CREATE TABLE `test_2` (
`id` int(11) NOT NULL,
`blah_vc` varchar(2) DEFAULT NULL,
`blah_c` char(5) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 |
+--------+--------------------------------------------+
1 row in set (0.00 sec)
So :
no : varchar has not been transformed to char
no : char has not been transformed to varchar
yes : not null has been added to primary key
well, that one is funny : describe says it hasn't,
but show create table indicates it has...
Anyway : it makes sense, for a primary key column, to not be nullable.
yes : null is specified as default for columns that can be null.
That question is pretty old, written back in the days of MySQL 4.
As of MySQL 5.0, 1 & 2 from that list are no longer true.