If I execute this query:
CREATE TABLE `varchar_test1` (
`id` tinyint(1) NOT NULL,
`cloumn_1` varchar(21844) NOT NULL) ENGINE=InnoDB DEFAULT CHARSET=utf8;
it is ok.
If I then execute this:
ALTER TABLE `varchar_test1` ADD COLUMN `cloumn_2` varchar(21844) NOT NULL;
I get an error:
ERROR 1118 (42000): Row size too large. The maximum row size for the used table type, not counting BLOBs, is 65535. This includes storage overhead, check the manual. You have to change some columns to TEXT or BLOBs
If I execute this:
CREATE TABLE `varchar_test2` (
`id` int NOT NULL AUTO_INCREMENT,
`cloumn_1` varchar(21844) NOT NULL,
PRIMARY KEY (`id`)) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8;
I get:
ERROR 1118 (42000): Row size too large. The maximum row size for the used table type, not counting BLOBs, is 65535. This includes storage overhead, check the manual. You have to change some columns to TEXT or BLOBs
Why?
Running mysql --version returns
mysql Ver 14.14 Distrib 5.7.17, for macos10.12 (x86_64) using EditLine wrapper
Your problem is that your columns store multi-byte values and thus exceed the maximum row size.
As explained in the docs, MySQL tables have a maximum row size of 65,535 bytes, regardless of the engine you use. Adding your two varchar(21844) columns means you go over this limit. This happens because the CHARSET on your table is utf8 (currently an alias for utf8mb3), so each character of a varchar in your table takes a maximum of 3 bytes (plus 2 bytes to store the length). That's fine with one column of 21,844 characters, but not with two.
To fix this, use TEXT (or TINYTEXT, MEDIUMTEXT, etc.) instead of VARCHAR columns. This will cause the values to be stored separately, so each of your columns will actually only contribute a few bytes to the row size. (This is also explained in the docs.)
Also, FYI: the spelling is column, not cloumn.
Related
I have a problem to create i18n table for CakePHP 3 Translate Behavior. So I have my database in phpmyadmin and when I want to execute this piece of code from the official cookbook :
CREATE TABLE i18n (
id int NOT NULL auto_increment,
locale varchar(6) NOT NULL,
model varchar(255) NOT NULL,
foreign_key int(10) NOT NULL,
field varchar(255) NOT NULL,
content text,
PRIMARY KEY (id),
UNIQUE INDEX I18N_LOCALE_FIELD(locale, model, foreign_key, field),
INDEX I18N_FIELD(model, foreign_key, field)
);
PhpMyAdmin say :
1071 - Specified key was too long; max key length is 767 bytes
I'm in uft8_unicode_ci. Should I go for utf8_general_ci?
Thanks for your help.
There is no difference in size requirements between utf8_unicode and utf8_general, they only differ with regards to sorting.
By default the index (key prefix) limit is 767 bytes for InnoDB tables (and 1000 bytes for MyISAM), if applicable enable the innodb_large_prefix option (it is enabled by default as of MySQL 5.7) which raises the limit to 3072 bytes, or make the VARCHAR columns smaller, and/or change their collation, the locale column (which holds ISO locale/country codes) surely doesn't use unicode characters, and chances are that your model and column/field names also only use ASCII characters, and that their names are way below 255 characters in length.
With an ASCII collation the VARCHAR columns require only 1 byte per char, unlike with UTF-8, which can require up to 3 bytes (or 4 bytes for the mb4 variants), which alone already causes the index size limit to be exceeded (3 * 255 * 2 = 1530).
See also
MySQL 5.7 Manual > Character Sets and Collations
MySQL 5.7 Manual > Limits on InnoDB Tables > Maximums and Minimums
MySQL 5.7 Manual > InnoDB Startup Options and System Variables > innodb_large_prefix
I have limited my request with :
model varchar(85) NOT NULL,
field varchar(85) NOT NULL,
model and field at 85, I think it's enought, I mysql accept it.
Hope that will help someone.
I've created a view which uses GROUP_CONCAT to concatenate results from a query on products column with data type of 'varchar(7) utf8_general_ci' in a column named concat_products.
The problem is that MySQL truncates value of "concat_products" column.
phpMyAdmin says the data type of "concat_products" column is varchar(341) utf8_bin
Table products:
CREATE TABLE `products`(
`productId` tinyint(2) unsigned NOT NULL AUTO_INCREMENT,
`product` varchar(7) COLLATE utf8_general_ci NOT NULL,
`price` mediumint(5) unsigned NOT NULL,
PRIMARY KEY (`productId`)
) ENGINE=InnoDB AUTO_INCREMENT=28 DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci
The "concat_products_vw" view:
CREATE VIEW concat_products_vw AS
SELECT
`userId`,
GROUP_CONCAT(CONCAT_WS('_', `product`, `productId`, `price`)
ORDER BY `productId` ASC SEPARATOR '*') AS concat_products
FROM
`users`
LEFT JOIN `products`
ON `users`.`accountBalance` >= `product`.`price`
GROUP BY `productId`
According to MySQL manual:
Values in VARCHAR columns are variable-length strings
Length can be specified as a value from 1 to 255 before MySQL 4.0.2 and 0 to 255 as of MySQL 4.0.2.
EDIT
Values in VARCHAR columns are variable-length strings. The length can be specified as a value from 0 to 65,535.
Why MySQL specifies more than 255 characters for varchar "concat_products" column? (solved!)
Why uf8_bin instead of utf8_general_ci?
Is it possible to change the data type of a column in a view for example in my case to text for "concat_products" column?
If not what can I do to prevent MySQL from truncating "concat_products" column?
As I already wrote in an earlier comment, the MySQL manual says:
Values in VARCHAR columns are variable-length strings. The length can
be specified as a value from 0 to 65,535.
So the problem is not with the data type of the field.
The MySQL manual also says:
The result is truncated to the maximum length that is given by the
group_concat_max_len system variable, which has a default value of
1024. The value can be set higher, although the effective maximum length of the return value is constrained by the value of
max_allowed_packet. The syntax to change the value of
group_concat_max_len at runtime is as follows, where val is an
unsigned integer:
SET [GLOBAL | SESSION] group_concat_max_len = val;
Your options for changing the value of group_concat_max_len are:
changing the value at MySQL startup by appending this to the command:
--group_concat_max_len=your_value_here
adding this line in your MySQL configuration file (mysql.ini): group_concat_max_len=your_value_here
running this command after MySQL startup:
SET GLOBAL group_concat_max_len=your_value_here;
running this command after opening a MySQL connection:
SET SESSION group_concat_max_len=your_value_here;
Documentation: SET, Server System Variables: group_concat_max_len
As Jocelyn mentioned, the size of a GROUP_CONCAT() result is bounded by group_concat_max_len, however there is an additional interaction with ORDER BY that results in a further truncation to 1/3 of group_concat_max_len. For an example, see this related answer.
The default value for group_concat_max_len is 1024, and 1024 / 3 = 341 probably explains why the type of concat_products shows up as varchar(341) in the original example. If you were to remove the GROUP BY productId clause, concat_products should show up as varchar(1024).
I have not found this interaction between GROUP_CONCAT() and ORDER BY mentioned in the MySQL Manual, but it affects at least MySQL Server 5.1.
Year 2023, on Linux Server you can edit my.cnf file and then set:
[mysqld]
group_concat_max_len = 2048
I'm on a MYSQL database attempting to alter my table encoded in latin1 to UTF-8. The tables name is Journalist that has 12 columns listed as varchar with max length of 3000. This is the sql i'm running.
ALTER TABLE `journalist` CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;
The error I'm receiving
Row size too large. The maximum row size for the used table type, not counting BLOBs, is 65535. This includes storage overhead, check the manual. You have to change some columns to TEXT or BLOBs
Do I have to alter the size of the table before I run this conversion query?
and/or how I might accomplish this encoding alteration otherwise?
I did what #Wrikken suggested. I deleted my table and lowered varchar's max_length attributes to 1500 from 3000. I then ran this SQL on my new empty table
ALTER TABLE `table_name` CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;
from here, I repopulated it with my backup table, using a script.
To answer the question:
Lower varchar max_length limits
Or change varchar fields to LONGTEXT, or BLOBS
Error:
1071 - Specified key was too long; max key length is 1000 bytes
CREATE TABLE `phppos_modules_actions` (
`action_id` VARCHAR( 255 ) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL ,
`module_id` VARCHAR( 255 ) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL ,
`action_name_key` VARCHAR( 255 ) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL ,
`sort` INT NOT NULL ,
PRIMARY KEY ( `action_id` , `module_id` )
) ENGINE = INNODB CHARACTER SET utf8 COLLATE utf8_unicode_ci;
I know the error occurs because of 255x2x3 (3 bytes per character)
This doesn't happen on all installations. What setting can I change?
NO_ENGINE_SUBSTITUTION disabled with INNODB not active, a bad combination
Up to MySql 5.5 the sqlmode default was a blank string, that means that the sqlmode NO_ENGINE_SUBSTITUTION was not set by default
According to MySql docs (see https://dev.mysql.com/doc/refman/5.6/en/sql-mode.html#sqlmode_no_engine_substitution) this is the meaning of sqlmode NO_ENGINE_SUBSTITUTION:
Control automatic substitution of the default storage engine when a
statement such as CREATE TABLE or ALTER TABLE specifies a storage
engine that is disabled or not compiled in.
Because storage engines can be pluggable at runtime, unavailable
engines are treated the same way:
With NO_ENGINE_SUBSTITUTION disabled, for CREATE TABLE the default
engine is used and a warning occurs if the desired engine is
unavailable. For ALTER TABLE, a warning occurs and the table is not
altered.
With NO_ENGINE_SUBSTITUTION enabled, an error occurs and the table is
not created or altered if the desired engine is unavailable.
So: if NO_ENGINE_SUBSTITUTION is disabled AND INNODB is switched OFF, MySql will switch to MYISAM also if you specify INNODB in your CREATE TABLE statement.
If the table you are creating is OK for MYISAM, you just receive a warning and the table is created. That is not your case, your creation statement include an index that is beyond the 1000 bytes limit of MYISAM, then the creation fails with error 1071 reporting the error of MYISAM. That is because the working engine is MYISAM, not INNODB.
PROOF
MySql version 5.1.56 community
Case 1:
Options in my.cnf
sql-mode=""
default-storage-engine=MYISAM
skip-innodb uncommented (without#)
Return on execution of your create statement:
Error Code: 1071. Specified key was too long; max key length is 1000 bytes
Explanation: INNODB is not active, the engine is automatically switched to MYISAM
that returns this error as they key is longer than MYISAM 1000 bytes limit.
The key length is:
2 fields x 255 char x 3 bytes utf8 encoding + 2 x 1 length byte = 1532 bytes
Case 2:
Options in my.cnf
sql-mode="NO_ENGINE_SUBSTITUTION"
default-storage-engine=MYISAM
skip-innodb uncommented (without#)
Return on execution of your create statement:
Error Code: 1286. Unknown table engine 'INNODB'
Explanation: INNODB is not active but the engine substitution is not permitted
by sql mode therefore the DB returns an error about the attempt of using a disabled engine.
Case 3:
Options in my.cnf
sql-mode="NO_ENGINE_SUBSTITUTION"
default-storage-engine=MYISAM
skip-innodb commented (with#)
Return on execution of your create statement:
Table creation OK!
Explanation: INNODB is active (skip-innodb commented) and it is used also if
the default engine is MYISAM.
To reproduce the tests restart MySql after every change in my.cnf.
Since MySql version 5.6 sqlmode is no more empty by default and contains NO_ENGINE_SUBSTITUTION, moreover INNODB is the default engine, so the error is difficult to meet.
OTHER TESTS
No other way of reproducing the error:
Error Code: 1071. Specified key was too long; max key length is 1000 bytes
while trying to create an INNODB table has been found.
In INNODB you have two kinds of ERROR 1071:
Error Code: 1071. Specified key was too long; max key length is 767 bytes
this has nothing to do with innodb_large_prefix ON or OFF, but is only related to the size of a single VARCHAR column being used as an index.
Mysql store varchar utf8 with 3 bytes plus 1 byte for the length up 255 character and 2 after (see: http://dev.mysql.com/doc/refman/5.7/en/storage-requirements.html), so if you try to set a key with VARCHAR(256) utf8 you get:
256 x 3 + 2 = 770 bytes
and you get the previous error, as the max key length for a single column is 767 bytes for an InnoDB table. A VARCHAR(255) is ok, because:
255 x 3 + 1 = 766 bytes
I tested it on four installations of Mysql, version 5.1.56, 5.5.33, 5.6 and 5.7, and that is confirmed. No issue with your query with VARCHAR(255), issue with VARCHAR(256):
Error Code: 1071. Specified key was too long; max key length is 767 bytes
As you can see the message is different, because it is an INNODB message not a MYISAM one!
The other type of ERROR 1071 for INNODB tables is:
Error Code: 1071. Specified key was too long; max key length is 3072 bytes
This is related to keys with multiple columns. For those keys to be enabled you need to set innodb_large_prefix to on.
Anyway, if you try to run something like this:
CREATE TABLE `phppos_modules_actions` (
`action_id` VARCHAR( 255 ) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL ,
`module_id` VARCHAR( 255 ) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL ,
`action_name_key` VARCHAR( 255 ) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL ,
`action_name_key1` VARCHAR( 255 ) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL ,
`action_name_key2` VARCHAR( 255 ) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL ,
`sort` INT NOT NULL ,
PRIMARY KEY ( `action_id` , `module_id`, `action_name_key`, `action_name_key1`, `action_name_key2` )
) ENGINE = INNODB CHARACTER SET utf8 COLLATE utf8_unicode_ci;
With a key of 5 VARCHAR(255) utf8 columns, that is 3830 bytes, you will run into the:
Error Code: 1071. Specified key was too long; max key length is 3072 bytes
Exotic Hypothesis
During the search of the cause I formulated and tested different and pretty weird hypothesis:
ROW format
Tested REDUNDANT, COMPACT, COMPRESS, DYNAMIC: no impact on table creation with your statement.
FILE format
Tested Antelope and Barracuda: no impact on table creation with your statement.
MySql built
Tested 32bit and 64 bit MySql: no impact on table creation with your statement.
Others similar failures
Here you can find the same error in the same situation:
https://www.drupal.org/node/2466287
I tested that statement in the 3 test situations listed in PROOF and it reproduced exactly the same behavior as yours, so I can say the issue is the same. In that case they switched to other DB, but the problem is the mix of setting, not the DB version.
References
A very good article of indexing with INNODB is given here:
http://mechanics.flite.com/blog/2014/07/29/using-innodb-large-prefix-to-avoid-error-1071/
WARNING: disabling INNODB by uncommenting skip-innodb in my.cnf after the creation of INNODB table with index longer than 1000 will not permit the start of MySql service
Regards
I don't now how much flexibility you have, but here are some options:
Upgrade MySQL version to the latest release (I tested the code on
MySQL 5.5.25 and got no errors.)
Change from utf8 to latin1.
Reduce the size of the the fields that make up the primary key.
Create a separate auto_increment primary key field to replace the existing key. Create a separate index on the first field action_id (but not the second field)
Other than those options, you are pretty much stuck since there is no setting that you can change in MySQL that will enable an index key greater than 1000 bytes.
In MySQL 5.6.3+ with some limitations(need ROW_FORMAT=DYNAMIC, innodb_file_format=BARRACUDA and innodb_file_per_table=true) you can enable innodb_large_prefix for a 3072 byte key length limit.
You have (accidentally?) InnoDB turned off. It is creating the table as MyISAM and the entire key is more than 1000 bytes.
Check variables of whether you have InnoDB, and whether there is "no_engine_substitution" (if that existed way back in 5.1.56) Also check mysqld.err to see if it has some relevant errors on startup.
It seems to be related in the storage engine. My end user was using MyISAM which handles the storage of data differently causing that error.
As says, the total length of your index is too long.
The short answer is that you shouldn't be indexing such long VARCHAR columns anyway, because the index will be very bulky and inefficient.
The best practice is to use prefix indexes so you're only indexing a left substring of the data. Most of your data will be a lot shorter than 255 characters anyway.
767 bytes is the stated prefix limitation for InnoDB tables - its 1,000 bytes long for MyISAM tables.
According to the response to this issue, you can get the key to apply by specifying a subset of the column rather than the entire amount. Means You can declare a prefix length per column as you define the index.
For example:
KEY `key_name` (`action_id`(50),`module_id`(50))
Here is the example to explain it more clearly: I created a table and inserted some data in it.
CREATE TABLE `phppos_modules_actions` (
`action_id` VARCHAR( 255 ) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL ,
`module_id` VARCHAR( 255 ) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL ,
`action_name_key` VARCHAR( 255 ) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL ,
`sort` INT NOT NULL ,
PRIMARY KEY ( `action_id`(50) , `module_id`(50) )
) ENGINE = INNODB CHARACTER SET utf8 COLLATE utf8_unicode_ci;
But what's the best prefix length for a given column? Here's a method to find out:
SELECT
ROUND(SUM(LENGTH(`action_id`)<10)*100/COUNT(*),2) AS pct_length_10,
ROUND(SUM(LENGTH(`action_id`)<20)*100/COUNT(*),2) AS pct_length_20,
ROUND(SUM(LENGTH(`action_id`)<50)*100/COUNT(*),2) AS pct_length_50,
ROUND(SUM(LENGTH(`action_id`)<100)*100/COUNT(*),2) AS pct_length_100
FROM `phppos_modules_actions`;
+---------------+---------------+---------------+----------------+
| pct_length_10 | pct_length_20 | pct_length_50 | pct_length_100 |
+---------------+---------------+---------------+----------------+
| 42.86 | 80.20 | 100 | 100 |
+---------------+---------------+---------------+----------------+
This tells you that 80% of your strings are less than 20 characters, and all of your strings are less than 50 characters. So there's no need to index more than a prefix length of 50, and certainly no need to index the full length of 255 characters.
Tweak as you need to get the key to apply, but I wonder if it would be worth it to review your data model regarding this entity to see if there's improvements that would allow you to implement the intended business rules without hitting the MySQL limitation.
Some alternative way about settings in innodb can be found at http://mechanics.flite.com/blog/2014/07/29/using-innodb-large-prefix-to-avoid-error-1071/
Official doc 5.6
Limits on InnoDB Tables
The InnoDB internal maximum key length is 3500 bytes, but MySQL itself
restricts this to 3072 bytes. This limit applies to the length of the
combined index key in a multi-column index.
http://dev.mysql.com/doc/refman/5.7/en/innodb-restrictions.html
The MyISAM Storage Engine
The maximum key length is 1000 bytes. This can also be changed by
changing the source and recompiling. For the case of a key longer than
250 bytes, a larger key block size than the default of 1024 bytes is
used.
http://dev.mysql.com/doc/refman/5.7/en/myisam-storage-engine.html
However your table is declared as InnoDB. So I don't know what to think
There's also a clue at the end of this old bug
If you need this you should really look at MySQL 5.5 and the innodb_large_prefix option that is available from 5.5.14 (July 2011) onwards because it probably does what you are looking for:
"Enable this option to allow index key prefixes longer than 767 bytes
(up to 3072 bytes), for InnoDB tables that use the DYNAMIC and
COMPRESSED row formats. (Creating such tables also requires the
option values innodb_file_format=barracuda and
innodb_file_per_table=true.) See Section 13.3.15, “Limits on InnoDB
Tables” for the relevant maximums associated with index key prefixes
under various settings.
The maximum key length is 1000 bytes. This can also be changed by changing the source and recompiling. For the case of a key longer than 250 bytes, a larger key block size than the default of 1024 bytes is used.
In my case, MySQL was started without InnoDB support. And during DB backup importing - tried to create MyISAM tables.
At the same time, no errors were shown in the console during MySQL restart.
Only when checked MySQL log file - found this.
BTW an error was related to innodb_log_file_size = 4G, only when changed it to innodb_log_file_size = 1G - InnoDB support was enabled.
I've created a view which uses GROUP_CONCAT to concatenate results from a query on products column with data type of 'varchar(7) utf8_general_ci' in a column named concat_products.
The problem is that MySQL truncates value of "concat_products" column.
phpMyAdmin says the data type of "concat_products" column is varchar(341) utf8_bin
Table products:
CREATE TABLE `products`(
`productId` tinyint(2) unsigned NOT NULL AUTO_INCREMENT,
`product` varchar(7) COLLATE utf8_general_ci NOT NULL,
`price` mediumint(5) unsigned NOT NULL,
PRIMARY KEY (`productId`)
) ENGINE=InnoDB AUTO_INCREMENT=28 DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci
The "concat_products_vw" view:
CREATE VIEW concat_products_vw AS
SELECT
`userId`,
GROUP_CONCAT(CONCAT_WS('_', `product`, `productId`, `price`)
ORDER BY `productId` ASC SEPARATOR '*') AS concat_products
FROM
`users`
LEFT JOIN `products`
ON `users`.`accountBalance` >= `product`.`price`
GROUP BY `productId`
According to MySQL manual:
Values in VARCHAR columns are variable-length strings
Length can be specified as a value from 1 to 255 before MySQL 4.0.2 and 0 to 255 as of MySQL 4.0.2.
EDIT
Values in VARCHAR columns are variable-length strings. The length can be specified as a value from 0 to 65,535.
Why MySQL specifies more than 255 characters for varchar "concat_products" column? (solved!)
Why uf8_bin instead of utf8_general_ci?
Is it possible to change the data type of a column in a view for example in my case to text for "concat_products" column?
If not what can I do to prevent MySQL from truncating "concat_products" column?
As I already wrote in an earlier comment, the MySQL manual says:
Values in VARCHAR columns are variable-length strings. The length can
be specified as a value from 0 to 65,535.
So the problem is not with the data type of the field.
The MySQL manual also says:
The result is truncated to the maximum length that is given by the
group_concat_max_len system variable, which has a default value of
1024. The value can be set higher, although the effective maximum length of the return value is constrained by the value of
max_allowed_packet. The syntax to change the value of
group_concat_max_len at runtime is as follows, where val is an
unsigned integer:
SET [GLOBAL | SESSION] group_concat_max_len = val;
Your options for changing the value of group_concat_max_len are:
changing the value at MySQL startup by appending this to the command:
--group_concat_max_len=your_value_here
adding this line in your MySQL configuration file (mysql.ini): group_concat_max_len=your_value_here
running this command after MySQL startup:
SET GLOBAL group_concat_max_len=your_value_here;
running this command after opening a MySQL connection:
SET SESSION group_concat_max_len=your_value_here;
Documentation: SET, Server System Variables: group_concat_max_len
As Jocelyn mentioned, the size of a GROUP_CONCAT() result is bounded by group_concat_max_len, however there is an additional interaction with ORDER BY that results in a further truncation to 1/3 of group_concat_max_len. For an example, see this related answer.
The default value for group_concat_max_len is 1024, and 1024 / 3 = 341 probably explains why the type of concat_products shows up as varchar(341) in the original example. If you were to remove the GROUP BY productId clause, concat_products should show up as varchar(1024).
I have not found this interaction between GROUP_CONCAT() and ORDER BY mentioned in the MySQL Manual, but it affects at least MySQL Server 5.1.
Year 2023, on Linux Server you can edit my.cnf file and then set:
[mysqld]
group_concat_max_len = 2048