I have a myisam only dedicated 32 GB RAM mysql server that is running on default configuration. I want to change the engine type to InnoDB of one table in order to avoid table locks. It has 50 million records and size on disk is around 15 GB. I am using mysql version 5.5
I guess I will need to add the following options and restart mysql.
innodb_buffer_pool_size=1G
innodb_log_file_size=100M
innodb_file_per_table=1
What else is considered to be necessary while changing the engine type?
You'll actually be running a command to convert each table.
It goes faster to first sort the table:
ALTER TABLE tablename ORDER BY primary_key_column;
Then, run the alter command:
ALTER TABLE tablename ENGINE = INNODB;
It might take a while if the table is really large, and it will use a lot of your CPU....
First of all check if your database supports InnoDB engine (I bet it is supported ;)):
SHOW ENGINES\G
If so, there is already default innodb related parameters in place, check them with:
SHOW VARIABLES LIKE '%innodb%'
and try to understand them and alter the to your specific needs. Even if you use the default params, you are now fine to play arround with InnoDB tables.
If you want to create only InnoDB tables, you can change your default storage engine, either for your current session with: SET storage_engine=INNODB; or in your config using default-storage-engine option.
By the way, the fastest way to convert a table to InnoDB is not the above described way. One can do the following to convert a table to InnoDB by simply inserting the data:
CREATE TABE new AS SELECT * FROM old WHERE 1<>1;
ALTER TABLE new ENGINE = INNODB;
INSERT INTO new SELECT * FROM old;
Of course you have to add the indexes you need manually, but its usually worth the time (and pain) you save compared to the ALTER TABLE ... on slightly bigger tables.
Related
I had one table that I want drop. That table use something around 130GB. When I start drop/truncate that table, mysql server slowdown and I get a lot of issues about users using that service.
How can I drop a large table without affect mysql performance?
I already tried create a new table like that I want to drop, rename them and drop unused table (old table), but rename command hangs my mysql service.
I am using:
mysql Ver 15.1 Distrib 10.1.28-MariaDB, for Linux (x86_64) using readline 5.1
The OS is probably the problem. Some OSs take a long time to finish deleting a very large file. Sorry, but you simply have to wait.
When you DROP or TRUNCATE a table in MySQL or MariaDB and you have large buffer pool, there can be server lockup or stall of multiple seconds.The is because when a table is dropped, MySQL will perform a full scan of buffer pool looking for pages that belong to dropped table. This will lock the buffer-pool and you have to wait until it's unlock. If your buffer pool is smaller this will not effect much but if you have larger one, it becomes devastating.
One possible solution can be delete all rows before dropping a table, this will not cause a memory stall because there will be no data in memory.
Another solution can be use of non-InnoDB storage engine for temporary tables. You can specify ENGINE=MyISAM in your CREATE TABLE statement. If you have a large code base and don’t explicitly specify the storage engine, you can change the default storage engine to MyISAM by adjusting the following configuration option:
default_tmp_storage_engine = MyISAM
Of course there are some drawbacks with these solutions. You can find complete details of this bug, solution and drawbacks MySQL troubleshooting article.
The good news is that the fix itself has been implemented MySQL 8.0.23 and later.
The quickest way to empty out a large table is to create a new one with the same columns. Truncate will be a lot slower than just dropping
Disable foreign key checks
SET foreign_key_checks = 0;
Rename old table to be dropped
RENAME TABLE old TO temp;
Create the new empty table exactly like the older table
CREATE TABLE new LIKE old;
Drop old table
DROP TABLE temp
Enable foreign key checks
SET foreign_key_checks = 1;
I have MySQL 5.5.37 with InnoDB installed locally with apt-get on Ubuntu 13.10. My machine is i7-3770 + 32Gb memory + SSD hard drive on my desktop. For a table "mytable" which contains only 1.5 million records the following DDL query takes more than 20 min (!):
ALTER TABLE mytable ADD some_column CHAR(1) NOT NULL DEFAULT 'N';
Is there a way to improve it?
I checked
show processlist;
and it was showing that it is copying my table for some reason.
It is disturbingly inconvenient. Is there a way to turn off this copy?
Are there other ways to improve performance of adding a column to a large table?
Other than that my DB is relatively small with only 1.3Gb dump size. Therefore it should (in theory) fit 100% in memory.
Are there settings which can help?
Would migration to Precona change anything for me?
Add: I have
innodb_buffer_pool_size = 134217728
Are there other ways to improve performance of adding a column to a large table?
Short answer: no. You may add ENUM and SET values instantly, and you may add secondary indexes while locking only for writes, but altering table structure always requires a table copy.
Long answer: your real problem isn't really performance, but the lock time. It doesn't matter if it's slow, it only matters that other clients can't perform queries until your ALTER TABLE is finished. There are some options in that case:
You may use the pt-online-schema-change, from Percona toolkit. Backup your data first! This is the easiest solution, but may not work in all cases.
If you don't use foreign keys and it's slow because you have a lot of indexes, it might be faster for you to create a copy of the table with the changes you need but no secondary indexes, populate it with the data, and create all indexes with a single alter table at the end.
If it's easy for you to create replicas, like if you're hosted at Amazon RDS, you may create a master-master replica, run the alter table there, let it get back in sync, and switch instances after finished.
UPDATE
As others mentioned, MySQL 8.0 INNODB added support for instant column adds. It's not a magical solution, it has limitations and side-effects -- it can only be the last column, the table must not have a full text index, etc -- but should help in many cases.
You can specify explicit ALGORITHM=INSTANT LOCK=NONE parameters, and if an instant schema change isn't possible, MySQL will fail with an error instead of falling back to INPLACE or COPY. Example:
ALTER TABLE mytable
ADD COLUMN mycolumn varchar(36) DEFAULT NULL,
ALGORITHM=INPLACE, LOCK=NONE;
https://mysqlserverteam.com/mysql-8-0-innodb-now-supports-instant-add-column/
MariaDb 10.3, MySQL 8.0 and probably other MySQL variants to follow have an "Instant ADD COLUMN" feature whereby most columns (there are a few constraints, see docs) can be added instantly with no table rebuild.
MariaDb: https://mariadb.com/resources/blog/instant-add-column-innodb
MySQL: https://mysqlserverteam.com/mysql-8-0-innodb-now-supports-instant-add-column/
I know this is a rather old question but today i encountered a similar problem. I decided to create a new table and to import the old table in the new table. Something like:
CREATE TABLE New_mytable LIKE mytable ;
ALTER TABLE New_mytable ADD some_column CHAR(1) NOT NULL DEFAULT 'N';
insert into New_mytable select * from mytable ;
Then
START TRANSACTION;
insert into New_mytable select * from mytable where id > (Select max(id) from New_mytable) ;
RENAME TABLE mytable TO Old_mytable;
RENAME TABLE New_mytable TO mytable;
COMMIT;
This does not make the update process go any faster, but it does minimize downtime.
Hope this helps.
What about Online DDL?
http://www.tocker.ca/2013/11/05/a-closer-look-at-online-ddl-in-mysql-5-6.html
Maybe you would use TokuDB instead:
http://www.tokutek.com/products/tokudb-for-mysql/
There is no way to avoid copying the table when adding or removing columns because the structure changes. You can add or remove secondary indexes without a table copy.
Your table data doesn't reside in memory. The indexes can reside in memory.
1.5 million records is not a lot of rows, and 20 minutes seems quite long, but perhaps your rows are large and you have many indexes.
While the table is being copied, you can still select rows from the table. However, if you try to do any updates, they will be blocked until the ALTER is complete.
I am new to Mysql .
will it affect my data on server if i change mysql engine from Myisam to innodb.
Thanks
Changing engine from MyISAM to INNODB should not affect your data, but safe side you can easily take backup of your table before changine engine.
Taking backup:
CREATE TABLE backup_table LIKE your_table;
INSERT INTO backup_table SELECT * FROM your_table;
It may affect the performance of your queries. You need to configure Innodb specific System variables. e.g.
innodb_buffer_pool_size = 16G
innodb_additional_mem_pool_size = 2G
innodb_log_file_size = 1G
innodb_log_buffer_size = 8M
Changing engine to INNODB:
ALTER TABLE table_name ENGINE=INNODB;
I found two caveats when converting MyISAM tables to InnoDB: row size and support for full-text indexes
I ran into an issue converting some tables from an off-the-shelf application from MyISAM to InnoDB because of the maximum record size. MyISAM supports longer rows than InnoDB does. The maximum in a default InnoDB installation is about 8000 bytes. You can work around this by TRUNCATE'ing a table that fails, but this will bite you later on the INSERT. You might have to break your data up into multiple tables or restructure it with variable length column types such as TEXT (which can be slower).
A stock Innodb installation doesn't support FULLTEXT indexes. This may or may not impact your application. For one application's table I was converting, we decided to look in other fields for the data we needed rather than doing full text scans. (I did an "ALTER TABLE DROP INDEX..." on it to remove the FULLTEXT index before converting to InnoDB.) I wouldn't recommend full-text indexes for a write-heavy table anyway.
If converting a big table full of data with "ALTER TABLE..." works on the first try, you're probably okay.
http://dev.mysql.com/doc/refman/5.5/en/innodb-restrictions.html
Lastly, if you want to convert from MyISAM to InnoDB on a running system that is read heavy (where UPDATEs and INSERTs are rare), you can run this without interrupting your users. (The RENAME runs atomically. It will back out all of the changes if any of them don't work.)
CREATE TABLE mytable_ind AS SELECT * FROM mytable;
ALTER TABLE mytable_ind ENGINE=InnoDB;
RENAME TABLE mytable TO mytable_myi, mytable_ind TO mytable;
DROP TABLE mytable_myi;
I have MySQL running on my machine configured with MyISAM as its default tables.
Now I want to ask few of questions:
1) If I change the default table to InnoDB in the configuration file (my.conf), clear the log file and restart mysql, would that harm any of my previous database or tables?
2) If I alter few tables' engine to InnoDB using the following command, would that affect its data at all?
ALTER TABLE table_name ENGINE = InnoDB;
3) Is it a good idea to keep few tables as MyISAM (for read and write) and the rest as InnoDB (more for selecting data) or is it preferred to select one engine for all the tables in the database?
2) It will only affect the internal representation. Nothing that you will notice on the outside.
3) It is a perfectly good idea, if it enhances performance.
2) You can mix database types. i.e. innoDB and MyISAM.
3) innoDB supposedly keeps data safer. I think it is the default on latest versions of mySQL.
I am in the beginning stages of a project and I have so far been using the default MySQL database.
By the way, does the default database have name?
My question is how I can change the existing tables to be utf-8 and InnoDB without deleting the current ones and making new tables. Is there an alter table to make the table utf-8 and InnoDB?
Thanks,
Alex
MyISAM is the default Storage Engine for MySQL (until 5.5.5, at which point InnoDB became the default). There is no concept of a default database.
To make an existing table use InnoDB, use the following:
ALTER TABLE tbl_name ENGINE = InnoDB;
To change the character set of an existing table to utf8, use the following:
ALTER TABLE tbl_name CONVERT TO CHARACTER SET utf8;
If you want to play it safe with changing a table to InnoDB, I have a nice suggestion:
Create a new table as InnoDB and load it
For this example you have a MyISAM table called mytable in the mydb database. You can perform the following:
use mydb
CREATE TABLE mytable_innodb LIKE mytable;
ALTER TABLE mytable_innodb ENGINE=InnoDB;
INSERT INTO mytable_innodb SELECT * FROM mytable;
ALTER TABLE mytable RENAME mytable_myisam;
ALTER TABLE mytable_innodb RENAME mytable;
That's it. The new table is InnoDB plus you have a backup of the original table in MyISAM with its original contents and layout. You are free to perform whatever other conversions you need on the new InnoDB table.
CAVEAT
Make sure you optimize InnoDB
InnoDB performance tweaks
Howto: Clean a mysql InnoDB storage engine?
How to Safely Change InnoDB Log File Size
Major Differences Between InnoDB and MyISAM