Is it possible to drop a specific partition in InnoDB and later restore it again?
The reason behind this is to archive old partition and restore it again incase it needs to be access.
I been searching around and so far do not see anything promising.
If your are using mysql 5.6:
create table dest like source; alter table dest remove partitioning;
alter table source exchange partition XXX with table dest;
you can dump & load data later;
or you can use innobackupex to backup dest table and restore it later using transportable tablespaces
Related
My partition schemes looks something like:
ALTER TABLE my_table
PARTITION BY RANGE (integer_field) (
PARTITION p0 VALUES LESS THAN (100) DATA DIRECTORY = '/my_location/partitions/p0' ,
PARTITION p1 VALUES LESS THAN (200) DATA DIRECTORY = '/my_location/partitions/p1' ,
PARTITION p_other VALUES LESS THAN (MAXVALUE) DATA DIRECTORY = '/my_location/partitions/p_other'
);
As expected, data is being stored properly into the partitions, and into the proper directory.
Problem:
Now when I remove/move the directory from the location, like mv /my_location/partitions/p0 /some_other_location/ , the directory gets moved successfully, but data is still query-able from MySQL shell, even after restarting the shell.
My Solution:
To get it working as expected, after moving the directory including the .ibd file, I had to drop the partition explicitly:
ALTER TABLE my_table DROP PARTITION p0;
This removed the partition from the scheme as required and also cleared the data, verified it by again querying the same data.
Assumption / Understanding:
I think that MySQL is caching the data, not sure where and why exactly, which makes it query-able even after partition directory is moved away. Definitely cache is not at connection level as I closed and reopned the shell.
Question:
I expected the data to disappear as soon as the directory p0 was moved away. Is it really necessary to run the drop partition statement each time the directory is moved?
Constraints:
It is sure that p0 directory is moved away only when the p0 partition is no more used. So there will not be any more data required to enter into existing p0 partition
MySQL: 8.0.19
Windows or not?
Did you restart mysqld?
mv on Linux (and close relatives) is really a "rename". And, assuming the target is on the same filesystem, that rename can even involve a different directory.
When a program (eg, mysqld) has a file (eg, the table partition) "open" mysqld, it has control over it -- even if you run off to a shell and rm the file!.
I suspect that when you restart mysqld (for any reason, including reboot), the "data directory" will become messed up.
Aside from the filesystem's partitioning, you must tell MySQL when you "archive" a partition. Read up on "transportable tablespaces" for the version you are using. Here's a writeup for 5.6; 5.7 has some improvements.
I don't see the advantage of using a filesystem partition. With "transportable tablespaces", you can disconnect a MySQL partition from a PARTITIONed table. This turns the partition into a table. Then that table can be deleted, renamed, copied, etc, without impacting the partitioned table. Search for "transportable" in http://mysql.rjweb.org/doc.php/partitionmaint ; there are some links.
As pointed by #Rick James and #Solarflare, moving around the ibd file while it is still in Open state for MySQL does behave in weird way, and tablespaces gets messed up. Following their guidance, and the MySQL Docs, here is the final approach that successfully worked for me (Possibly, the right way to do it):
Lock the table
FLUSH TABLES my_table FOR EXPORT;
This prevents any update/write operation to the desired table and makes the ibd file safe to copy. Additionally, this creates a .cfg file. These steps are very well explained in MySQL Docs
Once the table is "Locked", copy the .ibd file to the desired location for archival. PS: Do not move / delete the source file yet.
cp -r /my_location/partitions/p0 /some_other_location/
Unlock tables to be able to alter the partition
UNLOCK TABLES;
Drop the required partition safely. This also informs the tablespaces.
ALTER TABLE my_table DROP PARTITION p0;
Note that this statement leads to removal of partition, plus the data corresponding to that partition.
One of my table's file size (a .MYD file) has increased by about 100%.
If I look into the data there is just a normal daily load during the last few days. What could have caused this file size increase?
myisamchk
As suggested by a user I tried sudo myisamchk -r tablename. The result of this operation was "Fixing index 1" "Fixing index 2" ... and the exact same file size.
Edit: After running the command a second time the file size decreased to the normal (half) size.
#e4c5 has the most likely answer. He is asking you to remove the deleted rows. MySQL will not reclaim the space from deleted rows. As more and more rows are deleted, the larger your .MYD becomes. The only way to reclaim the space is to rebuild the .MYD - either by recreating the table, using the optimize table command or running myisamchk -r.
The following command will rebuild the table.
optimize table $table_name;
The myisamchk -r command line approach works too - but be sure the table is locked to prevent users from accessing it (otherwise you run the very real risk of losing data). Be sure to flush table before using it again after running myisamchk repair or MySQL will detect the table has changed and run an unnecessary check against it. Of the three methods, myisamchk -r is the most prone to data lost (but still my preferred method- you just have to be careful and prepared for the worst).
It sounds like you are replacing the entire table every day. Here's an optimal way to do it; no bloat, no OPTIMIZE:
CREATE TABLE new LIKE real;
populate `new`
RENAME TABLE real TO old,
new TO real;
DROP TABLE old;
That way, you build a clean, new, table and swap it in. Note that the table is always available; that is, no downtime.
Based on #rick-james's answer I looked further and found another thread. This does the trick... very fast and no downtime (if the table is not written during the run).
-- table_org is the name of the table to 'optimize'
CREATE TABLE `table_new` LIKE `table_org`;
-- disable keys for quick inserts
ALTER TABLE `table_new` DISABLE KEYS;
INSERT INTO `table_new`
SELECT * FROM `table_org`;
ALTER TABLE `table_new` ENABLE KEYS;
RENAME TABLE `table_org` TO `table_old`, `table_new` TO `table_org`;
DROP TABLE `table_old`;
-- is not needed, but just to have an up-to-date data
ANALYZE TABLE `table_org`;
I have the innodb_file_per_table option enabled so I have an .frm and an .ibd file for each table. What's unusual is there are also some files that appear to be from temporary tables on the filesystem:
/db/mysql$ ls
<snip>
#sql2-25fe-5.frm
#sql2-25fe-5.ibd
#sql2-55f4-73.frm
#sql2-55f4-73.ibd
#sql-55f6_13b626.ibd
#sql-6abe_e3.ibd
</snip>
One of them is rather large so I'd like to get rid of them if it's safe. I tried dropping them like regular and temporary tables but it didn't work either way.
drop table `#sql2-25fe-5`;
drop temporary table `#sql2-25fe-5`;
Any ideas? Thanks
Just to clear this out:
Dumping and reloading all the tables freed up the space like I wanted it to (an innobackupex backup & restore did not work). I believe these permanent temporary tables were the result of one of two bugs which have been fixed in mysql:
InnoDB: If the server crashed at a precise moment during an ALTER TABLE operation that rebuilt the clustered index for an InnoDB table, the original table could be inaccessible afterward. An example of such an operation is ALTER TABLE ... ADD PRIMARY KEY The fix preserves the original table if the server halts during this operation. You might still need to rename the .ibd file manually to restore the original table contents: in MySQL 5.6 and higher, rename from #sql-ib$new_table_id.ibd to table_name.ibd within the database directory; prior to MySQL 5.6, the temporary file to rename is table_name#1 or #2. (Bug #14669848)
https://dev.mysql.com/doc/relnotes/mysql/5.5/en/news-5-5-30.html
or
InnoDB: For UPDATE statements in which an error occurred, it was possible for a temporary file opened during the update not to be closed. (Bug #15978766)
https://dev.mysql.com/doc/relnotes/mysql/5.5/en/news-5-5-32.html
I have a myisam only dedicated 32 GB RAM mysql server that is running on default configuration. I want to change the engine type to InnoDB of one table in order to avoid table locks. It has 50 million records and size on disk is around 15 GB. I am using mysql version 5.5
I guess I will need to add the following options and restart mysql.
innodb_buffer_pool_size=1G
innodb_log_file_size=100M
innodb_file_per_table=1
What else is considered to be necessary while changing the engine type?
You'll actually be running a command to convert each table.
It goes faster to first sort the table:
ALTER TABLE tablename ORDER BY primary_key_column;
Then, run the alter command:
ALTER TABLE tablename ENGINE = INNODB;
It might take a while if the table is really large, and it will use a lot of your CPU....
First of all check if your database supports InnoDB engine (I bet it is supported ;)):
SHOW ENGINES\G
If so, there is already default innodb related parameters in place, check them with:
SHOW VARIABLES LIKE '%innodb%'
and try to understand them and alter the to your specific needs. Even if you use the default params, you are now fine to play arround with InnoDB tables.
If you want to create only InnoDB tables, you can change your default storage engine, either for your current session with: SET storage_engine=INNODB; or in your config using default-storage-engine option.
By the way, the fastest way to convert a table to InnoDB is not the above described way. One can do the following to convert a table to InnoDB by simply inserting the data:
CREATE TABE new AS SELECT * FROM old WHERE 1<>1;
ALTER TABLE new ENGINE = INNODB;
INSERT INTO new SELECT * FROM old;
Of course you have to add the indexes you need manually, but its usually worth the time (and pain) you save compared to the ALTER TABLE ... on slightly bigger tables.
In a MySQL MASTER MASTER scenario using InnoDB
When dropping an index on one instance will the same table on the other instance be available?
What is the sequence of activities?
I assume the following sequence:
DROP INDEX on 1st instance
Added to the binary log
DROP INDEX on 2nd instance
Can anyone confirm?
I believe the following will happen:
Your DROP INDEX (which really runs an ALTER TABLE ... DROP INDEX) runs on the master
If the ALTER completes successfully the statement will then be added to the binlog and will be run on the slave
This means that the ALTER TABLE on the other machine won't start until the ALTER TABLE has successfully completed on the first (master) machine.
While the ALTER TABLE is running on either machine the table will be readable for a while and then neither readable/writeable as MySQL first makes a copy of the table internally then applies the changes.
From http://dev.mysql.com/doc/refman/5.0/en/alter-table.html
In most cases, ALTER TABLE works by
making a temporary copy of the
original table. The alteration is
performed on the copy, and then the
original table is deleted and the new
one is renamed. While ALTER TABLE is
executing, the original table is
readable by other sessions. Updates
and writes to the table are stalled
until the new table is ready, and then
are automatically redirected to the
new table without any failed updates.
The temporary table is created in the
database directory of the new table.
This can be different from the
database directory of the original
table if ALTER TABLE is renaming the
table to a different database.
In a MySQL MASTER MASTER scenario using InnoDB
In so far as I'm aware, such a thing is not possible. You'd need to use NDB, or be in a multi-master-slave environment with auto-incrementing fields configured to increment by 1/2/3/more. So assuming the latter. (Note: if you're aware of an InnoDB based solution, please share.)
When dropping an index on one instance will the same table on the other instance be available?
Dropping an index only means your index won't be available. Not the table. It'll be written (and propagated) to the binary log and begone with.