I have a rather large mysql database table where one field is longtext, I wish to use compress on this field, would this result in diskspace reduction? or is the storage space already allocated and it won't result in any space reduction. the storage engine is innodb.
InnoDB compressed row format has some prerequisites:
innodb_file_format=Barracuda in your my.cnf file.
innodb_file_per_table in your my.cnf file. The compression doesn't work for tables stored in the central tablespace (ibdata1).
Change a table to use compressed format.
ALTER TABLE MyTable ROW_FORMAT=COMPRESSED
Then the table should be stored in the compressed format, and it will take less space.
The ratio of compression depends on your data.
note: If you had a table stored in the ibdata1 central tablespace, and you restructure the table into file-per-table, ibdata1 will not shrink! The only way to shrink ibdata1 is to dump all your InnoDB data, shutdown MySQL, rm the tablespace, restart MySQL, and reload your data.
Re your comments:
No, it shouldn't change the way you do queries.
Yes, it will take a long time. How long depends on your system. I'd recommend trying it with a smaller table first so you get a feel for it.
It needs to spend some CPU resources to compress when you write, and decompress when you read. It's common for a database server to have extra CPU resources -- databases are more typically constrained by I/O. But the speed might become a bottleneck. Again, it depends on your system, your data, and your usage. I encourage you to test carefully to see if it's a net win or not.
Related
I need your advice. Need to reclaim disk space on live server with minimum downtime. We are using:
mysql -- 5.5
innodb table per file -- on
Was a huge table(70% insers/30% deletes -- it means that sometimes we delete rows from this table), this table was dropped with "drop table" command, as we expect, mysql didn't release disk space to OS, but now we need to release free space. OPTIMIZE command is possible way -- but now we don't 100% sure if we will have enough free space on disk to do this command, because previous table was huge. But in mysql documentation written:
You can run OPTIMIZE TABLE to compact or recreate a file-per-table tablespace. When you run an OPTIMIZE TABLE, InnoDB creates a new .ibd file with a temporary name, using only the space required to store actual data. When the optimization is complete, InnoDB removes the old .ibd file and replaces it with the new one.
I highlighted confusing moment in documentation, does it means, that "OPTIMIZE dropped_table;" will use only space for data that really needed, for this situation there is no actual data if we drop table early ?
If the table was created when innodb_file_per_table was OFF, then the table was written to ibdata1, and the space is not returned to the OS by DROP TABLE.
If it had been ON, then there would be a .ibd file. Did you see that file before the DROP? Does it still exist? (It should not still exist.)
When DROP removes the .ibd file, there is a slight lag (depending on the OS) while waiting for the data to be actually freed up by the OS. du and/or df reflects this lag.
OPTIMIZE TABLE will copy the table to tmpdir (I think), then drop the old table and play RENAME games. This requires extra space -- enough for a compete copy of the table. Assuming you really have a .ibd file, it will create another .ibd file (with a temp name) and shuffle files. If tmpdir points to a different "filesystem", the shuffle will involve a copy, not just a move.
If I have not explained things adequately, please provide more details -- actual commands you are proposing, size of table in question, whether the .ibd exists, etc.
I have some large InnoDB databases, close to 1TB.
In order to free up some space while working on an alternative storage, I deleted some unused InnoDB indexes hoping that it would free up some space.
It freed nothing.
As it's InnoDB, will the engine use the empty allocated space for further inserts and indexing?
InnoDB stores data in a tablespace. By default, there is one single tablespace and data of all the databases is stored in one file. This file has data dictionary, tables, as well as indexes in it. There is a global parameter innodb_data_file_path that defines this tablespace file. It has a syntax like ibdata1:256M:autoextend, this means at the beginning a file of size 256 MB will be created and then whenever the data size exceeds this, the file will be auto-extended. The innodb_autoextend_increment variable defines in MB's that by how much each increment should be.
How you will get your Index Space?
You should first backup all InnoDB tables and change setting in my.ini/my.cnf as innodb_file_per_table and restart MySQL server.
Now import those tables now each table will have it's own tablespace which can shrink size on deleting a table.
These are possible workarounds:
Separate Files per Table: InnoDB provides this option where data (data + indexes) for each table can be stored in a separate file through a global variable innodb_file_per_table.
Fixed Tablespace size: One way to work around with the tablespace file size problems is to fix the tablespace size (remove autoextend) to an extrapolated value. So, when you hit the limit, you know it is time to cleanup.
Move to MyISAM: For all the tables (or even databases), for which you feel data is not that critical to have transactions et al, move them to MyISAM.
Answer here: https://stackoverflow.com/a/8932289/82114
If you don't use innodb_file_per_table, reclaiming disk space is possible, but quite tedious, and requires a significant amount of downtime.
The "How To" can be found here: http://dev.mysql.com/doc/refman/5.6/en/innodb-data-log-reconfiguration.html
Be sure to also retain a copy of your schema in your dump.
Currently, you cannot remove a data file from the system tablespace.
To decrease the system tablespace size, use this procedure:
Use mysqldump to dump all your InnoDB tables.
Stop the server.
Remove all the existing tablespace files, including the ibdata and
ib_log files. If you want to keep a backup copy of the information,
then copy all the ib* files to another location before the removing
the files in your MySQL installation.
Remove any .frm files for InnoDB tables.
Configure a new tablespace.
Restart the server.
Import the dump files.
You should reorganize the InnoDB infrastructure. Why? Because ibdata1 never shrinks. With innodb_file_per_table disabled, it will bloat ibdata1 in a hurry? Aside from data and indexes, what lives in ibdata1?
Data dictionary
Double write buffer (handles Background Page Writes)
Insert Buffer (andles Changes to Secondary Indexes)
Rollback Segments
Undo Tablespace
That being said, you need to migrate your data out, delete all InnoDB relayed files, and reload with two things:
innodb_file_per_table enabled
bigger transaction logs
I wrote up how to do this back on Oct 29, 2010 : Howto: Clean a mysql InnoDB storage engine?
Going forward you could alwsy shrink individual InnoDB tables. For example, to shrink mydb.mytable:
ALTER TABLE mydb.mytable ENGINE=InnoDB;
My ibdata1 file for MySQL database grew to about 32GB over time. Recently I deleted about 10GB of data from my databases (and restarted mysql for good measure), but the file won't shrink. Is there any way to reduce the size of this file
The file size of InnoDB tablespaces will never reduce automatically, no matter how much data you delete.
What you could do, although it is a lot of effort, is to create one tablespace for every table by setting
innodb_file_per_table
The long part about this is, that you need to export ALL DATA from the mysql server (setting up a new server would be easier) and then reimport the data. Instead of one single ibdata1 file which holds the data for each and every table, you will find a lot of files called tablename.ibd which hold the data only for one single table.
Afterwards:
When you then delete a lot of data from tables, you can let mysql recreate the data-file by issuing
alter table <tablename> engine=myisam;
to switch to MyIsam (and have the InnoDB data file for this table deleted) and then
alter table <tablename> engine=innodb;
to recreate the table.
Unless you set innodb_file_per_table, ibdata1 holds all InnoDB tables plus undo.
This file never shrinks.
To shrink it, you should (at your own risk):
Backup and drop all InnoDB tables in all databases
Delete the file manually
Reinitialize InnoDB storage (by restarting mysqld) and
Restore the tables from backup.
If you set innodb_file_per_table, you'll still have to do this to reclaim the space, but in this case you'll be able to do this on per-table basis, without affecting the other tables.
Note that the undo is still held in ibdata, even with innodb_file_per_table set.
Adding, Removing, or Resizing InnoDB Data and Log Files
Run optimize table your_db.your_table; sql request
or use mysql workbench migration wizard and it will create database copy with reduced size
Microsoft SQL Server has a nice feature, which allows a database to be automatically expanded when it becomes full. In MySQL, I understand that a database is, in fact, a directory with a bunch of files corresponding to various objects. Does it mean that a concept of database size is not applicable and a MySQL database can be as big as available disk space allows without any additional concern? If yes, is this behavior the same across different storage engines?
It depends on the engine you're using. A list of the ones that come with MySQL can be found here.
MyISAM tables have a file per table. This file can grow to your file system's limit. As a table gets larger, you'll have to tune it as there's index and data size optimizations that limit the default size. Also, this MyISAM documentation page says:
There is a limit of 2^32 (~4.295E+09)
rows in a MyISAM table. If you build
MySQL with the --with-big-tables
option, the row limitation is
increased to (2^32)^2 (1.844E+19) rows.
See Section 2.16.2, “Typical configure
Options”. Binary distributions for
Unix and Linux are built with this
option.
InnoDB can operate in 3 different modes: using innodb table files, using a whole disk as a table file or using innodb_file_per_table.
Table files are pre-created per your MySQL instance. You typically create a large amount of space and monitor it. When it starts filling up, you need to configure another file and restart your server. You can also set it to autoextend, so that it will add a chunk of space to the last table file when it starts to fill up. I typically don't use this feature, as you never know when you'll take the performance hit for extending the table. This page talks about configuring it.
I've never used a whole disk as a table file, but it can be done. Instead of pointing to a file, I believe you point your InnoDB table files at the un-formatted, unmounted device.
innodb_file_per_table makes InnoDB tables act like MyISAM tables. Each table gets its own table file. Last time I used this, the table files did not shrink if you deleted rows from them. When a table is dropped or altered, the file resizes.
The Archive engine is a gzipped MyISAM table.
A memory table doesn't use disk at all. In fact, when a server restarts, all the data is lost.
Merge tables are like a poor man's partitioning for MyISAM tables. It causes a bunch of identical tables to be queried as if there were one. Aside from the FRM table definition, no files exist other than the MyISAM ones.
CSV tables are wrappers around CSV files. The usual file system limits apply here. They are not too fast, since they can't have indexes.
I don't think anyone uses BDB any more. At least, I've never used it. It uses a Berkly database as a back end. I'm not familiar with its restrictions.
Federated tables are used to connect to and query tables on other database servers. Again, there is only an FRM file.
The Blackhole engine doesn't store anything locally. It's used primarily for creating replication logs and not for actual data storage, since there is no data storage :)
MySQL Cluster is completely different: it stores just about everything in memory (recent editions allow disk storage) and is very different from all the other engines.
what you describe is roughly true for MyISAM tables. for InnoDB tables the picture is different, and more similar to what other DBMSs do: one (or a few) big file with complex internal structure for the whole server. to optimize it, you can use a whole disk (or partition) as a file. (at least in unix-like systems, where everything is a file)
I am running a mysql server. I would like to somehow make sure that the whole DB is loaded into the ram as I heard it would be alot faster.
Is this true? and how do I vertify it?
If you are using InnoDB, then you can set innodb_buffer_pool_size to be at least as large as your data and indexes. Then as you access data, it will be cached into memory and all subsequent accesses will occur via memory.
If you wanted to pre-cache the data, you could do some of your common queries, or full table/index scans to force it to load all of the data into memory.
There is an init-file option for mysqld which can have the server automatically execute some commands on startup, where you could include the pre-caching queries to execute.
Keep in mind you don't want to run out of memory, so be careful with setting innodb_buffer_pool_size too large.
Well, one option is to use a ramdrive.. though it's not permanent.
http://www.linuxforums.org/forum/misc/670-how-create-ram-drive-linux.html
(Just for completeness,) You could use HEAP as table engine. But keep in mind that whatever data you put in a HEAP table is gone when you reboot the server.
Tables in RAM are very fast.
You can change your existing tables by altering their storage engine. But make a backup if you want to use the data after a server restart.
ALTER TABLE test ENGINE = MEMORY;
Keep in mind, that MEMORY tables cannot contain BLOB or TEXT columns and data and index size is limited to available memory.