first off, not a DB guy. here is the problem, data drive for the database is 96% full. in the my.cnf there is a line that has the following, (only showing part due to space)
innodb_data_file_path=nmsdata1:4000M;nmsdata2:4000M;
going up to
nmsdata18:4000M:autoextend
so in the folder where the files are stored files 1-17 are 4gb in size, file 18 is 136gb as of today.
I inherited the system and it has no vendor support or much documentation.
I can see there are a few tables that are really large
Table_name NumRows Data Length
---------- ------- -----------
pmdata 100964536 14199980032
fault 310864227 63437946880
event 385910821 107896160256
I know ther is a ton of writes happening and there should be a cron job that tells it to only keep the last 3 months data but I am concerned the DB is fragmented and not releasing space back for use.
so my task is to free up space in the DB so the drive does not fill up.
This is a weakness of innodb: tablespaces never shrink. They grow, and even if you "defragment" the tables, they just get written internally to another part of the tablespace, leaving more of the tablespace "free" for use by other data, but the size of the file on disk does not shrink.
Even if you DROP TABLE, that doesn't free space to the drive.
This has been a sore point for InnoDB for a long time: https://bugs.mysql.com/bug.php?id=1341 (reported circa 2003).
The workaround is to use innodb_file_per_table=1 in your configuration, so each table has its own tablespace. Then when you use OPTIMIZE TABLE <tablename> it defragments by copying data to a new tablespace, in a more efficient, compact internal layout, and then drops the fragmented one.
But there's a big problem with this in your case. Even if you were to optimize tables after setting innodb_file_per_table=1, their data would be copied into new tablespaces, but that still wouldn't shrink or drop the old multi-table tablespaces like your nmsdata1 through 18. They would still be huge, but "empty."
What I'm saying is that you're screwed. There is no way to shrink these tablespaces, and since you're full up on disk space, there's no way to refactor them either.
Here's what I would do: Build a new MySQL Server. Make sure innodb_file_per_table=1 is configured. Also configure the default for the data file path: innodb_data_file_path=ibdata1:12M:autoextend. That will make the central tablespace small from the start. We'll avoid expanding it with data.
Then export a dump of your current database server, all of it. Import that into your new MySQL server. It will obey the file-per-table setting, and data will create and fill new tablespaces, one per table.
This is also an opportunity to build the new server with larger storage, given what you know about the data growth.
It will take a long time to import so much data. How long depends on your server performance specifications, but it will take many hours at least. Perhaps days. This is a problem if your original database is still taking traffic while you're importing.
The solution to that is to use replication, so your new server can "catch up" from the point where you created the dump to the current state of the database. This procedure is documented, but it may be quite a bit of learning curve for someone who is not a database pro, as you said: https://dev.mysql.com/doc/refman/8.0/en/replication-howto.html
You should probably get a consultant who knows how to do this work.
Related
I am using mysql5, and I want to shrink some 'deleted' spaces in ibd file. I already search for 'optimize table' option, but I cannot use it now, because it is a very critical table. It uses 19G in Mysql, but 33G in OS. I just figure out 33G is getting an increase.
I heard that it reuses in 33G spaces, like the black area. But why it increases every day?
MySQL does not reuse these free spaces? I mean, about 24G is never reused?
Thanks for the read, I hope not my poor English makes you are confused when you read it.
The option you are looking for is innodb_use_trim=1, in combination with innodb_file_per_table=1, if the version of MySQL you are using is new enough to have it. This will release back empty pages by punching holes in the tablespace files.
The valid parts of a table are scattered throughout the .ibd. file. The Operating System does not have a way to reuse freed up pieces from the middle of a table.
So, InnoDB never shrinks a tablespace, only grows it.
Meanwhile, new INSERTs will fill in some of the freed space.
What led to this situation? Did you DELETE lots of rows? If you expect to do another big DELETE, see the following advice for next time: http://mysql.rjweb.org/doc.php/deletebig
Meanwhile, if the table was created with innodb_file_per_table = 1, then OPTIMIZE TABLE will copy the table over and release the old copy to the OS. But... You need extra disk space to do the task. And it will block some operations against the table. (The details depend on which version of MySQL/MariaDB you are using.)
Why do you need to recoup the space? If you are not running low on disk space, then does it really matter?
Low on space?
Show us SHOW CREATE TABLE. There may be some tips on making the table smaller. (This does not solve the problem, but delays future problems.) For example, if an INT (4 bytes) can be changed to SMALLINT (2 bytes), that could help shrink the table. Or an INDEX might be redundant; DROPping it would help.
The ALTER to change such things might do the OPTIMIZE at the same time. (In MySQL 5.0, virtually all ALTERs are performed by copying the table over.)
Another tip when disk space is terribly tight... (Assuming innodb_file_per_table has always been on): OPTIMIZE smaller tables; Some of them may shrink a little bit, thereby giving you some extra disk space.
Another tip:
Determine which tables are in their own tablespaces.
Find out how much free space is in ibdata1.
SET innodb_file_per_table=OFF;
ALTER TABLE t ENGINE=InnoDB; for some of the small tables is step 1. But stop before Data_free of ibdata1 drops too small.
That tip will free up the space for those table that could be moved without causing ibdata1 to grow.
Mysql 5.05 that is hosting an older application that still gets lots of love from the users. Unfortunately, I'm not anything other than a hack dba at best, and am very hesitant in my skills to safely migrate to a new version of the database unless absolutely necessary. We are in the process of procuring a new application to take over responsibilities for the old application, but are probably a year out or so.
Anyway, I was patching the application the other day and added a column to a table, the command took a while to complete and in the meantime nearly filled up my drive hosting the datafiles. (table is roughly 25G) I believe this was a function of the creation of a temporary table. For reasons I'm not clear on, the space did not become free again after the column was added; i.e., I lost roughly 25G of disk space. I believe (?) this was due to the fact that the database was created with a single datafile; I'm not really sure on the whys, but I do know that I had to free up some space elsewhere to get the drive to an operable state.
That all being said, I've got the column added, but it is worthless to the application without an index. I held off adding the index trying to figure out if it is going to create another massive, persistent 'temporary' table at index creation time. Can anyone out there give me insight into:
Will a create index and or alter table create index statement result in the creation of a temporary table the same size as the existing table?
How can I recover the space that got added to ibdata1 when I added the column?
Any and all advice is greatly appreciated.
MySQL prior to version 5.1 adds/removes indices on InnoDB tables by building temporary tables. It's very slow and expensive. The only way around this is to either upgrade MySQL to 5.1, or to dump the table with e.g. mysqldump, drop it, recreate it with the new indices, and then restore it from the dump.
You can't shrink ibdata1 at all. Your only solution is to rebuild from scratch. It is possible to configure MySQL so it doesn't use one giant ibdata1 file for all the databases - read that answer and it will explain how to configure MySQL/InnoDB so this doesn't happen again, and also how to safely dump and recreate all your databases.
Ultimately, you probably want to
Make a complete dump of your database
Upgrade to MySQL 5.1 or newer
Turn on InnoDB one-file-per-table mode
Restore the dump.
I have a few very large MySql tables on an Amazon Std EBS 1TB Volume (the file-per-table flag is ON and each ibd file is about 150 GB). I need to move all these tables from database db1 to database db2. Alongwith this, I would also like to move the tables out to a different Amazon Volume (which I think is considered a different Partition/File-System, even if the FileSystem type is the same). The reason I am moving to another volume is so I can get another 1TB space.
Things I have tried:
RENAME TABLE db1.tbl1 TO db2.tbl1 does not help because I cannot move it out to a different volume. I cannot mount a Volume at db2 because then it is considered a different file-system and MYSQL fails with an error:
"Invalid cross-device link" error 18
Created a stub db2.tbl1, stopped mysql, deleted db2's tbl1 and copied over db1's tbl.ibd. Doesn't work (the db information is buried in the ibd?)
I do not want to try the obvious mysqldump-import OR selectinto-loadfile because each table takes a day and a half to move even with most optimizations (foreign-key checks off etc). If I take indexes out before the import , re-indexing takes long and the overall time taken is still too long.
Any suggestions would be much appreciated.
Usually what I would suggest in this case, is to create an ec2 snapshot of the volume and write that snapshot into your larger volume.
You'll need to resize the partition afterwards.
As a sidenote, if your database is that large, EBS might be a major bottleneck. You're better off getting locally attached storage, but unfortunately the process is a bit different.
You might want to use Percona xtrabackup for this:
https://www.percona.com/doc/percona-xtrabackup/LATEST/index.html
one question about NDBCLUSTER.
I inherited the writing of a web site basing on NDBCLUSTER 5.1 solution (LAMP platform).
Unfortunately, who designed the former solution didn't realize that this database engine has strong limits. One, the maximum number of fields a table can have is 128. The former programmer conceived tables with 369 fields in a single row, one for each day of the year plus some key field (he originally worked with MyISAM engine). Ok it must be refactored, anyways, I know.
What is more, the engine needs a lot of tuning: maximum number of attributes for a table (which defaults to 1000, a bit too few) and many other parameters, the misinterpretation or underestimation of which can lead to serious problems once you're in production with your database and you're forced to change something.
Even the fact that disk storage for NDBCLUSTER tables is kind of aleatory if not precisely configured: even if specified in CREATE TABLE statements, the engine seems to prefer keeping data in memory - which explains the speed - but can be a pain if your table on node 1 should suddenly collapse (as it did during testing). All table data lost on all nodes and table corrupted after 1000 records only.
We were on a server with 8Gb RAM, and the table had just 27 fields.
Please note that no ndb_mgm operation for nodes shutdown ran to compromise table data. It simply fell down, full stop. Our provider didn't understand why.
So the question is: would you recommend NDBCLUSTER as a stable solution for a large scale web service database?
We're talking about a database which should contain several millions of records, thousands of tables and thousands of catalogues.
If not that which database would you recommend as the best to accomplish the task of making a nation-level scale web service.
Thanks in advance.
I have a terrible experience with NDBCLUSTER. It's good replacement for memcached with range invalidation, nothing more. The stability and configurability does not exist for this solution. You can not force all processes to listen on specific ports, backup was working but I have to edit bkp files in vim to restore database etc..
Microsoft SQL Server has a nice feature, which allows a database to be automatically expanded when it becomes full. In MySQL, I understand that a database is, in fact, a directory with a bunch of files corresponding to various objects. Does it mean that a concept of database size is not applicable and a MySQL database can be as big as available disk space allows without any additional concern? If yes, is this behavior the same across different storage engines?
It depends on the engine you're using. A list of the ones that come with MySQL can be found here.
MyISAM tables have a file per table. This file can grow to your file system's limit. As a table gets larger, you'll have to tune it as there's index and data size optimizations that limit the default size. Also, this MyISAM documentation page says:
There is a limit of 2^32 (~4.295E+09)
rows in a MyISAM table. If you build
MySQL with the --with-big-tables
option, the row limitation is
increased to (2^32)^2 (1.844E+19) rows.
See Section 2.16.2, “Typical configure
Options”. Binary distributions for
Unix and Linux are built with this
option.
InnoDB can operate in 3 different modes: using innodb table files, using a whole disk as a table file or using innodb_file_per_table.
Table files are pre-created per your MySQL instance. You typically create a large amount of space and monitor it. When it starts filling up, you need to configure another file and restart your server. You can also set it to autoextend, so that it will add a chunk of space to the last table file when it starts to fill up. I typically don't use this feature, as you never know when you'll take the performance hit for extending the table. This page talks about configuring it.
I've never used a whole disk as a table file, but it can be done. Instead of pointing to a file, I believe you point your InnoDB table files at the un-formatted, unmounted device.
innodb_file_per_table makes InnoDB tables act like MyISAM tables. Each table gets its own table file. Last time I used this, the table files did not shrink if you deleted rows from them. When a table is dropped or altered, the file resizes.
The Archive engine is a gzipped MyISAM table.
A memory table doesn't use disk at all. In fact, when a server restarts, all the data is lost.
Merge tables are like a poor man's partitioning for MyISAM tables. It causes a bunch of identical tables to be queried as if there were one. Aside from the FRM table definition, no files exist other than the MyISAM ones.
CSV tables are wrappers around CSV files. The usual file system limits apply here. They are not too fast, since they can't have indexes.
I don't think anyone uses BDB any more. At least, I've never used it. It uses a Berkly database as a back end. I'm not familiar with its restrictions.
Federated tables are used to connect to and query tables on other database servers. Again, there is only an FRM file.
The Blackhole engine doesn't store anything locally. It's used primarily for creating replication logs and not for actual data storage, since there is no data storage :)
MySQL Cluster is completely different: it stores just about everything in memory (recent editions allow disk storage) and is very different from all the other engines.
what you describe is roughly true for MyISAM tables. for InnoDB tables the picture is different, and more similar to what other DBMSs do: one (or a few) big file with complex internal structure for the whole server. to optimize it, you can use a whole disk (or partition) as a file. (at least in unix-like systems, where everything is a file)