I have a large datatable of around 60GB. This table had alot of unused rows and after deleting about 6GB of them I have noticed that table size stayed the same (60GB) and their was an "optimize" message within phpmyadmin. So i clicked to optimize, but i didnt have enough space on the hd. So i had to halt the process and restarted mysql.
After i logged back in I happen to have a problem with my table, when I try to access it i get message similar to:
"Table 'table_name' is marked as crashed and last (automatic?) repair failed"
Right now I have 3.5GB of space to use on the hard drive. What would be the best way forward to repair, fix and shirt this particular table?
At the moment my plan is to download full database from the server onto a local hard drive; after which I will delete unused data (it will most likely be 59.99GB of it) and then to either copy or re-import data back into live database.
Thanks.
May you free space exporting only another heavy table and truncate this?
You get more space and after repair affected table you only must import a table, not full database.
Related
first off, not a DB guy. here is the problem, data drive for the database is 96% full. in the my.cnf there is a line that has the following, (only showing part due to space)
innodb_data_file_path=nmsdata1:4000M;nmsdata2:4000M;
going up to
nmsdata18:4000M:autoextend
so in the folder where the files are stored files 1-17 are 4gb in size, file 18 is 136gb as of today.
I inherited the system and it has no vendor support or much documentation.
I can see there are a few tables that are really large
Table_name NumRows Data Length
---------- ------- -----------
pmdata 100964536 14199980032
fault 310864227 63437946880
event 385910821 107896160256
I know ther is a ton of writes happening and there should be a cron job that tells it to only keep the last 3 months data but I am concerned the DB is fragmented and not releasing space back for use.
so my task is to free up space in the DB so the drive does not fill up.
This is a weakness of innodb: tablespaces never shrink. They grow, and even if you "defragment" the tables, they just get written internally to another part of the tablespace, leaving more of the tablespace "free" for use by other data, but the size of the file on disk does not shrink.
Even if you DROP TABLE, that doesn't free space to the drive.
This has been a sore point for InnoDB for a long time: https://bugs.mysql.com/bug.php?id=1341 (reported circa 2003).
The workaround is to use innodb_file_per_table=1 in your configuration, so each table has its own tablespace. Then when you use OPTIMIZE TABLE <tablename> it defragments by copying data to a new tablespace, in a more efficient, compact internal layout, and then drops the fragmented one.
But there's a big problem with this in your case. Even if you were to optimize tables after setting innodb_file_per_table=1, their data would be copied into new tablespaces, but that still wouldn't shrink or drop the old multi-table tablespaces like your nmsdata1 through 18. They would still be huge, but "empty."
What I'm saying is that you're screwed. There is no way to shrink these tablespaces, and since you're full up on disk space, there's no way to refactor them either.
Here's what I would do: Build a new MySQL Server. Make sure innodb_file_per_table=1 is configured. Also configure the default for the data file path: innodb_data_file_path=ibdata1:12M:autoextend. That will make the central tablespace small from the start. We'll avoid expanding it with data.
Then export a dump of your current database server, all of it. Import that into your new MySQL server. It will obey the file-per-table setting, and data will create and fill new tablespaces, one per table.
This is also an opportunity to build the new server with larger storage, given what you know about the data growth.
It will take a long time to import so much data. How long depends on your server performance specifications, but it will take many hours at least. Perhaps days. This is a problem if your original database is still taking traffic while you're importing.
The solution to that is to use replication, so your new server can "catch up" from the point where you created the dump to the current state of the database. This procedure is documented, but it may be quite a bit of learning curve for someone who is not a database pro, as you said: https://dev.mysql.com/doc/refman/8.0/en/replication-howto.html
You should probably get a consultant who knows how to do this work.
I have a MySQL table named requests. It's growing so fast. Currently, it has 3 million rows. Also, the current table engine is "InnoDB".
Few days ago, I got this error:
ERROR 1114 (HY000): The table is full
I've resolved the problem (temporary I guess) by adding this to MySQL configuration:
innodb_data_file_path = ibdata1:10M:autoextend
But still sometimes my serves goes to down and I have to restart it to make it available. Recently, I've made that table empty and I got rid of being down.
But it will be full again soon. Any idea how can I fix that issue? Should I use another engine for that table? Should I use compressed type-row for it? or what?
Noted that, sometimes I need to select from that table and it has one index on a column (in addition to pk)
You are limited by disk space. You must keep an eye on that and take action before you get to "table full".
If you store the data in a MyISAM table, you can go twice as long before getting "table full". If you simply write to a disk file (no database, no queries; etc), you can squeeze a little more in before "disk full".
Do you need to "purge" old data? That way you could continue to receive new data without ever hitting "table full". The best way to do that is via InnoDB and PARTITION BY RANGE(TO_DAYS(...)). If you purge anything over a month old, use daily partitions. For, say, 90 days, use weekly partitions. More discussion: http://mysql.rjweb.org/doc.php/partitionmaint
What will you do with the data? Analyze it? Search it? What SQL queries do you envision? Answer those; there could be other tips.
I accidentally wrote a code with infinity loop, and added A LOT OF ROWS IN A TABLE. The loop ran for around 20-60 sec (not sure).
So, i tried to TRUNCATE "lastrequestdone"; but, it is not working. I still the same amount of rows in the table.
I tried to drop the table and re-create it, but i found out the rows still exist even after dropping the table and recreating it.
I also started to get this message in PHPMyAdmin
The phpMyAdmin configuration storage is not completely configured, some extended features have been deactivated. Find out why.
Or alternately go to 'Operations' tab of any database to set it up there.
and when i click on "find out why"
I get this message
Configuration of pmadb… not OK
General relation features Disabled
Create a database named 'phpmyadmin' and setup the phpMyAdmin configuration storage there.
PHP is really fast, I have no idea, how many rows i added....
P.S: I tried to drop and re-create the table again after 10 minutes, and it is empty now. Sorry for rushing to stackoverflow, I was scared to death.
But, I still get those error in yellow background above.
Long story short. I had a very large corrupt InnoDB table. Tried a number of things to rebuild/recover the table. (I was finally successful) However, one of the things I tried was building a new table with Myisam engine with force innodb reovery on, however there was another crash along the way and I'm left with a .TMD file for that table.
Just curious if I'm safe to delete this file? Table does not show up anywhere in the database, via show tables, drop table doesn't do anything. etc. At this point its just taking up disk space in my data directory.
The .TMD file is an intermediate data file for a table that needs to recreate its data file.
So you can remove it because it's normally used as a temporarly file but you could rename the file and check what happens just in case.
I foolishly tried to add a column to a table that I did not have enough space on disk to copy and had to kill it and expand my RDS instance's storage capacity to avert a site crash. I would like to do it again (this time with enough disk space) but I can't seem to get back to my pre-query free storage levels. My query was to create a table like a giant table, add a column and then insert the entire contents of the old table together with null into the new table. I tried CALL mysql.rds_rotate_slow_log; and CALL mysql.rds_rotate_general_log; but judging by my AWS Cloudwatch panel, I'm still down ~10GB from my pre-query levels. No lines were successfully inserted into the new table. Is there some "clear hdd cache" command or something like that? Since it's RDS, I don't have access to the instance that's running it but I do have master user and RDS CLI access.
EDIT: It seems my problem may be related to giant ibdata files but since I don't have root access, I can't really execute the solutions mentioned in How to shrink/purge ibdata1 file in MySQL
The solution was to drop the new table. I didn't think that anything was stored in the new table because select count(*) from new_table; returned 0 but I guess the temporary data was tied in to the new table anyway. I'm not sure how exactly this works from a database structural point of view but fortunately it did what I wanted.
Bottom line: killed inserts still use storage space.
If somebody can explain why this is the case, it would be helpful for the future.