I have a server with limited storage
/apps: available 60GB ; used 59GB ; free 1GB
We have mysql datadirectory pointed to /apps
datadir /apps/mysql/data/
/apps/mysql/data/ibdata 50GB
Yesterday we observed the server shutdown with table full error which is a disc full basically.
So we have purged one old month data in one of the tables and have 10GB free space under innodb. But still there only 1GB free space on the file system.
Now if I try to insert again, whether the innodb again try to auto extend the ibdata file or it used the 9GB free space?
Related
What are the recommended configurations for allocating RAM to MySQL?
My environment: Ubuntu 14 machine with 96G RAM, MySQL Ver 14.14 Distrib 5.7.
And how to set those configurations?
Thank you in advance!
Assuming you are are using InnoDB as engine than please see here https://www.percona.com/blog/2013/09/20/innodb-performance-optimization-basics-updated/
MySQL Innodb Settings
From 5.5 InnoDB is the default engine, so these parameters are even more important for performance than before. The most important ones are:
innodb_buffer_pool_size: InnoDB relies heavily on the buffer pool and should be set correctly, so be sure to allocate enough memory to it. Typically a good value is 70%-80% of available memory. More precisely, if you have RAM bigger than your dataset setting it bit larger should be appropriate with that keep in account of your database growth and re-adjust innodb buffer pool size accordingly. Further, there is improvement in code for InnoDB buffer scalability if you are using Percona Server 5.1 or Percona Server 5.5 You can read more about it here.
innodb_buffer_pool_instances: Multiple innodb buffer pools introduced in InnoDB 1.1 and MySQL 5.5. In MySQL 5.5 the default value for it was 1 which is changed to 8 as new default value in MySQL 5.6. Minimum innodb_buffer_pool_instances should be lie between 1 (minimum) & 64 (maximum). Enabling innodb_buffer_pool_instances is useful in highly concurrent workload as it may reduce contention of the global mutexes.
Dump/Restore Buffer Pool: This feature speed up restarts by saving and restoring the contents of the buffer pool. This feature is first introduced in Percona Server 5.5 you can read about it here. Also Vadim benchmark this feature You can read more about it in this post. Oracle MySQL also introduced it in version 5.6, To automatically dump the database at startup and shutdown set innodb_buffer_pool_dump_at_shutdown & innodb_buffer_pool_load_at_startup parameters to ON.
innodb_log_file_size: Large enough InnoDB transaction logs are crucial for good, stable write performance. But also larger log files means that recovery process will slower in case of crash. However this is not such big issue since great improvements in 5.5. Default value has been changed in MySQL 5.6 to 50 MB from 5 MB (old default), but it’s still too small size for many workloads. Also, in MySQL 5.6, if innodb_log_file_size is changed between restarts then MySQL will automatically resize the logs to match the new desired size during the startup process. Combined log file size is increased to almost 512 GB in MySQL 5.6 from 4 GB. To get the optimal logfile size please check this blog post.
innodb_log_buffer_size: Innodb writes changed data record into lt’s log buffer, which kept in memory and it saves disk I/O for large transactions as it not need to write the log of changes to disk before transaction commit. 4 MB – 8 MB is good start unless you write a lot of huge blobs.
innodb_flush_log_at_trx_commit: When innodb_flush_log_at_trx_commit is set to 1 the log buffer is flushed on every transaction commit to the log file on disk and provides maximum data integrity but it also has performance impact. Setting it to 2 means log buffer is flushed to OS file cache on every transaction commit. The implication of 2 is optimal and improve performance if you are not concerning ACID and can lose transactions for last second or two in case of OS crashes.
innodb_thread_concurrency: With improvements to the InnoDB engine, it is recommended to allow the engine to control the concurrency by keeping it to default value (which is zero). If you see concurrency issues, you can tune this variable. A recommended value is 2 times the number of CPUs plus the number of disks. It’s dynamic variable means it can set without restarting MySQL server.
innodb_flush_method: DIRECT_IO relieves I/O pressure. Direct I/O is not cached, If it set to O_DIRECT avoids double buffering with buffer pool and filesystem cache. Given that you have hardware RAID controller and battery-backed write cache.
innodb_file_per_table: innodb_file_per_table is ON by default from MySQL 5.6. This is usually recommended as it avoids having a huge shared tablespace and as it allows you to reclaim space when you drop or truncate a table. Separate tablespace also benefits for Xtrabackup partial backup scheme.
I have a C# application performing some database operations on a MySQL 5.7 server. Once the complete system hunp up and I had to hard-reset it. When it comes to a specific table read/write operation now the database server crashes. The windows log shows
InnoDB: Trying to access page number 286720 in space 29,
space name myInstance/myTable, which is outside the tablespace bounds.
Byte offset 0, len 16384, i/o type read.
I tried to use mysqlcheck --repair but it fails because note : The storage engine for the table doesn't support repair.
I've read some advices that say I should start MySQL in a recovery mode, so I added
[mysqld]
innodb_force_recovery=4
to the my.ini config file whereupon I should be able to use mysqldump to export the affected database table. But unfortunatelly I am not.
mysqldump: Error 2013: Lost connection to MySQL server
during query when dumping table `myTable` at row: 1246
Edit:
I checked the error log again and found lots of entries saying
[ERROR] C:\Program Files\MySQL\MySQL Server 5.7\bin\mysqld.exe:
The table 'myTable' is full
I'm running the server on a Windows 32bit OS with an NTFS formatted partition. The myTable.ibd file size is around 4.5 GB, checking C.10.3 Limits on Table Size states a file size limit of "2TB (possibly larger)" for Win32 w/ NTFS.
While checking the reasons for my error the only possible cause I found was a full InnoDB tablespace. The solution might be "Changing the Number or Size of InnoDB Redo Log Files" although the coherence is a bit vague to me. Nevertheless I increased the size of the Redo Log Files from 48M to 100M. But nothing changed.
If I perform a SQL select * from myTable order by Id desc the server crashes instantly. Error log entry is exactly the same as above.
I checked the chapter 15.7.1 Resizing the InnoDB System Tablespace as well and found out that innodb_data_file_path is not explicitly specified.
Any ideas what I can do now? Thanks a lot!
InnoDB cannot repair corruption in tablespaces. This was never implemented and mysqlcheck won't help in any way.
The corruption is in space id 29 which is table myInstance.myTable. To repair it you need to dump all records from this table with innodb_force_recovery. Try all values from 4 to 6 until MySQL doesn't crash. Then drop the table and reload the dump.
If MySQL crashes even with innodb_force_recovery=6 then restore the table from backup.
If you don't have backup - use script http://bazaar.launchpad.net/~percona-dev/percona-data-recovery-tool-for-innodb/trunk/view/head:/fetch_data.sh . It will fetch as many records as it can.
One possible cause could be that the ib_logfile* files are corrupted.
To fix this, remove these files using rm ib_logfile*.
Where are these files?
These files are in mysql datadir. The location of datadir depends on the OS. Check my.cnf, in osx that should be in /usr/local/etc.
I am using innodb. I have a problem with app performance. When I run mysqlturner.pl I get:
[--] Data in InnoDB tables: 8G (Tables: 1890)
[!!] Total fragmented tables: 1890
Ok, i have run mysqlcheck -p --optimize db.
I have diceded that innodb_file_per_table is disabled. The last 2 years a database wasn't reindex. How I can do it?
Make mysqldump?
Stop mysql serice?
insert enable innodb_file_per_table into my.cnf?
Start mysql serice?
Import from mysqldump?
run mysqlcheck -p --optimize db?
Will everything be ok?
Tables fragmented
Bogus.
All InnoDB tables are always (according to that tool) fragmented.
In reality, only 1 table in 1000 needs to be defragmented.
OPTIMIZE TABLE foo; will defrag a single table. But, again, I say "don't bother".
ibdata1 bloated
On the other hand... If you are concerned about the size of ibdata1 and Data_free (SHOW TABLE STATUS) shows that a large chunk of ibdata1 is "free", then the only cure is painful: As already mentioned: dump, stop, remove ibdata1, restart, reload.
But... If you don't have enough disk space to dump everything, you are in deep weeds.
You have 8GB of tables? (Sum Data_length and Index_length across all InnoDB tables.) If ibdata1 is, say 10GB, don't bother. If it is 100GB, then you have a lot of wasted space. But if you are not running out of disk space, again I say "don't bother".
If you are running out of disk space and ibdata1 seems to have a lot "free", then do the dump etc. But additionally: If you have 1890 tables, probably most are "tiny". Maybe a few are somewhat big? Maybe some table fluctuates dramatically in size (eg, add a million rows, then delete most of them)? I suggest having innodb_file_per_table ON when creating big or fluctuating tables; OFF for tiny tables.
Tiny tables take less space when living in ibdata1; big/fluctuating tables can be cleaned up if needed via OPTIMIZE if ever needed.
Yes that would do it. Take a backup before you start and test the entire process out first on an copy of the host, esp. the exact mysqldump command, you don't want databases like 'information_schema' in the dump.
Also ensure that other services cannot connect to your db once you start the export - any data they change would be lost during the import.
There are detailed instructions in this previous question:
Howto: Clean a mysql InnoDB storage engine?
My ibdata file size is increasing in mysql5.5 version server and it has been set up with the master-slave replication. Server contains 6 databases. What are the precautions to be taken during the process of reducing the size of ibdata file in an master slave replication setup when compared to normal server. I am using the Mysql 5.5 version.
My ibdata file size is increasing in mysql5.5 version server and it has been set up with the master-slave replication. Server contains 6 databases. What are the precautions to be taken during the process of reducing the size of ibdata file in an master slave replication setup when compared to normal server.
They are the same precautions for both situations.
Use a separate file per table, by including
innodb_file_per_table
in your my.ini file (note there is no = or value)
Use ALTER TABLE ... ENGINE=INNODB to change existing tables to single file format
Use OPTIMIZE TABLE to reclaim space as needed.
I am using Ubuntu for PHP/MySQL development. Ubuntu is installed on VM and unfortunately VM has been assigned very low disk space (40 GB).
After 1st startup the disk had more than 20 GB disk space available.
Over past week I had to import and delete the db.sql.gz file quite some times. Now I only have 50 MB disk space left. Even though I have deleted all the databases using phpmyadmin, I suspect they still use the disk space.
How can I recover this disk space?