One of our users is using Cloud SQL(MySQL).
They turn on general logs flag and log_output is file.
They need these general logs because of some special circumstances.
MySQL generates about 8TB of general logs and these logs result a bigger use of the disk.
Here is the tricky part:
They want to remove these general logs file [1] to decrease the size of the disk.
However, this is their production database. They afraid this operation will impact their database's performance.
Since these log files are located in /var/log/mysql.log, the remove logs operation will execute on the OS level, right? -> This is the part we are not so sure.
If our user executes this truncateAPI, will this operation affect their database's performance?
Is there any best practice for this kind of situation?
P.S: Our user doesn't want to turn off general logs flag. They will try to truncate these logs once for a while. But for now, they need to truncate the huge amount of logs that they accumulated in the past few momths.
[1] https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances/truncateLog
I understand that you have turned on general logs flag and log_output is FILE and you want to remove these general logs files to decrease the size of the disk.
According to to the official documentation link:
To make your general or slow query logs available, enable the
corresponding flag and set the log_output flag to FILE. This makes the
log output available using the Logs Viewer in the Google Cloud
Platform Console. Note that Stackdriver logging charges apply.
If log_output is set to NONE, you will not be able to access the logs.
If you set log_output to TABLE, the log output is placed in a table in
your database. If this table becomes large, it can affect instance
restart time or cause the instance to lose its SLA coverage; for this
reason, the TABLE option is not recommended. If needed, you can
truncate your log tables by using the API. For more information, see
the instances.truncateLog reference page.
Instances: truncateLog truncate MySQL general and slow query log tables.
If I understood correctly, you can not "truncate the huge amount of logs that they accumulated in the past few months" because you did not set log_output to TABLE, therefore there are no tables to be truncated.
Regarding database performance: TRUNCATE TABLE Statement
On a system with a large InnoDB buffer pool and
innodb_adaptive_hash_index enabled, TRUNCATE TABLE operations may
cause a temporary drop in system performance due to an LRU scan that
occurs when removing an InnoDB table's adaptive hash index entries.
The problem was addressed for DROP TABLE in MySQL 5.5.23 (Bug
13704145, Bug #64284) but remains a known issue for TRUNCATE TABLE (Bug #68184).
Here you can check MySQL Server Log Maintenance.
Removing MySQL general_log FILES should not impact the performance of the database.
Related
I have a problem with MySQL in which the auto-generating of many dump file stored on the log file, and it cause a problem in my internal storage.
Please if you have any solution.
It sounds like you might have enabled the InnoDB Monitor, which outputs statistics and reports to the error log file every 15 seconds.
https://dev.mysql.com/doc/refman/8.0/en/innodb-enabling-monitors.html
Disable the monitors with these statements:
SET GLOBAL innodb_status_output=OFF;
SET GLOBAL innodb_status_output_locks=OFF;
In older versions of MySQL, instead of configuration options, they used an odd way of enabling the monitor. It was to create a table in the InnoDB engine named innodb_monitor or innodb_lock_monitor or innodb_tablespace_monitor. It didn't matter what columns were in this table or which schema you created it in. The existence of a table by those names would start the monitor output to the logs. You need to use either DROP TABLE or RENAME TABLE for each of those to disable the monitor output.
In addition, you should configure your MySQL Server to do log rotation, so the error log never fills the storage. There are many blogs describing how to do this, here's one example: https://scalegrid.io/blog/managing-mysql-server-logs-rotate-compress-retain-delete/
I have a MySQL database in my production environment.Which had about 430 million row, of which 190 million rows were not of any use, so I started deleting these rows range by range, in night, as it would have affected my apps performance in daytime.
Now when I am seeing in my monitoring app, I am seeing 100%IO, of which maximum is write (12-30MB/s). (400-500 writes/sec)
But when I am checking process list I don't find any INSERT or UPDATE query or any rollback.
What can be the possible issue or how can I find any hidden query which may be writing in MySQL.
(In IOTP, I found that write operations are being done by mysqld only)
One more thing, I can see write with 80MB/s in IOTOP , but when I am checking directory size in / , I don't see any rise in any directory size.
Back away slowly... and wait.
InnoDB doesn't change the actual data in the tablespace files with each DML query.
It writes the changes to memory, of course, and then the redo log, at which point they are effectively "live" and safely persisted to disk... but they are not yet applied to the actual data (tablespace) files. InnoDB then syncs the changes to the data files in the background but in the mean time, other queries use a combination of the tablespace and log contents to determine what the "official" table data currently contains. This is, of course, an oversimplification, but MVCC necessarily means the physical data is a superset, though not necessarily a proper superset, of the logical data.
That's very likely to be the explanation for what you are seeing now.
It makes sense that free/used disk space isn't changing, because finalizing the deletion of those rows will only really be marking the space inside the tablespace files as unused. They shouldn't grow or shrink.
Resist the temptation to try to "fix" it and whatever you do, don't restart the server... because the best possible outcome is that it will pick up where it left off because it still has work to do.
SHOW ENGINE INNODB STATUS takes some practice to interpret but will likely be another key to the puzzle.
Is the delete operation still undergoing? DELETE can be extremely slow and generate a lot of writes. It is often better to create a new identical table and copy the rows you want to KEEP over to it and then switch it with the production table instead of delete stuff in the production table directly.
If the DELETE has already finished and you suspect that there are other queries running, you can enable query log for a few seconds and see which queries are executed:
TRUNCATE TABLE mysql.general_log;
SET GLOBAL log_output = 'TABLE';
SET GLOBAL general_log = 'ON';
SELECT SLEEP(10);
SET GLOBAL general_log = 'OFF';
Then SELECT from mysql.general_log to see which queries executed during the 10 seconds sleep.
I have a datawarehouse on mySQL size being almost 1.4 TB
when i checked the number of files the mysql server daemon is accessing I was surprised.
cmd user : lsof | grep mysql
cnd2 used : lsof | grep mysql | wc -l
2598
Please suggest how can this be reduced.What is the impact of the same on mysql performance.
Very likely, it's possible to reduce the number of files held open by MySQL. But note that this will likely decrease performance. In terms of performance, what's more important (than the number of open files) is the closing and opening of files. (That's more expensive than holding files open.)
MySQL caches open tables, keeping recently used files open. This avoids the performance overhead associated with closing files, and then reopening them again later. The limit on the number of tables held open is set with the table_open_cache variable.
The limit on the number of files MySQL has open is set in the max_open_files variable.
Reference: http://dev.mysql.com/doc/refman/5.5/en/table-cache.html
Note that a MyISAM table is stored on disk as three files: tablename.frm, tablename.MYD and tablename.MYI. If the table is partitioned, each partition requires separate files.
With InnoDB engine, if the innodb_file_per_table option is set when a table is created, that's two files, tablename.frm and tablename.idb.
If multiple sessions are accessing a table, there will be multiple "open files" for that table.
There's also a couple of innodb log files, as well as the mysql log file (in logging is enable), the mysql general log, slow query log, etc.
Any temporary table, derived table or an intermediate result set that exceeds the limit on the size of tables stored "in memory", that's additional files will be spun out to disk, and potentially more open files.
If you are hitting a "too many open files" OS error, you can have the OS limit increased. If that's not possible, then decrease the max_open_files variable; you may also need to reduce some other related variables as well.
If you are looking at limiting the number of open files as a solution to slow performance, it's possible you are barking up the wrong tree, because reducing the number of files that MySQL can hold open will likely decrease performance.
I have a very large table with around 1M records. Due to bad performance, I have optimized the queries and needed to change the index.
I changed it using ALTER, now I am really not sure how this works in InnoDB. Do I need to restart MySQL server? If I need to restart MySQL server, how do I keep data integrity between tables (so that I don't miss data which is there in memory and did not get written to DB)?
I Googled and found that in the case of MySQL restart, I need to use global variable innodb_fast_shutdown -- what does it do when I set it and what if I don't? It is not very clear.
I am new to MySQL area with InnoDB. Any help is really appreciated.
So changed it using ALTER, now i am really not sure about how this works in innodb?
You are saying you added the index with ALTER TABLE ... ADD INDEX ... (or ADD KEY -- they are two ways of asking for exactly the same thing) presumably?
Once the ALTER TABLE finishes executing and your mysql> prompt returns, there is nothing else needed. At that point, the table has its new index and the index is fully populated.
You're done, and there is no need to restart the server.
Since you mentioned it, I'll also try to help clear up your misconceptions about innodb_fast_shutdown and the memory/disk divide in InnoDB.
InnoDB makes a one-time request for a block of memory the size of innodb_buffer_pool_size from the operating system when MySQL server starts up, in this example from the MySQL error log from one of my test servers:
130829 11:27:30 InnoDB: Initializing buffer pool, size = 4.0G
This is where InnoDB stores table and index data in memory, and the best performance is when this pool is large enough for all of your data and indexes. When rows are read, the pages from the tablespace files are read into the buffer pool first, then data extracted from there. If changes are made, the changes are written to the in-memory copies of table data and indexes in the buffer pool, and eventually they are flushed to disk. Pages in the pool are either "clean" -- meaning they are identical to what's on disk, because they've not been changed since they were loaded, or if changed, the changes have already been written to disk -- or "dirty" meaning they do not match what is on disk.
However, InnoDB is ACID-compliant -- and this could not be true if it only wrote the changes in memory and the changes were not persisted immediately somewhere prior to the in-memory changes even being made ... and that "somewhere" is the redo log -- on disk -- that stores what changes to be made in memory, immediately, in a format that allows this operation to be much faster than updating the actual tablespace files themselves in real-time would be.
In turn, the innodb_fast_shutdown variable determines whether MySQL finishes up everything written to the redo log before shutdown -- or after it starts back up. It works fine, either way, but if you need to shut the server down faster, it's faster and perfectly safe to let it pick everything up later, no matter what changes you have made.
Importantly, I don't know what you have read, but in routine operations, you never need to mess with the value of innodb_fast_shutdown unless you are shutting down in preparation for doing an upgrade to your version of MySQL server (and then it is primarily a safety precaution). The data on disk is always consistent with what is in memory, either because the tablespace files are already consistent with the memory representation of the data, or because the pending changes to the tablespace files are safely stored in the redo log, where they will be properly processed when the server comes back online.
In the case of ALTER TABLE anything pending for the table prior to the ALTER would have already been take care of, since InnoDB typically rebuilds entire the table in response to this command, so the only possible "pending" changes would be DML that occurred after the ALTER.
I server I am working on, general mysql log table is taking near 200GB space, which is huge. So, I am planning to clear it up with:
TRUNCATE table mysql.general_log
Is it ok? Will it cause any issue? I am concerned as the server is live and big application. Thanks.
It will definitely cause a problem unless it is disabled and then you truncate. If you truncate while it is enabled. Truncate will lock the table if table is huge, since mysql.general_log engine is either CSV or MyISAM ,meanwhile newly made entries will try to be written in general log table causing a lock. So for safety do like this
mysql> SET GLOBAL general_log=OFF;
mysql> TRUNCATE table mysql.general_log;
mysql> SET GLOBAL general_log=ON;
It won't cause problems, but you will lose all the old log entries, so Ravindra's advice above is good.
You can do a backup with:
mysqldump -p --lock_tables=false mysql general_log > genlog.sql
Do you need to have the general log on all the time? I usually only turn it on when I'm troubleshooting performance issues. MySQL logs EVERYTHING there (client connects, disconnects, and EVERY statement). In most systems, the logs get very big very quickly. There is also some performance overhead for this.