I am using Azure MySQL version 5.6
When i am trying to import a large MySQL dump file from linux enviroment into the PaaS of Azure (Azure Database for MySQL servers) using this command :
pv DBFILE.sql | mysql -u username#mysqlserver -h
mysqlservername.mysql.database.azure.com -pPassword DBNAME
I am getting this message:
"The size of BLOB/TEXT data inserted in one transaction is greater
than 10% of redo log size. Increase the redo log size using
innodb_log_file_size."
Is there any way to bypass this error?
I read on the Microsoft documentation that "innodb_log_file_size" is not configurable. Can i split this large dump file into smaller ones and import them all together? Does it make any difference?
The size of the dump file is not the problem. It won't help to split it up.
The problem is that the size of one BLOB or TEXT value on at least one row is greater than 1/10th the size of the innodb log file. You can't split the data to be less than a single BLOB or TEXT value.
According to the Azure documentation you linked to, the value of innodb_log_file_size is fixed at 256MB. This means you cannot import any row with a BLOB or TEXT value of more than 25.6MB. At least you can't import it to an InnoDB table.
The reason is that the redo log file has a fixed size, and the size of the log file creates a limit on the number of modified pages in the InnoDB buffer pool (not one-to-one though, because the format of redo log file records is not the same as pages in the buffer pool). It's kind of an arbitrary ratio, but the purpose of the limit on BLOB/TEXT values is meant to avoid a giant BLOB wrapping around and overwriting part of itself in a small redo log, which would leave the MySQL server in a state that could not recover from a crash. In MySQL 5.5, this was just a recommended limit. In MySQL 5.6, it became enforced by InnoDB, so an INSERT of a BLOB or TEXT that was too large would simply result in an error.
Amazon RDS used to have a similar restriction years ago. They only supported a fixed size for innodb_log_file_size, I recall it was 128MB. It was not configurable.
I was at an AWS event years ago in San Francisco, and I found an opportunity to talk to the Amazon RDS Product Manager in the hall between sessions. I gave him feedback that leaving this setting at a relatively small value without the ability to increase it was too limiting. It meant that one could only insert BLOB/TEXT of 12.8MB or less.
I'm sure I was not the only customer to give him that feedback. A few months later, an update to RDS allowed that variable to be changed. But you must restart the MySQL instance to apply the change, just like if you run MySQL yourself.
I'm sure that Azure will discover the same thing, and get an earful of feedback from their customers.
Related
I have a problem with MySQL in which the auto-generating of many dump file stored on the log file, and it cause a problem in my internal storage.
Please if you have any solution.
It sounds like you might have enabled the InnoDB Monitor, which outputs statistics and reports to the error log file every 15 seconds.
https://dev.mysql.com/doc/refman/8.0/en/innodb-enabling-monitors.html
Disable the monitors with these statements:
SET GLOBAL innodb_status_output=OFF;
SET GLOBAL innodb_status_output_locks=OFF;
In older versions of MySQL, instead of configuration options, they used an odd way of enabling the monitor. It was to create a table in the InnoDB engine named innodb_monitor or innodb_lock_monitor or innodb_tablespace_monitor. It didn't matter what columns were in this table or which schema you created it in. The existence of a table by those names would start the monitor output to the logs. You need to use either DROP TABLE or RENAME TABLE for each of those to disable the monitor output.
In addition, you should configure your MySQL Server to do log rotation, so the error log never fills the storage. There are many blogs describing how to do this, here's one example: https://scalegrid.io/blog/managing-mysql-server-logs-rotate-compress-retain-delete/
first off, not a DB guy. here is the problem, data drive for the database is 96% full. in the my.cnf there is a line that has the following, (only showing part due to space)
innodb_data_file_path=nmsdata1:4000M;nmsdata2:4000M;
going up to
nmsdata18:4000M:autoextend
so in the folder where the files are stored files 1-17 are 4gb in size, file 18 is 136gb as of today.
I inherited the system and it has no vendor support or much documentation.
I can see there are a few tables that are really large
Table_name NumRows Data Length
---------- ------- -----------
pmdata 100964536 14199980032
fault 310864227 63437946880
event 385910821 107896160256
I know ther is a ton of writes happening and there should be a cron job that tells it to only keep the last 3 months data but I am concerned the DB is fragmented and not releasing space back for use.
so my task is to free up space in the DB so the drive does not fill up.
This is a weakness of innodb: tablespaces never shrink. They grow, and even if you "defragment" the tables, they just get written internally to another part of the tablespace, leaving more of the tablespace "free" for use by other data, but the size of the file on disk does not shrink.
Even if you DROP TABLE, that doesn't free space to the drive.
This has been a sore point for InnoDB for a long time: https://bugs.mysql.com/bug.php?id=1341 (reported circa 2003).
The workaround is to use innodb_file_per_table=1 in your configuration, so each table has its own tablespace. Then when you use OPTIMIZE TABLE <tablename> it defragments by copying data to a new tablespace, in a more efficient, compact internal layout, and then drops the fragmented one.
But there's a big problem with this in your case. Even if you were to optimize tables after setting innodb_file_per_table=1, their data would be copied into new tablespaces, but that still wouldn't shrink or drop the old multi-table tablespaces like your nmsdata1 through 18. They would still be huge, but "empty."
What I'm saying is that you're screwed. There is no way to shrink these tablespaces, and since you're full up on disk space, there's no way to refactor them either.
Here's what I would do: Build a new MySQL Server. Make sure innodb_file_per_table=1 is configured. Also configure the default for the data file path: innodb_data_file_path=ibdata1:12M:autoextend. That will make the central tablespace small from the start. We'll avoid expanding it with data.
Then export a dump of your current database server, all of it. Import that into your new MySQL server. It will obey the file-per-table setting, and data will create and fill new tablespaces, one per table.
This is also an opportunity to build the new server with larger storage, given what you know about the data growth.
It will take a long time to import so much data. How long depends on your server performance specifications, but it will take many hours at least. Perhaps days. This is a problem if your original database is still taking traffic while you're importing.
The solution to that is to use replication, so your new server can "catch up" from the point where you created the dump to the current state of the database. This procedure is documented, but it may be quite a bit of learning curve for someone who is not a database pro, as you said: https://dev.mysql.com/doc/refman/8.0/en/replication-howto.html
You should probably get a consultant who knows how to do this work.
I want to be able to place limits on the storage size an individual MySQL database (for example 2GB per database). Was this able to be done? I'm unable to a find specific MySQL configuration that allows this.
Specifically I want to be able to do this on AWS RDS Instance - so any knowledge related to this would be helpful too.
Any help or guidance wether this could be done would be appreciated.
This question was asked almost 10 years ago: MySQL schema size
I won't close your question as a duplicate because it has been long enough that it deserves a fresh answer.
Nothing has changed. There is no option for MySQL to define a cap on the size of an individual database.
You can make a max size for the system tablespace, and all your tables and schemas will fill this if you set innodb_file_per_table=0. You can configure the max size in your my.cnf this way:
innodb_data_file_path=ibdata1:12M:autoextend:max:1G
That's just an example, it means the ibdata1 file is initially created at 12MB, it grows as needed, but will not grow larger than 1GB. See https://dev.mysql.com/doc/refman/8.0/en/innodb-init-startup-configuration.html for more information.
Even with the potential limit on the system tablespace, this doesn't give you control over how much space each schema can use. They all use the same tablespace.
If you use innodb_file_per_table=1, each table is stored in its own tablespace file. There is no option for defining the size of a table-specific tablespace.
Since MySQL 5.7, you can also create a General Tablespace, and assign multiple tables to it. For example, you could choose to put all tables for a specific schema into one general tablespace. But this also has no syntax for defining a size limit of a general tablespace.
I've seen some solutions on the net to revoke insert/update privileges on a schema if it exceeds its size limit. But this is not enforced by MySQL, it's an external solution.
I have periodic stream of incoming MySQL dumps for various experiments and I have not so much SSD space. I need only restore dump and import DB into completely different format based on pretty heavy SQL queries. Tables use InnoDB. I'd like to use compressed row format for all tables when doing restore.
Ability to use SSD pays off any CPU ovrhead.
I don't like to modify dumps itself (up to 60G).
There is no row formats specified in CREATE TABLE instructions inside dumps.
Any way to configure row format as COMPRESSED before restore process? Global or DB level configuration is OK.
Used MySQL version is 5.5.35 (Ubuntu 12.04). Option to upgrade MySQL if needed is not a problem.
I see one obvious option to modify dumps with added ROW FORMAT options by something like sed replace. It's worst case option as I'd like dumps to be kept untouched for other people around.
Microsoft SQL Server has a nice feature, which allows a database to be automatically expanded when it becomes full. In MySQL, I understand that a database is, in fact, a directory with a bunch of files corresponding to various objects. Does it mean that a concept of database size is not applicable and a MySQL database can be as big as available disk space allows without any additional concern? If yes, is this behavior the same across different storage engines?
It depends on the engine you're using. A list of the ones that come with MySQL can be found here.
MyISAM tables have a file per table. This file can grow to your file system's limit. As a table gets larger, you'll have to tune it as there's index and data size optimizations that limit the default size. Also, this MyISAM documentation page says:
There is a limit of 2^32 (~4.295E+09)
rows in a MyISAM table. If you build
MySQL with the --with-big-tables
option, the row limitation is
increased to (2^32)^2 (1.844E+19) rows.
See Section 2.16.2, “Typical configure
Options”. Binary distributions for
Unix and Linux are built with this
option.
InnoDB can operate in 3 different modes: using innodb table files, using a whole disk as a table file or using innodb_file_per_table.
Table files are pre-created per your MySQL instance. You typically create a large amount of space and monitor it. When it starts filling up, you need to configure another file and restart your server. You can also set it to autoextend, so that it will add a chunk of space to the last table file when it starts to fill up. I typically don't use this feature, as you never know when you'll take the performance hit for extending the table. This page talks about configuring it.
I've never used a whole disk as a table file, but it can be done. Instead of pointing to a file, I believe you point your InnoDB table files at the un-formatted, unmounted device.
innodb_file_per_table makes InnoDB tables act like MyISAM tables. Each table gets its own table file. Last time I used this, the table files did not shrink if you deleted rows from them. When a table is dropped or altered, the file resizes.
The Archive engine is a gzipped MyISAM table.
A memory table doesn't use disk at all. In fact, when a server restarts, all the data is lost.
Merge tables are like a poor man's partitioning for MyISAM tables. It causes a bunch of identical tables to be queried as if there were one. Aside from the FRM table definition, no files exist other than the MyISAM ones.
CSV tables are wrappers around CSV files. The usual file system limits apply here. They are not too fast, since they can't have indexes.
I don't think anyone uses BDB any more. At least, I've never used it. It uses a Berkly database as a back end. I'm not familiar with its restrictions.
Federated tables are used to connect to and query tables on other database servers. Again, there is only an FRM file.
The Blackhole engine doesn't store anything locally. It's used primarily for creating replication logs and not for actual data storage, since there is no data storage :)
MySQL Cluster is completely different: it stores just about everything in memory (recent editions allow disk storage) and is very different from all the other engines.
what you describe is roughly true for MyISAM tables. for InnoDB tables the picture is different, and more similar to what other DBMSs do: one (or a few) big file with complex internal structure for the whole server. to optimize it, you can use a whole disk (or partition) as a file. (at least in unix-like systems, where everything is a file)