does the optimize table of mysql really rebuild(recreate) table? - mysql

I have a question about optimize table of mysql,I have read many documents about this function,all of the document says:it can defragment an InnoDB table and reorganizes the physical storage of table data and associated index data, to reduce storage space,but no one can tell me the principle of optimize table and slove my questions,I have known those informations:
As of MySQL 5.6.17,OPTIMIZE TABLE uses online DDL for regular and partitioned InnoDB tables
,As of MySQL 5.6.17,The table rebuild triggered by OPTIMIZE TABLE and performed under the cover by ALTER TABLE ... FORCE is completed in place(ALGORITHM =inplace)
my questions:
does the optimize table of mysql really rebuild(recreate) table? because it completed in place(not ALGORITHM=copy),is ALGORITHM=inplace real rebuild(recreate) table?
thank you for your response.
https://dev.mysql.com/doc/refman/5.6/en/innodb-online-ddl-operations.html
enter image description here

Demo:
I created a table on MySQL 5.6.37 and filled it with about 1GB of random data.
mysql> select version();
+-----------------+
| version() |
+-----------------+
| 5.6.37-82.2-log |
+-----------------+
mysql> show table status\G
*************************** 1. row ***************************
Name: mytable
Engine: InnoDB
Version: 10
Row_format: Compact
Rows: 61914
Avg_row_length: 20926
Data_length: 1295613952
Max_data_length: 0
Index_length: 1589248
Data_free: 4194304
Auto_increment: 131056
Create_time: 2019-02-20 15:54:23
Update_time: NULL
Check_time: NULL
Collation: utf8mb4_general_ci
Checksum: NULL
Create_options:
Comment:
I can observe the file in my datadir and note the inode number (the leftmost number).
/usr/local/var/mysql/test ls -li
total 2590752
8625784850 -rw-rw---- 1 bkarwin admin 67 Feb 20 07:54 db.opt
8625784853 -rw-rw---- 1 bkarwin admin 8580 Feb 20 07:54 mytable.frm
8625784854 -rw-rw---- 1 bkarwin admin 1321205760 Feb 20 07:56 mytable.ibd
I run optimize table.
mysql> optimize table mytable;
While it's running, I check progress by listing files in the datadir again. I see it has created a new temp file and it's filling it with a copy of the data.
Note the inode number of the temp file.
/usr/local/var/mysql/test ls -li
total 4395064
8625785082 -rw-rw---- 1 bkarwin admin 8580 Feb 20 07:57 #sql-337_16.frm
8625785083 -rw-rw---- 1 bkarwin admin 914358272 Feb 20 07:57 #sql-ib2021-3689447301.ibd
8625784850 -rw-rw---- 1 bkarwin admin 67 Feb 20 07:54 db.opt
8625784853 -rw-rw---- 1 bkarwin admin 8580 Feb 20 07:54 mytable.frm
8625784854 -rw-rw---- 1 bkarwin admin 1321205760 Feb 20 07:56 mytable.ibd
Finally my optimize table finishes. It took almost 2 minutes. If it wasn't doing a real table copy, why would it have taken so long?
+--------------+----------+----------+-------------------------------------------------------------------+
| Table | Op | Msg_type | Msg_text |
+--------------+----------+----------+-------------------------------------------------------------------+
| test.mytable | optimize | note | Table does not support optimize, doing recreate + analyze instead |
| test.mytable | optimize | status | OK |
+--------------+----------+----------+-------------------------------------------------------------------+
2 rows in set (1 min 5.52 sec)
Looking at the datadir again, I see the temp file has been renamed to be the new tablespace, and the old file has been removed. The new file has the inode number of the temp file, indicating it's the copy.
/usr/local/var/mysql/test ls -li
total 2557984
8625784850 -rw-rw---- 1 bkarwin admin 67 Feb 20 07:54 db.opt
8625785082 -rw-rw---- 1 bkarwin admin 8580 Feb 20 07:57 mytable.frm
8625785083 -rw-rw---- 1 bkarwin admin 1304428544 Feb 20 07:58 mytable.ibd

Related

MariaDB with MyRocks engine: Many huge files after insertions

I am using MariaDB with MyRocks engine. After short time, I try to insert bunch of data into some tables (tables could be InnoDB and MyRocks engine), my disk grows very fast. I check MariaDB data directory and I see this:
drwxr-x--x 2 mysql mysql 12K Jun 26 12:07 #rocksdb
drwxr-xr-x 7 mysql mysql 4.0K Jun 26 12:07 .
drwxr-xr-x 6 root root 4.0K Jun 21 10:22 ..
-rw-rw---- 1 mysql mysql 1.1G Jun 25 21:39 1.000001
-rw-rw---- 1 mysql mysql 1.1G Jun 25 21:40 1.000002
-rw-rw---- 1 mysql mysql 1.6G Jun 25 21:44 1.000003
-rw-rw---- 1 mysql mysql 1.1G Jun 25 21:48 1.000004
-rw-rw---- 1 mysql mysql 1.1G Jun 25 21:50 1.000005
-rw-rw---- 1 mysql mysql 218M Jun 26 06:53 1.000006
-rw-rw---- 1 mysql mysql 357 Jun 26 06:56 1.000007
-rw-rw---- 1 mysql mysql 1.1G Jun 26 07:06 1.000008
-rw-rw---- 1 mysql mysql 1.1G Jun 26 07:09 1.000009
-rw-rw---- 1 mysql mysql 1.1G Jun 26 07:17 1.000010
-rw-rw---- 1 mysql mysql 1.1G Jun 26 07:19 1.000011
-rw-rw---- 1 mysql mysql 1.1G Jun 26 07:22 1.000012
-rw-rw---- 1 mysql mysql 1.1G Jun 26 07:24 1.000013
-rw-rw---- 1 mysql mysql 1.1G Jun 26 07:27 1.000014
-rw-rw---- 1 mysql mysql 1.1G Jun 26 07:29 1.000015
-rw-rw---- 1 mysql mysql 1.1G Jun 26 07:32 1.000016
-rw-rw---- 1 mysql mysql 1.1G Jun 26 07:34 1.000017
-rw-rw---- 1 mysql mysql 1.1G Jun 26 07:37 1.000018
-rw-rw---- 1 mysql mysql 1.1G Jun 26 07:40 1.000019
-rw-rw---- 1 mysql mysql 1.1G Jun 26 07:42 1.000020
-rw-rw---- 1 mysql mysql 1.1G Jun 26 07:45 1.000021
-rw-rw---- 1 mysql mysql 1.1G Jun 26 07:51 1.000022
-rw-rw---- 1 mysql mysql 1.1G Jun 26 08:00 1.000023
-rw-rw---- 1 mysql mysql 1.1G Jun 26 08:07 1.000024
-rw-rw---- 1 mysql mysql 1.1G Jun 26 08:15 1.000025
...
I see many files whose name contains only numbers. I don't know those files were created by MyRocks or MySQL, and can I delete them?
A guess: You configured the following, assuming it turned on binlogging:
log_bin = 1
and those are binlogs starting with the '1'.
A new file every few minutes implies a huge amount of writes (such as the inserts you mention).
If you don't need binlogging and change to this and restart, then the files should stop being created:
log_bin = OFF
The files (1.*) can be removed.
If they don't stop, then my guess is wrong.
If you do need binlogging (for replication and/or point-in-time backups, then we need to investigate the huge rapidity of them.
I'll bet you did a restart about Jun 26 06:53.
Rocksdb is probably not relevant to the question. (But I could be wrong.)

Cannot see MySQL tables after moving database dir with symbolic link

I'm running out of space on the partition where MySQL is installed so I tried moving my largest database to a new partition and symlinked this in the data directory. When I try to use this database I get the following error
mysql> use fb20;
Database changed
mysql> show tables;
ERROR 1018 (HY000): Can't read dir of './fb20/' (errno: 13)
I've checked that symlinks are enabled:
mysql> show variables like 'have_symlink';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| have_symlink | YES |
+---------------+-------+
All of the permissions look correct. The MySQL data directory is /data:
root> ll /data/
drwxr-x--- 243 mysql mysql 8192 Feb 21 08:02 mysql/
root> ll /data/mysql/ | grep fb20
lrwxrwxrwx 1 mysql mysql 19 Feb 21 08:02 fb20 -> /sandata/mysql/fb20/
and the new directory is /sandata:
root> ll /sandata/
drwxr-x--- 3 mysql mysql 60 Feb 20 20:32 mysql/
root> ll /sandata/mysql/
total 360
drwxr-x--- 3 mysql mysql 60 Feb 20 20:32 ./
drwxrwxr-x 35 root perma 4096 Feb 20 19:59 ../
drwx------ 2 mysql mysql 303104 Feb 15 12:20 fb20/
Not sure what I am doing wrong.
Based on the comment I added the following lines to the file /etc/apparmor.d/usr.sbin.mysqld
/sandata/mysql/ r,
/sandata/mysql/** rwk,
and then reloaded the profile (restarting apparmor did not work):
sudo apparmor_parser -r /etc/apparmor.d/usr.sbin.mysqld

why my MySQL binary log file size is so small?

I have a lot of small binary log files , each of them just only 126 Bytes
-rw-rw----. 1 mysql mysql 126 Jan 6 16:13 mysql-bin.000001
-rw-rw----. 1 mysql mysql 126 Jan 6 16:16 mysql-bin.000002
-rw-rw----. 1 mysql mysql 150 Jan 7 10:34 mysql-bin.000003
-rw-rw----. 1 mysql mysql 126 Jan 8 09:57 mysql-bin.000004
-rw-rw----. 1 mysql mysql 126 Jan 8 11:08 mysql-bin.000005
-rw-rw----. 1 mysql mysql 107 Jan 8 11:09 mysql-bin.000006
-rw-rw----. 1 mysql mysql 126 Jan 8 13:21 mysql-bin.000007
-rw-rw----. 1 mysql mysql 342 Jan 8 16:25 mysql-bin.index
how to let them grow bigger? and my "binlog_cache_size" is more than 126 Byte
mysql> show variables like 'binlog_cache_size';
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id: 1
Current database: test
+-------------------+-------+
| Variable_name | Value |
+-------------------+-------+
| binlog_cache_size | 32768 |
+-------------------+-------+
1 row in set (0.00 sec)
can anybody tell me why(⊙ˍ⊙) ? thank you very much :)

mySQL stopped running? Cant restart. Error: "Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock'"

My client has a server thats been running solidly for 2 years. He called yesterday b.c he noticed the server is down (it was down for a couple of days actually). I'm trying to figure out whats going on. I can't seem to start mysql.
When I run: /etc/init.d/mysqld start it fails. And running mysql results in
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
I tried tweaking the settings in /etc/my.cnf to change the socket to be in the tmp directory but that didn't work either. I think there's something else going on. Perhaps a memory issue? Or is something corrupt?
Below is my mysqld.log file.
140108 05:14:51 mysqld started
InnoDB: Log scan progressed past the checkpoint lsn 362 2789208032
140108 5:14:51 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 362 2789874367
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 1637 row operations to undo
InnoDB: Trx id counter is 0 138804224
140108 5:14:51 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Error: tried to read 344064 bytes at offset 6 41517056.
InnoDB: Was only able to read 339968.
140108 5:15:08 InnoDB: Operating system error number 0 in a file operation.
InnoDB: Error number 0 means 'Success'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/operating-system-error-codes.html
InnoDB: File operation call: 'read'.
InnoDB: Cannot continue operation.
140108 05:15:08 mysqld ended
My /var/lib/mysql directory looks like this
total 27654604
drwxr-xr-x 8 mysql mysql 4096 Jan 8 05:14 .
drwxr-xr-x 28 root root 4096 May 29 2012 ..
drwxr-xr-x 2 mysql mysql 24576 Apr 5 2013 db_stackme
drwxr-xr-x 2 mysql mysql 4096 May 30 2012 db_stackmeuser
-rwxr-xr-x 1 mysql mysql 28280094720 Dec 20 09:54 ibdata1
-rwxr-xr-x 1 mysql mysql 5242880 Jan 8 05:14 ib_logfile0
-rwxr-xr-x 1 mysql mysql 5242880 Dec 20 09:51 ib_logfile1
drwxr-xr-x 2 mysql mysql 4096 May 30 2012 mysql
drwxr-xr-x 2 mysql mysql 8192 May 30 2012 openx
drwxr-xr-x 2 mysql mysql 4096 Jun 1 2012 stackme_business
drwxr-xr-x 2 mysql mysql 4096 May 30 2012 test
I would GREATLY appreciate some input on how to solve this problem.

MYSQL data files are compressed to gz

I have drupal website, the database is about 10GB. I was checking the datadir of my drupal database and found that some files are .gz!! why is this? and does it affect performance??
the website is working fine and i can use those tables normally although there is no corresponding file for.gz (ie. I have content_type_company.MYD.gz in the data directory but there is no content_type_company.MYD)
the list is something like this:
ls -lhS *.gz
-rw-rw---- 1 mysql mysql 3.4M 2011-04-18 06:52 content_field_headline_image.MYI.gz
-rw-rw---- 1 mysql mysql 379K 2011-04-14 15:50 print_node_conf.MYD.gz
-rw-rw---- 1 mysql mysql 105K 2011-04-18 06:56 content_type_company.MYD.gz
-rw-rw---- 1 mysql mysql 22K 2011-04-14 15:50 nodewords.MYD.gz
-rw-rw---- 1 mysql mysql 18K 2011-04-18 06:59 content_field_company_fax.MYD.gz
-rw-rw---- 1 mysql mysql 18K 2011-04-18 06:59 content_field_company_email.MYI.gz
-rw-rw---- 1 mysql mysql 4.7K 2011-04-17 21:55 content_type_editor_choice.MYI.gz
-rw-rw---- 1 mysql mysql 4.2K 2011-02-01 09:15 content_node_field_instance.MYD.gz
-rw-rw---- 1 mysql mysql 2.7K 2011-01-31 14:03 content_node_field.MYD.gz
-rw-rw---- 1 mysql mysql 1.9K 2011-04-18 01:31 content_type_press_review.MYI.gz
-rw-rw---- 1 mysql mysql 1.4K 2011-04-18 06:59 users.MYI.gz
-rw-rw---- 1 mysql mysql 1.1K 2011-02-02 12:25 batch.MYD.gz
-rw-rw---- 1 mysql mysql 1.1K 2011-01-19 12:35 node_access.MYI.gz
-rw-rw---- 1 mysql mysql 801 2010-09-06 09:49 menu_custom.MYD.gz
-rw-rw---- 1 mysql mysql 679 2011-04-17 12:12 scheduler.MYI.gz
-rw-rw---- 1 mysql mysql 671 2011-01-12 11:24 users.frm.gz
-rw-rw---- 1 mysql mysql 659 2011-04-14 22:53 video_rendering.MYI.gz
-rw-rw---- 1 mysql mysql 570 2011-04-17 14:39 content_type_slideshow.MYD.gz
-rw-rw---- 1 mysql mysql 533 2011-02-02 09:46 permission.MYI.gz
-rw-rw---- 1 mysql mysql 471 2011-01-30 12:53 panels_display.MYD.gz
-rw-rw---- 1 mysql mysql 412 2010-09-07 14:41 content_field_slideshow_image.frm.gz
-rw-rw---- 1 mysql mysql 372 2011-01-12 11:24 date_formats.MYD.gz
-rw-rw---- 1 mysql mysql 355 2010-07-26 15:51 contact.frm.gz
-rw-rw---- 1 mysql mysql 351 2010-07-26 16:15 sessions.frm.gz
-rw-rw---- 1 mysql mysql 349 2011-03-28 18:02 cache_update.frm.gz
-rw-rw---- 1 mysql mysql 345 2011-03-28 18:03 ctools_object_cache.frm.gz
-rw-rw---- 1 mysql mysql 326 2010-07-26 15:54 print_mail_page_counter.frm.gz
-rw-rw---- 1 mysql mysql 317 2010-07-26 15:54 search_index.frm.gz
-rw-rw---- 1 mysql mysql 310 2010-07-26 15:51 i18n_strings.frm.gz
-rw-rw---- 1 mysql mysql 292 2010-07-26 15:49 authmap.frm.gz
-rw-rw---- 1 mysql mysql 291 2011-01-12 11:24 scheduler.frm.gz
-rw-rw---- 1 mysql mysql 286 2010-07-26 15:54 permission.frm.gz
-rw-rw---- 1 mysql mysql 275 2010-07-26 15:49 block_class.frm.gz
-rw-rw---- 1 mysql mysql 272 2010-07-26 15:51 captcha_points.frm.gz
-rw-rw---- 1 mysql mysql 269 2010-07-26 15:54 page_manager_weights.frm.gz
-rw-rw---- 1 mysql mysql 251 2011-03-28 18:03 imagecache_preset.frm.gz
-rw-rw---- 1 mysql mysql 249 2010-11-24 13:19 content_type_press_review.frm.gz
-rw-rw---- 1 mysql mysql 248 2011-01-26 09:21 aggregator_category_feed.frm.gz
-rw-rw---- 1 mysql mysql 243 2010-07-26 15:54 role.frm.gz
-rw-rw---- 1 mysql mysql 240 2010-07-26 16:15 variable.frm.gz
-rw-rw---- 1 mysql mysql 202 2010-08-09 13:20 stylizer.MYI.gz
-rw-rw---- 1 mysql mysql 192 2011-04-05 07:50 vocabulary_node_types.MYD.gz
-rw-rw---- 1 mysql mysql 152 2010-07-26 16:15 taxonomy_manager_merge.MYI.gz
-rw-rw---- 1 mysql mysql 78 2011-04-14 03:06 video_rendering.MYD.gz
-rw-rw---- 1 mysql mysql 58 2011-02-01 17:40 content_type_video_gallery.MYD.gz
The files are compressed to save space, of course, and gzip works like that - when a file called 'x' is gzipped, it's replaced by 'x.gz' the original file ''becomes'' the compressed file.
If the database works as expected, I wouldn't worry about it.
The size of gzipped files is much smaller than Your database size. Nothing to worry.