Dose any one knows why information_schema.tables.data_free in InnoDB always be 8388608, no matter how many rows in tables;
table_schema table_name table_rows data_free engine
g33v1 appraise 0 8388608 InnoDB
g33v1 areatype 12403 8388608 InnoDB
g33v1 atype 581982 8388608 InnoDB
g33v1 atype2 579700 8388608 InnoDB
thanks.
I think it is the temporary table maximum size allocated for doing sorting and other things which requires the space on HDD. As the same things appears in other cases also
mysql> show global variables like '%tmp%';
+----------------+-----------------------+
| Variable_name | Value |
+----------------+-----------------------+
| bdb_tmpdir | /usr/local/mysql/tmp/ |
| max_tmp_tables | 32 |
| tmp_table_size | 8388608 |
| tmpdir | /usr/local/mysql/tmp |
+----------------+-----------------------+
mysql> show global variables like '%myisam%';
+---------------------------------+---------------+
| Variable_name | Value |
+---------------------------------+---------------+
| myisam_data_pointer_size | 4 |
| myisam_max_extra_sort_file_size | 2147483648 |
| myisam_max_sort_file_size | 2147483647 |
| myisam_recover_options | OFF |
| myisam_repair_threads | 1 |
| myisam_sort_buffer_size | 4194304 |
| myisam_stats_method | nulls_unequal |
+---------------------------------+---------------
Related
I am currently debugging some performance issues related to my cloudsql mySQL database, and am looking to investigate its configuration parameters.
In the cloud sql docs the following note can be found:
"Maximum concurrent connections - Note:
To determine the maximum value that you can set for this flag, the
basic formulas are:
Available RAM = Global Buffers + (Thread Buffers x max_connections)
max_connections = (Available RAM - Global Buffers) / Thread Buffers
To get the list of buffers and their values: SHOW VARIABLES LIKE '%buffer%'; Allow some headroom for other processes requiring
memory."
However, running the sql command produces the following output:
mysql> SHOW VARIABLES LIKE '%buffer%';
+-------------------------------------+----------------+
| Variable_name | Value |
+-------------------------------------+----------------+
| bulk_insert_buffer_size | 8388608 |
| innodb_buffer_pool_chunk_size | 55574528 |
| innodb_buffer_pool_dump_at_shutdown | ON |
| innodb_buffer_pool_dump_now | OFF |
| innodb_buffer_pool_dump_pct | 25 |
| innodb_buffer_pool_filename | ib_buffer_pool |
| innodb_buffer_pool_in_core_file | ON |
| innodb_buffer_pool_instances | 1 |
| innodb_buffer_pool_load_abort | OFF |
| innodb_buffer_pool_load_at_startup | ON |
| innodb_buffer_pool_load_now | OFF |
| innodb_buffer_pool_size | 55574528 |
| innodb_change_buffer_max_size | 25 |
| innodb_change_buffering | all |
| innodb_log_buffer_size | 16777216 |
| innodb_sort_buffer_size | 1048576 |
| join_buffer_size | 262144 |
| key_buffer_size | 8388608 |
| myisam_sort_buffer_size | 8388608 |
| net_buffer_length | 16384 |
| preload_buffer_size | 32768 |
| read_buffer_size | 131072 |
| read_rnd_buffer_size | 262144 |
| sort_buffer_size | 262144 |
| sql_buffer_result | OFF |
+-------------------------------------+----------------+
25 rows in set (0.02 sec)
I do not understand which values to plug into the formula. Which values correspond to Global Buffers and Thread Buffers?
I found this blog article that lists which are the Global Buffer and Thread Buffer variables, which are:
Global Buffer:
key_buffer_size
innodb_buffer_pool_size
innodb_log_buffer_size
innodb_additional_mem_pool_size
net_buffer_size
query_cache_size
Thread Buffer:
sort_buffer_size
myisam_sort_buffer_size
read_buffer_size
join_buffer_size
read_rnd_buffer_size
thread_stack
I use mysql mariadb(Server version: 10.3.20-MariaDB-1:10.3.20+maria~stretch mariadb.org binary distribution).
I have ~700 000 records with columns:
id
html (mediumtext) with very big average length in field: ~150000
date
+2 small other
In html I have very long text (it's html's).
Now I need select * from table;, to analyse this html but this query takes over ~0.03819s per query (I tested on smaller part) so: total rows 700000*0.03819s per query = (700000*0.03819s)/60/60 = over 7 hours of selecting!
I have 8 cores and 60GB of RAM. Profiling query shows that time of transferring data is very very long.
How to speed it up? It's is possible, or that much of data it's too much for mysql and I need mongodb?
query_cache_limit = 64M
query_cache_size = 1024M
max_allowed_packet = 64M
net_buffer_length = 16384
max_connect_errors = 1000
thread_concurrency = 32
concurrent_insert = 2
read_rnd_buffer_size = 8M
bulk_insert_buffer_size = 8M
query_cache_limit = 64M
query_cache_size = 1024M
query_cache_type = 1
query_prealloc_size = 262144
query_alloc_block_size = 65536
transaction_alloc_block_size = 8192
transaction_prealloc_size = 4096
max_write_lock_count = 16
innodb_buffer_pool_size=30G
innodb_flush_log_at_trx_commit=2
innodb_thread_concurrency=16
innodb_flush_method=O_DIRECT
innodb_read_io_threads = 64
innodb_write_io_threads = 16
innodb_buffer_pool_instances = 20
MariaDB [db]> explain select id, href, html from raw limit 10;
+------+-------------+-------+------+---------------+------+---------+------+--------+-------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-------+------+---------------+------+---------+------+--------+-------+
| 1 | SIMPLE | raw | ALL | NULL | NULL | NULL | NULL | 658793 | |
+------+-------------+-------+------+---------------+------+---------+------+--------+-------+
1 row in set (0.227 sec)
after playing with indexes:
MariaDB [db]> show index from raw;
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| raw | 0 | PRIMARY | 1 | id | A | 658793 | NULL | NULL | | BTREE | | |
| raw | 1 | id | 1 | id | A | 658793 | NULL | NULL | | BTREE | | |
| raw | 1 | href | 1 | href | A | 658793 | NULL | NULL | YES | BTREE | | |
| raw | 1 | date | 1 | date | A | 131758 | NULL | NULL | YES | BTREE | | |
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
4 rows in set (3.724 sec)
38ms to fetch 150Kb from a spinning disk is quite fast.
query_cache_size = 1024M -- This is much too high. Stop at about 50M.
A PRIMARY KEY is a unique index. So, if id is the primary key, do not also say KEY(id).
It's is possible, or that much of data it's too much for mysql and I need mongodb?
Assuming you are running at disk speed, you cannot expect any other product to run faster.
What will the client do with 100GB of data in one batch? MySQL will be happy to deliver it, but the client will probably choke to death.
I have a server on Debian 6 using Mysql 5.1 and I'm having problems with big tables on Innodb. As Mysql was installed on its default configurations I'd like to know some suggestion about how to improve the performance of my server.
DB Mysql 5.1
Operative System: Debian6
I have 4G available on free memory according free -g command on Debian.
I have 6GB on swap on Debian OS.
mysql> SHOW VARIABLES LIKE '%buffer%';
+-------------------------+------------+
| Variable_name | Value |
+-------------------------+------------+
| bulk_insert_buffer_size | 8388608 |
| innodb_buffer_pool_size | 2097152000 |
| innodb_log_buffer_size | 1048576 |
| join_buffer_size | 131072 |
| key_buffer_size | 16777216 |
| myisam_sort_buffer_size | 8388608 |
| net_buffer_length | 16384 |
| preload_buffer_size | 32768 |
| read_buffer_size | 131072 |
| read_rnd_buffer_size | 262144 |
| sort_buffer_size | 2097144 |
| sql_buffer_result | OFF |
+-------------------------+------------+
12 rows in set
I have to mention that the only thing I have changed on my cnf file is innodb_buffer_pool_size to 2097152000 but I have no get good result about performance.
By the way on my server I have big tables with partitions and big tables on MyIsam as well.
I want to check the log in MySQL to see the queries that are being run by my application. How can I do this? I am using XAMPP and the directory to MySQL is C:\xampp\mysql.
This is what I get when I do show variables like '%log%';
mysql> show variables like '%log%';
+---------------------------------+------------------------------------+
| Variable_name | Value |
+---------------------------------+------------------------------------+
| back_log | 50 |
| binlog_cache_size | 32768 |
| binlog_format | STATEMENT |
| expire_logs_days | 0 |
| general_log | OFF |
| general_log_file | C:/xampp/mysql/data/mysql.log |
| innodb_flush_log_at_trx_commit | 1 |
| innodb_locks_unsafe_for_binlog | OFF |
| innodb_log_buffer_size | 8388608 |
| innodb_log_file_size | 5242880 |
| innodb_log_files_in_group | 2 |
| innodb_log_group_home_dir | C:\xampp\mysql\data\ |
| innodb_mirrored_log_groups | 1 |
| log | OFF |
| log_bin | OFF |
| log_bin_trust_function_creators | OFF |
| log_bin_trust_routine_creators | OFF |
| log_error | C:\xampp\mysql\data\mysql.err |
| log_output | FILE |
| log_queries_not_using_indexes | OFF |
| log_slave_updates | OFF |
| log_slow_queries | OFF |
| log_warnings | 1 |
| max_binlog_cache_size | 4294963200 |
| max_binlog_size | 1073741824 |
| max_relay_log_size | 0 |
| relay_log | |
| relay_log_index | |
| relay_log_info_file | relay-log.info |
| relay_log_purge | ON |
| relay_log_space_limit | 0 |
| slow_query_log | OFF |
| slow_query_log_file | C:/xampp/mysql/data/mysql-slow.log |
| sql_log_bin | ON |
| sql_log_off | OFF |
| sql_log_update | ON |
| sync_binlog | 0 |
+---------------------------------+------------------------------------+
37 rows in set (0.00 sec)
Enable general query log by the following query in mysql command line
SET GLOBAL general_log = 'ON';
Now open C:/xampp/mysql/data/mysql.log and check query log
If it fails, open your my.cnf file. For windows its my.ini file and enable it there. Just make sure its in the [mysqld] section
[mysqld]
general_log = 1
Note: In xampp my.ini file can be either found in xampp\mysql or in c:\windows directory
Set up the General Query Log.
Its given on OFFICIAL MYSQL website.
SET GLOBAL general_log = 'ON';
You can also use custom path:
[mysqld]
# Set Slow Query Log
long_query_time = 1
slow_query_log = 1
slow_query_log_file = "C:/slowquery.log"
#Set General Log
log = "C:/genquery.log"
Seems like the general query log is the file that you need. A good introduction to this is at http://dev.mysql.com/doc/refman/5.1/en/query-log.html
For me, general_log didn't worked. But adding this to my.ini worked
[mysqld]
log-output=FILE
slow_query_log = 1
slow_query_log_file = "d:/temp/developer.log"
My mysql only caters read requests. I thought, it will be a good idea to make use of the cache completely. I am running MySQL in a VM and that is the only application running inside the VM. I am allocating 2GB memory for that VM. I am using a 64 bit centos on the VM. If you think already it is using the maximum memory that can be used, I can also allocate more to that VM. I am not very good in understanding the mysql settings and finding out the memory footprint used by a process but I am interested in learning how to. Thanks a lot for the help.
These are some information regarding my MYSQL :
mysql> show global variables like "%cache%";
+------------------------------+----------------------+
| Variable_name | Value |
+------------------------------+----------------------+
| binlog_cache_size | 32768 |
| have_query_cache | YES |
| key_cache_age_threshold | 300 |
| key_cache_block_size | 1024 |
| key_cache_division_limit | 100 |
| max_binlog_cache_size | 18446744073709547520 |
| query_cache_limit | 1048576 |
| query_cache_min_res_unit | 4096 |
| query_cache_size | 0 |
| query_cache_type | ON |
| query_cache_wlock_invalidate | OFF |
| table_cache | 64 |
| thread_cache_size | 0 |
+------------------------------+----------------------+
13 rows in set (0.00 sec)
mysql> show global variables like "%buffer%";
+-------------------------------+---------+
| Variable_name | Value |
+-------------------------------+---------+
| bulk_insert_buffer_size | 8388608 |
| innodb_buffer_pool_awe_mem_mb | 0 |
| innodb_buffer_pool_size | 8388608 |
| innodb_log_buffer_size | 1048576 |
| join_buffer_size | 131072 |
| key_buffer_size | 8384512 |
| myisam_sort_buffer_size | 8388608 |
| net_buffer_length | 16384 |
| preload_buffer_size | 32768 |
| read_buffer_size | 131072 |
| read_rnd_buffer_size | 262144 |
| sort_buffer_size | 2097144 |
+-------------------------------+---------+
12 rows in set (0.00 sec)
mysql> show table status where name="items"
-> ;
+-------+--------+---------+------------+-------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+-------------------+----------+----------------+---------+
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment |
+-------+--------+---------+------------+-------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+-------------------+----------+----------------+---------+
| items | MyISAM | 10 | Dynamic | 42667 | 346 | 14775916 | 281474976710655 | 1970176 | 0 | 341337 | 2009-07-22 13:31:00 | 2010-10-20 15:37:18 | NULL | latin1_swedish_ci | NULL | | |
+-------+--------+---------+------------+-------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+-------------------+----------+----------------+---------+
This is the output of my ulimit -a
[sethu#work13 root]$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 8191
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 8191
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Please let me know if you need more information.
You can achieve a quick performance boost by enabling the query cache!
Add this to your my.cnf:
query_cache_type=1
query_cache_limit=1M
query_cache_size=32M
For basic statistics and recommendations you can start with mysqltuner.pl script. Do not apply recommendations blindly, as it might decrease performance.
MySQLTuner Github page
One-liner to fetch latest version of script and run it:
curl -sSL mysqltuner.pl | perl
Perhaps you could consider using memcached? However, going to InnoDB table should give an immediate improvement (as N.B. said in a comment)