MYQSL rep.
I don't know why my MYSQL is not working
DATA CONF:
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
port = 3306
basedir = /usr
lc-messages-dir =/usr/share/mysql
skip-external-locking//
gtid_mode = ON
enforce-gtid_consistency
log-slave-updates
binlog_format = mixed
CHANGE MASTER TOMASTER_HOST='',MASTER_PORT=3306,MASTER_USER='',MASTER_PASSWORD='',
MASTER_AUTO_POSITION=1,GET_MASTER_PUBLIC_KEY=1
I've setup a Docker-Compose file that creates a mariadb database and aspnet backend (both from docker files I've written). If I expose ports and run the containers outside of a compose (using localhost) or set network_mode: "host" in the docker-compose file then the app interacts as expected. However, using the default bridge network that the compose generates any command between containers seems to timeout.
I've attempted to run a networking container (nicolaka/netshoot) attached to the same network and ping the relevant containers via the DNS name. The name resolves to the IP address and then the ping hangs, I've also run and attached a mysql container and attempted connecting to the mariadb container which results in a timeout.
This all looks to be some networking issue in my setup. Config listed below, any help would be much appreciated!
MariaDb DockerFile
ARG VERSION=10.4
FROM mariadb/server:${VERSION}
ENV MYSQL_ROOT_PASSWORD ****
EXPOSE 3306
AspNet DockerFile
FROM mcr.microsoft.com/dotnet/aspnet:5.0
# Container SetUp
EXPOSE 80
# .NET specific
COPY ToDoListBackEnd.Api/bin/Release/net5.0/publish ToDoListBackEnd/
WORKDIR /ToDoListBackEnd
ENTRYPOINT ["dotnet", "ToDoListBackEnd.Api.dll"]
ComposeFile:
version: '3.3'
services:
db: # Service Hosting MariaDb
build: ./Database
container_name: todolist-database
ports:
- "6001:3306"
volumes:
- todo-db:/var/lib/mysql
restart: always
environment:
MYSQL_ROOT_PASSWORD: ***
MYSQL_DATABASE: ToDoList
MYSQL_USER: root
MYSQL_PASSWORD: ***
backend: # Backend C# Service
build: ./BackEnd
container_name: todolist-backend
ports:
- "5002:80"
restart: always
depends_on:
- db
links:
- db
volumes:
todo-db:
Network Config
{
"Name": "todolist_default",
"Id": "6dd07863467b35ece5e00f400ca03c0c1b9950a5f8d5a6538b47889ac2dcae68",
"Created": "2021-04-21T19:39:58.329215937+01:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.23.0.0/16",
"Gateway": "172.23.0.1"
}
]
},
"Internal": false,
"Attachable": true,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"01f10355356ea8d194b9de6c6e9ead0af5398ad8d10dc743a70927c88532f3ef": {
"Name": "todolist-database",
"EndpointID": "9b69958ff0a1c9ff7cf62f3478965fdef0622fd3fe98922bebee4ebe72aa9508",
"MacAddress": "02:42:ac:17:00:03",
"IPv4Address": "172.23.0.3/16",
"IPv6Address": ""
},
"356092a918beb6a0a021d369fd5f76e556e54ebad3c1b8fa626ce33fd66af579": {
"Name": "mysqlTest",
"EndpointID": "3ae06081e831c8f94bc36c59cd085561b81685da164011c5e6438ffd0d1bb6c2",
"MacAddress": "02:42:ac:17:00:02",
"IPv4Address": "172.23.0.2/16",
"IPv6Address": ""
},
"61bbc2047065f01732ddd0e7db492c0708144cb6ede7ee0b5466c708c36d9639": {
"Name": "todolist-backend",
"EndpointID": "06c46938e05ba48955c75a63a65268d4259c30ad9bb7a23f2d15c04299d644da",
"MacAddress": "02:42:ac:17:00:05",
"IPv4Address": "172.23.0.5/16",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {
"com.docker.compose.network": "default",
"com.docker.compose.project": "todolist",
"com.docker.compose.version": "1.29.1"
},
"CreatedTime": 1619030398329
}
my.cnf from inside the mariadb container
# MariaDB database server configuration file.
#
# You can copy this file to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html
# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
# Here is entries for some specific programs
# The following values assume you have at least 32M ram
# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
#
# * Basic Settings
#
#user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc_messages_dir = /usr/share/mysql
lc_messages = en_US
#skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
#bind-address = 127.0.0.1
#
# * Fine Tuning
#
max_connections = 100
connect_timeout = 5
wait_timeout = 600
max_allowed_packet = 16M
thread_cache_size = 128
sort_buffer_size = 4M
bulk_insert_buffer_size = 16M
tmp_table_size = 32M
max_heap_table_size = 32M
#
# * MyISAM
#
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched. On error, make copy and try a repair.
myisam_recover_options = BACKUP
key_buffer_size = 128M
#open-files-limit = 2000
table_open_cache = 400
myisam_sort_buffer_size = 512M
concurrent_insert = 2
read_buffer_size = 2M
read_rnd_buffer_size = 1M
#
# * Query Cache Configuration
#
# Cache only tiny result sets, so we can fit more in the query cache.
query_cache_limit = 128K
query_cache_size = 64M
# for more write intensive setups, set to DEMAND or OFF
#query_cache_type = DEMAND
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file = /var/log/mysql/mysql.log
#general_log = 1
#
# Error logging goes to syslog due to /etc/mysql/conf.d/mysqld_safe_syslog.cnf.
#
# we do want to know about network errors and such
#log_warnings = 2
#
# Enable the slow query log to see queries with especially long duration
#slow_query_log[={0|1}]
slow_query_log_file = /var/log/mysql/mariadb-slow.log
long_query_time = 10
#log_slow_rate_limit = 1000
#log_slow_verbosity = query_plan
#log-queries-not-using-indexes
#log_slow_admin_statements
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
# other settings you may need to change.
#server-id = 1
#report_host = master1
#auto_increment_increment = 2
#auto_increment_offset = 1
#log_bin = /var/log/mysql/mariadb-bin
#log_bin_index = /var/log/mysql/mariadb-bin.index
# not fab for performance, but safer
#sync_binlog = 1
expire_logs_days = 10
max_binlog_size = 100M
# slaves
#relay_log = /var/log/mysql/relay-bin
#relay_log_index = /var/log/mysql/relay-bin.index
#relay_log_info_file = /var/log/mysql/relay-bin.info
#log_slave_updates
#read_only
#
# If applications support it, this stricter sql_mode prevents some
# mistakes like inserting invalid dates etc.
#sql_mode = NO_ENGINE_SUBSTITUTION,TRADITIONAL
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
default_storage_engine = InnoDB
innodb_buffer_pool_size = 256M
innodb_log_buffer_size = 8M
innodb_file_per_table = 1
innodb_open_files = 400
innodb_io_capacity = 400
innodb_flush_method = O_DIRECT
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem
#
# * Galera-related settings
#
[galera]
# Mandatory settings
#wsrep_on=ON
#wsrep_provider=
#wsrep_cluster_address=
#binlog_format=row
#default_storage_engine=InnoDB
#innodb_autoinc_lock_mode=2
#
# Allow server to accept connections on all interfaces.
#
bind-address=0.0.0.0
#
# Optional setting
#wsrep_slave_threads=1
#innodb_flush_log_at_trx_commit=0
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
#no-auto-rehash # faster start of mysql but no tab completion
[isamchk]
key_buffer = 16M
#
# * IMPORTANT: Additional settings that can override those from this file!
# The files must end with '.cnf', otherwise they'll be ignored.
#
!include /etc/mysql/mariadb.cnf
!includedir /etc/mysql/conf.d/
Thanks again for any help!
So I seem to have fixed the issue. Turns out to be an issue with the OS (Arch Linux) not docker, I found this out by running the same files on a Windows system.
For any arch users I've listed my steps out at
https://bbs.archlinux.org/viewtopic.php?pid=1973008#p1973008
I use vertx java to build an app and writing tests for mysql queries. I can successful get results before but after I modified my.cnf because of other stuffs. I cannot no long get returned results but "Fail to read any response from the server, the underlying connection might get lost unexpectedly". I searched google, but I cannot figure out exactly where is wrong and how to fix it.
I would post my.cnf file:
[client]
port = 3306
socket = /tmp/mysql.sock
# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket = /tmp/mysql.sock
nice = 0
syslog
[mysqld]
# Basic Settings
port = 3306
user = root
pid-file = /usr/local/mysql/data/mysqld.local.pid
socket = /tmp/mysql.sock
port = 3306
basedir = /usr/local/mysql/
datadir = /usr/local/mysql/data
tmpdir = /tmp
lc-messages-dir = /usr/local/mysql/share
skip-external-locking
# Engine
default-storage-engine = InnoDB
# Listening IP
bind-address = 0.0.0.0
# Safety
max-connect-errors = 1000000
max_allowed_packet = 64M
skip-name-resolve
sysdate-is-now = 1
innodb = FORCE
innodb-strict-mode = 1
tls_version = TLSv1,TLSv1.1,TLSv1.2
wait_timeout = 28800
interactive_timeout = 10000
# Buffers
sort_buffer_size = 4M
read_buffer_size = 2M
join_buffer_size = 8M
read_rnd_buffer_size = 16M
# MyISAM
key-buffer-size = 32M
# CACHES AND LIMITS #
tmp-table-size = 128M
max-heap-table-size = 128M
# Bin logs
binlog-format = ROW
log_bin = /tmp/binlog
max_binlog_size = 100M
server-id = 1 # randomize it incase of multiple servers
# InnoDB
innodb-buffer-pool-size = 2048M
innodb_buffer_pool_instances = 8
innodb_log_buffer_size = 8M
innodb-log-files-in-group = 2
innodb-log-file-size = 256M
innodb-file-per-table = 1
innodb-flush-log-at-trx-commit = 1
innodb-flush-method = O_DIRECT
# With virtual synchrony redundancy, make write queries faster
innodb_doublewrite = 1
# LOGGING
general_log_file = /usr/local/mysql/mysql.log
log-error = /usr/local/mysql/data/mysqld.local.err
log-queries-not-using-indexes = 1
[mysqldump]
quick
quote-names
max_allowed_packet = 64M
[isamchk]
key_buffer = 16M
#
# * IMPORTANT: Additional settings that can override those from this file!
# The files must end with '.cnf', otherwise they'll be ignored.
#
Can you tell me what's wrong with the configuration and what should I change? Thanks
TokuDB alone beat InnoDB in our benchmarks by about 25% but when I turn master-slave on InnoDB now beats TokuDB by about 20%
Any idea what's going on
here's the conf that's running on a r4.8xlarge aws machine
[mysql_safe]
malloc-lib=/usr/include/jemalloc
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
server-id = 1
log_bin = /var/log/mysql/mysql-bin.log
binlog_do_db = mt
sync_binlog=1
binlog_format=ROW
binlog_row_image=FULL
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
explicit_defaults_for_timestamp
#malloc-lib=/usr/include/jemalloc
key_buffer_size= 250G
read_buffer_size=2G
read_rnd_buffer_size=50M
join_buffer_size=25M
tmp_table_size = 5G
sort_buffer_size = 2G
query_cache_limit = 10M
query_cache_size = 100M
innodb_buffer_pool_instances=64
tokudb_fanout = 128
tokudb_commit_sync = 0
tokudb_fsync_log_period = 1000
tokudb_directio = 1
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
innodb-flush-method = O_DIRECT
innodb-log-files-in-group = 2
innodb-log-file-size = 512M
innodb-flush-log-at-trx-commit = 2
innodb-file-per-table = 1
innodb-buffer-pool-size = 200G
log-error = /var/log/mysql/error.log
# Recommended in standard MySQL setup
sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_ALL_TABLES
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
Hi my server have 24 cores and 32GB of memory.
Am doing multiple "INSERT INTO SELECT" of 50 millions row at a time.
This takes about 15h a query but it is ticking along at 100% of only one CPU, I'am trying to get mySQL(5.5)(InnoDB) to use more of the resources.
I have read multiple threads about it, but I do not get it to work.
Most info is about adding innodb_thread_concurrency = 0
But I still get no results.
port = 3306
socket = /var/run/mysqld/mysqld.sock
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /media/ssd/db
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
innodb_buffer_pool_size=26G
innodb_thread_concurrency = 0
bind-address = 127.0.0.1
#
# * Fine Tuning
#
key_buffer = 1000M
max_allowed_packet = 160M
thread_stack = 192K
thread_cache_size = 8
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover = BACKUP
table_cache = 800
query_cache_limit = 5000M
query_cache_size = 1600M
join_buffer_size = 1000M
log_error = /var/log/mysql/error.log
# Here you can see queries with especially long duration
log_slow_queries = /var/log/mysql/mysql-slow.log
long_query_time = 2
Try this parameters:
innodb_io_capacity=5000 (or even 20000 depending on your IO subsystem)
innodb_buffer_pool_size=4G (for example)
innodb_log_file_size=1G
innodb_write_io_threads = 64
innodb_read_io_threads = 64
innodb_thread_concurrency = 0