I want to achieve 6000/sec requests and trying to setup HaProxy with multi proc settings but getting this error:
cpu-map expects a process number including 'all', 'odd', 'even', or a
number from 1 to 64, followed by a list of CPU ranges with numbers
from 0 to 63
Following are the configurations I'm using.
global
daemon
maxconn 200000
maxsslconn 200000
#stats socket /run/haproxy/admin.sock mode 660 level admin
stats socket 127.0.0.1:14567
nbproc 6
cpu-map auto:1/all 0
cpu-map auto:2/all 1
cpu-map auto:3/all 2
cpu-map auto:4/all 3
cpu-map auto:5/all 4
cpu-map auto:6/all 5
stats bind-process 6
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
# https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3
tune.ssl.default-dh-param 2048
and
global
daemon
maxconn 200000
maxsslconn 200000
#stats socket /run/haproxy/admin.sock mode 660 level admin
stats socket 127.0.0.1:14567
nbproc 6
cpu-map 1 0
cpu-map 2 1
cpu-map 3 2
cpu-map 4 3
cpu-map 5 4
cpu-map 6 5
stats bind-process 6
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
# https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3
tune.ssl.default-dh-param 2048
Nothing is working as expected.
Related
im trying to enable SSL on a mariadb in a docker container
i have made following changes in /etc/mysql/my.cnf file
ssl-ca=/etc/certs/client-cert.pem
ssl-cert=/etc/certs/server-cert.pem
ssl-key=/etc/certs/server-key.pem
ssl_ca=/etc/certs/client-cert.pem
ssl_cert=/etc/certs/server-cert.pem
ssl_key=/etc/certs/server-key.pem
Note: i have used variables names with - and _ because i am not sure which scheme is uses
also i have provided certificates in the above mentioned path
here is the result of "show variables like '%ssl%';"
MariaDB [(none)]> show variables like '%ssl%';
+---------------------+----------------------------+
| Variable_name | Value |
+---------------------+----------------------------+
| have_openssl | YES |
| have_ssl | YES |
| ssl_ca | /etc/certs/client-cert.pem |
| ssl_capath | |
| ssl_cert | /etc/certs/server-cert.pem |
| ssl_cipher | |
| ssl_crl | |
| ssl_crlpath | |
| ssl_key | /etc/certs/server-key.pem |
| version_ssl_library | OpenSSL 1.1.0g 2 Nov 2017 |
+---------------------+----------------------------+
10 rows in set (0.001 sec)
but this is what i get when i run "status" command on mysql command line
MariaDB [(none)]> status
--------------
mysql Ver 15.1 Distrib 10.3.9-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
Connection id: 42
Current database:
Current user: root#localhost
SSL: Not in use
Current pager: stdout
Using outfile: ''
Using delimiter: ;
Server: MariaDB
Server version: 10.3.9-MariaDB-1:10.3.9+maria~bionic mariadb.org binary distribution
Protocol version: 10
Connection: Localhost via UNIX socket
Server characterset: latin1
Db characterset: latin1
Client characterset: latin1
Conn. characterset: latin1
UNIX socket: /var/run/mysqld/mysqld.sock
Uptime: 5 min 52 sec
it says SSL not in use, now im not sure if SSL is enabled or not, or am i missing any flag? also how can i find out if its enabled or not?
here is the complete my.cnf file
# MariaDB database server configuration file.
#
# You can copy this file to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html
# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
# Here is entries for some specific programs
# The following values assume you have at least 32M ram
# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
#
# * Basic Settings
#
#user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc_messages_dir = /usr/share/mysql
lc_messages = en_US
skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
#bind-address = 127.0.0.1
#
# * Fine Tuning
#
max_connections = 100
connect_timeout = 5
wait_timeout = 600
max_allowed_packet = 16M
thread_cache_size = 128
sort_buffer_size = 4M
bulk_insert_buffer_size = 16M
tmp_table_size = 32M
max_heap_table_size = 32M
#
# * MyISAM
#
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched. On error, make copy and try a repair.
myisam_recover_options = BACKUP
key_buffer_size = 128M
#open-files-limit = 2000
table_open_cache = 400
myisam_sort_buffer_size = 512M
concurrent_insert = 2
read_buffer_size = 2M
read_rnd_buffer_size = 1M
#
# * Query Cache Configuration
#
# Cache only tiny result sets, so we can fit more in the query cache.
query_cache_limit = 128K
query_cache_size = 64M
# for more write intensive setups, set to DEMAND or OFF
#query_cache_type = DEMAND
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file = /var/log/mysql/mysql.log
#general_log = 1
#
# Error logging goes to syslog due to /etc/mysql/conf.d/mysqld_safe_syslog.cnf.
#
# we do want to know about network errors and such
#log_warnings = 2
#
# Enable the slow query log to see queries with especially long duration
#slow_query_log[={0|1}]
slow_query_log_file = /var/log/mysql/mariadb-slow.log
long_query_time = 10
#log_slow_rate_limit = 1000
#log_slow_verbosity = query_plan
#log-queries-not-using-indexes
#log_slow_admin_statements
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
# other settings you may need to change.
#server-id = 1
#report_host = master1
#auto_increment_increment = 2
#auto_increment_offset = 1
#log_bin = /var/log/mysql/mariadb-bin
#log_bin_index = /var/log/mysql/mariadb-bin.index
# not fab for performance, but safer
#sync_binlog = 1
expire_logs_days = 10
max_binlog_size = 100M
# slaves
#relay_log = /var/log/mysql/relay-bin
#relay_log_index = /var/log/mysql/relay-bin.index
#relay_log_info_file = /var/log/mysql/relay-bin.info
#log_slave_updates
#read_only
#
# If applications support it, this stricter sql_mode prevents some
# mistakes like inserting invalid dates etc.
#sql_mode = NO_ENGINE_SUBSTITUTION,TRADITIONAL
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
default_storage_engine = InnoDB
# you can't just change log file size, requires special procedure
#innodb_log_file_size = 50M
innodb_buffer_pool_size = 256M
innodb_log_buffer_size = 8M
innodb_file_per_table = 1
innodb_open_files = 400
innodb_io_capacity = 400
innodb_flush_method = O_DIRECT
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
ssl-ca=/etc/certs/client-cert.pem
ssl-cert=/etc/certs/server-cert.pem
ssl-key=/etc/certs/server-key.pem
ssl_ca=/etc/certs/client-cert.pem
ssl_cert=/etc/certs/server-cert.pem
ssl_key=/etc/certs/server-key.pem
#
# * Galera-related settings
#
[galera]
# Mandatory settings
#wsrep_on=ON
#wsrep_provider=
#wsrep_cluster_address=
#binlog_format=row
#default_storage_engine=InnoDB
#innodb_autoinc_lock_mode=2
#
# Allow server to accept connections on all interfaces.
#
#bind-address=0.0.0.0
#
# Optional setting
#wsrep_slave_threads=1
#innodb_flush_log_at_trx_commit=0
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
#no-auto-rehash # faster start of mysql but no tab completion
[isamchk]
key_buffer = 16M
#
# * IMPORTANT: Additional settings that can override those from this file!
# The files must end with '.cnf', otherwise they'll be ignored.
#
!include /etc/mysql/mariadb.cnf
!includedir /etc/mysql/conf.d/
I can't reproduce the problem:
$ mysql -u user -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 1
Server version: 10.3.9-MariaDB-1:10.3.9+maria~bionic-log mariadb.org binary distribution
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> \! ls -alF /path/to/MariaDB/ssl/
total 20
drwxr-xr-x 2 user user 140 Jan 01 00:01 ./
drwxr-xr-x 5 user user 240 Jan 01 00:01 ../
-r-----r-- 1 user user 1273 Jan 01 00:01 ca.pem
-r-----r-- 1 user user 1143 Jan 01 00:01 client-cert.pem
-r-----r-- 1 user user 1679 Jan 01 00:01 client-key.pem
-r-----r-- 1 user user 1147 Jan 01 00:01 server-cert.pem
-r-----r-- 1 user user 1679 Jan 01 00:01 server-key.pem
MariaDB [(none)]> \! cat /path/to/MariaDB/my.cnf
...
[client]
...
## MariaDB Client Configuration ##
ssl-ca=/path/to/MariaDB/ssl/ca.pem
ssl-cert=/path/to/MariaDB/ssl/client-cert.pem
ssl-key=/path/to/MariaDB/ssl/client-key.pem
### This option is disabled by default ###
ssl-verify-server-cert
...
...
[mysqld]
...
#
# * Security Features
#
ssl
ssl-ca=/path/to/MariaDB/ssl/ca.pem
ssl-cert=/path/to/MariaDB/ssl/server-cert.pem
ssl-key=/path/to/MariaDB/ssl/server-key.pem
ssl-cipher=TLSv1.2
...
MariaDB [(none)]> \s
--------------
mysql Ver 15.1 Distrib 10.3.9-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
Connection id: 1
Current database:
Current user: user#localhost
SSL: Cipher in use is ECDHE-RSA-AES256-GCM-SHA384
Current pager: stdout
Using outfile: ''
Using delimiter: ;
Server: MariaDB
Server version: 10.3.9-MariaDB-1:10.3.9+maria~bionic-log mariadb.org binary distribution
Protocol version: 10
Connection: Localhost via UNIX socket
Server characterset: utf8
Db characterset: utf8
Client characterset: utf8
Conn. characterset: utf8
UNIX socket: /var/run/mysqld/mysqld.sock
Uptime: 11 sec
Threads: 8 Questions: 61 Slow queries: 0 Opens: 32 Flush tables: 1 Open tables: 26 Queries per second avg: 5.545
--------------
MariaDB [(none)]> SHOW VARIABLES LIKE '%ssl%';
+---------------------+--------------------------------------+
| Variable_name | Value |
+---------------------+--------------------------------------+
| have_openssl | YES |
| have_ssl | YES |
| ssl_ca | /path/to/MariaDB/ssl/ca.pem |
| ssl_capath | |
| ssl_cert | /path/to/MariaDB/ssl/server-cert.pem |
| ssl_cipher | TLSv1.2 |
| ssl_crl | |
| ssl_crlpath | |
| ssl_key | /path/to/MariaDB/ssl/server-key.pem |
| version_ssl_library | OpenSSL 1.1.0g 2 Nov 2017 |
+---------------------+--------------------------------------+
10 rows in set (0.001 sec)
I have converted tables of a database from InnoDB to TokuDB and i noticed that with TokuDB, reads are using way too much CPU. Why is this?
To be more specific, the server with TokuDB tables is a slave of a server with InnoDB which is part of the PXC. The slave just used regular percona server and not PXC. But the slave seems to be using way too much CPU and i do not know why?
Below is my my.cnf config:
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
[mysqld_safe]
thp-setting=never
socket = /var/run/mysqld/mysqld.sock
nice = 0
flush_caches
numa_interleave
core-file-size = unlimited
open_files_limit = 1024
[mysqld]
back_log = 65535
bind-address = 0.0.0.0
binlog_format = ROW
character_set_server = utf8
collation_server = utf8_general_ci
core_file
basedir = /usr
datadir = /var/lib/mysql
#default_storage_engine = InnoDB
enforce-gtid-consistency = 1
expand_fast_index_creation = 1
expire_logs_days = 7
gtid_mode = ON
innodb_autoinc_lock_mode = 2
innodb_buffer_pool_instances = 1
innodb_buffer_pool_populate = 1
innodb_buffer_pool_size = 512M
innodb_data_file_path = ibdata1:64M;ibdata2:64M:autoextend
innodb_file_format = Barracuda
innodb_file_per_table
innodb_force_recovery = 1
innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
innodb_io_capacity = 1600
innodb_large_prefix
innodb_locks_unsafe_for_binlog = 1
innodb_log_file_size = 64M
innodb_print_all_deadlocks = 1
innodb_read_io_threads = 64
innodb_stats_on_metadata = FALSE
innodb_support_xa = FALSE
innodb_write_io_threads = 64
lc-messages-dir = /usr/share/mysql
log-bin = mysqld-bin
log-queries-not-using-indexes
log-slave-updates
long_query_time = 1
master_info_repository = TABLE
max_allowed_packet = 64M
max_connect_errors = 4294967295
max_connections = 2500
max_user_connections = 2550
min_examined_row_limit = 1000
open_files_limit = 1024
port = 3306
relay_log_info_repository = TABLE
relay-log-recovery = TRUE
relay-log-recovery = 1
skip-external-locking
skip-name-resolve
slave_parallel_workers = 8
slow_query_log = 1
slow_query_log_timestamp_always = 1
socket = /var/run/mysqld/mysqld.sock
table_open_cache = 4096
thread_cache = 1024
tmpdir = /srv/tmp
transaction_isolation = REPEATABLE-READ
updatable_views_with_limit = 0
user = mysql
wait_timeout = 60
server-id = 2
# TokuDB fine tuning
default_storage_engine = TokuDB
tokudb_analyze_time = 5
#tokudb_cache_size = 6G
tokudb_directio = 1
tokudb_commit_sync = 0
tokudb_fsync_log_period = 1000
tokudb_load_save_space =1
tokudb_alter_print_error=0
tokudb_block_size = 4MB
tokudb_bulk_fetch = 1
tokudb_disable_slow_alter = 1
tokudb_last_lock_timeout = empty
tokudb_row_format = tokudb_quicklz
#tokudb_data_dir = /var/lib/tokudb
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
#no-auto-rehash # faster start of mysql but no tab completition
[isamchk]
key_buffer = 16M
!includedir /etc/mysql/conf.d/
The following replication message was being reported by our monitoring system xymon when tokudb_cache_size when initially set to 80% of total RAM.
2016-02-25 16:42:04 9604 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--relay-log=db-kdb-slave-6-relay-bin' to avoid this problem.
2016-02-25 16:42:05 9604 [Warning] Recovery from master pos 552554502 and file mysqld-bin.001163. Previous relay log pos and relay log file had been set to 552554714, ./db-kdb-slave-6-relay-bin.002933 respectively.
2016-02-25 16:42:05 9604 [Warning] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
------More info about the Master server running InnoDB and part of PXC-----------
## Results from top
top - 10:05:12 up 14 days, 7:56, 2 users, load average: 2.16, 2.31, 2.39
Tasks: 413 total, 1 running, 412 sleeping, 0 stopped, 0 zombie
%Cpu(s): 8.9 us, 0.6 sy, 0.0 ni, 89.9 id, 0.3 wa, 0.0 hi, 0.2 si, 0.0 st
KiB Mem: 65704012 total, 63553216 used, 2150796 free, 169832 buffers
KiB Swap: 975868 total, 809892 used, 165976 free. 16304268 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2485 mysql 20 0 60.146g 0.045t 2.612g S 314.9 73.3 27762:43 mysqld
## disk info
george#db-erp-3:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 32G 8.0K 32G 1% /dev
tmpfs 6.3G 1.2M 6.3G 1% /run
/dev/sda2 274G 2.1G 258G 1% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 32G 0 32G 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/nvme0n1p1 1.1T 542G 503G 52% /srv
na1:/vol/yphome 4.5T 3.7T 875G 82% /net/account
## Memory info
george#db-erp-3:~$ free -g
total used free shared buffers cached
Mem: 62 60 2 0 0 15
-/+ buffers/cache: 44 17
Swap: 0 0 0
george#db-erp-3:~$
## Database info
+--------------------+----------------------+
| Data Base Name | Data Base Size in MB |
+--------------------+----------------------+
| information_schema | 0.00976563 |
| dberp | 347143.32031250 |
| mysql | 2.11562061 |
| performance_schema | 0.00000000 |
+--------------------+----------------------+
4 rows in set (0.13 sec)
+--------------------+----------------------+------------------+
| Data Base Name | Data Base Size in MB | Free Space in MB |
+--------------------+----------------------+------------------+
| information_schema | 0.00976563 | 0.00000000 |
| dberp | 347143.32031250 | 6270.00000000 |
| mysql | 2.11562061 | 4.00199127 |
| performance_schema | 0.00000000 | 0.00000000 |
+--------------------+----------------------+------------------+
4 rows in set (0.03 sec)
Your CPU will be higher for reads because TokuDB data needs to be decompressed to be used. Also, if this slave is processing any activity from the master than it's also doing compression for the insert/update/delete activity.
Couple of ideas.
1. Reduce the value of tokudb_block_size. While 4MB is great for compression it means that your point queries need to decompress a lot more data than they have to. Try using 256KB and see how CPU and performance changes. You might have to rebuild your slave to accomplish this easily (I'm now over a year away from working at TokuDB).
2. Look at your tokudb_cache_size. It defaults to 50% of RAM, but if nothing else is on this server you should up it to somewhere between 75% and 80%. This will mean less reads and decompression since more data will be in your cache.
I am trying to setup postfix mail server using ansible playbook. I am puzzled to understand different ways to setup postfix with mysql. I am not able to hold on to a complete schema. The one I got here and here
are some examples.
Using the schema from ansible role above I have setup postfix. And the schema tables include:
show tables;
+-------------------+
| Tables_in_mail |
+-------------------+
| alias |
| alias_destination |
| domain |
| domain_users |
| mailbox |
| mailbox_usage |
| user |
+-------------------+
7 rows in set (0.00 sec)
Now I want to test the postfix server with a test user /mailbox and domain.So I am looking for a way to init the schema for a test user. Any kind of help is appreciated
Edit
Tried to add a test user:
mysql> select * from user;
+----+-----------+---------------+--------+-------------+---------------------+---------------------+
| id | username | password | active | super_admin | created_at | modified_at |
+----+-----------+---------------+--------+-------------+---------------------+---------------------+
| 1 | sakhunzai | dTzdFMm0DSvmM | 1 | 0 | 0000-00-00 00:00:00 | 0000-00-00 00:00:00 |
+----+-----------+---------------+--------+-------------+---------------------+---------------------+
1 row in set (0.00 sec)
mysql> select * from domain;
+-------------+--------+-----------+-----------------------+---------------------+---------------------+
| name | active | backup_mx | default_mailbox_quota | created_at | modified_at |
+-------------+--------+-----------+-----------------------+---------------------+---------------------+
| example.com | 1 | 0 | 1073741824 | 0000-00-00 00:00:00 | 0000-00-00 00:00:00 |
+-------------+--------+-----------+-----------------------+---------------------+---------------------+
1 row in set (0.00 sec)
mysql> select * from mailbox;
+-----------+-------------+------------+------+--------+--------------+----------------+---------------------+---------------------+---------------+
| username | domain | local_part | name | active | quota_mbytes | quota_messages | created_at | modified_at | password |
+-----------+-------------+------------+------+--------+--------------+----------------+---------------------+---------------------+---------------+
| sakhunzai | example.com | | NULL | 1 | 0 | 0 | 0000-00-00 00:00:00 | 0000-00-00 00:00:00 | wZQ9cRLxshVEY |
+-----------+-------------+------------+------+--------+--------------+----------------+---------------------+---------------------+---------------+
1 row in set (0.00 sec)
Testing from telnet:
telnet mail.example.com 25
Trying 127.0.1.1...
Connected to mail.example.com.
Escape character is '^]'.
220 mail.example.com ESMTP Postfix
HELO test.example.com
250 mail.example.com
MAIL FROM: <sakhunzai#example.com>
250 2.1.0 Ok
RCPT TO:<sakhunzai#example.com>
550 5.1.1 <sakhunzai#example.com>: Recipient address rejected: User unknown in virtual mailbox table
Edit 2
After updating username I am able to complete the test:
update user set username="sakhunzai#example.com";
update mailbox set username="sakhunzai#example.com";
telnet mail.example.com 25
Trying 127.0.1.1...
Connected to mail.example.com.
Escape character is '^]'.
220 mail.example.com ESMTP Postfix
HELO test.example.com
250 mail.example.com
MAIL FROM: <sakhunzai#example.com>
250 2.1.0 Ok
RCPT TO:<sakhunzai#example.com>
250 2.1.5 Ok
DATA
354 End data with <CR><LF>.<CR><LF>
Subject: test message
.
250 2.0.0 Ok: queued as 3BEEB40940
QUIT
221 2.0.0 Bye
Connection closed by foreign host.
But with some errors in mail.log file:
to=<sakhunzai#example.com>, relay=none, delay=44, delays=44/0/0/0, dsn=4.3.0, status=deferred (mail transport unavailable)
Nov 10 08:13:51 postfix/smtpd[32013]: connect from localhost[127.0.0.1]
Nov 10 08:14:09 postfix/smtpd[32013]: 3BEEB40940: client=localhost[127.0.0.1]
Nov 10 08:14:49 postfix/cleanup[32020]: 3BEEB40940: message-id=<20151110081409.3BEEB40940#mail.example.com>
Nov 10 08:14:49 postfix/qmgr[8096]: 3BEEB40940: from=<sakhunzai#example.com>, size=337, nrcpt=1 (queue active)
Nov 10 08:14:49 postfix/qmgr[8096]: warning: connect to transport private/smtp-amavis: No such file or directory
Nov 10 08:14:49 postfix/error[32021]: 3BEEB40940: to=<sakhunzai#example.com>, relay=none, delay=44, delays=44/0/0/0, dsn=4.3.0, status=deferred (mail transport unavailable)
Nov 10 08:14:56 postfix/smtpd[32013]: disconnect from localhost[127.0.0.1]
postconf -n
alias_database = hash:/etc/aliases
alias_maps = hash:/etc/aliases
append_dot_mydomain = no
biff = no
broken_sasl_auth_clients = yes
config_directory = /etc/postfix
content_filter = smtp-amavis:[127.0.0.1]:10024
delay_warning_time = 4h
dovecot_destination_recipient_limit = 1
inet_interfaces = all
inet_protocols = all
mailbox_size_limit = 0
message_size_limit = 51200000
mime_header_checks = regexp:/etc/postfix/mime_header_checks.regexp
mydestination =
myhostname = mail.example.com
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
myorigin = /etc/mailname
queue_directory = /var/spool/postfix
readme_directory = no
recipient_delimiter = +
relay_domains = proxy:mysql:/etc/postfix/mysql_relay_domains_maps.cf
relayhost =
smtp_header_checks = pcre:/etc/postfix/smtp_header_checks.pcre
smtp_tls_security_level = may
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtpd_banner = $myhostname ESMTP $mail_name
smtpd_client_restrictions =
smtpd_helo_restrictions =
smtpd_recipient_restrictions = reject_unauth_destination reject_unlisted_recipient reject_rbl_client zen.spamhaus.org permit
smtpd_sasl_auth_enable = yes
smtpd_sasl_authenticated_header = yes
smtpd_sasl_path = private/auth
smtpd_sasl_security_options = noanonymous
smtpd_sasl_type = dovecot
smtpd_sender_restrictions =
smtpd_tls_cert_file = /etc/ssl/private/mail.example.com.crt
smtpd_tls_key_file = /etc/ssl/private/mail.example.com.key
smtpd_tls_loglevel = 1
smtpd_tls_received_header = yes
smtpd_tls_security_level = may
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
virtual_alias_maps = proxy:mysql:/etc/postfix/mysql_virtual_alias_mailbox_maps.cf proxy:mysql:/etc/postfix/mysql_virtual_alias_maps.cf
virtual_mailbox_domains = proxy:mysql:/etc/postfix/mysql_virtual_domains_maps.cf
virtual_mailbox_maps = proxy:mysql:/etc/postfix/mysql_virtual_mailbox_maps.cf
virtual_transport = dovecot
/etc/postfix/master.cf
#
# Postfix master process configuration file. For details on the format
# of the file, see the master(5) manual page (command: "man 5 master").
#
# Do not forget to execute "postfix reload" after editing this file.
#
# ==========================================================================
# service type private unpriv chroot wakeup maxproc command + args
# (yes) (yes) (yes) (never) (100)
# ==========================================================================
smtp inet n - - - - smtpd
#smtp inet n - - - 1 postscreen
#smtpd pass - - - - - smtpd
#dnsblog unix - - - - 0 dnsblog
#tlsproxy unix - - - - 0 tlsproxy
submission inet n - n - - smtpd
-o content_filter=smtp-amavis:[127.0.0.1]:10026
-o syslog_name=postfix/submission
-o smtpd_tls_security_level=encrypt
-o smtpd_sasl_auth_enable=yes
-o smtpd_recipient_restrictions=permit_sasl_authenticated,permit_mynetworks,reject
# -o milter_macro_daemon_name=ORIGINATING
#smtps inet n - - - - smtpd
# -o syslog_name=postfix/smtps
# -o smtpd_tls_wrappermode=yes
# -o smtpd_sasl_auth_enable=yes
# -o smtpd_client_restrictions=permit_sasl_authenticated,reject
# -o milter_macro_daemon_name=ORIGINATING
#628 inet n - - - - qmqpd
pickup fifo n - - 60 1 pickup
-o receive_override_options=no_header_body_checks
-o content_filter=smtp-amavis:[127.0.0.1]:10026
# Coment the above line if you do not want to sign localy originated mail from the pickup service
cleanup unix n - - - 0 cleanup
qmgr fifo n - n 300 1 qmgr
#qmgr fifo n - n 300 1 oqmgr
tlsmgr unix - - - 1000? 1 tlsmgr
rewrite unix - - - - - trivial-rewrite
bounce unix - - - - 0 bounce
defer unix - - - - 0 bounce
trace unix - - - - 0 bounce
verify unix - - - - 1 verify
flush unix n - - 1000? 0 flush
proxymap unix - - n - - proxymap
proxywrite unix - - n - 1 proxymap
smtp unix - - - - - smtp
relay unix - - - - - smtp
# -o smtp_helo_timeout=5 -o smtp_connect_timeout=5
showq unix n - - - - showq
error unix - - - - - error
retry unix - - - - - error
discard unix - - - - - discard
local unix - n n - - local
virtual unix - n n - - virtual
lmtp unix - - - - - lmtp
anvil unix - - - - 1 anvil
scache unix - - - - 1 scache
#
# ====================================================================
# Interfaces to non-Postfix software. Be sure to examine the manual
# pages of the non-Postfix software to find out what options it wants.
#
# Many of the following services use the Postfix pipe(8) delivery
# agent. See the pipe(8) man page for information about ${recipient}
# and other message envelope options.
# ====================================================================
#
# maildrop. See the Postfix MAILDROP_README file for details.
# Also specify in main.cf: maildrop_destination_recipient_limit=1
#
#maildrop unix - n n - - pipe
# flags=DRhu user=vmail argv=/usr/bin/maildrop -d ${recipient}
#
# ====================================================================
#
# Recent Cyrus versions can use the existing "lmtp" master.cf entry.
#
# Specify in cyrus.conf:
# lmtp cmd="lmtpd -a" listen="localhost:lmtp" proto=tcp4
#
# Specify in main.cf one or more of the following:
# mailbox_transport = lmtp:inet:localhost
# virtual_transport = lmtp:inet:localhost
#
# ====================================================================
#
# Cyrus 2.1.5 (Amos Gouaux)
# Also specify in main.cf: cyrus_destination_recipient_limit=1
#
#cyrus unix - n n - - pipe
# user=cyrus argv=/cyrus/bin/deliver -e -r ${sender} -m ${extension} ${user}
#
# ====================================================================
# Old example of delivery via Cyrus.
#
#old-cyrus unix - n n - - pipe
# flags=R user=cyrus argv=/cyrus/bin/deliver -e -m ${extension} ${user}
#
# ====================================================================
#
# See the Postfix UUCP_README file for configuration details.
#
#uucp unix - n n - - pipe
# flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail ($recipient)
#
# Other external delivery methods.
#
#ifmail unix - n n - - pipe
# flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient)
#bsmtp unix - n n - - pipe
# flags=Fq. user=bsmtp argv=/usr/lib/bsmtp/bsmtp -t$nexthop -f$sender $recipient
#scalemail-backend unix - n n - 2 pipe
# flags=R user=scalemail argv=/usr/lib/scalemail/bin/scalemail-store ${nexthop} ${user} ${extension}
#mailman unix - n n - - pipe
# flags=FR user=list argv=/usr/lib/mailman/bin/postfix-to-mailman.py
# ${nexthop} ${user}
# Dovecot
dovecot unix - n n - - pipe
flags=DRhu user=vmail:vmail argv=/usr/lib/dovecot/dovecot-lda -f ${sender} -d ${user}#${domain} -m ${extension}
# Amavisd-new
smtp-amavis unix - - - - 2 smtp
-o smtp_data_done_timeout=1200
-o smtp_send_xforward_command=yes
-o disable_dns_lookups=yes
-o max_use=20
127.0.0.1:10025 inet n - - - - smtpd
-o content_filter=
-o local_recipient_maps=
-o relay_recipient_maps=
-o smtpd_restriction_classes=
-o smtpd_delay_reject=no
-o smtpd_client_restrictions=permit_mynetworks,reject
-o smtpd_helo_restrictions=
-o smtpd_sender_restrictions=
-o smtpd_recipient_restrictions=permit_mynetworks,reject
-o smtpd_data_restrictions=reject_unauth_pipelining
-o smtpd_end_of_data_restrictions=
-o mynetworks=127.0.0.0/8
-o smtpd_error_sleep_time=0
-o smtpd_soft_error_limit=1001
-o smtpd_hard_error_limit=1000
-o smtpd_client_connection_count_limit=0
-o smtpd_client_connection_rate_limit=0
-o receive_override_options=no_header_body_checks,no_unknown_recipient_checks,no_address_mappings
Based upon this and this similar issues:
Comment this line
-o content_filter=smtp-amavis:[127.0.0.1]:10026
and try
I have setup Percona Xtradb cluster with 3 nodes. The first node starts fine with bootstrap, but when I try to start the second node to join the cluster, I get the following error:
2015-08-27 18:08:08 25990 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (a6b3fced-4ca1-11e5-b5da-d69fa186273c): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():463. IST will be unavailable.
2015-08-27 18:08:08 25990 [Note] WSREP: Member 0.0 (db-gc-pxc2) requested state transfer from 'any'. Selected 1.0 (db-gc-pxc1)(SYNCED) as donor.
2015-08-27 18:08:08 25990 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 0)
2015-08-27 18:08:08 25990 [Note] WSREP: Requesting state transfer: success, donor: 1
2015-08-27 18:08:08 25990 [Warning] WSREP: 1.0 (db-gc-pxc1): State transfer to 0.0 (db-gc-pxc2) failed: -12 (Cannot allocate memory)
2015-08-27 18:08:08 25990 [ERROR] WSREP: gcs/src/gcs_group.cpp:int gcs_group_handle_join_msg(gcs_group_t*, const gcs_recv_msg_t*)():731: Will never receive state. Need to abort.
2015-08-27 18:08:08 25990 [Note] WSREP: gcomm: terminating thread
2015-08-27 18:08:08 25990 [Note] WSREP: gcomm: joining thread
2015-08-27 18:08:08 25990 [Note] WSREP: gcomm: closing backend
Below is my cluster config in my.cnf file:
# Galera COnfig
wsrep_cluster_name = pxc
wsrep_cluster_address = gcomm://192.168.2.100,192.168.2.101,10.168.1.102
wsrep_node_address = 10.1.0.101
wsrep_provider = /usr/lib/libgalera_smm.so
wsrep_provider_options = "gcache.size=4G"
wsrep_slave_threads = 32
wsrep_sst_auth = "user:userpass"
wsrep_node_name = node2
#wsrep_sst_method = xtrabackup_throttle
wsrep_sst_method = xtrabackup-v2
What would be causing this error?
FYI, I do have the user and password for wsrep_sst_auth created in the database.
Here is the remainder of the my.cnf it it helps:
back_log = 65535
binlog_format = ROW
character_set_server = utf8
collation_server = utf8_general_ci
datadir = /var/lib/mysql
#default_storage_engine = InnoDB
expand_fast_index_creation = 1
expire_logs_days = 7
innodb_autoinc_lock_mode = 2
innodb_buffer_pool_instances = 6
innodb_buffer_pool_populate = 1
innodb_buffer_pool_size = 6G # XXX 64GB RAM, 80%
#innodb_data_file_path = ibdata1:64M;ibdata2:64M:autoextend
innodb_file_format = Barracuda
innodb_file_per_table
innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
innodb_io_capacity = 1600
innodb_large_prefix
innodb_locks_unsafe_for_binlog = 1
#innodb_log_file_size = 64M
innodb_print_all_deadlocks = 1
innodb_read_io_threads = 64
innodb_stats_on_metadata = FALSE
innodb_support_xa = FALSE
innodb_write_io_threads = 64
log-bin = mysqld-bin
log-queries-not-using-indexes
log-slave-updates
long_query_time = 1
max_allowed_packet = 64M
max_connect_errors = 4294967295
max_connections = 4096
min_examined_row_limit = 1000
performance-schema-instrument='%=ON'
port = 3306
relay-log-recovery = TRUE
skip-name-resolve
slow_query_log = 1
slow_query_log_timestamp_always = 1
table_open_cache = 4096
thread_cache = 1024
tmpdir = /srv/tmp
transaction_isolation = REPEATABLE-READ
updatable_views_with_limit = 0
user = mysql
wait_timeout = 60
This would seem to be the root cause:
2015-08-27 18:08:08 25990 [Warning] WSREP: 1.0 (db-gc-pxc1): State transfer to 0.0 (db-gc-pxc2) failed: -12 (Cannot allocate memory)
The new node tries to join the cluster. The new node has no state currently (local UUID is zeroes), and so an IST is not available - this means it needs to run a full SST form the donor node.
Node pxc2 is the joiner and pxc1 is the selected donor; however we get an error from pxc1 that the state transfer failed, which causes the joining to fail.
You should check the logs on the donor node (pxc1) for more detail; but the log we have indicates that it has insufficient memory to run the export of the database. Not knowing your hardware configuration, I can't give a definite response, but most likely your my.cnf is configured to be too memory hungry for your available memory and so the xtrabackup process cannot run, or else the database is too large. Add more memory to the node, or else reduce the allocations in the my.cnf.
It appears for some of our logs, the percona toolkit pt-query-digest tool works fine, but for others we get the following output:
# Files: /.../mysqld_slow.log
# Overall: 0 total, 1 unique, 0 QPS, 0x concurrency ______________________
# Attribute total min max avg 95% stddev median
# ============ ======= ======= ======= ======= ======= ======= =======
# Query size 18.19M 18.19M 18.19M 18.19M 18.19M 0 18.19M
# Profile
# Rank Query ID Response time Calls R/Call Apdx V/M Ite
# ========== ========== ========== ========== ========== ==== ===== ======
$
Anyone know what could be wrong with my log file? it appears to be valid, having the following first 10 lines:
Sep 28 00:00:37 gcdb-master mysqld_slow_log: SELECT `companies`.* FROM `companies` WHERE `companies`.`id` = 576 LIMIT 1;
Sep 28 00:00:37 gcdb-master mysqld_slow_log: # User#Host: db_one[db_one] # ip-127.0.0.1.ec2.internal [127.0.0.1]
Sep 28 00:00:37 gcdb-master mysqld_slow_log: # Query_time: 0.041188 Lock_time: 0.000046 Rows_sent: 1 Rows_examined: 46418
Sep 28 00:00:37 gcdb-master mysqld_slow_log: SET timestamp=1348790434;
Sep 28 00:00:37 gcdb-master mysqld_slow_log: SELECT `companies`.* FROM `companies` WHERE `companies`.`id` = 286358 LIMIT 1;
Sep 28 00:00:37 gcdb-master mysqld_slow_log: # User#Host: db_one[db_one] # ip-127.0.0.1.ec2.internal [127.0.0.1]
Sep 28 00:00:37 gcdb-master mysqld_slow_log: # Query_time: 0.030769 Lock_time: 0.000050 Rows_sent: 1 Rows_examined: 46583
Sep 28 00:00:37 gcdb-master mysqld_slow_log: SET timestamp=1348790434;
Sep 28 00:00:37 gcdb-master mysqld_slow_log: SELECT `companies`.* FROM `companies` WHERE `companies`.`id` = 286679 LIMIT 1;
Sep 28 00:00:37 gcdb-master mysqld_slow_log: # User#Host: db_one[db_one] # ip-127.0.0.1.ec2.internal [127.0.0.1]
Sep 28 00:00:37 gcdb-master mysqld_slow_log: # Query_time: 0.594351 Lock_time: 0.000038 Rows_sent: 12 Rows_examined: 342673
I have run some tests with your sample output, and I suspect that your file is not valid. This file, obtained from yours by cutting the syslog-like portion of each line, and supplemented with the two missing #description-lines before the first query, appears to work
# User#Host: db_one[db_one] # ip-127.0.0.1.ec2.internal [127.0.0.1]
# Query_time: 0.041188 Lock_time: 0.000046 Rows_sent: 1 Rows_examined: 46418
SELECT `companies`.* FROM `companies` WHERE `companies`.`id` = 576 LIMIT 1;
# User#Host: db_one[db_one] # ip-127.0.0.1.ec2.internal [127.0.0.1]
# Query_time: 0.041188 Lock_time: 0.000046 Rows_sent: 1 Rows_examined: 46418
SET timestamp=1348790434;
SELECT `companies`.* FROM `companies` WHERE `companies`.`id` = 286358 LIMIT 1;
# User#Host: db_one[db_one] # ip-127.0.0.1.ec2.internal [127.0.0.1]
# Query_time: 0.030769 Lock_time: 0.000050 Rows_sent: 1 Rows_examined: 46583
SET timestamp=1348790434;
SELECT `companies`.* FROM `companies` WHERE `companies`.`id` = 286679 LIMIT 1;
# User#Host: db_one[db_one] # ip-127.0.0.1.ec2.internal [127.0.0.1]
# Query_time: 0.594351 Lock_time: 0.000038 Rows_sent: 12 Rows_examined: 342673
as does the same file, where I have removed the first line, so as to start with a #description line, and output:
# 240ms user time, 20ms system time, 24.59M rss, 87.74M vsz
# Current date: Fri Nov 2 22:03:02 2012
# Hostname: mintaka
# Files: orig.log
# Overall: 3 total, 1 unique, 0 QPS, 0x concurrency ______________________
# Attribute total min max avg 95% stddev median
# ============ ======= ======= ======= ======= ======= ======= =======
# Exec time 113ms 31ms 41ms 38ms 40ms 5ms 40ms
# Lock time 142us 46us 50us 47us 49us 2us 44us
# Rows sent 3 1 1 1 1 0 1
# Rows examine 136.15k 45.33k 45.49k 45.38k 44.45k 0.00 44.45k
# Query size 234 76 79 78 76.28 1.50 76.28
# Profile
# Rank Query ID Response time Calls R/Call Apdx V/M Item
# ==== ================== ============= ===== ====== ==== ===== ==========
# 1 0x0C756AF10BC44B0D 0.1131 100.0% 3 0.0377 1.00 0.00 SELECT companies
# Query 1: 0 QPS, 0x concurrency, ID 0x0C756AF10BC44B0D at byte 226 ______
# This item is included in the report because it matches --limit.
# Scores: Apdex = 1.00 [1.0]*, V/M = 0.00
# Query_time sparkline: | ^ |
# Attribute pct total min max avg 95% stddev median
# ============ === ======= ======= ======= ======= ======= ======= =======
# Count 100 3
# Exec time 100 113ms 31ms 41ms 38ms 40ms 5ms 40ms
# Lock time 100 142us 46us 50us 47us 49us 2us 44us
# Rows sent 100 3 1 1 1 1 0 1
# Rows examine 100 136.15k 45.33k 45.49k 45.38k 44.45k 0.00 44.45k
# Query size 100 234 76 79 78 76.28 1.50 76.28
# String:
# Hosts ip-127.0.0.1.ec2.internal
# Users db_one
# Query_time distribution
# 1us
# 10us
# 100us
# 1ms
# 10ms ################################################################
# 100ms
# 1s
# 10s+
# Tables
# SHOW TABLE STATUS LIKE 'companies'\G
# SHOW CREATE TABLE `companies`\G
# EXPLAIN /*!50100 PARTITIONS*/
SELECT `companies`.* FROM `companies` WHERE `companies`.`id` = 286358 LIMIT 1\G
So I guess that the problem is connected with log file format and possibly rotation (e.g. the file is truncated so that the initial #description is lost).
Also, I've looked a bit into the Percona utility code. The default parser (slowlog) searches for a #Time line to get timestamp, which I find in my slow-log, but is not present in yours. This does not influence reading per se, but it might skew the results.
It also uses ";\n#" as input record separator, so the syslog format is a definite "no" for the default --type slowlog. The others also do not appear to work with syslog+slowlog format.
I've tried changing the input record separator and adding a hack to remove the syslog part of each line, and the result appears to work, but I fear it does not because the results on a known slowlog aren't consistent.
I am afraid the simplest thing to do is to chop off the beginning of the file, then cut off the beginning of each line, before feeding it to the utility:
sed -e '/.*: #/,$b' -e 'd' < slow.log \
| cut -d' ' -f6- \
| pt-query-digest -
It seems like the tool should never have an output like that, so you might consider filing a bug report.
The error may or may not be with your log file. You might try splitting your log file into a few smaller files and seeing if the files can be parsed -- this might at least narrow down the source of the bug. If all of the smaller files are parsed correctly, maybe it is an issue related to a lack of resources or pt-query-digest being killed for taking up too many resources.