Related
I'm trying to import some exported migration data into a MariaDB database.
I could successfully do the import into the H2 database.
But when trying to import in the MariaDB one, it creates 87 tables in the database instead of 91 tables, and also ends up in error:
2018-04-22 14:13:33,275 INFO [org.keycloak.connections.jpa.updater.liquibase.LiquibaseJpaUpdaterProvider] (ServerService Thread Pool -- 58) Initializing database schema. Using changelog META-INF/jpa-changelog-master.xml
2018-04-22 14:18:22,393 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'add' at address '[
("core-service" => "management"),
("management-interface" => "http-interface")
]'
This new log chunk shows it takes almost 5 mn. It's way too long.
More from the stacktrace:
16:16:55,690 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 58) MSC000001: Failed to start service jboss.undertow.deployment.default-server.default-host./auth: org.jboss.msc.service.StartException in service jboss.undertow.deployment.default-server.default-host./auth: java.lang.RuntimeException: RESTEASY003325: Failed to construct public org.keycloak.services.resources.KeycloakApplication(javax.servlet.ServletContext,org.jboss.resteasy.core.Dispatcher)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:84)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at org.jboss.threads.JBossThread.run(JBossThread.java:320)
Caused by: java.lang.RuntimeException: RESTEASY003325: Failed to construct public org.keycloak.services.resources.KeycloakApplication(javax.servlet.ServletContext,org.jboss.resteasy.core.Dispatcher)
at org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:162)
at org.jboss.resteasy.spi.ResteasyProviderFactory.createProviderInstance(ResteasyProviderFactory.java:2298)
at org.jboss.resteasy.spi.ResteasyDeployment.createApplication(ResteasyDeployment.java:340)
at org.jboss.resteasy.spi.ResteasyDeployment.start(ResteasyDeployment.java:253)
at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.init(ServletContainerDispatcher.java:120)
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.init(HttpServletDispatcher.java:36)
at io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:117)
at org.wildfly.extension.undertow.security.RunAsLifecycleInterceptor.init(RunAsLifecycleInterceptor.java:78)
at io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:103)
at io.undertow.servlet.core.ManagedServlet$DefaultInstanceStrategy.start(ManagedServlet.java:250)
at io.undertow.servlet.core.ManagedServlet.createServlet(ManagedServlet.java:133)
at io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:565)
at io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:536)
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42)
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
at io.undertow.servlet.core.DeploymentManagerImpl.start(DeploymentManagerImpl.java:578)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:100)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:81)
... 6 more
Caused by: java.lang.RuntimeException: Failed to update database
at org.keycloak.connections.jpa.updater.liquibase.LiquibaseJpaUpdaterProvider.update(LiquibaseJpaUpdaterProvider.java:102)
at org.keycloak.connections.jpa.updater.liquibase.LiquibaseJpaUpdaterProvider.update(LiquibaseJpaUpdaterProvider.java:67)
at org.keycloak.connections.jpa.DefaultJpaConnectionProviderFactory.update(DefaultJpaConnectionProviderFactory.java:322)
at org.keycloak.connections.jpa.DefaultJpaConnectionProviderFactory.migration(DefaultJpaConnectionProviderFactory.java:292)
at org.keycloak.connections.jpa.DefaultJpaConnectionProviderFactory.lambda$lazyInit$0(DefaultJpaConnectionProviderFactory.java:179)
at org.keycloak.models.utils.KeycloakModelUtils.suspendJtaTransaction(KeycloakModelUtils.java:544)
at org.keycloak.connections.jpa.DefaultJpaConnectionProviderFactory.lazyInit(DefaultJpaConnectionProviderFactory.java:130)
at org.keycloak.connections.jpa.DefaultJpaConnectionProviderFactory.create(DefaultJpaConnectionProviderFactory.java:78)
at org.keycloak.connections.jpa.DefaultJpaConnectionProviderFactory.create(DefaultJpaConnectionProviderFactory.java:56)
at org.keycloak.services.DefaultKeycloakSession.getProvider(DefaultKeycloakSession.java:163)
at org.keycloak.models.jpa.JpaRealmProviderFactory.create(JpaRealmProviderFactory.java:51)
at org.keycloak.models.jpa.JpaRealmProviderFactory.create(JpaRealmProviderFactory.java:33)
at org.keycloak.services.DefaultKeycloakSession.getProvider(DefaultKeycloakSession.java:163)
at org.keycloak.models.cache.infinispan.RealmCacheSession.getDelegate(RealmCacheSession.java:144)
at org.keycloak.models.cache.infinispan.RealmCacheSession.getMigrationModel(RealmCacheSession.java:137)
at org.keycloak.migration.MigrationModelManager.migrate(MigrationModelManager.java:76)
at org.keycloak.services.resources.KeycloakApplication.migrateModel(KeycloakApplication.java:246)
at org.keycloak.services.resources.KeycloakApplication.migrateAndBootstrap(KeycloakApplication.java:187)
at org.keycloak.services.resources.KeycloakApplication$1.run(KeycloakApplication.java:146)
at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:227)
at org.keycloak.services.resources.KeycloakApplication.<init>(KeycloakApplication.java:137)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:150)
... 28 more
Caused by: liquibase.exception.MigrationFailedException: Migration failed for change set META-INF/jpa-changelog-2.1.0.xml::2.1.0::bburke#redhat.com:
Reason: liquibase.exception.UnexpectedLiquibaseException: java.sql.SQLException: IJ031040: Connection is not associated with a managed connection: org.jboss.jca.adapters.jdbc.jdk8.WrappedConnectionJDK8#55194ba1
at liquibase.changelog.ChangeSet.execute(ChangeSet.java:573)
at liquibase.changelog.visitor.UpdateVisitor.visit(UpdateVisitor.java:51)
at liquibase.changelog.ChangeLogIterator.run(ChangeLogIterator.java:73)
at liquibase.Liquibase.update(Liquibase.java:210)
at liquibase.Liquibase.update(Liquibase.java:190)
at liquibase.Liquibase.update(Liquibase.java:186)
at org.keycloak.connections.jpa.updater.liquibase.LiquibaseJpaUpdaterProvider.updateChangeSet(LiquibaseJpaUpdaterProvider.java:135)
at org.keycloak.connections.jpa.updater.liquibase.LiquibaseJpaUpdaterProvider.update(LiquibaseJpaUpdaterProvider.java:88)
... 53 more
Caused by: liquibase.exception.UnexpectedLiquibaseException: java.sql.SQLException: IJ031040: Connection is not associated with a managed connection: org.jboss.jca.adapters.jdbc.jdk8.WrappedConnectionJDK8#55194ba1
at liquibase.database.jvm.JdbcConnection.getURL(JdbcConnection.java:79)
at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:62)
at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:122)
at liquibase.database.AbstractJdbcDatabase.execute(AbstractJdbcDatabase.java:1247)
at liquibase.database.AbstractJdbcDatabase.executeStatements(AbstractJdbcDatabase.java:1230)
at liquibase.changelog.ChangeSet.execute(ChangeSet.java:548)
... 60 more
Caused by: java.sql.SQLException: IJ031040: Connection is not associated with a managed connection: org.jboss.jca.adapters.jdbc.jdk8.WrappedConnectionJDK8#55194ba1
at org.jboss.jca.adapters.jdbc.WrappedConnection.lock(WrappedConnection.java:164)
at org.jboss.jca.adapters.jdbc.WrappedConnection.getMetaData(WrappedConnection.java:913)
at liquibase.database.jvm.JdbcConnection.getURL(JdbcConnection.java:77)
... 65 more
The export command was:
$KEYCLOAK_HOME/bin/standalone.sh -Dkeycloak.migration.action=export -Dkeycloak.migration.provider=dir -Dkeycloak.migration.dir=exported_realms -Dkeycloak.migration.strategy=OVERWRITE_EXISTING
The import command that fails is:
$KEYCLOAK_HOME/bin/standalone.sh -Dkeycloak.migration.action=export -Dkeycloak.migration.provider=dir -Dkeycloak.migration.dir=exported_realms -Dkeycloak.migration.strategy=OVERWRITE_EXISTING
Here is the data source used in the standalone/configuration/standalone.xml file:
<datasource jndi-name="java:/jboss/datasources/KeycloakDS" pool-name="KeycloakDS" enabled="true">
<connection-url>jdbc:mysql://localhost:3306/keycloak?useSSL=false&characterEncoding=UTF-8</connection-url>
<driver>mysql</driver>
<pool>
<min-pool-size>5</min-pool-size>
<max-pool-size>15</max-pool-size>
</pool>
<security>
<user-name>keycloak</user-name>
<password>xxxxxx</password>
</security>
<validation>
<valid-connection-checker class-name="org.jboss.jca.adapters.jdbc.extensions.mysql.MySQLValidConnectionChecker"/>
<validate-on-match>true</validate-on-match>
<exception-sorter class-name="org.jboss.jca.adapters.jdbc.extensions.mysql.MySQLExceptionSorter"/>
</validation>
</datasource>
I'm using keycloak-3.4.1.Final and mariadb-10.1.24 on a java version 1.8.0_60.
Running the ./mysqltuner.pl utility shows:
-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[OK] InnoDB buffer pool / data size: 2.0G/222.6M
[OK] Ratio InnoDB log file size / InnoDB Buffer pool size: 256.0M * 2/2.0G should be equal 25%
[OK] InnoDB buffer pool instances: 2
[--] InnoDB Buffer Pool Chunk Size not used or defined in your version
[!!] InnoDB Read buffer efficiency: 63.85% (802 hits/ 1256 total)
[!!] InnoDB Write Log efficiency: 0% (1 hits/ 0 total)
[OK] InnoDB log waits: 0.00% (0 waits / 1 writes)
General recommendations:
Control warning line(s) into /home/stephane/programs/install/mariadb/mariadb.error.log file
1 CVE(s) found for your MySQL release. Consider upgrading your version !
MySQL started within last 24 hours - recommendations may be inaccurate
Dedicate this server to your database for highest performance.
Reduce or eliminate unclosed connections and network issues
Consider installing Sys schema from https://github.com/mysql/mysql-sys
Variables to adjust:
query_cache_size (=0)
query_cache_type (=0)
query_cache_limit (> 1M, or use smaller result sets)
After installing the MySQLTuner utility:
wget https://raw.githubusercontent.com/major/MySQLTuner-perl/master/mysqltuner.pl
wget https://raw.githubusercontent.com/major/MySQLTuner-perl/master/basic_passwords.txt -O basic_passwords.txt
wget https://raw.githubusercontent.com/major/MySQLTuner-perl/master/vulnerabilities.csv -O vulnerabilities.csv
chmod +x mysqltuner.pl
./mysqltuner.pl
I understood my database server was poorly configured and writes were too slow, resulting in a timeout of the import operation.
I then configured the my.cnf file with the directives:
skip-name-resolve = 1
performance_schema = 1
innodb_log_file_size = 256M
innodb_buffer_pool_size = 2G
innodb_buffer_pool_instances = 2
innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
thread_cache_size = 4
The one directive that allowed the import to complete successfully was:
innodb_flush_log_at_trx_commit = 2
UPDATE: I commented out the innodb_flush_log_at_trx_commit = 2 directive, so as to trigger the error again. I could then collect the additional information as requested by the bellow comment.
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 14761
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 14761
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
$ free -m
total used free shared buff/cache available
Mem: 3743 2751 116 115 875 700
Swap: 4450 74 4376
The complete my.cnf file:
[mysqld]
sql_mode = NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION # This is strict mode: NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES
socket = /home/stephane/programs/install/mariadb/tmp/mariadb.sock
user = stephane
basedir = /home/stephane/programs/install/mariadb
datadir = /home/stephane/programs/install/mariadb/data
log-bin = /home/stephane/programs/install/mariadb/mariadb.bin.log
log-error = /home/stephane/programs/install/mariadb/mariadb.error.log
general-log-file = /home/stephane/programs/install/mariadb/mariadb.log
slow-query-log-file = /home/stephane/programs/install/mariadb/mariadb.slow.queries.log
long_query_time = 1
log-queries-not-using-indexes = 1
innodb_file_per_table = 1
sync_binlog = 1
character-set-client-handshake = FALSE
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
wait_timeout = 28800 # amount of seconds during inactivity that MySQL will wait before it will close a connection on a non-interactive connection
interactive_timeout = 28800 # same, but for interactive sessions
max_allowed_packet = 128M
net_write_timeout = 180
skip-name-resolve = 1
thread_cache_size = 4
#skip-networking
#skip-host-cache
#bulk_insert_buffer_size = 1G
performance_schema = 1
innodb_log_file_size = 128M
innodb_buffer_pool_size = 1G
innodb_buffer_pool_instances = 2
#innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
[client]
socket = /home/stephane/programs/install/mariadb/tmp/mariadb.sock
default-character-set = utf8mb4
[mysql]
default-character-set = utf8mb4
The environment status and variables
The other commands output:
$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 1,9G 0 1,9G 0% /dev
tmpfs 375M 6,1M 369M 2% /run
/dev/sda1 17G 7,6G 8,4G 48% /
tmpfs 1,9G 21M 1,9G 2% /dev/shm
tmpfs 5,0M 4,0K 5,0M 1% /run/lock
tmpfs 1,9G 0 1,9G 0% /sys/fs/cgroup
/dev/sda5 438G 51G 365G 13% /home
tmpfs 375M 16K 375M 1% /run/user/1000
$ top - 19:22:22 up 1:13, 1 user, load average: 2,04, 1,27, 1,17
Tasks: 223 total, 1 running, 222 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2,9 us, 0,6 sy, 0,0 ni, 73,3 id, 23,1 wa, 0,0 hi, 0,1 si, 0,0 st
KiB Mem : 3833232 total, 196116 free, 2611360 used, 1025756 buff/cache
KiB Swap: 4557820 total, 4488188 free, 69632 used. 985000 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
10399 stephane 20 0 4171240 448816 25724 S 3,6 11,7 0:39.78 java
8110 stephane 20 0 1217152 111392 40800 S 2,3 2,9 0:54.32 chrome
8290 stephane 20 0 1276140 148024 41360 S 2,0 3,9 0:43.52 chrome
1272 root 20 0 373844 45632 28108 S 1,0 1,2 1:37.31 Xorg
3172 stephane 20 0 729100 37060 22116 S 1,0 1,0 0:14.50 gnome-terminal-
11433 stephane 20 0 3163040 324680 9288 S 1,0 8,5 0:05.09 mysqld
8260 stephane 20 0 1242104 142028 42292 S 0,7 3,7 0:31.93 chrome
8358 stephane 20 0 1252060 99884 40876 S 0,7 2,6 0:34.06 chrome
12580 root 20 0 1095296 78872 36456 S 0,7 2,1 0:10.29 dockerd
14 root rt 0 0 0 0 S 0,3 0,0 0:00.01 watchdog/1
2461 stephane 20 0 1232332 203156 74752 S 0,3 5,3 4:29.75 chrome
7437 stephane 20 0 3509576 199780 46004 S 0,3 5,2 0:20.66 skypeforlinux
8079 stephane 20 0 1243784 130948 38848 S 0,3 3,4 0:23.82 chrome
8191 stephane 20 0 1146672 72848 37536 S 0,3 1,9 0:12.41 chrome
8501 root 20 0 0 0 0 S 0,3 0,0 0:00.80 kworker/0:1
9331 stephane 20 0 46468 4164 3380 R 0,3 0,1 0:01.38 top
1 root 20 0 220368 8492 6404 S 0,0 0,2 0:02.26 systemd
2 root 20 0 0 0 0 S 0,0 0,0 0:00.00 kthreadd
4 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 kworker/0:0H
6 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 mm_percpu_wq
$ iostat -x
Linux 4.13.0-39-generic (stephane-ThinkPad-X201) 22/05/2018 _x86_64_ (4 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
8,65 0,92 2,17 8,73 0,00 79,53
Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
sda 42,68 23,49 816,72 905,13 8,40 36,83 16,45 61,06 17,18 52,96 1,98 19,14 38,53 4,31 28,53
$ free -m
total used free shared buff/cache available
Mem: 3743 2571 137 107 1034 924
Swap: 4450 68 4382
The top, iostat and free commands were executed during the import migration script execution.
The full MySQL Tuner output
The root cause of this error is because the linux Server needs increment the number of open files.
Really, you need first tune your bdd, because when is very slowly this cause a timeout.
Check the attribute "open files" with the command:
ulimit -n
In my case I use 200000.
Please use this example:
# maximum capability of system
user#ubuntu:~$ cat /proc/sys/fs/file-max
708444
# available limit
user#ubuntu:~$ ulimit -n
1024
# To increase the available limit to say 200000
user#ubuntu:~$ sudo vim /etc/sysctl.conf
# add the following line to it
fs.file-max = 200000
# run this to refresh with new config
user#ubuntu:~$ sudo sysctl -p
# edit the following file
user#ubuntu:~$ sudo vim /etc/security/limits.conf
# add following lines to it
* soft nofile 200000
* hard nofile 200000
www-data soft nofile 200000
www-data hard nofile 200000
root soft nofile 200000
root hard nofile 200000
# edit the following file
user#ubuntu:~$ sudo vim /etc/pam.d/common-session
# add this line to it
session required pam_limits.so
# logout and login and try the following command
user#ubuntu:~$ ulimit -n
200000
# now you can increase no.of.connections per Nginx worker
# in Nginx main config /etc/nginx/nginx.conf
worker_connections 200000;
worker_rlimit_nofile 200000;
Your ulimit -a report indicates 'open files' are at 1024
ulimit -n 10000 would expand your capacity to better accommodate MySQL.
In the 1,318 seconds reported in SHOW GLOBAL STATUS, there we
33 com_rollback items counted and
1 handler_rollback possibly all the result of the java failure documented above.
Suggestions to consider for your my.cnf-ini [mysqld] section to possibly speed up the processing.
Suggestions by Back To Basics, Inc. 05/24/2018 for this IMPORT processing
max_connect_errors=10 # why tolerate 100 hacker/cracker attempts?
thread_cache_size=30 # from 4 to ensure threads ready to go
innodb_io_capacity_max=10000 # from 2000 default, for SSD vs HDD
innodb_io_capacity=5000 # from 200 default, for SSD vs HDD
have_symlink=NO # to protect server from RANSOMWARE crowd
innodb_flush_neighbors=0 # from 1, no need when SSD - no rotational delay
innodb_lru_scan_depth=512 # from 1024 to conserve CPU see v8 refman
innodb_print_all_deadlocks=ON # from OFF in error log for proactive correction
innodb_purge_threads=4 # from 1 to speed purge processing
log_bin=OFF # from ON unless you need to invest the resources during import
log_warnings=2 # from 1 for addl info on aborted_connection in error log
max_join_size=1000000000 # from upper limit of 4 Billion rows
max_seeks_for_key=32 # rather than allowing optimizer to search 4 Billion ndx's.
max_write_lock_count=16 # to allow RD after nn lcks rather than 4 Billion
performance_schema=OFF # from ON for this IMPORT processing speed
log_queries_not_using_indexes=0 # not likely to look at these, for import
Copy and paste at END of [mysqld} for quick test, weed out duplicate
same variable names when time permits from the top of [mysqld].
These will not resolve the error documented but should speed up the processing.
PLease provide feedback when time permits.
I've successfully configured my Ubuntu server so MySQL can use hugepages. Problem is when I want to enable memlock option in my.cnf and /proc/sys/vm/hugetlb_shm_group is set to another group (MySQL is a member of this group).
# uname -a
Linux hostname 3.13.0-32-generic #57~precise1-Ubuntu SMP Tue Jul 15 03:51:20 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
# mysqld --version
mysqld Ver 5.5.53-0ubuntu0.12.04.1 for debian-linux-gnu on x86_64 ((Ubuntu))
# egrep ^Huge /proc/meminfo
HugePages_Total: 15000
HugePages_Free: 15000
HugePages_Rsvd: 0
HugePages_Surp: 0
# id mysql
uid=107(mysql) gid=113(mysql) groups=113(mysql),117(hugetlb)
If /proc/sys/vm/hugetlb_shm_group is set to 113 everything goes well no matter of memlock option.
But if I set /proc/sys/vm/hugetlb_shm_group to 117 it works only without the memlock option. If I enable it I get this error when MySQL starts:
InnoDB: HugeTLB: Warning: Failed to allocate 21978152960 bytes. errno 1
InnoDB HugeTLB: Warning: Using conventional memory pool
Any idea what can be wrong?
Turned out to be a problem with ulimit. Settings in /etc/security/limits.conf are ignored so I got it working with simple edit in /etc/init/mysql.conf:
exec /usr/bin/mysqld
changed to
script
ulimit -l unlimited
/usr/bin/mysqld
end script
I searched about the topic subject and tested options, but I still cant increase the open-files-limit on my mariadb server that is used as remote database server for cpanel/whm server.
here is s good reference
http://duntuk.com/how-raise-ulimit-open-files-and-mysql-openfileslimit
I increased it in
/etc/my.cnf
open-files-limit=65550
here is some results
#ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 63471
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65535
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 65535
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
#ulimit -Hn -Sn
open files (-n) 65535
open files (-n) 65535
cat /etc/systemd/system/mariadb.service.d/limits.conf
[Service]
LimitNOFILE=65500
cat /usr/lib/systemd/system/mariadb.service
[Unit]
Description=MariaDB database server
After=syslog.target
After=network.target
[Service]
Type=simple
User=mysql
Group=mysql
LimitNOFILE=infinity
LimitMEMLOCK=infinity
and still in mysql I get
show global variables like 'open%';
+------------------+-------+
| Variable_name | Value |
+------------------+-------+
| open_files_limit | 1024 |
+------------------+-------+
1 row in set (0.00 sec)
and im getting following error over and over after restart
60108 16:30:02 [ERROR] mysqld: Can't open file: './username_wp/wp_users.frm' (errno: 24)
160108 16:30:02 [ERROR] Error in accept: Too many open files
160108 16:30:04 [ERROR] Error in accept: Too many open files
160108 16:30:06 [ERROR] Error in accept: Too many open files
160108 16:30:11 [ERROR] mysqld: Can't open file: './username_db/strikes.frm' (errno: 24)
and here is the startup log of mariadb
[Warning] Could not increase number of max_open_files to more than 1024 (request: 132107)
RH/CentOS mariadb integration is quiet confusing, using in some places mariadb as its name, mysqld elsewhere...
You should edit(or create) a .conf file in
/etc/systemd/system/mariadb.service.d/
For eg: /etc/systemd/system/mariadb.service.d/centreon.conf
Then edit it as kujiy suggested
[Service]
LimitNOFILE=320000
Don't forget to reload systemd services files:
# systemctl daemon-reload
Cedric
This is an OS problem.
I think you have to increase the "hard limit" in /etc/security/limits.conf, something like this:
* hard nofile 65536
* soft nofile 16384
then use ulimit to increase the "soft limit".
finally i find solution but without any reasonable cause !
i was running MariaDB 5.5 and all settings where fine but the soft limit did not goes more than 1024 !
i was thinking my mariadb is 10.0 as i freshly installed it
after i find out that its 5.5 i tried to upgrade it to v10.0 ! and then BOOM ! problem solved without any extra action or setting ( kernel setting allows maximum files limit as described in reference link )
i hope it help other peoples have same problem ( but after try all other required settings )
You can see the official instruction in the mariadb.service file;
[root#db1 system]# cat /usr/lib/systemd/system/mariadb.service | grep exam -A 5
# For example, if you want to increase mariadb's open-files-limit to 10000,
# you need to increase systemd's LimitNOFILE setting, so create a file named
# "/etc/systemd/system/mariadb.service.d/limits.conf" containing:
# [Service]
# LimitNOFILE=10000
Though I think this should be written in the official manual...
I have CentOS 7.3 with MariaDB 10.0.29 installed from official repo.
For some reason, systemd unit is named mysql and I had to create /etc/systemd/system/mysql.service.d/oioki.conf:
[Service]
LimitNOFILE=500000
Don't forget to run systemctl daemon-reload after that.
open_files_limit is dynamic value that depends on next params:
#my.cnf file
# max connections
max_connections = 64
# table_open_cache = max_connections * tables used in one transaction + 5
table_open_cache = 800
# table_definition_cache = all tables(50) + max_connections + 5
table_definition_cache = 400
# open_files_limit = table_open_cache * 1.4
open_files_limit = 1120
To check the value of open_files_limit run this command from mysql:
mysql> select ##open_files_limit;
...it will output:
+--------------------+
| ##open_files_limit |
+--------------------+
| 12903 |
+--------------------+
1 row in set (0.000 sec)
I had this problem running RedHat Enterprise Linux 7.2 with Software Collections (SCL) and SystemD
rh-mariadb100-mariadb-server-10.0.20-1.el7.x86_64
In my case the hard limit was ok, but the soft limit was stuck on 1024
Summary of the old and new
old : RHEL6.x : MySQL : init.d : mysql_safe : 'root', mysqld : 'mysql'
new : RHEL7.x : MariaDB : SystemD : mysql_safe : 'mysql', mysqld 'mysql'
Note that the old initl.d wrapper script runs as root, but the new SystemD wrapper script runs as 'mysql'. This can cause mysql_safe to not make the call to ulimit. In my case setting LimitNOFILE in SystemD didn't help - all it did was increase the file limit for the wrapper, but NOT for the daemon itself.
I found a two step process to get it working
Step 1 - re-instate the mysqld_safe section to the conf file so that mysql_safe will read it
sudo vi /etc/opt/rh/rh-mariadb100/my.cnf.d/mariadb-server.cnf
[mysqld_safe]
open_files_limit=2048
Step 2 - create a SystemD drop-in to start the wrapper as root (the daemon will still run as 'mysql'). Note that LimitNOFILE is commented out.
/etc/systemd/system
mkdir rh-mariadb100-mariadb.service.d
cd rh-mariadb100-mariadb.service.d
vi limits.conf
[Service]
#LimitNOFILE=2048
User=
Group=
Reload SystemD service files
systemctl daemon-reload
now restart the service, check value of open_files_limit in mysql client
it should now say 2048 (new soft limit)
The proper solution would probably be to get rid of the wrapper and use native SystemD.
UPDATE
it looks like this is fixed in the more recent version, which no longer uses the mysql_safe wrapper
rh-mariadb101-mariadb-10.1.19-6.el7.x86_64
To increase open-files-limit in MySQL 5.6 on Centos 7 you must:
nano /usr/lib/systemd/system/mysqld.service
add at the end of the file:
LimitNOFILE=65535
LimitNPROC=65535
then:
systemctl daemon-reload
systemctl restart mysqld
Done! , I assume that for MariaDB is the same thing
Ensure that no other files exist in the:
/etc/systemd/system/mariadb.service.d
folder that might be limiting the number of files.
That was my case, there was another file that was reducing the limit I set :)
I'm managing a site that does streaming video, the site is running off:
Intel(R) Xeon(R) CPU E31220 # 3.10GHz
16 GB RAM
2 x 2TB RAID1
1gbit unmetered bandwidth is not an issue as the usage is only 150-250mbit etc.
On this machine, I run the following:
- apache to host the [url]www.domain.com[/url]
- mysql server (apt-get install mysql-server)
Everything has been going fine except now there are 200 people streaming videos at any given time.. so now I'm noticing that the site loads slower.. I believe this is because mysql is using 100% cpu. Below I've pasted my top, and also my mysql settings.. Could someone help me with the mysql settings to slow this down?
I do realize that at some point I need to move the mysql server to another machine on its own, but I still think with the current settings im about to post, it would still use 100% cpu even on the standalone machine.. so i think the settings need to be changed? or can someone guide me. Also, I cannot lower the wait_timeout because when I do that, it causes errors on the video conversion script which fetches the videos and converts them and sometimes it can take awhile so i don't know if thats a problem or what
my.cnf:
# * Fine Tuning
#
max_allowed_packet = 16M
thread_stack = 1M
thread_cache_size = 50
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover = BACKUP
max_connections = 1000
wait_timeout = 20000
tmp_table_size = 500M
max_heap_table_size = 1000M
table_cache = 1000
#thread_concurrency = 10
#
# * Query Cache Configuration
#
query_cache_limit = 4M
query_cache_size = 64M
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file = /var/log/mysql/mysql.log
#general_log = 1
#
# Error log - should be very few entries.
#
log_error = /var/log/mysql/error.log
#
# Here you can see queries with especially long duration
#log_slow_queries = /var/log/mysql/mysql-slow.log
#long_query_time = 2
#log-queries-not-using-indexes
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
# other settings you may need to change.
#server-id = 1
#log_bin = /var/log/mysql/mysql-bin.log
expire_logs_days = 10
max_binlog_size = 100M
#binlog_do_db = include_database_name
#binlog_ignore_db = include_database_name
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
#no-auto-rehash # faster start of mysql but no tab completition
[isamchk]
key_buffer_size = 64M
----------------------------------------
>> MySQLTuner 1.1.1 - Major Hayden <major#mhtx.net>
>> Bug reports, feature requests, and downloads at [url]http://mysqltuner.com/[/url]
>> Run with '--help' for additional options and output filtering
[!!] Successfully authenticated with no password - SECURITY RISK!
-------- General Statistics --------------------------------------------------
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.5.37-0ubuntu0.13.10.1
[OK] Operating on 64-bit architecture
-------- Storage Engine Statistics -------------------------------------------
[--] Status: +Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
[--] Data in MyISAM tables: 1G (Tables: 41)
[--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 17)
[!!] InnoDB is enabled but isn't being used
[!!] Total fragmented tables: 4
-------- Security Recommendations -------------------------------------------
[OK] All database users have passwords assigned
-------- Performance Metrics -------------------------------------------------
[--] Up for: 22h 23m 46s (5M q [69.809 qps], 1M conn, TX: 8B, RX: 4B)
[--] Reads / Writes: 29% / 71%
[--] Total buffers: 716.0M global + 3.5M per thread (1000 max threads)
[OK] Maximum possible memory usage: 4.1G (26% of installed RAM)
[OK] Slow queries: 0% (2/5M)
[OK] Highest usage of available connections: 77% (775/1000)
[OK] Key buffer size / total MyISAM indexes: 8.0M/119.0M
[OK] Key buffer hit rate: 99.4% (33M cached / 208K reads)
[OK] Query cache efficiency: 67.6% (1M cached / 1M selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 921 sorts)
[!!] Temporary tables created on disk: 28% (244 on disk / 849 total)
[OK] Thread cache hit rate: 98% (17K created / 1M connections)
[OK] Table cache hit rate: 62% (428 open / 681 opened)
[OK] Open file limit used: 9% (476/5K)
[!!] Table locks acquired immediately: 55%
-------- Recommendations -----------------------------------------------------
General recommendations:
Add skip-innodb to MySQL configuration to disable InnoDB
Run OPTIMIZE TABLE to defragment tables for better performance
MySQL started within last 24 hours - recommendations may be inaccurate
Enable the slow query log to troubleshoot bad queries
Temporary table size is already large - reduce result set size
Reduce your SELECT DISTINCT queries without LIMIT clauses
Optimize queries and/or use InnoDB to reduce lock wait
-------------------------------------------------------
It says run optimize table, but how do i do that? On the web I've tried to search and found to run the mysql check command but i get
# mysqlcheck -u root -p --auto-repair --check --optimize --all-databases
Error: mysqlcheck doesn't support multiple contradicting commands.
Further, is it safe to just optimize the database? It wont hurt it or anything? Of course ill back up first
I've enabled slow query log now, but so far nothing yet..
At one point, Mysqltuner was saying i had reached 996/1000 connections.. but when i went to go raise the max_connections to 2000 and then i restarted the mysql server, the site became even slower than before.. Perhaps i should not be restarting the mysql server and just adjusting globally ?
top - 23:27:42 up 11 days, 5:28, 3 users, load average: 2.41, 4.40, 5.97
Tasks: 269 total, 3 running, 265 sleeping, 0 stopped, 1 zombie
%Cpu(s): 24.3 us, 5.6 sy, 0.0 ni, 69.3 id, 0.3 wa, 0.0 hi, 0.5 si, 0.0 st
KiB Mem: 16408692 total, 16237624 used, 171068 free, 9552 buffers
KiB Swap: 15624184 total, 8220 used, 15615964 free, 15072644 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
29178 mysql 20 0 2662m 73m 7280 S 100.4 0.5 4:49.03 /usr/sbin/mysqld
29428 daemon 20 0 544m 19m 7048 S 2.7 0.1 0:07.81 /usr/sbin/apache2 -k start
29943 daemon 20 0 543m 17m 5428 S 2.0 0.1 0:02.57 /usr/sbin/apache2 -k start
29945 daemon 20 0 543m 17m 5352 S 2.0 0.1 0:00.76 /usr/sbin/apache2 -k start
29672 daemon 20 0 543m 16m 4696 S 1.7 0.1 0:00.63 /usr/sbin/apache2 -k start
29792 daemon 20 0 544m 17m 5396 S 1.3 0.1 0:00.15 /usr/sbin/apache2 -k start
183 root 20 0 0 0 0 S 1.0 0.0 8:51.20 [md0_raid1]
29445 daemon 20 0 543m 16m 4700 S 0.7 0.1 0:01.04 /usr/sbin/apache2 -k start
29744 daemon 20 0 543m 16m 5384 S 0.7 0.1 0:00.23 /usr/sbin/apache2 -k start
7705 root 35 15 0 0 0 D 0.3 0.0 2:27.65 [md0_resync]
29435 daemon 20 0 543m 17m 5596 S 0.3 0.1 0:00.37 /usr/sbin/apache2 -k start
29451 daemon 20 0 543m 17m 5356 S 0.3 0.1 0:00.44 /usr/sbin/apache2 -k start
29453 daemon 20 0 0 0 0 Z 0.3 0.0 0:00.07 [apache2] <defunct>
29501 daemon 20 0 543m 16m 5324 S 0.3 0.1 0:00.10 /usr/sbin/apache2 -k start
29518 daemon 20 0 543m 17m 5948 S 0.3 0.1 0:00.31 /usr/sbin/apache2 -k start
29534 daemon 20 0 543m 17m 5456 S 0.3 0.1 0:00.42 /usr/sbin/apache2 -k start
29539 daemon 20 0 543m 16m 5348 S 0.3 0.1 0:00.24 /usr/sbin/apache2 -k start
29542 daemon 20 0 543m 16m 4680 S 0.3 0.1 0:00.51 /usr/sbin/apache2 -k start
29549 daemon 20 0 543m 17m 5352 S 0.3 0.1 0:00.90 /usr/sbin/apache2 -k start
29656 daemon 20 0 543m 16m 4792 S 0.3 0.1 0:00.25 /usr/sbin/apache2 -k start
29673 daemon 20 0 543m 16m 5392 R 0.3 0.1 0:00.20 /usr/sbin/apache2 -k start
29682 daemon 20 0 543m 16m 4704 S 0.3 0.1 0:00.58 /usr/sbin/apache2 -k start
29791 daemon 20 0 543m 17m 5392 S 0.3 0.1 0:00.81 /usr/sbin/apache2 -k start
29793 daemon 20 0 543m 17m 5712 S 0.3 0.1 0:00.12 /usr/sbin/apache2 -k start
29926 daemon 20 0 543m 16m 4704 S 0.3 0.1 0:00.44 /usr/sbin/apache2 -k start
29956 daemon 20 0 543m 16m 4700 S 0.3 0.1 0:00.33 /usr/sbin/apache2 -k start
=====================================
# free -tm
total used free shared buffers cached
Mem: 16024 15861 162 0 4 14351
-/+ buffers/cache: 1506 14517
Swap: 15257 8 15249
Total: 31282 15870 15411
Additional question:
If I do move the mysql to its own machine, should the hard drive be ssds ? is that better for mysql ? and would 32 gb of ram be sufficient
The biggest thing in the database is the sessions table which can get as big as 1.5 gb so far.. but i clear it and the database shrinks to under 100mb
I believe you cant --check and --optimize at the same time, because --check means "just check, don't touch anything".
It is not normal for MySQL to use a lot of CPU, as its main objective is to store and fetch information, not process it. It should use a lot of memory though, in fact it is good to allow MySQL to use as much memory as you have available. After all, unused memory is wasted memory. I see MySQL is using 0.5%. This is definitevely a bad sign. Tune your query cache up.
You may be overusing SQL queries in your site, but if I had to guess I believe your problem may be lack of proper indexing in your tables.
When you do things like SELECT * FROM parent, child WHERE child.parent_id = parent.id it is important that parent.id is a PRIMARY KEY in order to allow MySQL to perform it's task more efficiently.
I am importing a large csv file into mysql(30 million rows) and I had another terminal open to show the process list. I used to be able to see the row count on the process list but now, whenever I enter the command "show process list" it hangs. I had 20 million records imported. Do I have to start all over again?
iostat:
[user#gggggg ~]$ iostat
Linux 2.6.18-308.4.1.el5
avg-cpu: %user %nice %system %iowait %steal %idle
5.02 0.00 0.16 0.87 0.00 93.95
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 2.15 5.31 55.58 8644514 90425752
sda1 0.00 0.00 0.00 2138 5592
sda2 2.15 5.31 55.57 8640178 90419984
sda3 0.00 0.00 0.00 1582 176
top - 14:18:55 up 18 days, 20:00, 2 users, load average: 2.02, 2.09, 2.06
Tasks: 106 total, 3 running, 103 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.2%us, 0.0%sy, 0.0%ni, 49.9%id, 49.8%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 8174532k total, 7656780k used, 517752k free, 105904k buffers
Swap: 4257216k total, 88k used, 4257128k free, 6958020k cached
First, check the filesize from the OS
If the table is called mydb.mytable and it is MyISAM, do this:
cd /var/lib/mysql/mydb
watch ls -l mytable.*
If the filesize keeps growing, you are fine
Don't forget to check for diskspace
df -h
If you ran out of diskspace while loading a MyISAM table, mysqld is just sitting there. Why? According to MySQL 5.0 Certification Study Guide Page 408,409 Section 29.2 bulletpoint 11 says:
If you run out of disk space while adding rows to a MyISAM table, no
error occurs. The server suspends the operation until space becomes
available, and then completes the operation.
Therefore, if the data partition is out of diskspace, you must free up some space so mysqld can conitnue the LOAD DATA INFILE.
If everything seems frozen, you may have to kill mysqld as follows:
IDTOKILL=`ps -ef | grep mysqld_safe | grep -v grep | awk '{print $2}'`
kill -9 ${IDTOFKILL}
IDTOKILL=`ps -ef | grep mysqld | grep -v grep | awk '{print $2}'`
kill -9 ${IDTOFKILL}
Then, check your diskspace