I'm trying to import some exported migration data into a MariaDB database.
I could successfully do the import into the H2 database.
But when trying to import in the MariaDB one, it creates 87 tables in the database instead of 91 tables, and also ends up in error:
2018-04-22 14:13:33,275 INFO [org.keycloak.connections.jpa.updater.liquibase.LiquibaseJpaUpdaterProvider] (ServerService Thread Pool -- 58) Initializing database schema. Using changelog META-INF/jpa-changelog-master.xml
2018-04-22 14:18:22,393 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'add' at address '[
("core-service" => "management"),
("management-interface" => "http-interface")
]'
This new log chunk shows it takes almost 5 mn. It's way too long.
More from the stacktrace:
16:16:55,690 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 58) MSC000001: Failed to start service jboss.undertow.deployment.default-server.default-host./auth: org.jboss.msc.service.StartException in service jboss.undertow.deployment.default-server.default-host./auth: java.lang.RuntimeException: RESTEASY003325: Failed to construct public org.keycloak.services.resources.KeycloakApplication(javax.servlet.ServletContext,org.jboss.resteasy.core.Dispatcher)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:84)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at org.jboss.threads.JBossThread.run(JBossThread.java:320)
Caused by: java.lang.RuntimeException: RESTEASY003325: Failed to construct public org.keycloak.services.resources.KeycloakApplication(javax.servlet.ServletContext,org.jboss.resteasy.core.Dispatcher)
at org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:162)
at org.jboss.resteasy.spi.ResteasyProviderFactory.createProviderInstance(ResteasyProviderFactory.java:2298)
at org.jboss.resteasy.spi.ResteasyDeployment.createApplication(ResteasyDeployment.java:340)
at org.jboss.resteasy.spi.ResteasyDeployment.start(ResteasyDeployment.java:253)
at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.init(ServletContainerDispatcher.java:120)
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.init(HttpServletDispatcher.java:36)
at io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:117)
at org.wildfly.extension.undertow.security.RunAsLifecycleInterceptor.init(RunAsLifecycleInterceptor.java:78)
at io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:103)
at io.undertow.servlet.core.ManagedServlet$DefaultInstanceStrategy.start(ManagedServlet.java:250)
at io.undertow.servlet.core.ManagedServlet.createServlet(ManagedServlet.java:133)
at io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:565)
at io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:536)
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42)
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
at io.undertow.servlet.core.DeploymentManagerImpl.start(DeploymentManagerImpl.java:578)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:100)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:81)
... 6 more
Caused by: java.lang.RuntimeException: Failed to update database
at org.keycloak.connections.jpa.updater.liquibase.LiquibaseJpaUpdaterProvider.update(LiquibaseJpaUpdaterProvider.java:102)
at org.keycloak.connections.jpa.updater.liquibase.LiquibaseJpaUpdaterProvider.update(LiquibaseJpaUpdaterProvider.java:67)
at org.keycloak.connections.jpa.DefaultJpaConnectionProviderFactory.update(DefaultJpaConnectionProviderFactory.java:322)
at org.keycloak.connections.jpa.DefaultJpaConnectionProviderFactory.migration(DefaultJpaConnectionProviderFactory.java:292)
at org.keycloak.connections.jpa.DefaultJpaConnectionProviderFactory.lambda$lazyInit$0(DefaultJpaConnectionProviderFactory.java:179)
at org.keycloak.models.utils.KeycloakModelUtils.suspendJtaTransaction(KeycloakModelUtils.java:544)
at org.keycloak.connections.jpa.DefaultJpaConnectionProviderFactory.lazyInit(DefaultJpaConnectionProviderFactory.java:130)
at org.keycloak.connections.jpa.DefaultJpaConnectionProviderFactory.create(DefaultJpaConnectionProviderFactory.java:78)
at org.keycloak.connections.jpa.DefaultJpaConnectionProviderFactory.create(DefaultJpaConnectionProviderFactory.java:56)
at org.keycloak.services.DefaultKeycloakSession.getProvider(DefaultKeycloakSession.java:163)
at org.keycloak.models.jpa.JpaRealmProviderFactory.create(JpaRealmProviderFactory.java:51)
at org.keycloak.models.jpa.JpaRealmProviderFactory.create(JpaRealmProviderFactory.java:33)
at org.keycloak.services.DefaultKeycloakSession.getProvider(DefaultKeycloakSession.java:163)
at org.keycloak.models.cache.infinispan.RealmCacheSession.getDelegate(RealmCacheSession.java:144)
at org.keycloak.models.cache.infinispan.RealmCacheSession.getMigrationModel(RealmCacheSession.java:137)
at org.keycloak.migration.MigrationModelManager.migrate(MigrationModelManager.java:76)
at org.keycloak.services.resources.KeycloakApplication.migrateModel(KeycloakApplication.java:246)
at org.keycloak.services.resources.KeycloakApplication.migrateAndBootstrap(KeycloakApplication.java:187)
at org.keycloak.services.resources.KeycloakApplication$1.run(KeycloakApplication.java:146)
at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:227)
at org.keycloak.services.resources.KeycloakApplication.<init>(KeycloakApplication.java:137)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:150)
... 28 more
Caused by: liquibase.exception.MigrationFailedException: Migration failed for change set META-INF/jpa-changelog-2.1.0.xml::2.1.0::bburke#redhat.com:
Reason: liquibase.exception.UnexpectedLiquibaseException: java.sql.SQLException: IJ031040: Connection is not associated with a managed connection: org.jboss.jca.adapters.jdbc.jdk8.WrappedConnectionJDK8#55194ba1
at liquibase.changelog.ChangeSet.execute(ChangeSet.java:573)
at liquibase.changelog.visitor.UpdateVisitor.visit(UpdateVisitor.java:51)
at liquibase.changelog.ChangeLogIterator.run(ChangeLogIterator.java:73)
at liquibase.Liquibase.update(Liquibase.java:210)
at liquibase.Liquibase.update(Liquibase.java:190)
at liquibase.Liquibase.update(Liquibase.java:186)
at org.keycloak.connections.jpa.updater.liquibase.LiquibaseJpaUpdaterProvider.updateChangeSet(LiquibaseJpaUpdaterProvider.java:135)
at org.keycloak.connections.jpa.updater.liquibase.LiquibaseJpaUpdaterProvider.update(LiquibaseJpaUpdaterProvider.java:88)
... 53 more
Caused by: liquibase.exception.UnexpectedLiquibaseException: java.sql.SQLException: IJ031040: Connection is not associated with a managed connection: org.jboss.jca.adapters.jdbc.jdk8.WrappedConnectionJDK8#55194ba1
at liquibase.database.jvm.JdbcConnection.getURL(JdbcConnection.java:79)
at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:62)
at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:122)
at liquibase.database.AbstractJdbcDatabase.execute(AbstractJdbcDatabase.java:1247)
at liquibase.database.AbstractJdbcDatabase.executeStatements(AbstractJdbcDatabase.java:1230)
at liquibase.changelog.ChangeSet.execute(ChangeSet.java:548)
... 60 more
Caused by: java.sql.SQLException: IJ031040: Connection is not associated with a managed connection: org.jboss.jca.adapters.jdbc.jdk8.WrappedConnectionJDK8#55194ba1
at org.jboss.jca.adapters.jdbc.WrappedConnection.lock(WrappedConnection.java:164)
at org.jboss.jca.adapters.jdbc.WrappedConnection.getMetaData(WrappedConnection.java:913)
at liquibase.database.jvm.JdbcConnection.getURL(JdbcConnection.java:77)
... 65 more
The export command was:
$KEYCLOAK_HOME/bin/standalone.sh -Dkeycloak.migration.action=export -Dkeycloak.migration.provider=dir -Dkeycloak.migration.dir=exported_realms -Dkeycloak.migration.strategy=OVERWRITE_EXISTING
The import command that fails is:
$KEYCLOAK_HOME/bin/standalone.sh -Dkeycloak.migration.action=export -Dkeycloak.migration.provider=dir -Dkeycloak.migration.dir=exported_realms -Dkeycloak.migration.strategy=OVERWRITE_EXISTING
Here is the data source used in the standalone/configuration/standalone.xml file:
<datasource jndi-name="java:/jboss/datasources/KeycloakDS" pool-name="KeycloakDS" enabled="true">
<connection-url>jdbc:mysql://localhost:3306/keycloak?useSSL=false&characterEncoding=UTF-8</connection-url>
<driver>mysql</driver>
<pool>
<min-pool-size>5</min-pool-size>
<max-pool-size>15</max-pool-size>
</pool>
<security>
<user-name>keycloak</user-name>
<password>xxxxxx</password>
</security>
<validation>
<valid-connection-checker class-name="org.jboss.jca.adapters.jdbc.extensions.mysql.MySQLValidConnectionChecker"/>
<validate-on-match>true</validate-on-match>
<exception-sorter class-name="org.jboss.jca.adapters.jdbc.extensions.mysql.MySQLExceptionSorter"/>
</validation>
</datasource>
I'm using keycloak-3.4.1.Final and mariadb-10.1.24 on a java version 1.8.0_60.
Running the ./mysqltuner.pl utility shows:
-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[OK] InnoDB buffer pool / data size: 2.0G/222.6M
[OK] Ratio InnoDB log file size / InnoDB Buffer pool size: 256.0M * 2/2.0G should be equal 25%
[OK] InnoDB buffer pool instances: 2
[--] InnoDB Buffer Pool Chunk Size not used or defined in your version
[!!] InnoDB Read buffer efficiency: 63.85% (802 hits/ 1256 total)
[!!] InnoDB Write Log efficiency: 0% (1 hits/ 0 total)
[OK] InnoDB log waits: 0.00% (0 waits / 1 writes)
General recommendations:
Control warning line(s) into /home/stephane/programs/install/mariadb/mariadb.error.log file
1 CVE(s) found for your MySQL release. Consider upgrading your version !
MySQL started within last 24 hours - recommendations may be inaccurate
Dedicate this server to your database for highest performance.
Reduce or eliminate unclosed connections and network issues
Consider installing Sys schema from https://github.com/mysql/mysql-sys
Variables to adjust:
query_cache_size (=0)
query_cache_type (=0)
query_cache_limit (> 1M, or use smaller result sets)
After installing the MySQLTuner utility:
wget https://raw.githubusercontent.com/major/MySQLTuner-perl/master/mysqltuner.pl
wget https://raw.githubusercontent.com/major/MySQLTuner-perl/master/basic_passwords.txt -O basic_passwords.txt
wget https://raw.githubusercontent.com/major/MySQLTuner-perl/master/vulnerabilities.csv -O vulnerabilities.csv
chmod +x mysqltuner.pl
./mysqltuner.pl
I understood my database server was poorly configured and writes were too slow, resulting in a timeout of the import operation.
I then configured the my.cnf file with the directives:
skip-name-resolve = 1
performance_schema = 1
innodb_log_file_size = 256M
innodb_buffer_pool_size = 2G
innodb_buffer_pool_instances = 2
innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
thread_cache_size = 4
The one directive that allowed the import to complete successfully was:
innodb_flush_log_at_trx_commit = 2
UPDATE: I commented out the innodb_flush_log_at_trx_commit = 2 directive, so as to trigger the error again. I could then collect the additional information as requested by the bellow comment.
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 14761
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 14761
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
$ free -m
total used free shared buff/cache available
Mem: 3743 2751 116 115 875 700
Swap: 4450 74 4376
The complete my.cnf file:
[mysqld]
sql_mode = NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION # This is strict mode: NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES
socket = /home/stephane/programs/install/mariadb/tmp/mariadb.sock
user = stephane
basedir = /home/stephane/programs/install/mariadb
datadir = /home/stephane/programs/install/mariadb/data
log-bin = /home/stephane/programs/install/mariadb/mariadb.bin.log
log-error = /home/stephane/programs/install/mariadb/mariadb.error.log
general-log-file = /home/stephane/programs/install/mariadb/mariadb.log
slow-query-log-file = /home/stephane/programs/install/mariadb/mariadb.slow.queries.log
long_query_time = 1
log-queries-not-using-indexes = 1
innodb_file_per_table = 1
sync_binlog = 1
character-set-client-handshake = FALSE
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
wait_timeout = 28800 # amount of seconds during inactivity that MySQL will wait before it will close a connection on a non-interactive connection
interactive_timeout = 28800 # same, but for interactive sessions
max_allowed_packet = 128M
net_write_timeout = 180
skip-name-resolve = 1
thread_cache_size = 4
#skip-networking
#skip-host-cache
#bulk_insert_buffer_size = 1G
performance_schema = 1
innodb_log_file_size = 128M
innodb_buffer_pool_size = 1G
innodb_buffer_pool_instances = 2
#innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
[client]
socket = /home/stephane/programs/install/mariadb/tmp/mariadb.sock
default-character-set = utf8mb4
[mysql]
default-character-set = utf8mb4
The environment status and variables
The other commands output:
$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 1,9G 0 1,9G 0% /dev
tmpfs 375M 6,1M 369M 2% /run
/dev/sda1 17G 7,6G 8,4G 48% /
tmpfs 1,9G 21M 1,9G 2% /dev/shm
tmpfs 5,0M 4,0K 5,0M 1% /run/lock
tmpfs 1,9G 0 1,9G 0% /sys/fs/cgroup
/dev/sda5 438G 51G 365G 13% /home
tmpfs 375M 16K 375M 1% /run/user/1000
$ top - 19:22:22 up 1:13, 1 user, load average: 2,04, 1,27, 1,17
Tasks: 223 total, 1 running, 222 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2,9 us, 0,6 sy, 0,0 ni, 73,3 id, 23,1 wa, 0,0 hi, 0,1 si, 0,0 st
KiB Mem : 3833232 total, 196116 free, 2611360 used, 1025756 buff/cache
KiB Swap: 4557820 total, 4488188 free, 69632 used. 985000 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
10399 stephane 20 0 4171240 448816 25724 S 3,6 11,7 0:39.78 java
8110 stephane 20 0 1217152 111392 40800 S 2,3 2,9 0:54.32 chrome
8290 stephane 20 0 1276140 148024 41360 S 2,0 3,9 0:43.52 chrome
1272 root 20 0 373844 45632 28108 S 1,0 1,2 1:37.31 Xorg
3172 stephane 20 0 729100 37060 22116 S 1,0 1,0 0:14.50 gnome-terminal-
11433 stephane 20 0 3163040 324680 9288 S 1,0 8,5 0:05.09 mysqld
8260 stephane 20 0 1242104 142028 42292 S 0,7 3,7 0:31.93 chrome
8358 stephane 20 0 1252060 99884 40876 S 0,7 2,6 0:34.06 chrome
12580 root 20 0 1095296 78872 36456 S 0,7 2,1 0:10.29 dockerd
14 root rt 0 0 0 0 S 0,3 0,0 0:00.01 watchdog/1
2461 stephane 20 0 1232332 203156 74752 S 0,3 5,3 4:29.75 chrome
7437 stephane 20 0 3509576 199780 46004 S 0,3 5,2 0:20.66 skypeforlinux
8079 stephane 20 0 1243784 130948 38848 S 0,3 3,4 0:23.82 chrome
8191 stephane 20 0 1146672 72848 37536 S 0,3 1,9 0:12.41 chrome
8501 root 20 0 0 0 0 S 0,3 0,0 0:00.80 kworker/0:1
9331 stephane 20 0 46468 4164 3380 R 0,3 0,1 0:01.38 top
1 root 20 0 220368 8492 6404 S 0,0 0,2 0:02.26 systemd
2 root 20 0 0 0 0 S 0,0 0,0 0:00.00 kthreadd
4 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 kworker/0:0H
6 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 mm_percpu_wq
$ iostat -x
Linux 4.13.0-39-generic (stephane-ThinkPad-X201) 22/05/2018 _x86_64_ (4 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
8,65 0,92 2,17 8,73 0,00 79,53
Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
sda 42,68 23,49 816,72 905,13 8,40 36,83 16,45 61,06 17,18 52,96 1,98 19,14 38,53 4,31 28,53
$ free -m
total used free shared buff/cache available
Mem: 3743 2571 137 107 1034 924
Swap: 4450 68 4382
The top, iostat and free commands were executed during the import migration script execution.
The full MySQL Tuner output
The root cause of this error is because the linux Server needs increment the number of open files.
Really, you need first tune your bdd, because when is very slowly this cause a timeout.
Check the attribute "open files" with the command:
ulimit -n
In my case I use 200000.
Please use this example:
# maximum capability of system
user#ubuntu:~$ cat /proc/sys/fs/file-max
708444
# available limit
user#ubuntu:~$ ulimit -n
1024
# To increase the available limit to say 200000
user#ubuntu:~$ sudo vim /etc/sysctl.conf
# add the following line to it
fs.file-max = 200000
# run this to refresh with new config
user#ubuntu:~$ sudo sysctl -p
# edit the following file
user#ubuntu:~$ sudo vim /etc/security/limits.conf
# add following lines to it
* soft nofile 200000
* hard nofile 200000
www-data soft nofile 200000
www-data hard nofile 200000
root soft nofile 200000
root hard nofile 200000
# edit the following file
user#ubuntu:~$ sudo vim /etc/pam.d/common-session
# add this line to it
session required pam_limits.so
# logout and login and try the following command
user#ubuntu:~$ ulimit -n
200000
# now you can increase no.of.connections per Nginx worker
# in Nginx main config /etc/nginx/nginx.conf
worker_connections 200000;
worker_rlimit_nofile 200000;
Your ulimit -a report indicates 'open files' are at 1024
ulimit -n 10000 would expand your capacity to better accommodate MySQL.
In the 1,318 seconds reported in SHOW GLOBAL STATUS, there we
33 com_rollback items counted and
1 handler_rollback possibly all the result of the java failure documented above.
Suggestions to consider for your my.cnf-ini [mysqld] section to possibly speed up the processing.
Suggestions by Back To Basics, Inc. 05/24/2018 for this IMPORT processing
max_connect_errors=10 # why tolerate 100 hacker/cracker attempts?
thread_cache_size=30 # from 4 to ensure threads ready to go
innodb_io_capacity_max=10000 # from 2000 default, for SSD vs HDD
innodb_io_capacity=5000 # from 200 default, for SSD vs HDD
have_symlink=NO # to protect server from RANSOMWARE crowd
innodb_flush_neighbors=0 # from 1, no need when SSD - no rotational delay
innodb_lru_scan_depth=512 # from 1024 to conserve CPU see v8 refman
innodb_print_all_deadlocks=ON # from OFF in error log for proactive correction
innodb_purge_threads=4 # from 1 to speed purge processing
log_bin=OFF # from ON unless you need to invest the resources during import
log_warnings=2 # from 1 for addl info on aborted_connection in error log
max_join_size=1000000000 # from upper limit of 4 Billion rows
max_seeks_for_key=32 # rather than allowing optimizer to search 4 Billion ndx's.
max_write_lock_count=16 # to allow RD after nn lcks rather than 4 Billion
performance_schema=OFF # from ON for this IMPORT processing speed
log_queries_not_using_indexes=0 # not likely to look at these, for import
Copy and paste at END of [mysqld} for quick test, weed out duplicate
same variable names when time permits from the top of [mysqld].
These will not resolve the error documented but should speed up the processing.
PLease provide feedback when time permits.
Kubuntu 17
When I login and do "ps -ef | grep mysqld" I see two copies running, one as the system service daemon, and one as a user process.
ps -ef | grep mysqld
mysql 1953 1 0 09:02 ? 00:00:13 /usr/sbin/mysqld
richard 3233 3220 0 09:13 ? 00:00:35 /usr/sbin/mysqld --
defaults-file=/home/richard/.local/share/akonadi/mysql.conf --
datadir=/home/richard/.local/share/akonadi/db_data/ --
socket=/tmp/akonadi-richard.EK3Z9U/mysql.socket
richard 13309 3138 0 18:05 pts/1 00:00:00 grep --color=auto
mysqld
I don't see any script or command with "mysqld" in ~/.profle, ~/.autostart, ~/.bashrc, /etc/profile, /etc/init.d (except for the mysql start script presumably used by the system and owned by root.
Where else should I look for the errant command?
Any great ideas on how to look for it effectively?
I've successfully configured my Ubuntu server so MySQL can use hugepages. Problem is when I want to enable memlock option in my.cnf and /proc/sys/vm/hugetlb_shm_group is set to another group (MySQL is a member of this group).
# uname -a
Linux hostname 3.13.0-32-generic #57~precise1-Ubuntu SMP Tue Jul 15 03:51:20 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
# mysqld --version
mysqld Ver 5.5.53-0ubuntu0.12.04.1 for debian-linux-gnu on x86_64 ((Ubuntu))
# egrep ^Huge /proc/meminfo
HugePages_Total: 15000
HugePages_Free: 15000
HugePages_Rsvd: 0
HugePages_Surp: 0
# id mysql
uid=107(mysql) gid=113(mysql) groups=113(mysql),117(hugetlb)
If /proc/sys/vm/hugetlb_shm_group is set to 113 everything goes well no matter of memlock option.
But if I set /proc/sys/vm/hugetlb_shm_group to 117 it works only without the memlock option. If I enable it I get this error when MySQL starts:
InnoDB: HugeTLB: Warning: Failed to allocate 21978152960 bytes. errno 1
InnoDB HugeTLB: Warning: Using conventional memory pool
Any idea what can be wrong?
Turned out to be a problem with ulimit. Settings in /etc/security/limits.conf are ignored so I got it working with simple edit in /etc/init/mysql.conf:
exec /usr/bin/mysqld
changed to
script
ulimit -l unlimited
/usr/bin/mysqld
end script
SQLSTATE[HY000] [2001] Can't create UNIX socket (12)
On my local machine my project works fine. As soon as I pushed my changes to the server I get this error message.
The mysql log is totally empty.
Mysql config: my.cnf
However another site that uses the same mysql server works fine.
Here are some more information: My vServer is running ubuntu 10.04 and has 2 GB of RAM.
/root$ ps -le | grep mysqld
0 S 0 9502 1 0 85 0 - 1025 wait ? 00:00:00 mysqld_safe
4 S 109 9539 9502 0 76 0 - 77209 stext ? 00:01:02 mysqld
/root$ ps -le | grep mysqld
0 S 0 9502 1 0 85 0 - 1025 wait ? 00:00:00 mysqld_safe
4 S 109 9539 9502 0 75 0 - 77209 stext ? 00:01:03 mysqld
/root$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Maybe there is just to view ram?
This is CR_SOCKET_CREATE_ERROR, an error generated by the client code.
The 12 is probably your client's errno. What is error 12 on your system? For me, using the mysql perror command line utility, it's "OS error code 12: Cannot allocate memory"
I'm having an issue where MySQL 5.1.54 is restarting every 30 minutes on Ubuntu 11.04. When this occurs, the following appears in the MySQL log:
111030 12:01:52 [Note] /usr/sbin/mysqld: Normal shutdown
111030 12:01:52 [Note] Event Scheduler: Purging the queue. 0 events
111030 12:01:52 InnoDB: Starting shutdown...
111030 12:01:54 InnoDB: Shutdown completed; log sequence number 0 875122
111030 12:01:54 [Note] /usr/sbin/mysqld: Shutdown complete
111030 12:01:55 [Note] Plugin 'FEDERATED' is disabled.
111030 12:01:55 InnoDB: Initializing buffer pool, size = 256.0M
111030 12:01:55 InnoDB: Completed initialization of buffer pool
111030 12:01:55 InnoDB: Started; log sequence number 0 875122
111030 12:01:55 [Note] Event Scheduler: Loaded 0 events
111030 12:01:55 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.54-1ubuntu4-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu)
This occurs like clockwork every 30 minutes, so it's obviously some service restarting it.
I have checked the crontab of every user on the system (including system users), and none of them have a crontab setup, as you can see in the output below:
# awk -F: '{print $1}' /etc/passwd | xargs -n 1 -i crontab -u {} -l
no crontab for root
no crontab for daemon
no crontab for bin
no crontab for sys
no crontab for sync
no crontab for games
no crontab for man
no crontab for lp
no crontab for mail
no crontab for news
no crontab for uucp
no crontab for proxy
no crontab for www-data
no crontab for backup
no crontab for list
no crontab for irc
no crontab for gnats
no crontab for nobody
no crontab for libuuid
no crontab for syslog
no crontab for sshd
no crontab for landscape
no crontab for ubuntu
no crontab for statd
no crontab for myproxy
no crontab for condor
no crontab for messagebus
no crontab for avahi
no crontab for joe
no crontab for smmta
no crontab for smmsp
no crontab for postfix
no crontab for deploy
no crontab for mysql
no crontab for redis
My dmesg contains the following each time it is restarted. I'm not an apparmor expert, but I believe this is a normal message obtained each time the MySQL service starts:
[1165328.780405] type=1400 audit(1319976114.984:74): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=31985 comm="apparmor_parser"
Also, here are the contents of the MySQL upstart configuration in /etc/init/mysql.conf:
# MySQL Service
description "MySQL Server"
author "Mario Limonciello <superm1#ubuntu.com>"
start on (net-device-up
and local-filesystems
and runlevel [2345])
stop on runlevel [016]
respawn
env HOME=/etc/mysql
umask 007
# The default of 5 seconds is too low for mysql which needs to flush buffers
kill timeout 300
pre-start script
#Sanity checks
[ -r $HOME/my.cnf ]
[ -d /var/run/mysqld ] || install -m 755 -o mysql -g root -d /var/run/mysqld
/lib/init/apparmor-profile-load usr.sbin.mysqld
LC_ALL=C BLOCKSIZE= df --portability /var/lib/mysql/. | tail -n 1 | awk '{ exit ($4<4096) }'
end script
exec /usr/sbin/mysqld
post-start script
for i in `seq 1 30` ; do
/usr/bin/mysqladmin --defaults-file="${HOME}"/debian.cnf ping && {
exec "${HOME}"/debian-start
# should not reach this line
exit 2
}
sleep 1
done
exit 1
end script
Any idea what might be causing this? It doesn't cause any problems, other than Monit alerts stating that "PID changed Service mysqld" (I have Monit monitoring mysqld -- but it reports no errors with the mysqld process, other than the fact that every 30 minutes, it has its PID changed since MySQL is restarted).
Thanks in advance.
are you using chef or puppet, which might be doing something that triggers the reboot?
can you check (and probably post) the definition of your mysql job in upstart? (/etc/init/mysql.conf). OK = try removing the "respawn". It does not work as documented in the upstart documentation. Generally it's used to respawn the process if it's killed by other process, but it seems it does not function as expected. You can see why the apparmour is always loading - because of the pre-start stanza in the script.
As upstart is very new, and is still evolving, better is to use the SysV way.
You should try to run it without AppArmor: just run /usr/bin/mysqld_safe or /usr/bin/mysqld without using upstart and wait 30 minutes. If mysql does not auto-restart, then disable AppArmor in the /etc/init/mysql.conf file, or configure it differently.
If the problem is still there, read mysql's log. If the log is not enabled by default, you can use the option --log-error=/tmp/mysql.log --log-warnings when launching mysqld.
Seems converting the /etc/init.d/mysql start up script to sysV style rather than as an upstart script seemed to correct the problem for me.
I had this same problem upgrading to Ubuntu 14.04. I found this question because it mentions the AppArmor log message, so thanks! I may not have realised MySQL was restarting otherwise.
Upon investigating /var/log/daemon.log, I found the output of /etc/mysql/debian-start appearing repeatedly. The relevant part was this:
May 18 06:48:18 tom /etc/mysql/debian-start[15525]: Upgrading MySQL tables if necessary.
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: Looking for 'mysql' as: /usr/bin/mysql
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: Error: Server version (5.5.35-1ubuntu1) does not match with the version of
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: the server (5.5.37) with which this program was built/distributed. You can
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: use --skip-version-check to skip this check.
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: FATAL ERROR: Upgrade failed
I read through the /etc/mysql/debian-start script and tried running the upgrade command as in the script, hoping to debug it (MySQL server needs to be running at the time):
/usr/bin/mysql_upgrade --defaults-extra-file=/etc/mysql/debian.cnf
I then found that this worked without complaint and everything simply worked from that point onward. I don't know why it was failing in the first place, but that seemed to fix it. MySQL hasn't restarted itself since.