MariaDB server is down but I can still login with mysql client (?) - mysql

Question

MariaDB 10.3.8, Ubuntu 18 LTS.
When I check for the database status, I see that MariaDB is down. But I can login using the mysql client command. Is something wrong? How could I debug this?
sudo service mariadb status
mariadb.service - MariaDB 10.3.8 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; disabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf
Active: failed (Result: timeout) since Wed 2019-02-13 08:43:11 IST; 1min 0s ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 3293 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
Process: 3712 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
Process: 3706 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 3693 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Main PID: 3878 (mysqld)
Tasks: 13 (limit: 4915)
CGroup: /system.slice/mariadb.service
└─3878 /usr/sbin/mysqld
# lsof -i:3306
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
mysqld 2346 user_running_mariadb 31u IPv6 28910 0t0 TCP *:mysql (LISTEN)
ps uax | grep mysql
root 2049 0.0 0.0 4628 1812 ? S 08:40 0:00 /bin/sh /usr/bin/mysqld_safe
user_running_mariadb 2346 0.1 0.4 3005364 114444 ? Sl 08:40 0:02 /usr/sbin/mysqld --basedir=/usr --datadir=/my_db_files_dir --plugin-dir=/usr/lib/mysql/plugin --user=user_running_mariadb --skip-log-error --pid-file=/my_db_rundir/run/mysqld.pid --socket=/my_db_rundir/run/mysqld.sock --port=3306
#/my_db_rundir/run# ls -l
-rw-rw---- 1 user_running_mariadb group_running_mariadb 5 Feb 13 08:40 mysqld.pid
srwxrwxrwx 1 user_running_mariadb group_running_mariadb 0 Feb 13 08:40 mysqld.sock
#/my_db_rundir/run# cat mysqld.pid
2346
I can login:
mysql -u root -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 145
Server version: 10.3.8-MariaDB-1:10.3.8+maria~bionic-log mariadb.org
inary distribution
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input
tatement.
MariaDB [(none)]>
Additional outputs:
Error log:
2019-02-13 8:52:13 0 [ERROR] InnoDB: Unable to lock ./ibdata1 error: 11
2019-02-13 8:52:13 0 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2019-02-13 8:52:13 0 [Note] InnoDB: Unable to open the first data file
2019-02-13 8:52:13 0 [ERROR] InnoDB: Operating system error number 11 in a file operation.
2019-02-13 8:52:13 0 [ERROR] InnoDB: Error number 11 means 'Resource temporarily unavailable'
journalctl -u mariadb.service
Feb 13 08:51:31 my_host systemd[1]: mariadb.service: Start operation timed out. Terminating.
Feb 13 08:52:13 my_host systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Feb 13 08:52:13 my_host systemd[1]: mariadb.service: Failed with result 'timeout'.
Feb 13 08:52:13 my_host systemd[1]: Failed to start MariaDB 10.3.8 database server.

Answer 1

The service is timed out ('Result: timeout'), and the service has the systemd configuration SendSIGKILL=no, which means systemd has just let mysqld continue running.
Why it time out? take a look at journalctl -u mariadb.service.

Answer 2

I could fix the issue, however I'm not completely sure about the root cause and what really fixed the issue. Still, perhaps this information helps others facing the same issue.
Reinstalled MariaDB and upgraded to 10.3.12. This wasn't probably needed because I kept hitting the same issue after the upgrade.
After reboot, noticed that msyqld status showed as active while mariadb as inactive:
/ati/mariadb/logs# sudo service mysql status
Loaded: loaded (/etc/init.d/mysql; generated)
Active: active (running)
Docs: man:systemd-sysv-generator(8)
Process: 1650 ExecStart=/etc/init.d/mysql start (code=exited, status=0/SUCCESS)
Tasks: 39 (limit: 4915)
CGroup: /system.slice/mysql.service
─2003 /bin/sh /usr/bin/mysqld_safe
─2301 /usr/sbin/mysqld --basedir=/usr...
└─2302 logger -t mysqld -p daemon error
sudo service mariadb status
mariadb.service - MariaDB 10.3.12 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; disabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf
Active: inactive (dead)
In MariaDB web site, found the way of enabling mariadb service, with
systemctl enable mariadb
Now, it works again.