Status: "MariaDB server is down" - mysql

Question

Hope someone can help me:
I have many sites running on my server and I am using mariadb.
when I do:
sudo service mysql restart
I get:
Job for mariadb.service failed because the control process exited with
error code. See "systemctl status mariadb.service" and "journalctl -xe"
for details.
systemctl status mariadb.service
mariadb.service - MariaDB database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf
Active: failed (Result: exit-code) since Thu 2017-09-14 03:16:51 UTC; 1min 6s ago
Process: 13247 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=
Process: 13086 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`/usr/bin/gale
Process: 13070 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, s
Process: 13033 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, st
Main PID: 13247 (code=exited, status=1/FAILURE)
Status: "MariaDB server is down"
CGroup: /system.slice/mariadb.service
└─3803 /usr/sbin/mysqld
sudo journalctl -xe
Sep 14 03:35:51 ubuntu systemd[1]: Failed to start MariaDB database server.
-- Subject: Unit mariadb.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mariadb.service has failed.
--
-- The result is failed.
Sep 14 03:35:51 ubuntu systemd[1]: mariadb.service: Unit entered failed state.
Sep 14 03:35:51 ubuntu sudo[13566]: pam_unix(sudo:session): session closed for user root
Sep 14 03:35:51 ubuntu systemd[1]: mariadb.service: Failed with result 'exit-code'.
Sep 14 03:35:52 ubuntu sshd[13753]: Failed password for root from 123.183.209.136 port 38816 ssh2
Sep 14 03:35:52 ubuntu sshd[13753]: Received disconnect from 123.183.209.136 port 38816:11: [preauth]
Sep 14 03:35:52 ubuntu sshd[13753]: Disconnected from 123.183.209.136 port 38816 [preauth]
Sep 14 03:35:52 ubuntu sshd[13753]: PAM 1 more authentication failure; logname= uid=0 euid=0 tty=ssh ruser
Sep 14 03:36:34 ubuntu sshd[13756]: Connection closed by 123.183.209.136 port 60477 [preauth]
Sep 14 03:36:36 ubuntu postfix/pickup[12161]: 703AA5C0C5: uid=0 from=<root>
Sep 14 03:36:36 ubuntu postfix/cleanup[12162]: 703AA5C0C5: message-id=<20170914033636.703AA5C0C5#ubuntu.me
Sep 14 03:36:36 ubuntu postfix/cleanup[12162]: warning: 703AA5C0C5: write queue file: No space left on dev
Sep 14 03:36:36 ubuntu postfix/pickup[12161]: warning: maildrop/5152F5C0C4: error writing 703AA5C0C5: queu
Sep 14 03:37:17 ubuntu sshd[13760]: Received disconnect from 123.183.209.136 port 25567:11: [preauth]
Sep 14 03:37:17 ubuntu sshd[13760]: Disconnected from 123.183.209.136 port 25567 [preauth]
Sep 14 03:37:36 ubuntu postfix/pickup[12161]: 7764B5C0C5: uid=0 from=<root>
Sep 14 03:37:36 ubuntu postfix/cleanup[12162]: 7764B5C0C5: message-id=<20170914033736.7764B5C0C5#ubuntu.me
Sep 14 03:37:36 ubuntu postfix/cleanup[12162]: warning: 7764B5C0C5: write queue file: No space left on dev
Sep 14 03:37:36 ubuntu postfix/pickup[12161]: warning: maildrop/5152F5C0C4: error writing 7764B5C0C5: queu
Sep 14 03:37:47 ubuntu sudo[13765]: bob : TTY=pts/0 ; PWD=/home/bob ; USER=root ; COMMAND=/bin/journa
Sep 14 03:37:47 ubuntu sudo[13765]: pam_unix(sudo:session): session opened for user root by bob(uid=0)
lines 1238-1265/1265 (END)
I really don't want to lose my db.
What should I do next?
Many thanks
this is what df gives me:
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 49362256 46854648 0 100% /
devtmpfs 2019828 0 2019828 0% /dev
tmpfs 2021768 0 2021768 0% /dev/shm
tmpfs 2021768 17968 2003800 1% /run
tmpfs 5120 0 5120 0% /run/lock
tmpfs 2021768 0 2021768 0% /sys/fs/cgroup
tmpfs 404356 0 404356 0% /run/user/1000
tmpfs 404356 0 404356 0% /run/user/0

Answer 1

See the first filesystem (/dev/root) is 100% full! And there are no other volumes mounted besides tmpfs volumes. So mariadb is trying to write something to disk as it starts up, and can't, so it gives up and shuts itself down.
You have to figure out what's filling up your disk, and prune what you can. For example, as a guess I'd start by looking in the http server logs, since you say you host many sites on this server. I've seen many servers that filled up disk space by letting httpd log everything for months and months.
In the future, you might want to provision this server with more than one disk volume, so if one volume fills up it doesn't interfere with other volumes.
Also consider automating log rotation.
If the culprit for disk space usage is not logs, but is actually the mariadb data directory, then you might need to get an additional disk volume immediately, and move the mariadb content to it before you can bring your sites back online.
Unless you can find a little bit of other stuff elsewhere on the server that you can delete and make enough space for a short while.