apt upgrade mysqlserver freezes and will not upgrade (ubuntu server 20.04) - mysql

I had to kill the session and then the databases are inaccessible until I reboot the server.
When I log back in and issue ‘apt upgrade’ i get the following error.
E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem.
I run ‘dpkg --configure -a’ and get the following and then it just freezes
Setting up mysql-server-8.0 (8.0.22-0ubuntu0.20.04.3) ...
mysqld will log errors to /var/log/mysql/error.log
mysqld is running as pid 86150
On another tty
journalctl -xe gives this when the upgrade freezes.
-- The unit mysql.service has successfully entered the 'dead' state.
Jan 13 22:18:12 arpradio systemd[1]: Stopped MySQL Community Server.
-- Subject: A stop job for unit mysql.service has finished
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A stop job for unit mysql.service has finished.
--
-- The job identifier is 582 and the job result is done.
Jan 13 22:18:13 arpradio audit[6057]: AVC apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="/usr/sbin/mysqld" pid>
Jan 13 22:18:13 arpradio kernel: audit: type=1400 audit(1610572693.036:16): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="uncon>
When I reboot and login mysqld is running
I have tried the following.
# cp -r /etc/mysql /etc/mysql_old
# apt autoremove
E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem.
# dpkg --configure -a
Setting up mysql-server-8.0 (8.0.22-0ubuntu0.20.04.3) ...
mysqld will log errors to /var/log/mysql/error.log
mysqld is running as pid 7451
And we are back here with the freeze.
If I check the mysql processes I find no process 7451 but this is the output.
# ps aux | grep mysql
root 1679 0.0 0.0 8644 4688 ? Ssl 22:23 0:00 dockerize -template /tmp/db.cnf.tmpl:/etc/mysql/conf.d/db.cnf /usr/local/bin/docker-entrypoint.sh mysqld
systemd+ 2526 0.3 0.1 2446780 99836 ? Sl 22:23 0:03 mysqld
root 7370 0.3 0.0 27940 19836 pts/0 S+ 22:38 0:00 /usr/bin/perl -w /usr/share/debconf/frontend /var/lib/dpkg/info/mysql-server-8.0.postinst configure 8.0.22-0ubuntu0.20.04.2
root 7379 0.0 0.0 9756 3772 pts/0 S+ 22:38 0:00 /bin/bash /var/lib/dpkg/info/mysql-server-8.0.postinst configure 8.0.22-0ubuntu0.20.04.2
mysql 7657 1.9 0.5 1869164 358336 pts/0 Sl+ 22:39 0:01 mysqld --user=mysql --init-file=/var/lib/mysql-files/tmp.gQukBE7GtT --socket=/tmp/tmp.KUFQmDDseF/mysqld.sock --pid-file=/tmp/tmp.KUFQmDDseF/mysqld.pid
root 7759 0.0 0.0 9032 724 pts/1 S+ 22:39 0:00 grep --color=auto mysql
I checked the status of mysql.service
# systemctl status mysql.service
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Wed 2021-01-13 22:38:59 CET; 17min ago
Process: 649 ExecStart=/usr/sbin/mysqld (code=exited, status=0/SUCCESS)
Main PID: 649 (code=exited, status=0/SUCCESS)
Status: "Server shutdown complete"
Jan 13 22:21:43 arpradio systemd[1]: Starting MySQL Community Server...
Jan 13 22:21:45 arpradio systemd[1]: Started MySQL Community Server.
Jan 13 22:38:57 arpradio systemd[1]: Stopping MySQL Community Server...
Jan 13 22:38:59 arpradio systemd[1]: mysql.service: Succeeded.
Jan 13 22:38:59 arpradio systemd[1]: Stopped MySQL Community Server.

I fixed it, what I did was while the upgrade was hanging I ran
apt update && apt upgrade
on another tty and it upgraded everything, rebooted, I checked again after reboot and everything looks good.
# apt update
Hit:1 http://asi-fs-n.contabo.net/ubuntu focal InRelease
Hit:2 http://asi-fs-n.contabo.net/ubuntu focal-updates InRelease
Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease
Hit:4 https://download.docker.com/linux/ubuntu focal InRelease
Hit:5 http://asi-fs-n.contabo.net/ubuntu focal-backports InRelease
Get:6 http://security.ubuntu.com/ubuntu focal-security InRelease [109 kB]
Fetched 109 kB in 1s (112 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
root#arpradio:~# apt upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
root#arpradio:~# ps aux | grep mysql
mysql 657 1.2 0.6 2067208 378684 ? Ssl 23:09 0:02 /usr/sbin/mysqld
root 2148 0.1 0.0 9796 4784 ? Ssl 23:11 0:00 dockerize -template /tmp/db.cnf.tmpl:/etc/mysql/conf.d/db.cnf /usr/local/bin/docker-entrypoint.sh mysqld
systemd+ 2930 0.8 0.1 2380944 99508 ? Sl 23:11 0:00 mysqld
root 5507 0.0 0.0 9032 732 pts/0 S+ 23:12 0:00 grep --color=auto mysql
root#arpradio:~# systemctl status mysql.service
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2021-01-13 23:09:06 CET; 3min 26s ago
Process: 560 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
Main PID: 657 (mysqld)
Status: "Server is operational"
Tasks: 38 (limit: 72284)
Memory: 419.9M
CGroup: /system.slice/mysql.service
└─657 /usr/sbin/mysqld
Jan 13 23:09:04 arpradio systemd[1]: Starting MySQL Community Server...
Jan 13 23:09:06 arpradio systemd[1]: Started MySQL Community Server.

Related

gunicorn does not start after reboot

The advice given in gunicorn does not start after boot
does not solve my similar problem.
I'm using ubuntu server 20.04.1
my unit file: /etc/systemd/system/gunicorn.service
[Unit]
Description=Gunicorn Web Server as Unit Service Systemd - slimzulu.hopto.me
After=network.target
[Service]
User=nols
Group=nols
WorkingDirectory=/home/nols/fastapi
Environment="PATH=/home/nols/fastapi/env/bin"
ExecStart=/home/nols/fastapi/env/bin/gunicorn --config /home/nols/fastapi/gunicorn.py main:app
Restart=always
[Install]
WantedBy=multi-user.target
To activate the gunicorn.service
sudo systemctl daemon-reload
sudo systemctl enable gunicorn.service
sudo systemctl start gunicorn.service
The status of gunicorn.service after boot (sudo systemctl status gunicorn.service
)
● gunicorn.service - Gunicorn Web Server as Unit Service Systemd - slimzulu.hopto.me
Loaded: loaded (/etc/systemd/system/gunicorn.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2021-10-20 14:13:26 UTC; 3min 24s ago
Process: 681 ExecStart=/home/nols/fastapi/env/bin/gunicorn --config /home/nols/fastapi/gunicorn.py main:app (code=exited, status=3)
Main PID: 681 (code=exited, status=3)
Oct 20 14:12:52 ubuntuvirtual systemd[1]: Started Gunicorn Web Server as Unit Service Systemd - slimzulu.hopto.me.
Oct 20 14:13:26 ubuntuvirtual systemd[1]: gunicorn.service: Main process exited, code=exited, status=3/NOTIMPLEMENTED
Oct 20 14:13:26 ubuntuvirtual systemd[1]: gunicorn.service: Failed with result 'exit-code'.
~
Any ideas to figure out why the gunicorn service isn't starting after reboot?

ERROR: got error when setting up GCP MySQL server

Here is my steps that can replicate:
Compute Engine Creation:
1.select F1-Micro instance.
2.Select Ubuntu 20.04 LTS Minimal.
Install:
$sudo apt-get update
$sudo apt-get install mysql-server
My question is how can I get no error when setting up MySQL server in GCP?
Here is the error:
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: activating (start-pre) since Fri 2021-01-29 20:58:52 UTC; 3s ago
Cntrl PID: 2338 (mysql-systemd-s)
Tasks: 1 (limit: 682)
Memory: 1.7M
CGroup: /system.slice/mysql.service
├─2338 /bin/bash /usr/share/mysql/mysql-systemd-start pre
├─2356 /bin/bash /usr/share/mysql/mysql-systemd-start pre
├─2357 /bin/bash /usr/share/mysql/mysql-systemd-start pre
├─2358 [my_print_defaul]
├─2359 sed -n s/^--datadir=//p
└─2360 tail -n 1
Jan 29 20:58:49 instance-1 systemd[1]: mysql.service: Killing process 2233 (mysqld) with signal SIGKILL.
Jan 29 20:58:51 instance-1 systemd[1]: mysql.service: Main process exited, code=killed, status=9/KILL
Jan 29 20:58:51 instance-1 systemd[1]: mysql.service: Failed with result 'timeout'.
Jan 29 20:58:51 instance-1 systemd[1]: Failed to start MySQL Community Server.
Jan 29 20:58:52 instance-1 systemd[1]: mysql.service: Scheduled restart job, restart counter is at 1.
Jan 29 20:58:52 instance-1 systemd[1]: Stopped MySQL Community Server.
Jan 29 20:58:52 instance-1 systemd[1]: Starting MySQL Community Server...
dpkg: error processing package mysql-server-8.0 (--configure):
installed mysql-server-8.0 package post-installation script subprocess returned error exit status 1
Setting up libcgi-pm-perl (4.46-1) ...
Setting up libhtml-template-perl (2.97-1) ...
dpkg: dependency problems prevent configuration of mysql-server:
mysql-server depends on mysql-server-8.0; however:
Package mysql-server-8.0 is not configured yet.
dpkg: error processing package mysql-server (--configure):
dependency problems - leaving unconfigured
Setting up libcgi-fast-perl (1:2.15-1) ...
No apport report written because the error message indicates its a followup error from a previous failure.
Processin
g triggers for systemd (245.4-4ubuntu3.4) ...
Processing triggers for libc-bin (2.31-0ubuntu9.1) ...
Errors were encountered while processing:
mysql-server-8.0
mysql-server```

Can't start mysql server after updating Ubuntu 18.4

first i have tried to uninstall both of them
root#thaali:/etc/mysql# sudo apt-get remove --purge mysql*
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package mysql.cnf
E: Couldn't find any package by glob 'mysql.cnf'
E: Couldn't find any package by regex 'mysql.cnf'
E: Unable to locate package mysql.conf.d
E: Couldn't find any package by glob 'mysql.conf.d'
E: Couldn't find any package by regex 'mysql.conf.d'
Then i have tried to install them
root#thaali:/etc/mysql# sudo apt-get install mysql-server mysql-client
Reading package lists... Done
Building dependency tree... 50%
Building dependency tree
Reading state information... Done
mysql-client is already the newest version (5.7.26-0ubuntu0.19.04.1).
mysql-server is already the newest version (5.7.26-0ubuntu0.19.04.1).
0 upgraded, 0 newly installed, 0 to remove and 13 not upgraded.
This is what happen when i tried to restart
root#thaali:/etc/mysql# sudo service mysql start
Job for mysql.service failed because the control process exited with error code.
See "systemctl status mysql.service" and "journalctl -xe" for details.
This is what happened when i try to systemctl
root#thaali:/etc/mysql# systemctl status mysql.service
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; bad; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2019-05-29 22:07:43 +0530; 6min ago
Process: 21283 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=1/FAILURE)
මැයි 29 22:07:43 thaali systemd[1]: mysql.service: Control process exited, code=exited, status=1/FAILURE
මැයි 29 22:07:43 thaali systemd[1]: mysql.service: Failed with result 'exit-code'.
මැයි 29 22:07:43 thaali systemd[1]: Failed to start MySQL Community Server.
මැයි 29 22:07:43 thaali systemd[1]: mysql.service: Service RestartSec=100ms expired, scheduling restart.
මැයි 29 22:07:43 thaali systemd[1]: mysql.service: Scheduled restart job, restart counter is at 5.
මැයි 29 22:07:43 thaali systemd[1]: Stopped MySQL Community Server.
මැයි 29 22:07:43 thaali systemd[1]: mysql.service: Start request repeated too quickly.
මැයි 29 22:07:43 thaali systemd[1]: mysql.service: Failed with result 'exit-code'.
මැයි 29 22:07:43 thaali systemd[1]: Failed to start MySQL Community Server.
This is the error that's throw me when i tried to connect
root#thaali:/etc/mysql# mysql -u root -p
Enter password:
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
Can any one please help me to solve this and i have seen in some places i have edit etc/my.cnf to fix this in my case i don't have such a file
This one is really weird for I was not dist upgrading. One day it just didn't work.
When I tried journalctl -xe I got this
...
Oct 05 08:51:26 <servername> mysql-systemd-start[10914]: ERROR: Unable to start MySQL server:
Oct 05 08:51:26 <servername> mysql-systemd-start[10914]: mysqld: Error on realpath() on '/var/lib/mysql-files' (Error 2 - No such file or directory)
...
I fixed it by doing this:
sudo mkdir /var/lib/mysql-files
sudo chown -R mysql:mysql /var/lib/mysql-files/
sudo chmod 700 /var/lib/mysql-files/
sudo service mysql restart
Restarting the service didn't actually fix all problems I was having so I ended with rebooting the machine.

MariaDB server is down but I can still login with mysql client (?)

MariaDB 10.3.8, Ubuntu 18 LTS.
When I check for the database status, I see that MariaDB is down. But I can login using the mysql client command. Is something wrong? How could I debug this?
sudo service mariadb status
mariadb.service - MariaDB 10.3.8 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; disabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf
Active: failed (Result: timeout) since Wed 2019-02-13 08:43:11 IST; 1min 0s ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 3293 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
Process: 3712 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
Process: 3706 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 3693 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Main PID: 3878 (mysqld)
Tasks: 13 (limit: 4915)
CGroup: /system.slice/mariadb.service
└─3878 /usr/sbin/mysqld
# lsof -i:3306
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
mysqld 2346 user_running_mariadb 31u IPv6 28910 0t0 TCP *:mysql (LISTEN)
ps uax | grep mysql
root 2049 0.0 0.0 4628 1812 ? S 08:40 0:00 /bin/sh /usr/bin/mysqld_safe
user_running_mariadb 2346 0.1 0.4 3005364 114444 ? Sl 08:40 0:02 /usr/sbin/mysqld --basedir=/usr --datadir=/my_db_files_dir --plugin-dir=/usr/lib/mysql/plugin --user=user_running_mariadb --skip-log-error --pid-file=/my_db_rundir/run/mysqld.pid --socket=/my_db_rundir/run/mysqld.sock --port=3306
#/my_db_rundir/run# ls -l
-rw-rw---- 1 user_running_mariadb group_running_mariadb 5 Feb 13 08:40 mysqld.pid
srwxrwxrwx 1 user_running_mariadb group_running_mariadb 0 Feb 13 08:40 mysqld.sock
#/my_db_rundir/run# cat mysqld.pid
2346
I can login:
mysql -u root -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 145
Server version: 10.3.8-MariaDB-1:10.3.8+maria~bionic-log mariadb.org
inary distribution
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input
tatement.
MariaDB [(none)]>
Additional outputs:
Error log:
2019-02-13 8:52:13 0 [ERROR] InnoDB: Unable to lock ./ibdata1 error: 11
2019-02-13 8:52:13 0 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2019-02-13 8:52:13 0 [Note] InnoDB: Unable to open the first data file
2019-02-13 8:52:13 0 [ERROR] InnoDB: Operating system error number 11 in a file operation.
2019-02-13 8:52:13 0 [ERROR] InnoDB: Error number 11 means 'Resource temporarily unavailable'
journalctl -u mariadb.service
Feb 13 08:51:31 my_host systemd[1]: mariadb.service: Start operation timed out. Terminating.
Feb 13 08:52:13 my_host systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Feb 13 08:52:13 my_host systemd[1]: mariadb.service: Failed with result 'timeout'.
Feb 13 08:52:13 my_host systemd[1]: Failed to start MariaDB 10.3.8 database server.
The service is timed out ('Result: timeout'), and the service has the systemd configuration SendSIGKILL=no, which means systemd has just let mysqld continue running.
Why it time out? take a look at journalctl -u mariadb.service.
I could fix the issue, however I'm not completely sure about the root cause and what really fixed the issue. Still, perhaps this information helps others facing the same issue.
Reinstalled MariaDB and upgraded to 10.3.12. This wasn't probably needed because I kept hitting the same issue after the upgrade.
After reboot, noticed that msyqld status showed as active while mariadb as inactive:
/ati/mariadb/logs# sudo service mysql status
Loaded: loaded (/etc/init.d/mysql; generated)
Active: active (running)
Docs: man:systemd-sysv-generator(8)
Process: 1650 ExecStart=/etc/init.d/mysql start (code=exited, status=0/SUCCESS)
Tasks: 39 (limit: 4915)
CGroup: /system.slice/mysql.service
─2003 /bin/sh /usr/bin/mysqld_safe
─2301 /usr/sbin/mysqld --basedir=/usr...
└─2302 logger -t mysqld -p daemon error
sudo service mariadb status
mariadb.service - MariaDB 10.3.12 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; disabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf
Active: inactive (dead)
In MariaDB web site, found the way of enabling mariadb service, with
systemctl enable mariadb
Now, it works again.

Status: "MariaDB server is down"

Hope someone can help me:
I have many sites running on my server and I am using mariadb.
when I do:
sudo service mysql restart
I get:
Job for mariadb.service failed because the control process exited with
error code. See "systemctl status mariadb.service" and "journalctl -xe"
for details.
systemctl status mariadb.service
mariadb.service - MariaDB database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf
Active: failed (Result: exit-code) since Thu 2017-09-14 03:16:51 UTC; 1min 6s ago
Process: 13247 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=
Process: 13086 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`/usr/bin/gale
Process: 13070 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, s
Process: 13033 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, st
Main PID: 13247 (code=exited, status=1/FAILURE)
Status: "MariaDB server is down"
CGroup: /system.slice/mariadb.service
└─3803 /usr/sbin/mysqld
sudo journalctl -xe
Sep 14 03:35:51 ubuntu systemd[1]: Failed to start MariaDB database server.
-- Subject: Unit mariadb.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mariadb.service has failed.
--
-- The result is failed.
Sep 14 03:35:51 ubuntu systemd[1]: mariadb.service: Unit entered failed state.
Sep 14 03:35:51 ubuntu sudo[13566]: pam_unix(sudo:session): session closed for user root
Sep 14 03:35:51 ubuntu systemd[1]: mariadb.service: Failed with result 'exit-code'.
Sep 14 03:35:52 ubuntu sshd[13753]: Failed password for root from 123.183.209.136 port 38816 ssh2
Sep 14 03:35:52 ubuntu sshd[13753]: Received disconnect from 123.183.209.136 port 38816:11: [preauth]
Sep 14 03:35:52 ubuntu sshd[13753]: Disconnected from 123.183.209.136 port 38816 [preauth]
Sep 14 03:35:52 ubuntu sshd[13753]: PAM 1 more authentication failure; logname= uid=0 euid=0 tty=ssh ruser
Sep 14 03:36:34 ubuntu sshd[13756]: Connection closed by 123.183.209.136 port 60477 [preauth]
Sep 14 03:36:36 ubuntu postfix/pickup[12161]: 703AA5C0C5: uid=0 from=<root>
Sep 14 03:36:36 ubuntu postfix/cleanup[12162]: 703AA5C0C5: message-id=<20170914033636.703AA5C0C5#ubuntu.me
Sep 14 03:36:36 ubuntu postfix/cleanup[12162]: warning: 703AA5C0C5: write queue file: No space left on dev
Sep 14 03:36:36 ubuntu postfix/pickup[12161]: warning: maildrop/5152F5C0C4: error writing 703AA5C0C5: queu
Sep 14 03:37:17 ubuntu sshd[13760]: Received disconnect from 123.183.209.136 port 25567:11: [preauth]
Sep 14 03:37:17 ubuntu sshd[13760]: Disconnected from 123.183.209.136 port 25567 [preauth]
Sep 14 03:37:36 ubuntu postfix/pickup[12161]: 7764B5C0C5: uid=0 from=<root>
Sep 14 03:37:36 ubuntu postfix/cleanup[12162]: 7764B5C0C5: message-id=<20170914033736.7764B5C0C5#ubuntu.me
Sep 14 03:37:36 ubuntu postfix/cleanup[12162]: warning: 7764B5C0C5: write queue file: No space left on dev
Sep 14 03:37:36 ubuntu postfix/pickup[12161]: warning: maildrop/5152F5C0C4: error writing 7764B5C0C5: queu
Sep 14 03:37:47 ubuntu sudo[13765]: bob : TTY=pts/0 ; PWD=/home/bob ; USER=root ; COMMAND=/bin/journa
Sep 14 03:37:47 ubuntu sudo[13765]: pam_unix(sudo:session): session opened for user root by bob(uid=0)
lines 1238-1265/1265 (END)
I really don't want to lose my db.
What should I do next?
Many thanks
this is what df gives me:
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 49362256 46854648 0 100% /
devtmpfs 2019828 0 2019828 0% /dev
tmpfs 2021768 0 2021768 0% /dev/shm
tmpfs 2021768 17968 2003800 1% /run
tmpfs 5120 0 5120 0% /run/lock
tmpfs 2021768 0 2021768 0% /sys/fs/cgroup
tmpfs 404356 0 404356 0% /run/user/1000
tmpfs 404356 0 404356 0% /run/user/0
See the first filesystem (/dev/root) is 100% full! And there are no other volumes mounted besides tmpfs volumes. So mariadb is trying to write something to disk as it starts up, and can't, so it gives up and shuts itself down.
You have to figure out what's filling up your disk, and prune what you can. For example, as a guess I'd start by looking in the http server logs, since you say you host many sites on this server. I've seen many servers that filled up disk space by letting httpd log everything for months and months.
In the future, you might want to provision this server with more than one disk volume, so if one volume fills up it doesn't interfere with other volumes.
Also consider automating log rotation.
If the culprit for disk space usage is not logs, but is actually the mariadb data directory, then you might need to get an additional disk volume immediately, and move the mariadb content to it before you can bring your sites back online.
Unless you can find a little bit of other stuff elsewhere on the server that you can delete and make enough space for a short while.