I have a backup script that uses mysqldump to dump a Mediawiki database, then archives it with gzip. It seems to be working okay, but I am curious why the size of the archives appear to grow and shrink at random. It's not a very active site, so large amounts of data aren't being added or deleted on the daily.
-rw-r--r-- 1 root root 91M Mar 27 11:46 wiki_data_20220325.sql.gz
-rw-r--r-- 1 root root 93M Mar 27 11:46 wiki_data_20220326.sql.gz
-rw-r--r-- 1 root root 92M Mar 27 11:56 wiki_data_20220327.sql.gz
-rw-r--r-- 1 root root 110M Mar 28 03:15 wiki_data_20220328.sql.gz
-rw-r--r-- 1 root root 99M Mar 29 03:15 wiki_data_20220329.sql.gz
-rw-r--r-- 1 root root 103M Mar 30 03:15 wiki_data_20220330.sql.gz
-rw-r--r-- 1 root root 107M Mar 31 03:15 wiki_data_20220331.sql.gz
-rw-r--r-- 1 root root 78M Mar 27 11:47 wiki_html_20220320.tar.gz
-rw-r--r-- 1 root root 173M Mar 27 11:47 wiki_xml_20220321.xml
-rw-r--r-- 1 root root 173M Mar 27 11:47 wiki_xml_20220322.xml
-rw-r--r-- 1 root root 173M Mar 27 11:47 wiki_xml_20220323.xml
-rw-r--r-- 1 root root 173M Mar 27 11:47 wiki_xml_20220324.xml
The size difference persists after extracting the archives.
-rw-rw-r-- 1 user user 280M Mar 31 10:27 wiki0328.sql
-rw-r--r-- 1 user user 110M Mar 31 10:26 wiki0328.sql.gz
-rw-rw-r-- 1 user user 267M Mar 31 10:27 wiki0329.sql
-rw-r--r-- 1 user user 99M Mar 31 10:26 wiki0329.sql.gz
It's not necessarily a problem, but I am curious. Is this common / normal behavior for databases dumped from complex software like Mediawiki?
Here's the relevant chunk of the backup script, in case it matters...
echo "## Set ReadOnly on"
echo "\$wgReadOnly = 'Dumping Database, Access will be restored shortly';" >> $localSet
echo "## Dumping XML..."
php $dumpXML --full --quiet > $saveLoc/"wiki_xml_"$(date +%Y%m%d)".xml"
echo "## Dumping database..."
mysqldump my_wiki | gzip -f > $saveLoc/"wiki_data_"$(date +%Y%m%d)".sql.gz"
echo "## Set ReadOnly off"
tail -n 1 "$localSet" | wc -c | xargs -I {} truncate "$localSet" -s -{}
Thanks in advance for any info!
Summary of the comments above: the objectcache table in a Wordpress database varies in size, and this is normal. Therefore it will cause the database backup to vary in size. To minimize the size of the backup, some people omit the objectcache table from backups.
Related
I am using a raspberry pi 3 with OSMC as the operating system along with Debian Stretch and nginx, and installed manually mariaDB 10.2 following some instructions I found somewhere a while back.
I have changed the datadir for mariadb to /media/USBHDD2/shared/mysql
When I boot, or reboot, the pi, mariaDB fails to start. Before, when I had the default datadir = /var/lib/mysql it was all fine. If I change it back it is fine.
However, if I login to the console I can successfully start it by using
service mysql start
Note that I am using 'service' rather than 'systemctl' - the latter does not work. The files mariadb.service and mysql.service do not exist anywhere.
In /etc/init.d I find two files: mysql and myswql which seem to be identical. If I remove the myswql from the directory mariadb won't start at all. I have tried editing these by putting, for example, a sleep 15 at the beginning, but to no avail. I have read all sorts of solutions about trying to test if the USBHDD2 is mounted, eg using
while ! test -f /media/USBHDD2/shared/test.txt
do
sleep 1
done
which I tried in the /etc/init.d/mysql and myswql files, and also in rc.local before calling for the start of mysql.
But that doesn't work either.
I also renamed the links in rc?.d to S99mysql so is starts after everything else, still no joy.
I have spent two full days on this to no avail. What do I need to do to get this working so that mysql starts on boot?
Files system is ntfs
output from ls -la //media/USBHDD2/shared/mysql is as follows:
total 176481
drwxrwxrwx 1 root root 4096 Mar 27 11:41 .
drwxrwxrwx 1 root root 4096 Mar 27 13:06 ..
-rwxrwxrwx 1 root root 16384 Mar 27 11:41 aria_log.00000001
-rwxrwxrwx 1 root root 52 Mar 27 11:41 aria_log_control
-rwxrwxrwx 1 root root 0 Nov 3 2016 debian-10.1.flag
-rwxrwxrwx 1 root root 12697 Mar 27 11:41 ib_buffer_pool
-rwxrwxrwx 1 root root 50331648 Mar 27 11:41 ib_logfile0
-rwxrwxrwx 1 root root 50331648 Mar 26 22:02 ib_logfile1
-rwxrwxrwx 1 root root 79691776 Mar 27 11:41 ibdata1
drwxrwxrwx 1 root root 32768 Mar 25 18:37 montegov_admin
-rwxrwxrwx 1 root root 0 Nov 3 2016 multi-master.info
drwxrwxrwx 1 root root 20480 Sep 3 2019 mysql
drwxrwxrwx 1 root root 0 Sep 3 2019 performance_schema
drwxrwxrwx 1 root root 86016 Mar 25 20:06 rentmaxpro_wp187
drwxrwxrwx 1 root root 0 Sep 3 2019 test
drwxrwxrwx 1 root root 32768 Nov 3 2016 trustedhomerenta_admin
drwxrwxrwx 1 root root 32768 Nov 3 2016 trustedhomerenta_demo
drwxrwxrwx 1 root root 40960 Mar 25 21:05 trustedhomerenta_meta
drwxrwxrwx 1 root root 36864 Mar 25 21:25 trustedhomerenta_montego
drwxrwxrwx 1 root root 36864 Mar 26 20:37 trustedhomerenta_testmontego
The problem is that the external drive is configured as ntfs.
Mysql requires the files and directory to be owned by mysql:mysql but since the ntfs does not have the same system of owners and groups as linux, the linux mount process asigns its own owner and group to the filestructure when mounting the drive. By defualt this ends up being root:root so mysql cannot use them.
ntfs does not allow CHOWN to work, so there is no way to change the ownership away from root.
One solution is to backup all the files, repartition as EXT4, and then restore all the files.
The solution I finally used was to specify mysql as the owner and group at the time that the drive is being mounted. Thus my /etc/fstab file was changed to:
ID=C2CA68D9CA68CB6D /media/USBHDD2 ntfs users,exec,uid=mysql,gid=mysql 0 2
and now mysql starts properly at boot.
phew ;-)
Thanks #danblack for getting me thinking in the right direction
I'm working in company where by they are using kvm virtualisation
[root#601 log]# virsh list --all --title
Id Name State Title
----------------------------------------------------------------------------------
2 reporting-pilosa07 running 10.3.6.172
3 reporting-pilosa09 running 10.3.6.173
4 reporting-pilosa11 running 10.3.6.174
5 reporting-pilosa13 running 10.3.6.175
6 reporting-pilosa05 running 10.3.6.171
the VMs are running, but time to time, they dead for some reason and I would like to look at individual VM logs
[root#601 qemu]# ls -ltr
total 32
-rw------- 1 root root 2341 Oct 15 2018 reporting-pilosa07.log
-rw------- 1 root root 2341 Oct 15 2018 reporting-pilosa09.log
-rw------- 1 root root 2341 Oct 15 2018 reporting-pilosa11.log
-rw------- 1 root root 2341 Oct 15 2018 reporting-pilosa13.log
-rw------- 1 root root 4885 Nov 12 2018 reporting-pilosa05.log
-rw------- 1 root root 7181 Jul 25 04:14 offlineonboarder02.log
[root#601 qemu]# pwd
/var/log/libvirt/qemu
The logs are not logging since a year back. Where can I enable back the logs so that I can observe the cause to why the VMs went dead?
Thanks.
I wish to change the default character sets and collations to utf8mb4 instead of latin1_swedish_ci, and quickly found the answer at Change MySQL default character set to UTF-8 in my.cnf?.
Okay, which my.cnf should be used? MySQL comes with /etc/my.cnf with default settings, however, this does not seem to the case with MariaDB.
Reading the manual
Option File Locations on Linux, Unix, Mac
On a Linux, Unix or Mac server, MariaDB looks for the MariaDB option file file in the following locations:
...
MariaDB starting with 10.0.13
....
When SYSCONFDIR is defined (for example, in RPM packages it is /etc)
Location Scope
SYSCONFDIR/my.cnf Global
$MYSQL_HOME/my.cnf Server
defaults-extra-file File specified with --defaults-extra-file=path, if any
~/.my.cnf User
MYSQL_HOME is the environment variable containing the path to the directory holding the server-specific my.cnf file. If MYSQL_HOME is not set, and the server is started with mysqld_safe, MYSQL_HOME is set as follows:
If there is a my.cnf file in the MariaDB data directory, but not in the MariaDB base directory, MYSQL_HOME is set to the MariaDB data directory.
Else, MYSQL_HOME is set to the MariaDB base directory.
Okay, it is either SYSCONFDIR/my.cnf or $MYSQL_HOME/my.cnf, right? But where are these?
[michael#devserver doctrine]$ ls -l /etc/my*
-rw-r--r-- 1 root root 202 Dec 30 23:29 /etc/my.cnf
/etc/my.cnf.d:
total 12
-rw-r--r-- 1 root root 763 Dec 30 23:29 enable_encryption.preset
-rw-r--r-- 1 root root 232 Dec 30 23:29 mysql-clients.cnf
-rw-r--r-- 1 root root 1080 Dec 30 23:29 server.cnf
[michael#devserver doctrine]$ cat /etc/my.cnf
#
# This group is read both both by the client and the server
# use it for options that affect everything
#
[client-server]
#
# include all files from the config directory
#
!includedir /etc/my.cnf.d
[michael#devserver doctrine]$ cat /etc/my.cnf.d/*
#
# !include this file into your my.cnf (or any of *.cnf files in /etc/my.cnf.d)
# and it will enable data at rest encryption. This is a simple way to
# ensure that everything that can be encrypted will be and your
# data will not leak unencrypted.
#
# DO NOT EDIT THIS FILE! On MariaDB upgrades it might be replaced with a
# newer version and your edits will be lost. Instead, add your edits
# to the .cnf file after the !include directive.
#
# NOTE that you also need to install an encryption plugin for the encryption
# to work. See https://mariadb.com/kb/en/mariadb/data-at-rest-encryption/#encryption-key-management
#
[mariadb]
aria-encrypt-tables
encrypt-binlog
encrypt-tmp-disk-tables
encrypt-tmp-files
loose-innodb-encrypt-log
loose-innodb-encrypt-tables
#
# These groups are read by MariaDB command-line tools
# Use it for options that affect only one utility
#
[mysql]
[mysql_upgrade]
[mysqladmin]
[mysqlbinlog]
[mysqlcheck]
[mysqldump]
[mysqlimport]
[mysqlshow]
[mysqlslap]
#
# These groups are read by MariaDB server.
# Use it for options that only the server (but not clients) should see
#
# See the examples of server my.cnf files in /usr/share/mysql/
#
# this is read by the standalone daemon and embedded servers
[server]
# this is only for the mysqld standalone daemon
[mysqld]
#
# * Galera-related settings
#
[galera]
# Mandatory settings
#wsrep_on=ON
#wsrep_provider=
#wsrep_cluster_address=
#binlog_format=row
#default_storage_engine=InnoDB
#innodb_autoinc_lock_mode=2
#
# Allow server to accept connections on all interfaces.
#
#bind-address=0.0.0.0
#
# Optional setting
#wsrep_slave_threads=1
#innodb_flush_log_at_trx_commit=0
# this is only for embedded server
[embedded]
# This group is only read by MariaDB servers, not by MySQL.
# If you use the same .cnf file for MySQL and MariaDB,
# you can put MariaDB-only options here
[mariadb]
# This group is only read by MariaDB-10.2 servers.
# If you use the same .cnf file for MariaDB of different versions,
# use this group for options that older servers don't understand
[mariadb-10.2]
[michael#devserver doctrine]$ ls -l /usr/share/mysql/
total 1272
-rwxr-xr-x 1 root root 1153 Dec 30 23:34 binary-configure
drwxr-xr-x 2 root root 4096 Jan 26 00:53 charsets
drwxr-xr-x 2 root root 23 Jan 26 00:55 czech
drwxr-xr-x 2 root root 23 Jan 26 00:55 danish
drwxr-xr-x 2 root root 23 Jan 26 00:55 dutch
drwxr-xr-x 2 root root 23 Jan 26 00:55 english
-rw-r--r-- 1 root root 541061 Dec 30 23:29 errmsg-utf8.txt
drwxr-xr-x 2 root root 23 Jan 26 00:55 estonian
-rw-r--r-- 1 root root 583827 Dec 30 23:29 fill_help_tables.sql
drwxr-xr-x 2 root root 23 Jan 26 00:55 french
drwxr-xr-x 2 root root 23 Jan 26 00:55 german
drwxr-xr-x 2 root root 23 Jan 26 00:55 greek
drwxr-xr-x 2 root root 23 Jan 26 00:55 hindi
drwxr-xr-x 2 root root 23 Jan 26 00:55 hungarian
-rw-r--r-- 1 root root 15313 Dec 30 23:29 install_spider.sql
drwxr-xr-x 2 root root 23 Jan 26 00:55 italian
drwxr-xr-x 2 root root 23 Jan 26 00:55 japanese
drwxr-xr-x 2 root root 23 Jan 26 00:55 korean
-rw-r--r-- 1 root root 1328 Dec 30 23:29 magic
-rw-r--r-- 1 root root 1627 Dec 30 23:34 maria_add_gis_sp_bootstrap.sql
-rw-r--r-- 1 root root 1649 Dec 30 23:34 maria_add_gis_sp.sql
-rw-r--r-- 1 root root 4881 Dec 30 23:34 my-huge.cnf
-rw-r--r-- 1 root root 20062 Dec 30 23:34 my-innodb-heavy-4G.cnf
-rw-r--r-- 1 root root 4868 Dec 30 23:34 my-large.cnf
-rw-r--r-- 1 root root 4882 Dec 30 23:34 my-medium.cnf
-rw-r--r-- 1 root root 2808 Dec 30 23:34 my-small.cnf
-rwxr-xr-x 1 root root 1061 Dec 30 23:34 mysqld_multi.server
-rwxr-xr-x 1 root root 868 Dec 30 23:34 mysql-log-rotate
-rw-r--r-- 1 root root 1955 Dec 30 23:29 mysql_performance_tables.sql
-rwxr-xr-x 1 root root 12113 Dec 30 23:34 mysql.server
-rw-r--r-- 1 root root 4517 Dec 30 23:29 mysql_system_tables_data.sql
-rw-r--r-- 1 root root 27622 Dec 30 23:29 mysql_system_tables.sql
-rw-r--r-- 1 root root 10411 Dec 30 23:29 mysql_test_data_timezone.sql
-rw-r--r-- 1 root root 1210 Dec 30 23:29 mysql_to_mariadb.sql
drwxr-xr-x 2 root root 23 Jan 26 00:55 norwegian
drwxr-xr-x 2 root root 23 Jan 26 00:55 norwegian-ny
drwxr-xr-x 4 root root 35 Dec 31 00:28 policy
drwxr-xr-x 2 root root 23 Jan 26 00:55 polish
drwxr-xr-x 2 root root 23 Jan 26 00:55 portuguese
drwxr-xr-x 2 root root 23 Jan 26 00:55 romanian
drwxr-xr-x 2 root root 23 Jan 26 00:55 russian
drwxr-xr-x 2 root root 23 Jan 26 00:55 serbian
drwxr-xr-x 2 root root 23 Jan 26 00:55 slovak
drwxr-xr-x 2 root root 23 Jan 26 00:55 spanish
drwxr-xr-x 2 root root 23 Jan 26 00:55 swedish
drwxr-xr-x 2 root root 62 Jan 26 00:55 systemd
drwxr-xr-x 2 root root 23 Jan 26 00:55 ukrainian
-rw-r--r-- 1 root root 3479 Dec 30 23:34 wsrep.cnf
-rwxr-xr-x 1 root root 2230 Dec 30 23:34 wsrep_notify
[michael#devserver doctrine]
I'm using OSX and running docker over a Boot2docker VM.
I've been trying to figure out how to persist a container's data (MySQL official docker image) to the host but without much success.
I keep receiving an error stating that the /var/lib/mysql directory that the MySQL service is trying to write to is not accessible.
docker run -e MYSQL_ROOT_PASSWORD=12345 -v "$(pwd)/.docker-volumes/mysql:/var/lib/mysql" mysql:5.6
Looking at the permissions of the mounted library in the container, this is what I see:
root#mysql:/# ls -la /var/lib/
total 44
drwxr-xr-x 16 root root 4096 Jan 27 18:35 .
drwxr-xr-x 18 root root 4096 Jan 27 18:35 ..
drwxr-xr-x 7 root root 4096 Jan 27 18:35 apt
drwxr-xr-x 14 root root 4096 Jan 27 18:35 dpkg
drwxr-xr-x 2 root root 4096 Jul 14 2013 initscripts
drwxr-xr-x 2 root root 4096 Jul 14 2013 insserv
drwxrwsr-x 2 libuuid libuuid 4096 Dec 11 2012 libuuid
drwxr-xr-x 2 root root 4096 Dec 24 13:41 misc
drwxr-xr-x 1 1000 staff 102 Feb 4 15:10 mysql
drwxr-xr-x 2 root root 4096 Jan 27 16:48 pam
drwxr-xr-x 2 root root 4096 Nov 23 2012 update-rc.d
drwxr-xr-x 2 root root 4096 Jul 14 2013 urandom
As you can see, the mysql directory is owned by 1000 and belongs to the group 'staff'.
My assumption is that the service process running MySQL is probably set to another user (mysql) and therefore I get this error.
I've read that this specific issue can be solved using volume data containers, but since they persist the data only until the last container actually uses their volume, It's not a good solution for me.
Am I approaching this in the wrong way?
Thanks.
You're definitely better off using a data-volume container, I do the same thing with local psql and couchdb databases. The data actually persists, it's just not accessible unless you link the volume to a container. To actually force the volume to be removed you have to specify docker rm -v, that will remove the data volume if no other container is linked to it.
I have installed the following packages --
libevent-2.0.21
memcached-1.4.17
libmemcached-0.34
memcached_functions_mysql_1.1
All of the above have been installed successfully.
The output of the ldconfig -v command (the part where it shows that libmemcached libraries have been included) is as follows:
$ldconfig -v
/usr/local/libmemcached/lib:
libmemcached.so.3 -> libmemcached.so.3.0.0
libmemcachedprotocol.so.0 -> libmemcachedprotocol.so.0.0.0
libmemcachedutil.so.0 -> libmemcachedutil.so.0.0.0
But when I try to load UDF's into mysql using the install_functions.sql that is shipped with memcache_functions it throws the following error:
ERROR 1126 (HY000) at line 38: Can't open shared library 'libmemcached_functions_mysql.so' (errno: 0 libmemcached.so.3: cannot open shared object file: No such file or directory)
And the contents of the plugin directory are:
-rw-r--r-- 1 root root 6.1K Jan 21 13:49 adt_null.so
-rw-r--r-- 1 root root 11K Jan 21 13:49 auth.so
-rw-r--r-- 1 root root 6.0K Jan 21 13:49 auth_socket.so
-rw-r--r-- 1 root root 6.2K Jan 21 13:49 auth_test_plugin.so
-rw-r--r-- 1 root root 35K Jan 21 13:49 ha_example.so
-rw-r--r-- 1 root root 10K Jan 21 13:49 libdaemon_example.so
-rw-r--r-- 1 root root 361K Feb 13 02:47 libmemcached_functions_mysql.a
-rwxr-xr-x 1 root root 1.1K Feb 13 02:47 libmemcached_functions_mysql.la
-rwxr-xr-x 1 root root 167K Feb 13 02:47 libmemcached_functions_mysql.so
-rwxr-xr-x 1 root root 167K Feb 13 02:47 libmemcached_functions_mysql.so.0
-rwxr-xr-x 1 root root 167K Feb 13 02:47 libmemcached_functions_mysql.so.0.0.0
-rw-r--r-- 1 root root 11K Jan 21 13:49 mypluglib.so
-rw-r--r-- 1 root root 5.9K Jan 21 13:49 qa_auth_client.so
-rw-r--r-- 1 root root 11K Jan 21 13:49 qa_auth_interface.so
-rw-r--r-- 1 root root 6.0K Jan 21 13:49 qa_auth_server.so
-rw-r--r-- 1 root root 39K Jan 21 13:49 semisync_master.so
-rw-r--r-- 1 root root 15K Jan 21 13:49 semisync_slave.so
Installing memcached on Ubuntu 12.04.1 LTS (64 bit)
Python - 2.6.8
MySQL – 5.5.28
Zope – 2.12.19
1. Install memcached
apt-get install memcached
2. Install libmemcached -0 .34 (Version is very important. May or may not work with other versions).
Download it from https://launchpad.net/libmemcached/
tar xvf libmemcached-0.34.tar.gz
sudo ./configure --prefix=/usr/lib/libmemcached --with-memcached=/usr/bin/memcached
sudo make
sudo make install.
3. Install memcached_functions_mysql (Version 1.1 used at the time of the installation. (To create UDF’s that are invoked by triggers to manipulate the cache).
Downloading from https://launchpad.net/memcached-udfs
sudo ./configure --prefix=/usr/local/memcached_mysql --libdir=/usr/lib/mysql/plugin --with-mysql=/usr/bin/mysql_config --with-libmemcached=/usr/lib/libmemcached
sudo make
sudo make install
Navigate to the “sql” folder inside the memcached_mysql_functions directory.
mysql –u <username> -p < install_functions.sql
4. For Zope users
Install python-memcached-1.53
Download it from https://pypi.python.org/pypi/python-memcached/
Navigate to the extracted python-memcached directory.
/home/zope/zope/bin/python setup.py install
5. Edit the script containing which imports the modules.
allow_module('memcache')
from memcache import Client
allow_class(Client)
allow_module('memcache.Client.get')
allow_module('memcache.Client.set')
This is done so that memcache can be imported and used in your Restricted Python scripts.
If an external method is used to handle the above case, then the file does need to be updated with the above content.