Memory usage issues with VPS (ubuntu): MySQL process dies - mysql

I'm running a VPS, with specs:
Ubuntu 12.04.5 LTS (GNU/Linux 3.13.0-32-generic x86_64)
512mb RAM
1 CPU
20gb SSD
If you're wondering it's a DigitalOcean droplet. It's running TS3, LAMP (with wordpress), OpenVPN, BYOBU, and OwnCloud.
Now my problem is with mySQL dying on me after like 30m to 1hour. Usually after a reboot, the memory usage is 54% and mySQL doesn't have a problem, but as the memory usage goes towards 80-89% I start to get issues.
System load: 0.01 Users logged in: 0
Usage of /: 22.1% of 19.56GB IP address for eth0: *****
Memory usage: 90% IP address for as0t0: *****
Swap usage: 0% IP address for as0t1: *****
Processes: 93
As you can see, the memory usage is VERY high, and I've noticed the trend that mySQL process dies as the memory usage gets higher. However the swap usage is 0%.
Is there a way to make mySQL and the other processes to use the swap?
Would letting mySQL make use of the swap stop letting it die after my memory usage gets so high?
After the high memory usage, the process dies and I get this error:
[2002] SQLSTATE[HY000] [2002] Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
The processor load never goes above 25% in most cases. The server also runs a fast SSD, so it wouldn't be a problem to use a swap, and I don't have that much traffic.

Fixed it, by making a swap file of size 256mb. mySQL doesn't stop now after having no available memory to work in.
After following this tutorial by Etel Sverdlov:
https://www.digitalocean.com/community/tutorials/how-to-add-swap-on-ubuntu-12-04
I was able to make a swap file. I'll copy the tutorial for the sake it gets deleted.
How To Add Swap on Ubuntu 12.04
About Linux Swapping
Linux RAM is composed of chunks of memory called pages. To free up pages of RAM, a “linux swap” can occur and a page of memory is copied from the RAM to preconfigured space on the hard disk. Linux swaps allow a system to harness more memory than was originally physically available.
However, swapping does have disadvantages. Because hard disks have a much slower memory than RAM, virtual private server performance may slow down considerably. Additionally, swap thrashing can begin to take place if the system gets swamped from too many files being swapped in and out.
Check for Swap Space
Before we proceed to set up a swap file, we need to check if any swap files have been enabled on the VPS by looking at the summary of swap usage.
sudo swapon -s
An empty list will confirm that you have no swap files enabled:
Filename Type Size Used Priority
Check the File System
After we know that we do not have a swap file enabled on the virtual server, we can check how much space we have on the server with the df command. The swap file will take 256MB— since we are only using up about 8% of the /dev/sda, we can proceed.
df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda 20907056 1437188 18421292 8% /
udev 121588 4 121584 1% /dev
tmpfs 49752 208 49544 1% /run
none 5120 0 5120 0% /run/lock
none 124372 0 124372 0% /run/shm
Create and Enable the Swap File
Now it’s time to create the swap file itself using the dd command :
sudo dd if=/dev/zero of=/swapfile bs=1024 count=256k
“of=/swapfile” designates the file’s name. In this case the name is swapfile.
Subsequently we are going to prepare the swap file by creating a linux swap area:
sudo mkswap /swapfile
The results display:
Setting up swapspace version 1, size = 262140 KiB
no label, UUID=103c4545-5fc5-47f3-a8b3-dfbdb64fd7eb
Finish up by activating the swap file:
sudo swapon /swapfile
You will then be able to see the new swap file when you view the swap summary.
swapon -s
Filename Type Size Used Priority
/swapfile file 262140 0 -1
This file will last on the virtual private server until the machine reboots. You can ensure that the swap is permanent by adding it to the fstab file.
Open up the file:
sudo nano /etc/fstab
Paste in the following line:
/swapfile none swap sw 0 0
Swappiness in the file should be set to 10. Skipping this step may cause both poor performance, whereas setting it to 10 will cause swap to act as an emergency buffer, preventing out-of-memory crashes.
You can do this with the following commands:
echo 10 | sudo tee /proc/sys/vm/swappiness
echo vm.swappiness = 10 | sudo tee -a /etc/sysctl.conf
To prevent the file from being world-readable, you should set up the correct permissions on the swap file:
sudo chown root:root /swapfile
sudo chmod 0600 /swapfile
All credit to: Etel Sverdlov at: https://www.digitalocean.com/community/tutorials/how-to-add-swap-on-ubuntu-12-04

Related

How to build and boot Linux aarch64 with U-Boot with Buildroot on QEMU?

I tried:
git clone git://git.buildroot.net/buildroot
cd buildroot
git checkout 2019.08
make qemu_aarch64_virt_defconfig
make menuconfig
In menuconfig, I set:
Bootloaders
U-Boot configuration (Using an in-tree board defconfig file)
qemu_arm64
Kernel
Install kernel image to /boot in target
and finally:
make BR2_JLEVEL="$nproc"
Now, I can boot fine without U-Boot with the command line mentioned at: How to download the Torvalds Linux Kernel master, (re)compile it, and boot it with QEMU?
./output/host/usr/bin/qemu-system-aarch64 -M virt -cpu cortex-a57 -nographic -smp 1 -kernel output/images/Image -append "root=/dev/vda console=ttyAMA0" -netdev user,id=eth0 -device virtio-net-device,netdev=eth0 -drive file=output/images/rootfs.ext4,if=none,format=raw,id=hd0 -device virtio-blk-device,drive=hd0
but that is not using U-Boot.
When I do:
ls -l output/images/
it contains:
-rw-r--r-- 1 ciro ciro 6.5M 2019-09-20_13:36:23 Image
-rw-r--r-- 1 ciro ciro 60M 2019-09-20_13:39:02 rootfs.ext2
lrwxrwxrwx 1 ciro ciro 11 2019-09-20_13:36:25 rootfs.ext4 -> rootfs.ext2
-rw-r--r-- 1 ciro ciro 583K 2019-09-20_13:34:15 u-boot.bin
so there is a U-Boot binary there: u-boot.bin, but how do I use that with QEMU?
I tried as mentioned at: Can ARM qemu system emulator boot from card image without kernel param? to remove -kernel and -append and add -bios u-boot.bin:
./output/host/usr/bin/qemu-system-aarch64 -M virt -cpu cortex-a57 -nographic -smp 1 -bios output/images/u-boot.bin -netdev user,id=eth0 -device virtio-net-device,netdev=eth0 -drive file=output/images/rootfs.ext4,if=none,format=raw,id=hd0 -device virtio-blk-device,drive=hd0
Now I do get the U-Boot shell, but boot fails and leaves me no the U-Boot prompt:
U-Boot 2019.07 (Sep 20 2019 - 13:34:10 +0100)
DRAM: 128 MiB
Flash: 128 MiB
*** Warning - bad CRC, using default environment
In: pl011#9000000
Out: pl011#9000000
Err: pl011#9000000
Net:
Warning: virtio-net#31 using MAC address from ROM
eth0: virtio-net#31
Hit any key to stop autoboot: 0
starting USB...
No working controllers found
USB is stopped. Please issue 'usb start' first.
scanning bus for devices...
Device 0: unknown device
Device 0: QEMU VirtIO Block Device
Type: Hard Disk
Capacity: 60.0 MB = 0.0 GB (122880 x 512)
... is now current device
** No partition table - virtio 0 **
starting USB...
No working controllers found
BOOTP broadcast 1
DHCP client bound to address 10.0.2.15 (2 ms)
Using virtio-net#31 device
TFTP from server 10.0.2.2; our IP address is 10.0.2.15
Filename 'boot.scr.uimg'.
Load address: 0x40200000
Loading: *
TFTP error: 'Access violation' (2)
Not retrying...
BOOTP broadcast 1
DHCP client bound to address 10.0.2.15 (0 ms)
Using virtio-net#31 device
TFTP from server 10.0.2.2; our IP address is 10.0.2.15
Filename 'boot.scr.uimg'.
Load address: 0x40400000
Loading: *
TFTP error: 'Access violation' (2)
Not retrying...
=>
so it appears that U-Boot cannot handle the VirtIO device? Or according to Peter, I have to create a partition table. I couldn't find that automatically in Buildroot, but I could do it manually, here is one approach: https://unix.stackexchange.com/questions/209566/how-to-format-a-partition-inside-of-an-img-file/527132#527132
Another approach would be to keep -kernel -append and let QEMU put the kernel into memory as done without U-Boot, and then use the booti U-Boot command I've found on help:
booti - boot Linux kernel 'Image' format from memory
so I just need to find out its address. But that is kind of cheating since I want U-boot to do the hard work rather than cheat with QEMU.
My goal is to reach a good setup to develop U-Boot and QEMU's early boot stuff.
Given that u-boot correctly detects the virtio block device, I think it is unlikely that it cannot handle it. The error printed is "** No partition table - virtio 0 **", which is correct, because you've set up the block device to contain just rootfs.ext4, which will be a filesystem image. That suggests that you'll have more luck if you create a disk image with a partition table and write the rootfs to a partition within the disk image.
I followed Peter's advice and put it into a partitioned image with the sfdisk-fs-to-img command from https://unix.stackexchange.com/questions/209566/how-to-format-a-partition-inside-of-an-img-file/527132#527132
I am now able to read the root filesystem with:
ls virtio 0 /boot
and that contains Image file.
Now I think there are only some U-Boot specifics to resolve, which I'm not very familiar with:
load /boot/Image into memory with something like load virtio 0 0x100000 /boot/Image. TODO which address is valid? This arbitrary choice gave ** Reading file would overwrite reserved memory **
find out how to load the DTB and kernel CLI arguments. The DTB would need to be auto-generated with QEMU with qemu-system-aarch64 -machine dumpdtb=dtb.dtb
boot it with something like: booti 0x100000
I was hoping Buildroot would have automated things a bit more for me sadface.

import of file failed not enough space

I don't have a tech background, but decided to create a Google Compute Engine instance to be able to create a 1.6GB database. I use PostgreSQL (Pgadmin3) to access the database on GCE. It has been working fine so far, but now I am trying to import a new file and am getting the error message
"Panic: could not write to file "pg_xlog/xlogtemp.1293": No space left on the device"
I looked at the documentation in Google for resizing the disk. I did a df -h in SSH and got
/dev/sda1 9.8G 9.2G 16M 100% /
I also did
sudo lsblk --output NAME,size,state
NAME SIZE STATE
sda 20G running
└─sda1 10G
So then I did
sudo resize2fs /dev/disk/by-id/google-myddisk-part1
resize2fs 1.42.12 (29-Aug-2014)
The filesystem is already 2620928 (4k) blocks long. Nothing to do!
Anyone have any ideas?
As this is the root disk you have to reboot the instance. It should then automatically resize the disk during boot. Please check details here: https://cloud.google.com/compute/docs/disks/create-root-persistent-disks#repartitionrootpd

Amazon EC2 MySQL crashing issue because of less memory, how to sort it out?

I am new to Amazon EC2, recently I found mysql is not working properly, continuously crashing.
I think issue is less space or something..
Here is the some of the outputs:
free -k
total used free shared buffers cached
Mem: 1020536 254744 765792 0 54028 83748
-/+ buffers/cache: 116968 903568
Swap: 82628 0 82628
swapon -s
Filename Type Size Used Priority
/swapfile file 82628 0 -1
I am not able to start the mysql.
Can someone please suggest me the solutions?
Check the InnoDB cache size in my.cnf file. If it is more than the size of your system's memory you found your problem. Try to reduce that and try to start MySQL.

NTFS/GPT Mount exited with Exit Code 13

This is a duplicated post since I didn't get any help on askubuntu.com.
I have a 1TB external hard drive that I recently formatted to NTFS. It was mounting on my Ubuntu 11.10 fine until just now. I didn't make any changes to affect my OS or my exhdd.
The error that I get is:
Error mounting: mount exited with exit code 13: $MFTMirr does not match $MFT (record 0).
Failed to mount '/dev/sdb2': Input/output error
NTFS is either inconsistent, or there is a hardware fault, or it's a
SoftRAID/FakeRAID hardware. In the first case run chkdsk /f on Windows
then reboot into Windows twice. The usage of the /f parameter is very
important! If the device is a SoftRAID/FakeRAID then first activate
it and mount a different device under the /dev/mapper/ directory, (e.g.
/dev/mapper/nvidia_eahaabcc1). Please see the 'dmraid' documentation
for more details.
I did read this and this. But neither helped.
I tried installing ntfsfix but no such package exists anymore.
I have never used this HDD on a windows machine. If I need to use an other machine to do stuff to fix this, I have access to a mac.
Any advice?
This is my sudo fdisk -l output:
What in the world is GPT? I didn't do that. It used to be NTFS.
Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000586fb
Device Boot Start End Blocks Id System
/dev/sda1 * 2148 961320312 480659082+ 83 Linux
/dev/sda2 961320313 976773167 7726427+ 5 Extended
/dev/sda5 961320314 976773167 7726427 83 Linux
WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk doesn't support GPT. Use GNU Parted.
Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xcfd88605
Device Boot Start End Blocks Id System
/dev/sdb1 1 1953525167 976762583+ ee GPT
This is the thing that worked:
I first needed to get ntfs-3g (sudo apt-get install ntfs-3g)
Run sudo fdisk -l to figure out where the mount point is. Mine was /dev/sdb1
I ran ntfsfix -b /dev/sdb1 and that fixed the problem.
Error mounting: mount exited with exit code 13: $MFTMirr does not match $MFT (record 0). Failed to mount '/dev/sda1': Input/output error
NTFS is either inconsistent, or there is a hardware fault, or it's a SoftRAID/FakeRAID hardware. In the first case run chkdsk /f on Windows then reboot into Windows twice. The usage of the /f parameter is very important! If the device is a SoftRAID/FakeRAID then first activate it and mount a different device under the /dev/mapper/ directory, (e.g. /dev/mapper/nvidia_eahaabcc1).
Please see the 'dmraid' documentation for more details.
Solution :-
sudo fdisk -l
sudo ntfsfix /dev/select_disk_name
To find Disk name:
Go dashboard -> Disk utility -> Click disk -> then show Device /Dev/***

MySQL/Writing file error (Errcode 28)

I have the following error with one of our web applications -
Query3 failed: Error writing file '/tmp/MY1fnqpm' (Errcode: 28) ... INSERT MailList... (removed the rest of the query for security reasons)
Any ideas - is this some hard disk space issue on my server?
Use the perror command:
$ perror 28
OS error code 28: No space left on device
Unless error codes are different on your system, your file system is full.
We have experienced similar issue, and the problem was MySQL used /tmp directory for its needs (it's default configuration). And /tmp was located on its own partition, that had too few space for big MySQL requests.
For more details take a look for this answer:
https://stackoverflow.com/a/3716778/994302
I had same problem but disk space was okay (only 40% full).
Problem were inodes, I had too many small files and my inodes were full.
You can check inode status with df -i
The error means that you dont have enough space to create temp files needed by MySQL.
The first thing you can try is to expand the size of your /tmp/ partition. If you are under LVM, check the lvextend command.
If you are not able to increase the size of your partition /tmp/ you can work in the MySQL configuration, edit the my.cnf (typically on /etc/mysql/my.cnf) file and look for this line:
tmpdir = /tmp/
Change it for whatever you want (example /var/tmp/). Just be sure to have space and assign write permission for the mysql user in the new directory.
Hope this helps!
Run the following code:
du -sh /var/log/mysql
Perhaps mysql binary logs filled the memory, If so, follow the removal of old logs and restart the server. Also add in my.cnf:
expire_logs_days = 3
This error occurs when you don't have enough space in the partition. Usually MYSQL uses /tmp on linux servers. This may happen with some queries because the lookup was either returning a lot of data, or possibly even just sifting through a lot of data creating big temp files.
Edit your /etc/mysql/my.cnf
tmpdir = /your/new/dir
e.g
tmpdir = /var/tmp
Should be allocated with more space than /tmp that is usually in it's own partition.
I had this same error and the problem was simply not enough space on my virtual machine. I deleted some unnecessary files and it started working again.
my
memory/disk space allocation looked something like this
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 37G 37G 127M 100% /
...
For those like me who don't have a full disk or full inodes and that are using MySQL 8, you should take a look at your /var/lib/mysql directory to check if there is some large binlog.XXXXXX files.
It was the case for me and it prevented a query to be executed until the end because it seemed like this query was generating a large binlog file.
I just had to turn off the logbin generation by adding this parameter in /etc/mysql/mysql.conf.d/mysqld.cnf :
disable_log_bin
Then restart mysql service :
service mysql restart
After that i didn't have the "No space left on device" error anymore!
You can also try using this line if the other doesn't work:
du -sh /var/lib/mysql/database_Name
You may also want to check with your host and see how big they allow your databases to be.
For xampp users: on my experience, the problem was caused by a file, named '0' and located in the 'mysql' folder. The size was tooooo huge (mine exploded to about 256 Gb). Its removal fixed the problem.
Today. I have same problem... my solution:
1) check inode: df -i
I saw:
root#vm22433:/etc/mysql# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 124696 304 124392 1% /dev
tmpfs 127514 452 127062 1% /run
/dev/vda1 1969920 1969920 0 100% /
tmpfs 127514 1 127513 1% /dev/shm
tmpfs 127514 3 127511 1% /run/lock
tmpfs 127514 15 127499 1% /sys/fs/cgroup
tmpfs 127514 12 127502 1% /run/user/1002
2) I began to look what folders use the maximum number of inods:
for i in /*; do echo $i; find $i |wc -l; done
soon I found in /home/tomnolane/tmp folder, which contained a huge number of files.
3) I removed /home/tomnolane/tmp folder
PROFIT.
4) checked:
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 124696 304 124392 1% /dev
tmpfs 127514 454 127060 1% /run
/dev/vda1 1969920 450857 1519063 23% /
tmpfs 127514 1 127513 1% /dev/shm
tmpfs 127514 3 127511 1% /run/lock
tmpfs 127514 15 127499 1% /sys/fs/cgroup
tmpfs 127514 12 127502 1% /run/user/1002
it's ok.
5) restart mysql service - it's ok!!!!
I had the same problem and after little research found that the snap directory was occupying most of the space. So executed following command to get rid of it:
sudo apt autoremove --purge snapd
After that ran following command to get rid of useless/dev/loop mounts:
`sudo apt purge snapd ubuntu-core-launcher squashfs-tools`
After that ran following command to restart mysql:
sudo service mysql restart
It worked for me!