I've created a 10 GB HDD and 3.75 GB RAM instance in Google Cloud and hosted a quite heavy DB transaction application's backend/API there. The OS is Ubuntu 14.04 LTS and I'm using Apache web server with PHP and MySQL for the backend. The problem here is that the HDD space has almost run out of memory very quickly.
Using Linux (Ubuntu) commands, I've found that my source code (/var/www/html) size is about 200 MB and the MySQL DB folder (/var/lib/mysql) size is 3.7 GB (around 20,000,000 records in my project DB). I'm confused how rest of my HDD space is occupied (except OS files). As of today, I only have 35 MB left. Once for testing purpose, I copied the source code to another folder. Even then I had the same problem. When I realized that my HDD space is running out, I deleted that folder and freed around 200 MB. But later (around 10 minutes) that freed space has also gone!!!
I figured that some log file like Apache error log, access log, MySQL error log or CakePHP debug log may occupy that space but I've disabled and truncated those files long ago and checked if these file are creating again but it doesn't. So how????????
I'm seriously worried about this project to continue with this instance. I thought about adding additional HDD to remedy this situation but I need to be sure how my HDD space is being occupied first. Any help will be highly appreciated.
You can start by searching all the largest files in your system.
On the / directory type:
sudo find . -type f -size +5000k -exec ls -lh {} \;
Once you find the files you can start to troubleshoot.
If you get many file you can increase +5000k to aim for the larger files.
Related
I would like to know how MySQL uses disk space. Consider the following scenario.
I tried to import a large mysqldump (1.6 GB) using the command mysql -u root dbname < mydump.sql. But the import was failed and I got table xxx is full error.
I found that it was due to no memory left in C: drive. Actually, before importing the database, I had 4.1 GB free in C: drive. But after running the command, only 13 MB was left in C: drive.
To free up some space in C: drive, I dropped the database which I tried to import (as half of the total tables were imported already). But after deleting the database it freed up only 2 GB in C:drive now.
I have got a few questions here
Before importing I had 4.1 GB. After dropping the database I have 2.1 GB. So what is occupying the remaining 2GB data in my desk?
Is there a way I can find and clean that space? (I tried clearing %temp% system cache, and ran FLUSH QUERY CACHE. But nothing works)
If the mysqldump file is 1.96 GB, I thought the imported database will also be having almost the same size. If it is not, how much memory actually the database would occupy in the disk.
I managed to import the database after removing cache tables from the database. But I would like to know how to free up memory in my desk.
I have an issue that is driving me (and my customer) up the wall. They are running MySQL on Windows - I inherited this platform, I didn't design it and changing to MSSQL on Windows or migrating the MySQL instance to a *Nix VM isn't an option at this stage.
The Server is a single Windows VM, reasonably specced (4 vCores, 16 Gb RAM etc.)
Initially - they had a single Disk for the OS, MySQL and the MySQL backup location and they were getting inconsistent backups, regularly failing with the error message:
mysqldump: Got errno 22 on write
Eventually we solved this by simply moving the Backup destination to a second virtual disk (even though it is is on the same underlying Storage network, we believed that the above error was being caused by the underlying OS)
And life was good....
For about 2-3 months
Now we have a different (but my gut is telling me related) issue:
The MySQL Dump process is taking increasingly longer (over the last 4 days, the time taken for the dump has increased by about 30 mins per backup).
The Database itself is a bit large - 58 Gb, however the delta in size is only about 100 mb per day (and unless I'm missing something - it shouldn't take 30 minutes extra to dump 100 mb of Data).
Initially we thought that this was the underlying Storage network I/O - however as part of the backup script, once the .SQL file is created, it gets zipped up (to about 8.5 Gb) - and this process is very consistent in the time taken - which leads me not to suspect the disk I/O (as my presumption is that the Zip time would also increase if this were the case).
the script that I use to invoke the backup is this:
%mysqldumpexe% --user=%dbuser% --password=%dbpass% --single-transaction --databases %databases% --routines --force=true --max_allowed_packet=1G --triggers --log-error=%errorLogPath% > %backupfldr%FullBackup
the version of MySQLDump is C:\Program Files\MySQL\MySQL Server 5.7\bin\mysqldump.exe
Now to compound matters - I'm primarily a Windows guy (so have limited MySQL experience) and all the Linux Guys at the office won't touch it because it's on Windows.
I suspect the cause for the increase time is that there is something funky happening with the row Locks (possibly due to the application that uses the MySQL instance) - but I'm not sure.
Now for my questions: Has anyone experienced anything similar with a gradual increase of time for a MySQL dump over time?
Is there a way on Windows to natively monitor the performance of MySQLdump to find where the bottlenecks/locks are?
Is there a better way of doing a regular MySQL backup on a Windows platform?
Should I just tell the customer it's unsupported and to migrate it to a supported Platform?
Should I simply utter a 4 letter word and go to the Pub and drown my sorrows?
Eventually found the culprit was the underlying storage network.
I don't know what's going on. My server has been fine for probably a year. Now I'm having a severe problem with MariaDB/MySQL. The DB server keeps crashing. When it does and I bring it back online I get errors, several tables are marked crashed and I have to repair them. Here are the server specs...
CloudLinux Server release 6.6 installed on Centos 6.5 (x64)
WHM/Cpanel 11.50.1 Build 1 (Current)
MariaDB 10.0.21
RAM: 3,820MB (3750MB+ in use)
Swap: 1,023MB (1,023MB in use)
4 Cores (Low idle load)
Available Disk Space: 26GB
I suspect it has to do with memory. Here's a memory alert I get in WHM:
Here's what I get when I try to visit a web site on my server that uses MySQL (As expected):
Warning: mysql_connect(): Connection refused in /home/mysite/public_html/index.php on line 19
Unable to connect to server.
Here's a link to the main error log of my database server (Too much to post here): http://wikisend.com/download/182056/proton.myserver.com.err.txt
This is what happens when I restart my database server from WHM. Each time I restart the db server, random tables are marked as crashed. Sometimes a lot of tables, sometimes just a few and then I have to repair them:
Here is the contents of the /etc/my.cnf file:
root#proton [~]# cat /etc/my.cnf
[mysqld]
default-storage-engine=MyISAM
innodb_file_per_table=1
max_allowed_packet=268435456
open_files_limit=10000
innodb_buffer_pool_size=123731968
The only thing I've tried to fix this is setting this option in WHM:
I only have a handful of sites on the server. Any help is greatly appreciated.
SHOW VARIABLES LIKE '%buffer%';
Do you have other products running in the same VM/server? How much of the 3750MB are they using? Consider increasing RAM as a quick fix. Otherwise, lets look for what is chewing up RAM.
You are probably no using any InnoDB tables? If not then change this to 0:
innodb_buffer_pool_size=123731968
For MyISAM, the most important factor is key_buffer_size; it should be no more than about 500M for your case.
What is WHM?
Abrupt stops of mysql (for any reason) leads to the need to REPAIR MyISAM tables ("marked crashed"). (Consider moving to InnoDB to avoid this recurring nuisance.)
I am using Ubuntu for PHP/MySQL development. Ubuntu is installed on VM and unfortunately VM has been assigned very low disk space (40 GB).
After 1st startup the disk had more than 20 GB disk space available.
Over past week I had to import and delete the db.sql.gz file quite some times. Now I only have 50 MB disk space left. Even though I have deleted all the databases using phpmyadmin, I suspect they still use the disk space.
How can I recover this disk space?
we have a web application (racktables) that's giving us grief on our production box. whenever users try to run a search, it gives the following error:
Pdo exception: PDOException
SQLSTATE[HY000]: General error: 5 Out of memory (Needed 2057328 bytes) (HY000)
I cannot recreate the issue on our backup server. The servers match except for the fact that in production we have 16GB RAM and our backup we have 8GB. It's a moot point though because both are running 32 bit os's and so are only using 4GB of RAM. we also have set up a swap partition...
Here's what i get back from the "free -m" command in production:
prod:/etc# free -m
total used free shared buffers
Mem: 3294 1958 1335 0 118
-/+ buffers: 1839 1454
Swap: 3817 109 3707
prod:/etc#
I've checked to make sure that my.cnf on both boxes match. The database from production was replicated onto the backup server... so the data matches as well.
I guess our options are to:
A) convert the o/s to 64 bit so we can use more RAM.
B) start tweaking some of the innodb settings in my.cnf.
But before I try either A or B, I wanted to know if there's anything else I should compare between the two servers... seeing how the backup is working just fine. There must be a difference somewhere that we are not accounting for.
Any suggestions would be appreciated.
I created a script to simulate load on the backup server and was able to then to recreate the out of memory error message.
In the end, i added the "join_buffer_size" setting to my.cnf and set it 3 MBs. That has resolved the issue.
ps. I downloaded and ran tuning-primer.sh as well as mysqltuner.pl to narrow down the issues.