I am trying to stuff as many lamp stack containers inside docker as possible that are running wordpress. According to many sources it is possible to stuff over 500 containers on a docker server, for example:
Is there a maximum number of containers running on a Docker host?
However, they most likely discuss empty docker containers and not LAMP stack ones (apache+mysql).
I have managed to stuff 120 containers running wordpress and don't see a siginifcant issue with RAM or CPU. I should be able to fit more but performance goes downhill in response times after that point.
A colleague has mentioned that mysql performs significant I/O operations which could cause issues with file descriptors / open file limits / and fs.io.max.nr .
Could this be the case? Could it be that the number of open files of each containers limits the total number of containers?
Related
When I run my docker containers in Google Cloud Run, any disk space they use comes from the available memory.
I'm running several self-hosted github action runners on a single local server, and they have worn out my SSD over the past year. The thing is, all the data they are saving is pointless. None of it needs to be saved. It exists for a few minutes during the build, and then is deleted.
Is it possible for me to instead run all these docker machines using memory for the disk space? That would improve performance and stop putting unnecessary wear on a hard drive.
I have a container running a Flask server and a container running a MySQL server with a service link in the same stack on the same host.
The Flask server need to run subprocesses, which read data (>100MB) from database and I use pandas.read_sql to do that, but it seems that the networking between the containers becomes the bottleneck. The speed of a single thread is only about 500kbps as I saw in the rancher web interface.
I run SHOW PROCESSLIST; and found that the database spent large amount of time "sending data". The queries are fast. Maybe I need to do some configurations.
Though I guess it's a networking issue, I can provide further information about other aspects if needed.
I've been trying Docker for a few days. I'm using a Drupal image (docker4drupal) which basically contains MySQL (MariaDB), PHP (php-fpm) and NGINX.
Almost everytime I do a database import to the database container, on a VPS with 512MB RAM, the container with MariaDB dies... and messages like "MySQL server has gone away" appear... And this does not happen when my VPS has 1GB o 2GB RAM.
So, this seems to be a memory problem, but I need the evidence! I don't know where is the log that tells me that my container died because wasn't enough memory.
I checked MariaDB logs but I can't find anything... it's log only say somethign like "the database was not normally shutdown" and thaen "it's starting" and then "wating for connections"...
So, independently of my MariaDB config (which is not proper for a 512MB VPS)... Where can I find explicitly the reason of why the container with the database server died?
Any help is welcome.
Thanks a lot.
PD: I execute mysql cli from the PHP container, that's why despite the database container dies I still can see the output that something wrong happened.
Could be the kernel terminating most memory-consuming process on 'lack-of-memory' event. Some entries may be there in host system log. Lack of such entries doesn't guarantee it wasn't kernel who killed your DB, though.
Exact filename depends on host system configuration (meaning the VPS, in your case). Could be /var/log/{system.log,error.log, ...}.
As long as docker container is not an isolated VM but a wrapper over kernel-driven cgroups, kernel events are handled by host system loggin daemon
Hi Beto we can see the logs in docker checkout the below commands:
The docker logs --follow command will continue streaming the new output from the container’s STDOUT and STDERR.
That is probably too much to cram in a minuscule 512MB. Do one of
Increase RAM available. ("And this does not happen when my VPS has 1GB")
Split applications across multiple tiny Dockers.
Tune each app to use less RAM. (Didn't I answer your question recently?)
How many tables do you have? Hopefully not a lot, as in https://dba.stackexchange.com/questions/60888/mysql-runs-out-of-memory-when-importing-innodb-database
I have an AWS t2.micro EC2 instance with docker on it, and I bring up the following instances;
jwilder/nginx-proxy
mysql
wordpress
Which results in something like this docker stats;
CONTAINER MEM USAGE/LIMIT MEM %
wordpress 331.9 MB/1.045 GB 31.77%
nginx 18.32 MB/1.045 GB 1.75%
mysql 172.1 MB/1.045 GB 16.48%
Then, I run siege's default 15 concurrent connections against it, which spawns multiple apache processes, reaching the memory limit of the EC2 instance, crashing docker and bash due to no more memory, requiring my intervention to get it all running again.
I have a couple of questions regarding this.
Am I expecting too much? Should this setup be able to handle 15 concurrent connections? If so, what changes* need to be made?
How can I automate recovery from this? Is there a way to detect that memory is reaching capacity and do something (like reject requests or similar) until memory usage decreases? Is there a way to keep the system stable during the high request volume so once it's over it does not require my intervention to bring it back up?
* I've already done this to drop mysql memory from 22% to 15%.
Given a t2.micro only has 1GB total, and each of those containers has a 1GB limit on its own, have you tried limiting the max memory usage on each container (as per http://docs.docker.com/engine/reference/run/#user-memory-constraints) such that the total memory limit doesn't exceed 1GB?
The biggest impact, which stopped the EC2 instance from falling over, was limiting the memory a docker container can use with the -m option per #palfrey's answer.
Some additionally tweaks were required to reduce the memory footprint and have the service respond to 15 concurrent users, albeit somewhat slowly. This included;
MySQL
Disabling performance_schema
Using a minimal config
WordPress
Disabling KeepAlive
Limiting servers:
<IfModule mpm_prefork_module>
StartServers 1
MinSpareServers 1
MaxSpareServers 3
MaxRequestWorkers 10
MaxConnectionsPerChild 3000
</IfModule>
Docker
I created some docker images that extend the default images to include these optimisations;
mysql-minimal
wordpress-minimal
Further details in my blog post.
Probably, a micro only has 1GB of ram. You can run this configuration without docker just fine, but you do have to adjust for memory limitations. Docker probably adds some overhead. Is there a reason for running both nginx and apache?
Generally you test and limit your threads to what the system can handle, there are probably things you can do with caching that will help improve performance. Apache, nginx, php-fpm all have settings that can control the number of threads that are allowed to be created.
I run a pretty customized cluster for processing large amounts of scientific data based on a basic LAMP design. In general, I run a separate MySQL server with around 128GB of ram and about 1TB of storage. Separately, I run a head node that serves as an nfs mount point for the data input of my process, and a webserver to display results. Finally, I trypically have a few compute nodes that get their jobs from a mysql table, get the data from NFS, do some heavy lifting, then put results into mysql.
I have come across a dataset I would like to process which is pretty large (1TB of input data), and I don't really have the hardware on hand to handle it. As a result, I began investigating google compute engine etc, and the prospect of scaling instances to process these data rapidly with the results stored in a mysql instance. Upon completion the mysql tables could be dumped from the cloud and brought up locally for analysis. I would have no problem deploying a MySQL server, along with the rest of the LAMP pieces and the compute nodes, but I can't quite figure out how I would do this in the cloud.
A major sticking point seems to be the lack of read/write NFS which would allow me to get the data onto several instances, crunch it, then push the results to MySQL. This is a necessary step for me as I could queue hundreds of jobs from the webserver, then have the instances (as many as 50-100) pick the jobs up by connecting to a centralized mysql instance to find out what jobs an instance needs to do and where the data is. Process the data (there is a file conversion that happens which make the write part necessary), crunch the data, then load results to mysql. I hope I'm explaining my situation clearly. This seems like a great example of a CPU intensive process that would scale nicely in the cloud, I just can't seem to put all the pieces together... Any input is appreciated!
It sounds quite possible; I've been doing similar things in GCE for a while now.
NFS mount - you just need configure it as you would normally. Set up the NFS server on the head node, and then configure the clients on the slave nodes to mount it. Here and here are some basic configuration instructions for Centos 6 I used to get NFS up and running.
Setting up a LAMP stack is very straightforward. These machines run pretty much vanilla Linux distros, so you can just use yum or apt-get to install components.
For the cluster, you will probably end up having an image for the head node you use once, and then another image for the slave nodes that you replicate for each one.
For the scheduler, I've used Condor and Sge successfully, but I'm sure the other ones would work just as well.
Hope this helps.