How to build sidekiq and rails image with docker? - mysql

I am trying another way to build a rails application into a docker image.
The structure of my services:
redis -- from official docker hub registry
fluentd -- from official docker hub registry
mysql -- from official docker hub registry
sidekiq -- build myself(maybe there isn't a official image for this)
web -- build myself
I created two Dockerfiles like:
Dockerfile.sidekiq
Dockerfile.web
Dockerfile.sidekiq
FROM ruby:2.2.2
ENV APP_HOME /myapp
RUN mkdir $APP_HOME
WORKDIR $APP_HOME
ADD Gemfile $APP_HOME/Gemfile
ADD Gemfile.lock $APP_HOME/Gemfile.lock
ADD config/sidekiq.yml $APP_HOME/config/sidekiq.yml
ADD init_sidekiq.sh $APP_HOME/
RUN export LANG=C.UTF-8 && bundle install
ADD . $APP_HOME
CMD ["sh", "init_sidekiq.sh"]
init_sidekiq.sh
#!/bin/sh
bundle exec sidekiq -C config/sidekiq.yml
Dockerfile.web
FROM rails:4.2.1
ENV APP_HOME /myapp
RUN mkdir $APP_HOME
WORKDIR $APP_HOME
ADD Gemfile $APP_HOME/Gemfile
ADD Gemfile.lock $APP_HOME/Gemfile.lock
ADD init_web.sh $APP_HOME/
RUN export LANG=C.UTF-8 && bundle install
ADD . $APP_HOME
CMD ["sh", "init_web.sh"]
init_web.sh
#!/bin/sh
bundle exec rake db:create db:migrate
bundle exec rails server -b 0.0.0.0
Use them I built two images:
myapp_web
myapp_sidekiq
Then run these containers:
$ docker run --name redis -d redis
$ docker run --name fluentd -d -p 24224:24224 fluent/fluentd
$ docker run --name mysql -e MYSQL_ROOT_PASSWORD=my_password -d mysql
Make env.list
RAILS_ENV=production
DATABASE_URL=mysql2://root:my_password#172.17.0.4/myapp?checkout_timeout=20000
Go on run these containers:
$ docker run --name web -d -p 3000:3000 --link mysql:mysql --env-file ./env.list myapp_web
$ docker run --name sidekiq -d --link mysql:mysql --env-file ./env.list myapp_sidekiq
The result:
redis -- success
fluentd -- success
mysql -- success
web -- success
sidekiq -- failure
The sidekiq log:
$ docker logs sidekiq
Unknown database 'myapp'
/usr/local/bundle/gems/activerecord-4.2.1/lib/active_record/connection_adapters/mysql2_adapter.rb:23:in `rescue in mysql2_connection'
I used the same method both web and sidekiq to connect mysql. I believe that in the mysql container there exists a myapp database. But why it can't find it?
Is it a wrong way to make them been two containers? How to run sidekiq correctly?

I think the problem there is how are you connecting to redis? The message appears to come from sidekiq and somehow it can't connect to your redis server. (and I think trying to connect to some bogus db server/database)
So I think you need to link your sidekiq container to both your db container and also your redis container.
docker run --name sidekiq -d --link mysql:mysql --link redis:redis --env-file ./env.list myapp_sidekiq
Also, would be nice if you can share your env.list

Related

Dockerfile to install Nginx and MySQL in same image

I want to install nginx and mysql in same image. I start out with a mysql image with the plan to install docker using dockerfile.
Here is my dockerfile:
FROM mysql:latest
ENV MYSQL_ROOT_PASSWORD=HelloWorld \
MYSQL_DATABASE=content
RUN apt update
RUN apt install nginx -y
COPY nginx.conf /etc/nginx/nginx.conf
This starts the mysql db perfectly and nginx also gets installed. Unfortunately, nginx doesn't start. To start nginx I also added another command in the docker file:
CMD service nginx start
After adding this line in the dockerfile, the container closes after creation. What am I doing wrong here?
I am using below command to start container with above image:
docker run -it -p 3306:3306 -p 8080:80 -p 8081:443 --name mycontainer myimage
it's best to run each process in a separate container. but if you wanna do that, you should create a bash file to start MySQL and Nginx. finally, you should use that bash file as the ENTRYPOINT of your image/container

Docker image with mysql and .war file on ECS

I am totally new in Docker community and I am trying to create a custom container image with mysql and a .war file inside and to run it on a AWS EC2 instance. I've tried a lot but I cannot figure this out..
To build the container image I rum this
docker build -t <name-of-image> -f Dockerfile
I suppose Docker file content should contain something like
FROM mysql:latest
ENV TARGETD /opt/apache-tomcat-9.0.35
ENV WAR /target/NewWebApp.war
RUN apt-get -y update
RUN apt-get -y upgrade
# Create database
RUN mkdir /usr/sql
#RUN CHMOD 644 /usr/sql
ADD db.sql /usr/sql/db.sql
RUN mysql -h localhost -P 3306 --protocol=tcp -u root start && \
mysql -u root -e < /usr/sql/db.sql
EXPOSE 3306
ADD ${WAR} ${TARGETD}/webapps
And to run(deploy) image I use
docker run -d -p 8080:3306 <name-of-image>:latest
I have already installed Tomcat on 8080
What can I do in order to run this image and to be able to access it through AWS EC2?

Deploy mysql during build from Dockerfile

I am building an application which has parent and child dependency and to build of my application which is the final stage of build i need to connect to mysql for it during build stage itself.
In this i am getting the error:
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
I have mentioned my docker file code i am using and for mysql i have pulled image from dockerhub following instructions from below link:
https://dev.mysql.com/doc/mysql-installation-excerpt/5.5/en/docker-mysql-getting-started.html
And i was planning to run this as a separate container using bridge to communicate with my above container using below command:
docker run -d -name app-container-name --link mysql-container-name app-image-name
FROM maven:3.5.4-jdk-8 as maven
COPY ZP ZP
COPY CommonApp CommonApp
RUN cd ZP && mvn clean install
RUN cd CommonApp && mvn clean install package -U && mvn install:install-file -Dfile=/CommonApp/target/commonapp-0.0.1-SNAPSHOT.jar -DgroupId=com.z -DartifactId=commonapp -Dversion=0.0.1-SNAPSHOT -Dpackaging=jar;
FROM mysql:5.7
# ROOT PASSWORD
ENV MYSQL_ROOT_PASSWORD=root
ENV MYSQL_USER=root
ENV MYSQL_PASSWORD=root
ENV MYSQL_DATA_DIR=/var/lib/mysql \
MYSQL_RUN_DIR=/run/mysqld \
MYSQL_LOG_DIR=/var/log/mysql
RUN /etc/init.d/mysql start && \
mysql -u root -p$MYSQL_ROOT_PASSWORD -e "GRANT ALL PRIVILEGES ON *.* TO 'root'#'%' IDENTIFIED BY 'root';FLUSH PRIVILEGES;"
#PORT
EXPOSE 3306
FROM maven:3.5.4-jdk-8
COPY ZCApp ZCApp
RUN cd ZCApp && mvn clean package -U
How should i approach this problem. How can i build mysql along with the application itself using dockerfile.?
Had the same issue when built maven project. What makes this different from similar requests is that here you don't link two running containers but instead you link docker daemon, preforming build process, to running container.
For Docker to get access to database during build you have to expose ports of database. Using --link will have no effect because it links containers (and you dont have second container at the moment) and btw is considered as obsolete technique.
You have to explicitly start database container before build process and somehow expose its ports for docker daemon to access them.
Option 1 - using host networking.
First start database:
docker run -d --network=host mysql
Then build:
docker built -t foo .
Docker will see database on localhost during build process because database uses host's network and doesn't need any port exposion.
Option 2 - Expose ports
First start database:
docker run -d -p 3306:3306 mysql
Then build:
docker built -t foo .
Docker will again see database on localhost during build process because port is exposed.
What you have to double check is your connection string in mvn. It has to use localhost and default tcp port 3306

Docker, how to run .sql file in an image?

It's my first time working with Docker an I am not sure if I am doing things well.
I have a rails applications that depends on a Mysql database, so I've configured the docker-compose.yml file like this:
db:
image: library/mysql:5.6
environment:
MYSQL_ALLOW_EMPTY_PASSWORD: "yes"
expose:
- "3306"
ports:
- "3306:3306"
rails-app:
build: .
dockerfile: Dockerfile
environment:
RAILS_ENV: development
links:
- db
volumes:
- ".:/home/app"
volumes_from:
- bundle
... omitted lines ...
Then, if I run the following:
$ docker-compose run db mysql --host=$DOCKER_LOCALHOST --port=3306 --protocol=tcp -u root < shared/create_common_tables.sql
I get this error:
ERROR 2003 (HY000): Can't connect to MySQL server on '192.168.99.100' (111)
This sounds normal, because I suspect that I have to build before some container that links to db.
I know this because if I run this in this order:
$ docker-compose build rails-app
$ docker-compose run -e RAILS_ENV=development rails-app bundle
$ docker-compose run -e RAILS_ENV=development rails-app bundle exec rake db:create
$ docker-compose run db mysql --host=$DOCKER_LOCALHOST --port=3306 --protocol=tcp -u root < shared/create_common_tables.sql
It works fine.
But, how can I do to execute this sql before creating any container?
You can load the sql file during the build phase of the image. To do this you create a Dockerfile for the db service that will look something like this:
FROM mysql:5.6
COPY setup.sh /mysql/setup.sh
COPY setup.sql /mysql/setup.sql
RUN /mysql/setup.sh
where setup.sh looks something like this:
#!/bin/bash
set -e
service mysql start
mysql < /mysql/setup.sql
service mysql stop
And in your docker-compose.yml you'd change image to build: ./db or the path where you put your files.
Now this works if you have all your sql in a raw .sql file, but this wont be the case if you're using rails or a similar framework where the sql is actually stored in code. This leaves you with two options.
Instead of using FROM mysql:5.6 you can use FROM your_app_image_that_has_the_code_in_it and apt-get install mysql .... This leaves you with a larger image that contains both mysql and your app, allowing you to run the ruby commands above. You'd replace the mysql < /mysql/setup/sql with the rails-app bundle exec rake db:create lines. You'd also have to provide an app config that hits a database on localhost:3306 instead of db:3306
My preferred option is to create a script which exports the sql into a .sql file, which you can then use to build your database container. This is a bit more work, but is a lot nicer. It means that instead of running rails-app bundle exec rake db:create you'd just run the script to load a db.
Such a script would look something like this:
#!/bin/bash
set -e
docker-compose build rails-app
docker run -d --name mysql_empty mysql:5.6
docker run --link mysql_empty:db -v $PWD:/output project_rails-app export.sh
where export.sh looks something like this:
#!/bin/bash
set -e
RAILS_ENV=development
rails-app bundle exec rake db:create
mysqldump > /output/setup.sql
You could also replace the docker run script with a second compose file if you wanted to.

Docker Cannot link to a non running container

I need to create Rails and Mysql containers with docker-compose. When I try to create links between containers with docker-compose up, I get
Cannot start container
9b271c58cf6aecaf017dadaf5b Cannot link to a non running container:
/puma_db_1 AS /puma_web_1/db
Files
Dockerfile
FROM ubuntu:14.04
RUN apt-get -y update
RUN apt-get -y install git curl zlib1g-dev build-essential libssl-dev libreadline-dev libyaml-dev libsqlite3-dev sqlite3 libxml2-dev libxslt1-dev libcurl4-openssl-dev python-software-properties libffi-dev
RUN apt-get -y install libmysqlclient-dev
RUN git clone https://github.com/sstephenson/rbenv.git /root/.rbenv
RUN git clone https://github.com/sstephenson/ruby-build.git /root/.rbenv/plugins/ruby-build
RUN echo 'eval "$(rbenv init -)"' >> $HOME/.profile
RUN echo 'eval "$(rbenv init -)"' >> $HOME/.bashrc
RUN rbenv install 2.1.5
RUN rbenv global 2.1.5
RUN gem install rails -v 4.0.11
ADD app.tar.gz /home/
WORKDIR /home/app
RUN bundle install
EXPOSE 3000
CMD ["rails", "server", "-b", "0.0.0.0"]
docker-compose.yml
db:
image: mysql:latest
environment:
MYSQL_DATABASE: app_development
MYSQL_USER: mysql
DATABASE_PASSWORD: onetwo
ROOT_PASSWORD: onetwo
web:
build: .
command: bundle exec rails s -p 3000 -b '0.0.0.0'
ports:
- "4000:3000"
links:
- db
Most likely the db container fails to start.
Make sure it works fine by starting only the db service. You can do that with the following command:
docker-compose up db
If it appears the MySQL service is not running after this command, then you found the origin of your problem.
Not specifically related to MySQL but more the message ERROR: for <service> Cannot link to a non running container: /b2f21b869ccc_<dependency>_1 AS /<service>_1/<dependency>_1
I found that the dependency container had a different id than the one given (b2f21b869ccc in my example above)
Solved simply by running
docker-compose up -d --force-recreate <service>
which caused it to recreate the dependency and fix the link to the correct docker id
For me, it did not help running docker-compose up db.
This did the trick for me:
sudo service docker restart
and then continuing with docker-compose up (-d)
You might try out the new features of docker networking, To do this, You must remove the link parameter in your docker-compose.yml , and initialize the container with the --x-networking option.
docker-compose --x-networking up -d
To prevent docker generate random names for the containers, which are added to the /etc/hosts file of the respective network for every container, you can use the container_name: key in the docker-compose.yml
db:
container_name: db
image: mysql:latest
environment:
MYSQL_DATABASE: app_development
MYSQL_USER: mysql
DATABASE_PASSWORD: onetwo
ROOT_PASSWORD: onetwo
web:
container_name: web
build: .
command: bundle exec rails s -p 3000 -b '0.0.0.0'
ports:
- "4000:3000"
Issue:
I have gotten this error whenever docker-compose successfully
builds a set of Images, but one of those Imagesfails to
run (e.g. launch into its own Container).
In this case, I suspect the Image, underlying your
puma_db_1 Container, is failing to run. You can
find the name of this Image by running docker ps -a. That said, its name is most likely puma_db
Solution:
To get at the cause, you can try docker-compose up
<service_name> or docker-compose up db
Alternatively, I find the error message by running docker run
<image_name> more useful. In this case, that would be docker
run puma_db
I had the same problem for mssql.link, as I am not using local database (rather using the one we have on staging), all I had to do is just comment that line out by editing Dockerfile script:
# DOCKER_ARGS="${DOCKER_ARGS} --link mssql-server-linux:mssql.link"
This solution may help someone or may be no one, but it sorted it for me :)
If you started the container lets say X with a link --link keen_visvesvaraya and then once X is up the linked container was stopped, but X kept running . Now if you try to docker exec into X you get this error.
Yah solution is to restart.
I had the same problem with elasticsearch - symfony - and docker
Can not link to a non-running container:/43c1d3b410db_myindex_elasticsearch_1 AS /myindex_apache_1/elasticsearch
the solution is to delete the content of the data volume
  elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:5.5.2
Volumes:
     - ./docker/volume/elasticsearch: /usr/share/elasticsearch/data
and run docker-composer up -d again.
You can use below command Because It's working for me
docker run --name standlone-mysql -e MYSQL_ROOT_PASSWORD=password -e MYSQL_DATABASE=test -e MYSQL_USER=root -e MYSQL_ROOT_PASSWORD=root -d mysql:5.6
you need to modify the db: in yml file to include "POSTGRES_HOST_AUTH_METHOD" in environment section
db:
environment:
POSTGRES_HOST_AUTH_METHOD: trust
I got same error when restart the service.
Cannot link to a non running container: /c7e8ba2cc034_<service1>_1 AS /<service2>/<srvice1>
In my case there is Exited service2, so remove the container (docker rm) and start the service 2.