To install MySQL client on my EC2 instance I do the following:
# Update all packages
sudo yum update
# Install mysql client
sudo yum install mysql
I would like to know how can I provision that using CloudFormation when I create my EC2 instance?
Is there any samples which I can refer to?
To run a simple script on an EC2 instance provisioned using CloudFormation, you use the UserData property of the AWS::EC2::Instance resource.
Example:
Description: Run a bash script using the UserData property.
Mappings:
# amzn-ami-hvm-2016.09.1.20161221-x86_64-gp2
RegionMap:
us-east-1:
"64": "ami-9be6f38c"
Resources:
Instance:
Type: AWS::EC2::Instance
Properties:
ImageId: !FindInMap [ RegionMap, !Ref "AWS::Region", 64]
InstanceType: m3.medium
UserData:
"Fn::Base64":
!Sub |
#!/bin/bash
# Update all packages
yum -y update
# Install mysql client
yum -y install mysql
Just want to share also on top of the answer above.
I later found out my EC2 is deployed via EB and UserData does not seem to work.
In that case, I have to use .ebextension's container_commands.
So posting it here hopefully it will help someone as well:
files:
"/usr/local/bin/eb_mysql.sh":
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash
yum -y update
yum -y install mysql
container_commands:
040_install_mysql_client:
command: "chmod a+rx /usr/local/bin/eb_mysql.sh && /usr/local/bin/eb_mysql.sh"
Install MySQL Client:
sudo yum install -y https://dev.mysql.com/get/mysql57-community-release-el7-11.noarch.rpm
sudo rpm --import https://repo.mysql.com/RPM-GPG-KEY-mysql-2022
sudo yum install -y mysql-community-client
Related
I was trying to install and configure apache airflow on dev Hadoop cluster of a three nodes with below configurations/version:
Operating System: Red Hat Enterprise Linux Server 7.7
python 3.7.3
anaconda 2
spark 2.45
a)sudo yum install gcc gcc-c++ -y
b)sudo yum install libffi-devel mariadb-devel cyrus-sasl-devel -y
c)pip install 'apache-airflow[all]'
d)airflow initdb -- airflow.cfgfile was created with SQLlite
Then I followed below set of commands to configure it with mysql
a) rpm -Uvh https://repo.mysql.com/mysql80-community-release-el7-3.noarch.rpm
b) sed -i 's/enabled=1/enabled=0/' /etc/yum.repos.d/mysql-community.repo
c) yum --enablerepo=mysql80-community install mysql-community-server
d) systemctl start mysqld.service
Done below things at mysql
a) CREATE DATABASE airflow CHARACTER SET utf8 COLLATE utf8_unicode_ci;
b) create user 'airflow'#'localhost' identified by 'Airflow123';
c) grant all privileges on * . * to 'airflow'#'localhost';
here are some details from my airflow.cfg file
broker_url = sqla+mysql://airflow:airflow#localhost:3306/airflow
result_backend = db+mysql://airflow:airflow#localhost:3306/airflow
sql_alchemy_conn = mysql://airflow:Airflow123#localhost:3306/airflow
executor = CeleryExecutor
I am getting below error while running airflow initdb commands
ImportError: /home/xyz/anaconda2/envs/python3.7.2/lib/python3.7/site-packages/_mysql.cpython-37m-x86_64-linux-gnu.so: symbol mysql_real_escape_string_quote,
version libmysqlclient_18 not defined in file libmysqlclient.so.18 with link time reference
have set up the .bashrc file as:
export AIRFLOW_HOME=~/airflow
here's my directory created:
[xyz#innolx5984 airflow]$ pwd
/home/xyz/airflow
When I look for this file "libmysqlclient" I have found these many instances.
[xyz#innolx5984 airflow]$ find /home/xyz/ -name "*libmysqlclient*"
/home/xyz/anaconda2/pkgs/mysql-connector-c-6.1.11-h597af5e_1/lib/libmysqlclient.so
/home/xyz/anaconda2/pkgs/mysql-connector-c-6.1.11-h597af5e_1/lib/libmysqlclient.a
/home/xyz/anaconda2/pkgs/mysql-connector-c-6.1.11-h597af5e_1/lib/libmysqlclient.so.18
/home/xyz/anaconda2/pkgs/mysql-connector-c-6.1.11-h597af5e_1/lib/libmysqlclient.so.18.4.0
/home/xyz/anaconda2/lib/libmysqlclient.a
/home/xyz/anaconda2/lib/libmysqlclient.so
/home/xyz/anaconda2/lib/libmysqlclient.so.18
/home/xyz/anaconda2/lib/libmysqlclient.so.18.4.0
Just adding few more details in case it helps.
[xyz#innolx5984 airflow]$ mysql_config
Usage: /home/xyz/an
aconda2/bin/mysql_config [OPTIONS]
Options:
--cflags [-I/home/xyz/anaconda2/include ]
--cxxflags [-I/home/xyz/anaconda2/include ]
--include [-I/home/xyz/anaconda2/include]
--libs [-L/home/xyz/anaconda2/lib -lmysqlclient ]
--libs_r [-L/home/xyz/anaconda2/lib -lmysqlclient ]
--plugindir [/home/xyz`/anaconda2/lib/plugin]
--socket [/tmp/mysql.sock]
--port [0]
--version [6.1.11]
--variable=VAR VAR is one of:
pkgincludedir [/home/xyz/anaconda2/include]
pkglibdir [/home/xyz/anaconda2/lib]
plugindir [/home/xyz/anaconda2/lib/plugin]
Looking for some help and suggestion to resolve this
issue. I am not too sure whether heading into right direction.
Follow these steps to install Apache Airflow with MySQL using Anaconda3
1) Install Pre-requisites
yum install gcc gcc-c++ -y
yum install libffi-devel mariadb-devel cyrus-sasl-devel -y
dnf install redhat-rpm-config
2) Install Anaconda3 (comes with Python 3.7.6)
yum install libXcomposite libXcursor libXi libXtst libXrandr alsa-lib mesa-libEGL libXdamage mesa-libGL libXScrnSaver
wget https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh
chmod +x Anaconda3-2020.02-Linux-x86_64.sh
./Anaconda3-2020.02-Linux-x86_64.sh
Make sure you do conda initialize when prompted during installation.
This will make sure the correct version of python and pip are used in the subsequent steps.
3) Install Apache Airflow
pip install apache-airflow[mysql,celery]
You can add other subpackages as required. I have included only the ones required for Airflow to use MySQL database as backend.
4) Initialize Airflow
export AIRFLOW_HOME=~/airflow
airflow initdb
From here, I have mimicked the steps you have followed to configure MySQL Server
5) Install MySQL Server
rpm -Uvh https://repo.mysql.com/mysql80-community-release-el7-3.noarch.rpm
sed -i 's/enabled=1/enabled=0/' /etc/yum.repos.d/mysql-community.repo
yum --enablerepo=mysql80-community install mysql-server
systemctl start mysqld.service
6) Login to MySQL and configure database for Airflow
mysql> CREATE DATABASE airflow CHARACTER SET utf8 COLLATE utf8_unicode_ci;
mysql> CREATE user 'airflow'#'localhost' identified by 'Airflow123';
mysql> GRANT ALL privileges on *.* to 'airflow'#'localhost';
7) Update Airflow configuration file (~/airflow/airflow.cfg)
sql_alchemy_conn = mysql://airflow:Airflow123#localhost:3306/airflow
executor = CeleryExecutor
8) Initialize Airflow
airflow initdb
I have a complex Dockerfile which install much more than just mysql-server so I cannot start from an existing mysql container.
When removing all the extra-stuff I get this Dockerfile
FROM ubuntu:latest
ENV MYSQL_ROOT_PASSWORD=root
ENV MYSQL_ROOT_USER=root
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update
RUN apt-get install -y apt-utils
RUN apt-get install -y mysql-server
RUN usermod -d /var/lib/mysql/ mysql
RUN service mysql start
Unfortunately, mysql does not want to start:
---> 57a66bd64c2c
Step 8/9 : RUN usermod -d /var/lib/mysql/ mysql
---> Running in 596df248c2e4
---> ee78442bcc56
Step 9/9 : RUN service mysql start
---> Running in 0d9e5803cf33
* Starting MySQL database server mysqld
...fail!
The command '/bin/sh -c service mysql start' returned a non-zero code: 1
What is my mistake?
Looks like you've removed the most important parts of your docker file. Here is the Official MySQL repo Docker file.
FROM oraclelinux:7-slim
ENV PACKAGE_URL https://repo.mysql.com/yum/mysql-8.0-community/docker/x86_64/mysql-community-server-minimal-8.0.2-0.1.dmr.el7.x86_64.rpm
# Install server
RUN rpmkeys --import http://repo.mysql.com/RPM-GPG-KEY-mysql \
&& yum install -y $PACKAGE_URL \
&& yum install -y libpwquality \
&& rm -rf /var/cache/yum/*
RUN mkdir /docker-entrypoint-initdb.d
VOLUME /var/lib/mysql
COPY docker-entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
EXPOSE 3306 33060
CMD ["mysqld"]
You need to include a proper source with correct version to pull the image from.
and expose right ports, separate out volumes for MySQL to run. your container maybe failing due to any of this. I'd say remove the MySQL part out of your dockerfile and run the rest of the container.
Use the official mySQL image and install it in separate container. and then you can connect the Database with other apps.
I created a Dockerfile to create an image to run web based application. When I running that file and it tries to collect mysqlclient==1.3.7 ,following error is occured.
"mysql_config raise EnvironmentError("%s not found" %(mysql_config.path,))
EnvironmentError: mysql_config not found
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-ME9Fq7/mysqlclient/
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command. "
This is Dockerfile
############################################################
# Dockerfile to build Python WSGI Application Containers
# Based on Ubuntu
############################################################
# Set the base image to Ubuntu
FROM ubuntu:latest
# File Author / Maintainer
MAINTAINER Brian
# Add the application resources URL
#RUN echo "deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -sc) main universe" >> /etc/apt/sources.list
# Update the sources list
RUN apt-get update -y;
# Install basic applications
RUN apt-get install -y tar git curl nano wget dialog net-tools build-essential
# Install Python and Basic Python Tools
RUN apt-get install -y python python3-dev python-distribute python-pip libssl-dev
# Copy the application folder inside the container
ADD /WebApp /WebApp
#RUN pip install mysql
# Get pip to download and install requirements:
RUN pip install -r /WebApp/requirements.txt
# Expose ports
EXPOSE 80
# Set the default directory where CMD will execute
WORKDIR /WebApp
# Set the default command to execute
# when creating a new container
# i.e. using CherryPy to serve the application
CMD ./start.sh
How can I solve this problem?
I believe that you are trying to install the mysql driver for Python, but you don't have MySQL installed. Try to install MySQL before installing mysql driver:
RUN apt-get install mysql-server
Hope it helps!
I just started with Packer, and want to build AMI and Docker images.
Problem is - that Docker's Ubuntu image have no sudo, thus - build failed with an error:
docker: /tmp/script_2085.sh: 3: /tmp/script_2085.sh: sudo: not found
Is there any way to add some "conditions" for a shell provisioner to run cmds with or without sudo, depending on builder?
Currently - my template have:
...
"provisioners": [{
"type": "shell",
"inline": [
"sleep 30",
"sudo apt-get update",
"sudo apt-get install -y nginx"
]
}]
...
Add something like:
if which sudo; then sudo apt-get update && sudo apt-get install -y nginx; else apt-get update && apt-get install -y nginx; fi
Looks too weird solution... I'm sure - Packer has facility to do it in a better way.
Why don't you just install sudo in the docker box.
You could do this one of two ways:
Running a script to install sudo, only on your docker image
You can use the only parameter in the provisioner to install sudo on an image:
...
"provisioners": [{
"type": "shell",
"inline": [
"apt-get install sudo"
],
"only": ["docker"]
},
... other provisioners here ...
]
...
More info in the packer docs, here, under "Run On Specific Builds".
Installing sudo at the beginning of your provision script
At the beginning of your first provision script, add the command to install sudo if it is not already installed. Use the following (taken from here):
dpkg -l | grep -qw sudo || apt-get install sudo
to check the list of installed packages and install if sudo is not present.
Personally, I'd be inclined to go for the first option, since it explicitly states that something should be done only on certain boxes. No room for confusion.
Parting notes
Bear in mind that your provisioners will run in the order you define them in your packer file. So make sure you install sudo in the first one.
You may also need to run apt-get update before you do the install.
EDIT: My goal is to be able to emit metrics from my spring-boot application and have them sent to a Graphite server. For that I am trying to set up statsd. If you can suggest a cleaner approach, that would be better.
I have a Beanstalk application which requires statsd to run as a background process. I was able to specify commands and packages through ebextensions config file as follows:
packages:
yum:
git: []
commands:
01_nodejs_install:
command: sudo yum -y install nodejs npm --enablerepo=epel
ignoreErrors: true
02_mkdir_statsd:
command: mkdir /home/ec2-user/statsd
03_fetch_statsd:
command: git clone https://github.com/etsy/statsd.git /home/ec2-user/statsd
ignoreErrors: true
04_run_statsd:
command: node stats.js exampleConfig.js
cwd: /home/ec2-user/statsd
When I try to deploy the application to a new environment, the EC2 node never comes up fully. I logged in to check what might be going on and noticed in /var/log/cfn-init.log that 01_nodejs_install, 02_mkdir_statsd and 03_fetch_statsd were executed successfully. So I guess the system was stuck on the fourth command (04_run_statsd).
2016-05-24 01:25:09,769 [INFO] Yum installed [u'git']
2016-05-24 01:25:37,751 [INFO] Command 01_nodejs_install succeeded
2016-05-24 01:25:37,755 [INFO] Command 02_mkdir_statsd succeeded
2016-05-24 01:25:38,700 [INFO] Command 03_fetch_statsd succeeded
cfn-init.log (END)
I need help with the following:
If there is a better way to install and run statsd while instantiating an environment, I would appreciate if you could provide details on that approach. This current scheme seems hacky.
If this is the approach I need to stick with, how can I run the fourth command so that statsd can be run as a background process?
Tried a few things and found that the following ebextensions configs work:
packages:
yum:
git: []
commands:
01_nodejs_install:
command: sudo yum -y install nodejs npm --enablerepo=epel
ignoreErrors: true
02_mkdir_statsd:
command: mkdir /home/ec2-user/statsd
03_fetch_statsd:
command: git clone https://github.com/etsy/statsd.git /home/ec2-user/statsd
ignoreErrors: true
04_change_config:
command: cat exampleConfig.js | sed 's/2003/<graphite server port>/g' | sed 's/graphite.example.com/my.graphite.server.hostname/g' > config.js
cwd: /home/ec2-user/statsd
05_run_statsd:
command: setsid node stats.js config.js >/dev/null 2>&1 < /dev/null &
cwd: /home/ec2-user/statsd
Note that I added another command (04_change_config) so that I may configure my own Graphite server and port in statsd configs. This change is not needed to address the original question, though.
The actual run command uses setsid to run the command as a daemon.