Bitbucket pipeline import mysql schema - mysql

I'm trying to import database schema to mysql service through following statment
mysql -h 127.0.0.1 -u $DB_USERNAME -p$DB_PASSWORD $DB_DATABASE < DB_Schema.sql
and it return mysql: not found. I have even tried the following command
docker exec -i mysql mysql -h 127.0.0.1 -u $DB_USERNAME -p$DB_PASSWORD $DB_DATABASE < DB_Schema.sql
Even though received error + docker exec -i mysql mysql --user=$DB_USERNAME --password=$DB_PASSWORD 5i < DB_Schema.sql
Error: No such container: mysql
What would be the best way to use mysql so that I can import a stance of DB into it for testing purpose and how?
Please find the .yml file below.
# This is a sample build configuration for PHP.
# Check our guides at https://confluence.atlassian.com/x/e8YWN for more examples.
# Only use spaces to indent your .yml configuration.
# -----
# Specify a docker image from Docker Hub as your build environment.
# All of your pipeline scripts will be executed within this docker image.
image: php:8.0-fpm-alpine
# All of your Pipelines will be defined in the `pipelines` section.
# You can have any number of Pipelines, but they must all have unique
# names. The default Pipeline is simply named `default`.
pipelines:
default:
# Each Pipeline consists of one or more steps which each execute
# sequentially in separate docker containers.
# name: optional name for this step
# script: the commands you wish to execute in this step, in order
- parallel:
- step:
name: Installing Dependancies and Composer
caches:
- composer
script:
# Your Pipeline automatically contains a copy of your code in its working
# directory; however, the docker image may not be preconfigured with all
# of the PHP/Laravel extensions your project requires. You may need to install
# them yourself, as shown below.
- apt-get update && apt-get install -qy git curl libmcrypt-dev unzip libzip-dev libpng-dev zip git gnupg gnupg2 php-mysql
- docker-php-ext-configure gd --enable-gd --with-freetype --with-jpeg --with-webp && \
- docker-php-ext-install gd && \
- docker-php-ext-install exif && \
- docker-php-ext-install zip && \
- docker-php-ext-install pdo pdo_mysql
- rm -rf ./vendor
- curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
- composer install --ignore-platform-reqs
- composer dump-autoload
# Here we create link between the .env.pipelines file and the .env file
# so that our database can retreieve all the variables inside .env.pipelines
- ln -f -s .env.pipelines .env
artifacts:
- vendor/**
- step:
name: Installing and Running npm
image: node:16
caches:
- node
script:
- npm install -g grunt-cli
- npm install
- npm run dev
artifacts:
- node_modules/**
- step:
name: Running Test
deployment: local
script:
# Start up the php server so that we can test against it
- php artisan serve &
# # Give the server some time to start
- sleep 5
# - php artisan migrate
- docker ps
- docker container ls
- mysql -h 127.0.0.1 -u $DB_USERNAME -p$DB_PASSWORD $DB_DATABASE < DB_Schema.sql
# - docker exec -i mysql mysql -h 127.0.0.1 -u $DB_USERNAME -p$DB_PASSWORD -e "SHOW DATABASES"
- php artisan optimize
- php artisan test
services:
- mysql
- docker
# You might want to create and access a service (like a database) as part
# of your Pipeline workflow. You can do so by defining it as a service here.
definitions:
services:
mysql:
image: mysql:latest
environment:
MYSQL_DATABASE: $DB_DATABASE
MYSQL_USER: $DB_USERNAME
MYSQL_PASSWORD: $DB_PASSWORD
MYSQL_ROOT_PASSWORD: $DB_PASSWORD
SERVICE_TAGS: mysql
SERVICE_NAME: mysql

You cannot install/update/change you main image in the first step for them to be there in the last step. Make your custom Docker image with all those installations, which will make it faster to run the pipeline and will let you use other tools you need in your pipeline.

I prefer to use the "mysql" client outside Docker and have it reach into the Docker container based on the port mapping set up. Then, conceptually, it is like reading to a "mysqld" server on a separate "server".
LOAD DATA INFILE and INSERT, including use of mysql ... < dump.sql works fine.

Related

How to setup connection to MySql database with gitlab CI/CD

I'm trying to set up automatic testing of django project using CI/CD gitlab. The problem is, I can't connect to the Mysql database in any way.
gitlab-ci.yml
services:
- mysql:5.7
variables:
MYSQL_DATABASE: "db_name"
MYSQL_ROOT_PASSWORD: "dbpass"
MYSQL_USER: "username"
MYSQL_PASSWORD: "dbpass"
stages:
- test
test:
stage: test
before_script:
- apt update -qy && apt-get install -qqy --no-install-recommends default-mysql-client
- mysql --user=$MYSQL_USER --password=$MYSQL_PASSWORD --database=$MYSQL_DATABASE --host=$MYSQL_HOST --execute="SHOW DATABASES; ALTER USER '$MYSQL_USER'#'%' IDENTIFIED WITH mysql_native_password BY '$MYSQL_PASSWORD'"
script:
- apt update -qy
- apt install python3 python3-pip virtualenvwrapper -qy
- virtualenv --python=python3 venv/
- source venv/bin/activate
- pwd
- pip install -r requirement.txt
- python manage.py test apps
With this file configuration, I get error
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
What have I tried to do
add to mysql script tcp connection unstead socket
mysql --protocol=TCP --user=$MYSQL_USER --password=$MYSQL_PASSWORD --database=$MYSQL_DATABASE --host=$MYSQL_HOST --execute="SHOW DATABASES; ALTER USER '$MYSQL_USER'#'%' IDENTIFIED WITH mysql_native_password BY '$MYSQL_PASSWORD'"
And in this case I got
ERROR 2002 (HY000): Can't connect to MySQL server on 'localhost' (99)
How do I set up properly?
There can be multiple reasons for your issue:
Incorrect MySQL version.
Solution: Use mysql:5.7 instead of mysql:latest
MySQL host is missing.
Solution: add MYSQL_HOST in the variables with the hostname of the MySQL server. (Should be mysql when using mysql:5.7 in services key)
Django uses different credentials of DB.
Solution: check that the credentials in the variables section of your .gitlab-ci.yml and compare against Django's settings.py. They should be the same.
MySQL client not installed.
Solution: install the mysql-client in the script section and check if it is able to connect.
Here is a sample script that installs MySQL client and connects to the database in a debian based image (or a python:latest image):
script:
- apt-get update && apt-get install -y git curl libmcrypt-dev default-mysql-
- mysql --version
- sleep 20
- echo "SHOW tables;"| mysql -u root -p"$MYSQL_ROOT_PASSWORD" -h "${MYSQL_HOST}" "${MYSQL_DATABASE}"
Here is a complete and valid example of using MySQL 5.7 as a service and a python image with mysql-client installed successfully connecting to the MySQL database:
stages:
- test
variables:
MYSQL_DATABASE: "db_name"
MYSQL_ROOT_PASSWORD: "dbpass"
MYSQL_USER: "username"
MYSQL_PASSWORD: "dbpass"
MYSQL_HOST: mysql
test:
image: python:latest
stage: test
services:
- mysql:5.7
script:
- apt-get update && apt-get install -y git curl libmcrypt-dev default-mysql-client
- mysql --version
- sleep 20
- echo "SHOW tables;" | mysql -u root -p"$MYSQL_ROOT_PASSWORD" -h "${MYSQL_HOST}" "${MYSQL_DATABASE}"
- echo "Database host is '${MYSQL_HOST}'"
You need to use service name as database hostname. In this case MYSQL_HOST should be mysql.
You can see example on Gitlab page and read about how services are linked to the job
I see there is an accepted answer but with mysql 8.0 and python3:buster some things broke. The Python Debian images ship with mariadb and it is not easy to set up the standard mysql-client packages, resulting in the error:
"django.db.utils.OperationalError: 2059, “Authentication plugin..."
I got a working YAML below, using Ubuntu as the base image and mysql 8.0 as a service. You could either use the root user in both the .gitlab-ci and the test_settings or give the MYSQL user the privileges to create new databases and alter existing ones.
The initial MYSQL_DB _USER and _PASS variables can be set in Gitlab under Settings -> CI/CD -> Variables.
.gitlab-ci.yml:
variables:
# "When using a service (e.g. mysql) in the GitLab CI that needs environtment variables
# to run, only variables defined in .gitlab-ci.yml are passed to the service and
# variables defined in GitLab GUI are unavailable."
# https://gitlab.com/gitlab-org/gitlab/-/issues/30178
# DJANGO_CONFIG: "test"
MYSQL_DATABASE: $MYSQL_DB
MYSQL_ROOT_PASSWORD: $MYSQL_PASS
MYSQL_USER: $MYSQL_USER
MYSQL_PASSWORD: $MYSQL_PASS
# -- In your django settings file for the test environment you could put:
# DATABASES = {
# 'default': {
# 'ENGINE': 'django.db.backends.mysql',
# 'NAME': os.environ.get('MYSQL_DATABASE'),
# 'USER': os.environ.get('MYSQL_USER'),
# 'PASSWORD': os.environ.get('MYSQL_PASSWORD'),
# 'HOST': 'mysql',
# 'PORT': '3306',
# 'CONN_MAX_AGE':60,
# },
# }
# -- You could us '--settings' to specify a custom settings file on the command line
# -- below or use an environment variable to trigger an include in your settings:
# if os.environ.get('DJANGO_CONFIG')=='test':
# from .settings_test import * # or specific overrides
#
default:
image: ubuntu:20.04
# -- Pick zero or more services to be used on all builds.
# -- Only needed when using a docker container to run your tests in.
# -- Check out: http://docs.gitlab.com/ee/ci/docker/using_docker_images.html#what-is-a-service
services:
- mysql:8.0
# This folder is cached between builds
# http://docs.gitlab.com/ee/ci/yaml/README.html#cache
# cache:
# paths:
# - ~/.cache/pip/
before_script:
- echo -e "Using Database $MYSQL_DB with $MYSQL_USER"
- apt --assume-yes update
- apt --assume-yes install apt-utils
- apt --assume-yes install net-tools python3.8 python3-pip mysql-client libmysqlclient-dev
# - apt --assume-yes upgrade
- pip3 install -r requirements.txt
djangotests:
script:
# -- The MYSQL user gets only permissions for MYSQL_DB and therefor cant create a test_db.
- echo "GRANT ALL on *.* to '${MYSQL_USER}';"| mysql -u root --password="${MYSQL_ROOT_PASSWORD}" -h mysql
# -- use python3 explicitly. see https://wiki.ubuntu.com/Python/3
- python3 manage.py test
migrations:
script:
- python3 manage.py makemigrations
- python3 manage.py makemigrations myapp
- python3 manage.py migrate
- python3 manage.py check

Can't connect a go app to mysql (both inside a gitlab runner)

The go app simply inserts a harcoded value in a mysql table and spits it back out. This is done using this database driver. It works fine on linux servers, but during gitlab's CI this returns:
dial tcp 127.0.0.1:3306: connect: connection refused
This is the .gitlab-ci.yml
image: mysql
services:
- mysql:latest
variables:
MYSQL_DATABASE: storage
MYSQL_ROOT_PASSWORD: root
MYSQL_HOST: mysql
MYSQL_USER: root
MYSQL_TRANSAPORT: tcp
MYSQL_ADDRESS: "127.0.0.1:3306"
job:
script:
- apt-get update -qq && apt-get install -qq curl && apt-get install -qq git
- echo "SHOW GLOBAL VARIABLES LIKE 'PORT';" | mysql --user="$MYSQL_USER" --password="$MYSQL_ROOT_PASSWORD" --host="$MYSQL_HOST" "$MYSQL_DATABASE"
- curl -O https://dl.google.com/go/go1.10.1.linux-amd64.tar.gz
- tar -C /usr/local -xzf go1.10.1.linux-amd64.tar.gz
- rm go1.10.1.linux-amd64.tar.gz
- echo "export PATH=\$PATH:/usr/local/go/bin" >> ~/.bashrc
- echo "export GOPATH=\$HOME/go" >> ~/.bashrc
- echo "export PATH=\$PATH:\$GOPATH/bin" >> ~/.bashrc
- source ~/.bashrc
- go get github.com/go-sql-driver/mysql
- go build main.go
- ./main
Is there a standard way to use mysql from golang during CI?

docker pentaho mysql driver issue

I'm using Docker on Windows 10 to create a pentaho and mysql image that will run as containers on a network I define with docker network create.
The intention is that (as a first step) I will run a .KTR file with pan.sh that will read DB connection parameters from a .csv file and place these into the environment;
Get the DB connection parameters
Next a second .KTR checks to see if the DB exists using the above environment params;
Check DB exists
The problem is when I "Spin up" my project with docker-compose, step two fails with a driver not found issue. I've placed the drivers I require in the pentaho container's lib dir but I'm guessing this is not correct?
Ultimately, the intention is for a transformation to occur where data read from an OpenEdge DB is process via a series of steps in pentaho and written to the mysql DB.
Here's the supporting files;
Dockerfile;
FROM java:8-jre
MAINTAINER M Beynon
# Set required environment vars
ENV PDI_RELEASE=7.1 \
PDI_VERSION=7.1.0.0-12 \
CARTE_PORT=8181 \
PENTAHO_JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \
PENTAHO_HOME=/home/pentaho
# Create user
RUN mkdir ${PENTAHO_HOME} && \
groupadd -r pentaho && \
useradd -s /bin/bash -d ${PENTAHO_HOME} -r -g pentaho pentaho && \
chown pentaho:pentaho ${PENTAHO_HOME}
# Add files
RUN mkdir $PENTAHO_HOME/docker-entrypoint.d
COPY docker-entrypoint.sh $PENTAHO_HOME/scripts/
RUN chown -R pentaho:pentaho $PENTAHO_HOME
RUN apt-get update && apt-get install -y libwebkitgtk-1.0-0
RUN apt-get update && apt-get install -y dos2unix
RUN dos2unix $PENTAHO_HOME/scripts/docker-entrypoint.sh && apt-get --purge remove -y dos2unix && rm -rf /var/lib/apt/lists/*
# Switch to the pentaho user
USER pentaho
# Download PDI
RUN /usr/bin/wget \
--progress=dot:giga \
http://downloads.sourceforge.net/project/pentaho/Data%20Integration/${PDI_RELEASE}/pdi-ce-${PDI_VERSION}.zip \
-O /tmp/pdi-ce-${PDI_VERSION}.zip && \
/usr/bin/unzip -q /tmp/pdi-ce-${PDI_VERSION}.zip -d $PENTAHO_HOME && \
rm /tmp/pdi-ce-${PDI_VERSION}.zip
ENV KETTLE_HOME=$PENTAHO_HOME/data-integration \
PATH=$KETTLE_HOME:$PATH
WORKDIR $KETTLE_HOME
ENTRYPOINT ["../scripts/docker-entrypoint.sh"]
The entrypoint;
#!/bin/bash
# based on https://github.com/aloysius-lim/docker-pentaho-di/blob/master/docker/Dockerfile
#exit script if any command fails (non-zero value)
set -e
cd resources
cp mysql-connector-java-5.1.42-bin.jar ../lib/
cp PROGRESS_DATADIRECT_JDBC_OE_ALL.jar ../lib
cd ../
echo 'Drivers copied!'
echo ''
echo 'Running transformation!'
#run a transformation (get db credentials)
./pan.sh -file=resources/Read-DBs.ktr
#run a transformation (does the db exist)
./pan.sh -file=resources/GoldBi-Exists.ktr
#redirect input variables
exec "$#"
The Docker compose file;
version: "2"
services:
db:
image: mysql:latest
networks:
- my-pdi-network
environment:
- MYSQL_ROOT_PASSWORD=tbitter
- MYSQL_DATABASE=mysql-db
ports:
- "3307:3306"
volumes:
- ./goldbi:/var/lib/mysql
pdi:
image: my-pdi-image:latest
networks:
- my-pdi-network
volumes:
- C:\Docker-Pentaho\resource:/home/pentaho/data-integration/resources
networks:
my-pdi-network:
The error coming from pentaho;
2017/05/30 15:28:56 - Table exists.0 - Error occurred while trying to connect to the database
2017/05/30 15:28:56 - Table exists.0 -
2017/05/30 15:28:56 - Table exists.0 - Error connecting to database: (using class org.gjt.mm.mysql.Driver)
2017/05/30 15:28:56 - Table exists.0 - Communications link failure
2017/05/30 15:28:56 - Table exists.0 -
2017/05/30 15:28:56 - Table exists.0 - The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
Many Thanks.
P.S. Does anyone know how to prevent build from rebuilding everything, even if it's only a small change to the dockerfile or entrypoint file?
I seem to have found the solution;
There were two issues. The first seems to be that the ENV vars set in the first transformation are not being utilised in the second transformation. The second was that the host name was wrong in the second transformation (DB-Exists). It should have been 'db' which is the name specified for the container in the docker-compose file. As the contains are both running on a custom network I'd specified, they can automatically 'talk' to each other via their service names...reverse DNS?

Gitlab CI Runner | Custom mysql conf

I'm trying to set my own mysql conf for Gitlab CI Runner.
I found in the documentation how to set my own php.ini :
before_script:
- cp ci/php.ini /usr/local/etc/php/conf.d/test.ini
I didn't find informations about how setting my.cnf, I tried :
before_script:
- cp ci/my.cnf /usr/local/etc/mysql/conf.d/my.cnf
But /usr/local/etc/mysql/ doesn't exist in the generated environment.
This is all my gitlab.ci :
services:
- mysql:latest
variables:
# Configure mysql environment variables
WITH_XDEBUG: "1"
MYSQL_ROOT_PASSWORD: password
MYSQL_DATABASE: symfony
cache:
paths:
- vendor/
before_script:
# Install dependencies
- bash ci/docker_install.sh > /dev/null
- cp ci/php.ini /usr/local/etc/php/conf.d/test.ini
- cp ci/my.cnf /usr/local/etc/mysql/conf.d/my.cnf
- mv app/config/parameters.gitlab.yml app/config/parameters.yml
- mv app/config/config_test.gitlab.yml app/config/config_test.yml
- composer clear-cache
- composer install
- php bin/console doctrine:schema:update --force
- php bin/console doctrine:fixtures:load
test:
image: php:7.0
stage: test
script:
- echo "Running PHPUnit Tests"
- vendor/bin/phpunit --configuration phpunit.xml.dist --colors --debug --coverage-text
And my docker_install.sh :
#!/bin/bash
# We need to install dependencies only for Docker
[[ ! -e /.dockerenv ]] && [[ ! -e /.dockerinit ]] && exit 0
set -xe
# Install git (the php image doesn't have it) which is required by composer
apt-get update -yqq
apt-get install git -yqq
apt-get install wget -yqq
apt-get install zip unzip -yqq
# Install composer
curl -sS https://getcomposer.org/installer | php
mv composer.phar /usr/local/bin/composer
# Install mysql driver
# Here you can install any other extension that you need
docker-php-ext-install pdo_mysql mbstring
Thanks.
I don't think it is possible to easily mount the config inside of the service container using the Gitlab CI config. It is probably easier to just create a Dockerfile that extends the image that you use for testing and set the the configs you need there.
You can find an example of such a small Dockerfile here, where the author copies a config file to set the encoding to utf8:
https://hub.docker.com/r/dnhsoft/mysql-utf8/dockerfile
Or you can simply append your configs for example like this if you don't want to write a Dockerfile that copies files:
FROM mysql:8.0
RUN echo '[mysqld]' >> /etc/mysql/conf.d/mysql.cnf
RUN echo 'default-authentication-plugin = mysql_native_password' >> /etc/mysql/conf.d/mysql.cnf
RUN echo 'collation-server = utf8mb4_general_ci' >> /etc/mysql/conf.d/mysql.cnf
RUN echo 'character-set-server = utf8mb4' >> /etc/mysql/conf.d/mysql.cnf
Once your Dockerfile has the configs you need you can publish it to Dockerhub under your username. Then in your gitlab-ci.yml you can use it like this:
services:
- {name: 'mydockerusername/mysql:1', alias: 'mysql'}
In your scripts in gitlab-ci.yml you can then connect to your custom database container like this:
script:
- 'mysql -u gitlabci -h mysql --password="gitlabci" -e "SHOW DATABASES"'

how can i create a database on mysql docker service

I'm trying to run moodle phpunit on my gitlab ci server. Using gitlab-ci.yml file i'm creating a container with php 5.6 and mysql service.
# Services
services:
- mysql:latest
before_script:
- mysql -e 'CREATE DATABASE gitlab_ci_test DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_bin;' ;
I'm getting ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) and not sure how to proceed.
Tomasz this is my gitlab-ci.yml file you will need something like this:
# Select image from https://hub.docker.com/r/_/php/
image: php:7.0.0
services:
- mysql:5.7
variables:
# Configure mysql environment variables (https://hub.docker.com/r/_/mysql/)
MYSQL_DATABASE: symfony
MYSQL_ROOT_PASSWORD: qwerty
# Composer stores all downloaded packages in the vendor/ directory.
# Do not use the following if the vendor/ directory is commited to
# your git repository.
cache:
paths:
- vendor/
before_script:
# Install dependencies
- bash ci/docker_install.sh > /dev/null
- cp ci/parameters.yml app/config/parameters.yml
- composer install
test:app:
script:
- phpunit
And this is my docker_install.sh inside ci folder
#!/bin/bash
# We need to install dependencies only for Docker
[[ ! -e /.dockerenv ]] && [[ ! -e /.dockerinit ]] && exit 0
set -xe
# Install git (the php image doesn't have it) which is required by composer
apt-get update -yqq
apt-get install git -yqq
apt-get install wget -yqq
apt-get install zip unzip -yqq
# Install composer
curl -sS https://getcomposer.org/installer | php
mv composer.phar /usr/local/bin/composer
# Install phpunit, the tool that we will use for testing
curl -o /usr/local/bin/phpunit https://phar.phpunit.de/phpunit.phar
chmod +x /usr/local/bin/phpunit
# Install mysql driver
# Here you can install any other extension that you need
docker-php-ext-install pdo pdo_mysql mbstring