Connect with Liquibase (Docker) to a SSH tunneled database - mysql

I constructed an SSH tunnel to a MySQL database (I know that this can be done without a password, but this is not the question).
jfabianmeier#JFM-HP-2018:~$ sshpass -p mySuperPassword ssh -o StrictHostKeyChecking=no -M -S my-ctrl-socket -fNT -L 3306:mysql5:3306 myUserName#alfa3031.alfahosting-server.de
jfabianmeier#JFM-HP-2018:~$ ssh -S my-ctrl-socket -O check myUserName#alfa3031.alfahosting-server.de
Master running (pid=405)
Connecting to MySQL seems to work, at least without SSL (I don't know why, maybe the server only supports old protocols).
jfabianmeier#JFM-HP-2018:~$ mysql -h 127.0.0.1 -P 3306 -u web1444 -pmyDBPassword --ssl-mode=DISABLED
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 140637890
Server version: 5.7.25 MySQL Community Server (GPL)
Copyright (c) 2000, 2022, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> SHOW DATABASES;
+--------------------+
| Database |
+--------------------+
| information_schema |
| usr_web1444_1 |
| usr_web1444_2 |
| usr_web1444_3 |
| usr_web1444_4 |
| usr_web1444_5 |
+--------------------+
6 rows in set (0.26 sec)
mysql> exit
Bye
Liquibase does not work, though. I tried below Docker command, but cannot figure out what is wrong about it.
jfabianmeier#JFM-HP-2018:~$ docker run -e INSTALL_MYSQL=true --rm -v $(pwd):$(pwd) liquibase/liquibase:4.15 --url=jdbc:mysql://localhost:3306/usr_web1444_4?autoReconnect=true&useSSL=false --changeLogFile=~/changelogs/changelog.sql --username=web144 --password=myDBPassword update
[1] 408
-bash: --changeLogFile=~/changelogs/changelog.sql: No such file or directory
jfabianmeier#JFM-HP-2018:~$ Checksum verified. Installing mysql-connector-java-8.0.30.jar to /liquibase/lib/
mysql-connector-java-8.0.30.jar successfully installed in classpath.
####################################################
## _ _ _ _ ##
## | | (_) (_) | ##
## | | _ __ _ _ _ _| |__ __ _ ___ ___ ##
## | | | |/ _` | | | | | '_ \ / _` / __|/ _ \ ##
## | |___| | (_| | |_| | | |_) | (_| \__ \ __/ ##
## \_____/_|\__, |\__,_|_|_.__/ \__,_|___/\___| ##
## | | ##
## |_| ##
## ##
## Get documentation at docs.liquibase.com ##
## Get certified courses at learn.liquibase.com ##
## Free schema change activity reports at ##
## https://hub.liquibase.com ##
## ##
####################################################
Starting Liquibase at 09:56:38 (version 4.15.0 #4001 built at 2022-08-05 16:17+0000)
Liquibase Version: 4.15.0
Liquibase Community 4.15.0 by Liquibase
Missing required subcommand
Usage: liquibase [GLOBAL OPTIONS] [COMMAND] [COMMAND OPTIONS]
Command-specific help: "liquibase <command-name> --help"
Global Options
....
I would like to know whether my Liquibase command is just wrong, or the URL is wrong, or if the issue might be with the SSL problem above. I honestly do not understand the error message and was not able to find something helpful through Google.

If you are able to connect with the mysql client probably the tunnel is correctly configured.
The error seems to be motivated because Liquibase doesn't understand the command provided when invoking Docker from the command line:
docker run -e INSTALL_MYSQL=true --rm -v $(pwd):$(pwd) liquibase/liquibase:4.15 --url=jdbc:mysql://localhost:3306/usr_web1444_4?autoReconnect=true&useSSL=false --changeLogFile=~/changelogs/changelog.sql --username=web144 --password=myDBPassword update
As you can see, it is printing by console the error and the typical usage help:
Missing required subcommand
Usage: liquibase [GLOBAL OPTIONS] [COMMAND] [COMMAND OPTIONS]
Command-specific help: "liquibase <command-name> --help"
I tested the code and it seems to be motivated by the url argument.
Please, try providing that value enclosed within quotes:
docker run -e INSTALL_MYSQL=true --rm -v $(pwd):$(pwd) liquibase/liquibase:4.15 --url="jdbc:mysql://localhost:3306/usr_web1444_4?autoReconnect=true&useSSL=false" --changeLogFile=~/changelogs/changelog.sql --username=web144 --password=myDBPassword update
In addition, bash is reporting that Liquibase is unable to find your change logs:
-bash: --changeLogFile=~/changelogs/changelog.sql: No such file or directory
The Liquibase official Docker image documentation mentions the following when describing how to provide the necessary change logs:
The docker image has a /liquibase/changelog volume in which the directory
containing the root of your changelog tree can be mounted. Your
-- changeLogFile argument should list paths relative to this.
Following that advice, please, try providing the following docker command for running your migration (assuming your current directory contains changelogs/changelog.sql):
docker run -e INSTALL_MYSQL=true --rm -v $(pwd):/liquibase/changelog liquibase/liquibase:4.15 --url="jdbc:mysql://localhost:3306/usr_web1444_4?autoReconnect=true&useSSL=false" --changeLogFile=changelog/changelogs/changelog.sql --username=web144 --password=myDBPassword update
Please, note again the use of double quotes to enclose the url parameter.
If your tunnel is not working properly Liquibase will report something like the following:
####################################################
## _ _ _ _ ##
## | | (_) (_) | ##
## | | _ __ _ _ _ _| |__ __ _ ___ ___ ##
## | | | |/ _` | | | | | '_ \ / _` / __|/ _ \ ##
## | |___| | (_| | |_| | | |_) | (_| \__ \ __/ ##
## \_____/_|\__, |\__,_|_|_.__/ \__,_|___/\___| ##
## | | ##
## |_| ##
## ##
## Get documentation at docs.liquibase.com ##
## Get certified courses at learn.liquibase.com ##
## Free schema change activity reports at ##
## https://hub.liquibase.com ##
## ##
####################################################
Starting Liquibase at 17:22:56 (version 4.15.0 #4001 built at 2022-08-05 16:17+0000)
Liquibase Version: 4.15.0
Liquibase Community 4.15.0 by Liquibase
Unexpected error running Liquibase: Connection could not be created to jdbc:mysql://localhost:3306/usr_web1444_4?autoReconnect=true&useSSL=false with driver com.mysql.cj.jdbc.Driver. Could not create connection to database server. Attempted reconnect 3 times. Giving up.
- Caused by: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
- Caused by: Connection refused
For more information, please use the --log-level flag

Related

Issues with helm install Orion Context Broker

I'm trying to install FIWARE Orion on AKS using your Helm chart. I installed MongoDB using
helm repo add azure-marketplace https://marketplace.azurecr.io/helm/v1/repo
helm install my-release azure-marketplace/mongodb
Consequently I configured the MongoDB in values.yaml as follows:
## database configuration
db:
# -- configuration of the mongo-db hosts. if multiple hosts are inserted, its assumed that mongo is running as a replica set
hosts: [my-release-mongodb]
# - my-release-mongodb
# -- the db to use. if running in multiservice mode, its used as a prefix.
name: orion
# -- Database authentication (not needed if MongoDB doesn't use --auth)
auth:
# --user for connecting mongo
user: root
# -- password to be used on mongo
password: mypasswd
# -- the MongoDB authentication mechanism to use in the case user and password is set
#mech: SCRAM-SHA-1
I use the command : helm install test orion
As I see this error in the pod logging I suppose something is wrong;
kubectl logs test-orion-7dfcc9c7fb-8vbgw
time=2021-05-28T19:50:29.737Z | lvl=ERROR | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=mongocContextCachePersist.cpp[59]:mongocContextCachePersist | msg=Database Error (persisting context: command insert requires authentication)
Can you help me with this please?
Kind regards,
Johan,
you should assure that mongo-db is actually available at "my-release-mongodb:27017", you can use "kubectl get services" for that. Beside that, assure that "root:mypasswd" are actually the credentials setup at mongodb.

openstack compute service list --service nova-compute empty

After the installation of nova-compute on compute node, it failed to start and this command from the controller node return an empty result
openstack compute service list --service nova-compute
And the nova-compute.log file contain these two messages:
018-11-19 12:06:05.446 986 INFO os_vif [-] Loaded VIF plugins: ovs, linux_bridge
2018-11-19 12:30:13.784 1140 INFO os_vif [-] Loaded VIF plugins: ovs, linux_bridge
openstack compute service list :
return three service components for the controller with a down state
+----+------------------+------------+----------+---------+-------+----------------------------+
| ID | Binary | Host | Zone | Status | State | Updated At
+----+------------------+------------+----------+---------+-------+----------------------------+
| 2 | nova-conductor | Controller | internal | enabled | down | 2018-11-17T17:32:48.000000 |
| 4 | nova-scheduler | Controller | internal | enabled | down | 2018-11-17T17:32:49.000000 |
| 5 | nova-consoleauth | Controller | internal | enabled | down | None
+----+------------------+------------+----------+---------+-------+----------------------------+
service nova-compute status :
Active
How can i resolve these problems ?
This is because you might have missed to create the databases for nova_cell0.
# mysql -u root -p
MariaDB [(none)]> CREATE DATABASE nova_cell0;
MariaDB [(none)]> GRANT ALL PRIVILEGES ON nova_cell0.* TO 'nova'#'localhost' \ IDENTIFIED BY 'NOVA_DBPASS';
MariaDB [(none)]> GRANT ALL PRIVILEGES ON nova_cell0.* TO 'nova'#'%' \ IDENTIFIED BY 'NOVA_DBPASS';
#su -s /bin/sh -c "nova-manage cell_v2 map_cell0" nova
# su -s /bin/sh -c "nova-manage cell_v2 create_cell --name=cell1 --verbose" nova
109e1d4b-536a-40d0-83c6-5f121b82b650
# su -s /bin/sh -c "nova-manage db sync" nova
# nova-manage cell_v2 list_cells
#su -s /bin/sh -c "nova-manage api_db sync" nova
make sure in /etc/nova/nova.conf in compute node you have added following configuration:
[DEFAULT]
enabled_apis = osapi_compute,metadata
transport_url = rabbit://openstack:RABBIT_PASS#controller
Then restart compute services.
the try the command openstack compute service list.
this solution also holds good when openstack compute service list is empty or nova hypervisor list is empty.

How to use nc for transmit data in one command or in a shell script?

For remote backup,I want to use nc utility to transmit data for a streaming backup with xtrabackup.As Follows,
nc -l 1025 > /data/backups/backup.tar &
ssh root#192.168.0.110 "innobackupex --stream=tar ./ | bzip2 - | nc 192.168.0.100 1025"
When I run these commands seperately,it ended successfully.
#!/bin/bash
nc -l 1025 > /data/backups/backup.tar &
ssh root#192.168.0.110 "innobackupex --stream=tar ./ | bzip2 - | nc 192.168.0.100 1025"
But when I write them together in a shell script and execute the script, there came across the following errors,

How to connect to MySQL running on Docker from the host machine

I have already googled on this subject and found few threads. Based on these threads I have followed the following steps. But I am facing a problem.
Basically, I want to create a docker image for mysql and then connect to it from my host machine (Mac OS X).
Based on this post , I have to share the mysql unix socket with the host. towards this I have done the following steps
1. Start docker quick terminal
2. docker run --name mysql -e MYSQL_ROOT_PASSWORD=password -d mysql/mysql-server:latest
3. docker exec -it mysql bash
4. mysql -uroot -p
5. create database MyDB;
6. GRANT ALL PRIVILEGES ON *.* TO 'root'#'%' IDENTIFIED BY 'password';
7. exit;
8. mkdir /Users/abhi/host
9. docker run -it -v /host:/shared mysql/mysql-server:latest
Now I get the error
MacBook-Pro:~$ docker run -it -v /Users/abhi/host:/shared mysql/mysql-server
error: database is uninitialized and password option is not specified
You need to specify one of MYSQL_ROOT_PASSWORD, MYSQL_ALLOW_EMPTY_PASSWORD and MYSQL_RANDOM_ROOT_PASSWORD
But you see that I have provided the password and initialized my database.
All I want is that from my host machine, I can connect to the mysql database running inside docker.
EDIT:: ----- solution which worked ------
Thanks RICO. Finally the steps which worked for me are
1. Start docker quick terminal
2. docker run --name mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=password -d mysql/mysql-server:latest
3. docker exec -it mysql bash
4. mysql -uroot -p
5. create database MyDB;
or:
CREATE USER 'root'#'%' IDENTIFIED BY 'root';
GRANT ALL PRIVILEGES ON *.* TO 'root'#'%' WITH GRANT OPTION;
6. GRANT ALL PRIVILEGES ON *.* TO 'root'#'%' IDENTIFIED BY 'password';
7. exit;
8. docker-machine env default
Use the IP address obtained in step 8. port is 3306, user is root, password is password, database is MyDB.
Connection is successful!
So you basically you need to expose the mysql port to your host:
docker run --name mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=password -d mysql/mysql-server:latest
Then you can access from your host using the mysql command line:
mysql -h127.0.0.1 -ppassword -uroot
Not sure why you are trying to run another container to connect (perhaps you meant linking two containers)
If you are using Mac (or Windows) with docker-machine you want to connect to the IP address of your docker-machine VM. For example:
$ docker-machine ssh default
## .
## ## ## ==
## ## ## ## ## ===
/"""""""""""""""""\___/ ===
~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ / ===- ~~~
\______ o __/
\ \ __/
\____\_______/
_ _ ____ _ _
| |__ ___ ___ | |_|___ \ __| | ___ ___| | _____ _ __
| '_ \ / _ \ / _ \| __| __) / _` |/ _ \ / __| |/ / _ \ '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__| < __/ |
|_.__/ \___/ \___/ \__|_____\__,_|\___/ \___|_|\_\___|_|
Boot2Docker version 1.9.0, build master : 16e4a2a - Tue Nov 3 19:49:22 UTC 2015
Docker version 1.9.0, build 76d6bc9
docker#default:~$ ifconfig eth1
eth1 Link encap:Ethernet HWaddr 08:00:27:E6:C7:20
inet addr:192.168.99.100 Bcast:192.168.99.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fee6:c720/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:18827 errors:0 dropped:0 overruns:0 frame:0
TX packets:10280 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1791527 (1.7 MiB) TX bytes:2242596 (2.1 MiB)
Then connect to:
mysql -h192.168.99.100 -ppassword -uroot
docker run -e MYSQL_ROOT_PASSWORD=pass --name sql-db -p 3306:3306 mysql
docker exec -it sql-db bash
mysql -u root -p

docker out of disk space

I'm hainvg trouble with docker and volume size. I'm running docker-machine with
three containers. The one giving me trouble is the MySQL container which has a
data-only container for persistance. When I try to import a mysql file, mysql
complains that the table is
full,
which really means that the disk is out of space. Looking at the system, I see
the problem, but don't know how to correct it:
_ _ ____ _ _
| |__ ___ ___ | |_|___ \ __| | ___ ___| | _____ _ __
| '_ \ / _ \ / _ \| __| __) / _` |/ _ \ / __| |/ / _ \ '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__| < __/ |
|_.__/ \___/ \___/ \__|_____\__,_|\___/ \___|_|\_\___|_|
Boot2Docker version 1.8.1, build master : 7f12e95 - Thu Aug 13 03:24:56 UTC
2015
Docker version 1.8.1, build d12ea79
docker#default:~$ sudo -i
Boot2Docker version 1.8.1, build master : 7f12e95 - Thu Aug 13 03:24:56 UTC
2015
Docker version 1.8.1, build d12ea79
root#default:~# df -h
Filesystem Size Used Available Use% Mounted on
tmpfs 896.6M 112.4M 784.2M 13% /
tmpfs 498.1M 136.0K 498.0M 0% /dev/shm
/dev/sda1 2.8G 2.7G 0 100% /mnt/sda1
cgroup 498.1M 0 498.1M 0% /sys/fs/cgroup
none 462.2G 32.8G 429.5G 7% /Users
/dev/sda1 2.8G 2.7G 0 100%
/mnt/sda1/var/lib/docker/aufs
none 2.8G 2.7G 0 100%
/mnt/sda1/var/lib/docker/aufs/mnt/0a90321d2e941e31385a4c4096e
none 2.8G 2.7G 0 100%
/mnt/sda1/var/lib/docker/aufs/mnt/bca6fc0c017233ed634e7e19284
none 2.8G 2.7G 0 100%
/mnt/sda1/var/lib/docker/aufs/mnt/7e432f020ee8fc9c6a211c810e5
I created the docker-machine like this, which I thought would give me the space I need, but it has not.
docker-machine create --virtualbox-disk-size 4000 -d virtualbox default
I'm creating the mysql image like this
#!/bin/bash
echo "- checking that the image exists"
IMAGE=$(docker images | grep cp_mysql)
if [ -z "$IMAGE" ]; then
echo '- image does not exist, building'
docker build -t cp_mysql -f Dockerfile-app .
fi
echo "- checking if the mysql data volume exists"
DATA_VOL=$(docker ps -a | grep cp_mysql_data)
# if empty
if [ -z "$DATA_VOL" ]; then
echo '- data volume is empty - building'
DATA_VOL=$(docker build -t data -f Dockerfile-data_vol . | tail -n 1 | awk '{print $3}')
docker run --name cp_mysql_data $DATA_VOL
echo "- build data volume: $DATA_VOL"
else
DATA_VOL=$(echo $DATA_VOL | awk '{print $1}')
echo "- data volume is not empty - using existing volume: $DATA_VOL"
fi
echo '- check if existing cp_mysql_app exists'
APP=$(docker ps -a | grep cp_mysql_app)
if [ -z "$APP" ]; then
echo '- the app does not exist, let us create it'
docker run \
--restart=always \
--name cp_mysql_app \
--restart=always \
--volumes-from=cp_mysql_data \
-e MYSQL_USER=cp \
-e MYSQL_PASSWORD=cp \
-e MYSQL_DATABASE=cp \
-e MYSQL_ROOT_PASSWORD=root \
-d \
-p 3306:3306 cp_mysql
else
echo '- the does exist, let us just run it'
docker start cp_mysql_app
fi
Here's my docker info for that machine
docker#default:~$ docker info
Containers: 0
Images: 0
Storage Driver: aufs
Root Dir: /mnt/sda1/var/lib/docker/aufs
Backing Filesystem: tmpfs
Dirs: 0
Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.0.9-boot2docker
Operating System: Boot2Docker 1.8.1 (TCL 6.3); master : 7f12e95 - Thu Aug 13 03:24:56 UTC 2015
CPUs: 1
Total Memory: 996.2 MiB
Name: default
ID: XLSE:62WR:VWCR:T2Z6:FSE3:NTLV:EQRT:WLW5:NLPF:HPQH:JQGR:K4LZ
Debug mode (server): true
File Descriptors: 9
Goroutines: 16
System Time: 2015-09-08T16:31:38.029737031Z
EventsListeners: 0
Init SHA1:
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
Labels:
provider=virtualbox
What I basically need is unbounded space for /dev/sda1, or to be able to specify a large disk size. I know this stems from my misunderstanding how docker mounts work, but I thought this thread would jump-start my research.
Thanks in advance.
The option --virtualbox-disk-size needs to be entered in MB (see Docker Machine Docs). You created your machine with 4GB. This gives docker 2.8GB for storing images and running containers on /dev/sda1.
The default value for disk size, which is used by docker-machine create, is 20GB (according to docs linked above). Running docker images -a will show you how much disk space is required by the images you pulled. This will give you an indication how much more space you need.
Usually 20GB is a good size to pull several different images and run some containers. So maybe you create a new machine with the default disk size and try to import the mysql file again.