Superset with Docker cannot add connection to external mysql - mysql

I am newbie on using Ubuntu, I am trying to install apache-superset and successfully installed it by using Docker by directly sudo docker pull apache/superset, but I am stuck at adding Database Connector to the running superset
As for my local database I am using MySQL and I happen to use SSH Tunnel in localhost to access it in server. So I think at Docker container perspective this must be an "external" databases
What I have tried:
I installed mysqlclient from pip3
By following this references: https://devopsheaven.com/docker/devops/add-host/link/2017/10/04/connect-external-services-from-docker-container.html
I tried to type: sudo docker run -it mysql -h 192.168.100.1 -P 33063 -u czjovan --password=mypw cz_payment_merged but then i get:
2021-03-04 11:34:53+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.23-1debian10 started.
2021-03-04 11:34:53+00:00 [ERROR] [Entrypoint]: mysqld failed while attempting to check config
command was: mysqld -h 192.168.100.1 -P 33063 -u czjovan --password=mypw cz_payment_merged --verbose --help --log-bin-index=/tmp/tmp.EV6L0jrspQ
2021-03-04T11:34:53.402148Z 0 [ERROR] [MY-010124] [Server] Fatal error: Can't change to run as user 'czjovan' ; Please check that the user exists!
2021-03-04T11:34:53.403355Z 0 [ERROR] [MY-010119] [Server] Aborting
By Following this also: From inside of a Docker container, how do I connect to the localhost of the machine?
I tried to type: sudo docker run --rm it --network=host mysql mysql -h 127.0.0.1 but got error 2003 (hy000) can't connect to mysql server on 127.0.0.1
I tried to add mysqlconnector to sqlalchemy uri, but the driver not found
I am not an expert by setting these, I lack of Docker mechanism.. appreciate it if anyone willingly to direct me how to step by step resolving this..
UPDATE-------------------------------------------------:
-> Following Mustafa Guler to add -p 3306:3306, the mysql container now starts..
but I still cannot add database in Superset, what should I do next?

You need to provide the IP of the docker host machine. If you're using default networking, use the static IP 172.17.0.1 for the host.
Alternatively you can try host.docker.internal as the hostname.

You need to define user and pass as environment to create new user during start of mysql container. Maybe problem is user and pass MYSQL_USER, MYSQL_PASSWORD Also you do not expose 3306 port on docker run. Please use -p 3306:3306

I found out that installing Superset manually from scratch than using Superset from Docker Container are more a solution to me, since configuring Superset from docker manually can be a little cumbersome, what I did:
#-- Install Superset and MySQL Locally from Scratch ---#
sudo pip install apache-superset (
sudo apt install mysql-client-core-8.0
sudo apt install mysql-server
sudo pip install mysqlclient (for Superset Database Connectors)
#NOTE: there is also a package called 'superset' in pip, in my case i uninstalled this to ensure that only apache-superset is used
service mysql start
when mysql start, try to set password for the first time:
sudo mysqladmin -u root password
then to test it: mysql -u root -p, enter a created password
if it can enter mysql normally, the password set is successful,
CREATE DATABASE superset (this will save all config that superset progress will be saved)
Editing config.py in apache superset, which usually located on /usr/local/lib/python3.8/dist-packages/superset/config.py (this depends on pip installation from no 1),
8a) Edit the sqlalchemy uri part in config.py, so it can connect to a local installed mysql in ubuntu, to something like 'mysql://root:#localhost:3306/superset'
8b) Ensure that the database part in config.py is superset, or the same name with database name created in MySQL
After all database, sqlalchemy uri, and database name is prepared, its good to go to follow with Superset configs stated from: https://superset.apache.org/docs/installation/installing-superset-from-scratch
A) sudo superset db upgrade
B) sudo superset fab create-admin
C) sudo superset load examples
D) sudo superset init
E) sudo superset run -p 8088 --with-threads --reload --debugger (run this to start superset daily)
Run a Private SSH Tunnel Connecting to Database (Optional, in my case I use SSH Tunnel to access database)
And then I am able to add connection locally in Superset defining sqlalchemy uri
A) towards the SSH Tunnel mysql://(server_user):(server_password)#127.0.0.1:33063/(database_name)
(this is outside from local MySQL)
B) towards Local Installed MySQL 'mysql://root:rootpwd#127.0.0.1:3306/database_name'

I will recommend to use docker than manual installation by pip. Package dependencies especially flask are real problems.
Do not follow docker compose as beginner.
I had a problems using docker compose. Port is closed was annoying problem due to networking. Host.docker.internal doesn’t worked for me on Ubuntu 22 on Google cloud. I switched to manual installation and it was package versions hell on python 3.8. I would like to recommend to not follow official doc and use better approach with single docker image to start. Instead of running 5 containers by compose, run everything in one. Use official docker image, here image. Than modify docker file as follows to install custom db driver:
FROM apache/superset
USER root
RUN pip install mysqlclient
RUN pip install sqlalchemy-redshift
USER superset
Second step is to build new image based on docker file description. To avoid networking problems start both containers on same network (superset, your db) easier is to use host network. I used this on Google cloud, example as follow:
docker run -d --network host --name superset supers
The same command to start container with your database. —network host. This solved my problems. More about in whole step to step tutorial: medium or here blog

Related

Laravel Docker Container not connecting to local mysql

I have an issue connecting to mysql running in the local machine in my DockerFile i have mentioned
FROM php:7
RUN apt-get update -y && apt-get install -y openssl zip unzip git
RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
RUN docker-php-ext-install pdo mbstring pdo_mysql
WORKDIR /home
COPY . /home
RUN composer install --ignore-platform-reqs
CMD php artisan serve --host=0.0.0.0 --port=8081
EXPOSE 8081
and this in my .env configuration
DB_HOST=localhost
DB_DATABASE=databasename
DB_USERNAME=root
DB_PASSWORD=testpassword
I have very less clue about where it is failing. Do i need to install mysql for Docker container also?
A much simpler solution (for Mac OSX & Docker for Windows) is to replace the host address from localhost
to host.docker.internal
DB_HOST=host.docker.internal
DB_DATABASE=databasename
DB_USERNAME=root
DB_PASSWORD=testpassword
Basically the DNS namehost.docker.internal will resolves to the internal IP address used by the host.
NB: If you have changed your address to host.docker.internal but you still receive connection refused error, it’s most probably because MySQL is currently configured to only listen to the local network.
To resolve that, please update the value of the bind_address to 0.0.0.0 in your my.cnf configuration file.
you are trying to connect to mysql in localhost, which is (surprisingly) the reference to the local host. since its a relative address, inside the container it is being resolved as the container own address, and no mysql is awaiting you there...
so to solve it - just give it your real host ip instead of localhost or 127.0.0.1.
step 1 - fix .env file:
DB_HOST=<your_host_ip> #run `ifconfig` and look for your ip on `docker0` network
DB_DATABASE=databasename
DB_USERNAME=laravel_server #not root, since we are going to allow this user remote access.
DB_PASSWORD=testpassword
step 2 - create dedicated user:
open your mysql: mysql -u root -p, give your root password, and run the following:
CREATE USER 'laravel_server'#'%' IDENTIFIED BY 'testpassword';
GRANT ALL PRIVILEGES ON databasename.* TO 'laravel_server'#'%';
FLUSH PRIVILEGES;
we created the user and gave it the permissions.
step 3 - open mysql to remote access:
we have to make it listening on all interfaces and not just localhost and therefore we run:
sudo sed 's/.*bind-address.*/bind-address=0.0.0.0/' /etc/mysql/mysql.conf.d/mysqld.cnf
(you will be prompted for password. this command is just replacing the line in mysql configuration file)
step 4 - updating:
in the project directory: php artisan config:cache
service mysql restart
then docker build and run a new container again. it should work for you.
I see two options -
Use the private IP of your docker host i.e where mysql server is running.
Use the host network mode while running container in case you want to use localhost.
docker container --net=host ...
In my case (Ubuntu 20.04 Desktop), I had MariaDB already running and using port 3306. So when the app inside the docker container was trying to start MySQL that was inside the container it failed because it was trying to listen to an already used port. I switched off the already-running MariaDB using the command below :
sudo systemctl stop mariadb.service
Then tried starting the docker app. It ran successfully because port 3306 was now free and used by MySQL inside the container. But since I intend to use both of them, a much more permanent solution would be to configure either of the Database systems i.e the one inside the docker container or the one outside the docker container to use a different port other than the default 3306.

How to install wordpress by docker?

Question:
I follow some guides to install the wordpress + mysql by docker, but found that not work... I tried to test by curl command, found no any output, I need your help for the issue...
(I just transfer my wordpress hosting to VPS)
docker run --name mysql -e MYSQL_ROOT_PASSWORD=123456 -d mysql
docker run --name wordpress --link mysql:mysql -e WORDPRESS_DB_PASSWORD=123456 -d wordpress:4.8.2-apache
[root#vps ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6bd3954390e0 wordpress:4.8.2-apache "docker-entrypoint..." 11 seconds ago Up 10 seconds 80/tcp wordpress
eaa1f6a2fb96 mysql "docker-entrypoint..." 25 seconds ago Up 24 seconds 3306/tcp mysql
Follow troubleshooting and test wordpress:
[root#vps ~]# curl localhost:80
curl: (7) Failed connect to localhost:80; Connection refused
[root#vps ~]# docker inspect --format='{{.NetworkSettings.IPAddress}}' 6bd3954390e0
172.17.0.3
[root#vps ~]# curl 172.17.0.3:80
[root#vps ~]#
Resolved:
Thanks #junius(who in docker forums), #VladoDemcak, #yamenk and #user4860092! The issue was resolved!
If I do “docker run xxxx” that should not work for me, that maybe was caused by command incorrect. Then I tried to do docker-compose, curl no any output, but Wordpress should work normal. So curl no any output should normal.
Now, I completed task that transfer my Wordpress to new VPS, share follow tips and experiences:
Suggest follow docker official guide if you want to install WP by docker.
If you want to mapping the mysql and wordpress, can add follow
config in compose:
If you not config port part in compose, that mean not expose any
port to outside of container, so you couldn't access the port from
outside, as follow:
When you change “docker-compose.yml”, please not only use
“docker-compose down” that will not delete all config/file, suggest
you do “docker-compose down --volumes” that as install guide.
If you change database name, please add “WORDPRESS_DB_NAME: xxx” in
environment part of Wordpress(docker-compose.yml), that should no
this config in official install guide. So wordpress default connect
database name is “wordpress”.
If you want to debug wordpress/mysql and check log, you can not add
“-d”, use this “docker-compose up”
In order to restore mysql database, you can install phpmyadmin by
docker, then add follow config to “docker-compose.yml”, and follow
guide by “https://hub.docker.com/r/phpmyadmin/phpmyadmin/”
If you want to add some software in docker of Wordpress, e.g: zip,
mailx, you can do follow:
You don't have exposed ports, so you are not able to access wordpress (from host) which is running on the port 80 in docker container.
Probably you will need to expose port to some other port (not 80). So try to change docker run command for wordpress as follows:
docker run --name wordpress --link mysql:mysql -e WORDPRESS_DB_PASSWORD=123456 -p 81:80 -d wordpress:4.8.2-apache
Please notice -p 81:80 parameter in the command - Docker documentation expose-incoming-ports.
After that wordpress should be available on localhost:81.
I would suggest you to create docker-compose for your services rather than maintain linking and with docker-compose you are also able to run both services with one command.
There is a very detailed explanation in the official docker docs to how to do this. Follow the link below, and you shall get wordpress up and running.
https://docs.docker.com/compose/wordpress/

docker to connect with mysql database of host system and dump the sql file into host system and then host a web application

I am new to Docker but i have read quite about it. Now my requirement is:
I will give my client a shell script which he would run on a base ubuntu os on a completely new system. The docker image should use the database of host system. The shell script will do all the prerequisites of installing docker, mysql, etc. and will run a docker image. As the image is not available locally, it will pull from docker repository.
Now my problem is that i dont want to give my client the sql dump file just like that. The dump is included in the image and once the images is run i want the image to connect to the host database and dump the data and then host the webapp.
My docker file is :
FROM ubuntu:14.04
MAINTAINER test_manoj
RUN pip install requirements.txt
ADD . /home/myapp/
RUN sudo apt-get install -y supervisor
WORKDIR /home/myapp/
EXPOSE 8000
EXPOSE 80
cmd ["supervisord", "-c", "/home/myapp/supervisord.conf"]
There are some more apt-get install-s but i didnt find it useful to mention here. So basically i am installing nginx, uwsgi, supervisor.
I have exposed port 8000 for socket uwsgi connections and port 80 for nginx.
My docker run command is :
docker run --detach --net=host -v /var/run/mysqld.sock:/var/run/mysqld/mysqld.sock manoj/mydocker
I am using -v to connect the host mysql to container's mysql.
I have already found a work around for my problem that is running
docker run --rm --detach --net=host -v /var/run/mysqld/mysqld.sock:/var/run/mysqld/mysqld.sock manoj/mydocker mysq -uroot -proot db_name < dump.sql
before the main run command. I know this works but is there any other way to do this? And is there any other way i can use host's mysql without providing -v tag?

How to start mysql server in docker container

I am creating docker container and base image is ubuntu:14.04. I have to start mysql server in that container and then I have to create databases and I have to give the permission to the users. Docker is new for me. I tried a lot but still whenever I go to that image and check for mysql server is running or not. Every time what I got is mysql server is stopped and my dbs are also not created.
This is my dockerfile
FROM ubuntu:14.04
MAINTAINER <name> <emailid>
RUN sudo apt-get update
RUN sudo apt-get install -y apache2 mysql-server libapache2-mod-auth-mysql php5-mysql php5 git
RUN sudo apt-get install -y vim
CMD sudo /usr/sbin/mysqld -u mysql
I tried a lot but i am not able to run mysql server in docker image.
Did you manually install it in your container?
Why do you not simply use:
docker run -d --name mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=secret -e MYSQL_DATABASE=mySchema mysql:5
That would start a container named mysql running a mysql daemon, setting the default root password to secret, creating a new schema called mySchema and expose the MySQL port 3306 to clients so that they could connect.
Then you could connect to that with any MySQL client using the root user with the specified password secret and create your tables within the created schema mySchema.
I had similar issue for ubuntu :14.04 while setting up druid cluster in the docker container, using CMD to start mysql fixed it for me.
CMD mysql start \
Druid related stuff
&& mysql stop
docker exec -it container_name/id bash
service mysql status to check on the service status
service mysql start to start the mysql service

how to setup and configure mysql-proxy on ubuntu on amazon ec2

i am trying to setup mysql-proxy on ubuntu on amazon ec2
i have done following:
sudo apt-get install mysql-proxy --yes
vi /etc/default/mysql-proxy
i put following content on "/etc/default/mysql-proxy"
ENABLED="true"
OPTIONS="--proxy-lua-script=/usr/share/mysql-proxy/rw-splitting.lua
--proxy-address=127.0.0.1:3306
--proxy-backend-addresses=private_ip_of_another_ec2_db_server:3306,private_ip_of_another_ec2_db_server:3306"
also tied with "--proxy-address=private_ip_or_public_ip_of_proxy-server:3306 or 4040"
and "--proxy-backend-addresses=public_ip_of_another_ec2_db_server:3306,public_ip_of_another_ec2_db_server:3306"
after that i tried to connect proxy server from another pc using mysql like:
mysql -u some_user -pxxxxx -h proxy_server_ip
or
mysql -u some_user -pxxxxx -h proxy_server_ip -P 4040
but its not working
its showing error:
ERROR 2003 (HY000): Can't connect to MySQL server on 'ip' (10061)
i want to tell you can connect the db server remotely where i allowed remote connection to any host
i also tried /etc/init.d/mysql-proxy start or /etc/init.d/mysql-proxy restart but no result
just to inform you that /etc/init.d/mysql-proxy stop is showing failed
can anyone please help me to setup and configure mysql-proxy on ubuntu
===
Edit
i found some help from other question of stackoverflow and also according to a suggestion in the comments, have done following procedure. and it seems its working now.
i installed mysql-client and mysql-server locally(on proxy server)
then i tried to run mysql-proxy using following command:
mysql-proxy --proxy-backend-addresses=10.73.151.244:3306 --proxy-backend-addresses=10.73.198.7:3306 --proxy-address=:4040 --admin-username=root --admin-password=root --admin-lua-script=>/usr/lib/mysql-proxy/lua/admin.lua
then i tried to connect remotely to the proxy server and its working.
but it seems i need to run this command under screen because when i close the terminal proxy stops working.
Can you please tell me that do i need to run this command under screen or is there any other way to make it alive all time?
There is no need to install Mysql client or Mysql Server on your mysql-proxy.
Installing mysql-proxy does have "full daemon capabilities" compiled into it.
If your are running Ubuntu Server, you may wish to use an UPSTART service script.
This script can be copied into /etc/init/mysql-proxy.conf
# mysql-proxy.conf (Ubuntu 14.04.1) Upstart proxy configuration file for AWS RDS
# mysql-proxy - mysql-proxy job file
description "mysql-proxy upstart script"
author "shadowbq <shadowbq#gmail.com>"
# Stanzas
#
# Stanzas control when and how a process is started and stopped
# See a list of stanzas here: http://upstart.ubuntu.com/wiki/Stanzas#respawn
# When to start the service
start on runlevel [2345]
# When to stop the service
stop on runlevel [016]
# Automatically restart process if crashed
respawn
# Essentially lets upstart know the process will detach itself to the background
expect daemon
# Run before process
pre-start script
[ -d /var/run/mysql-proxy ] || mkdir -p /var/run/mysql-proxy
echo "starting mysql-proxy"
end script
# Start the process
exec /usr/bin/mysql-proxy --plugins=proxy --proxy-lua-script=/usr/share/mysql-proxy/rw-splitting.lua --log-level=debug --proxy-backend-addresses=private_ip_of_another_ec2_db_server:3306,private_ip_of_another_ec2_db_server:3306 --daemon --log-use-syslog --pid-file=/var/run/mysql-proxy/mysql-proxy.pid
In the above example I hard coded the AWS RDS server into script, instead of fiddling with defaults and config file
Install Upgraded version 0.8.5
Note:
apt repo does not have 0.8.5 so we need to download tar from mysql official site
Prerequisite :-
Create file /etc/default/mysql-proxy with following content
ENABLED="true"
OPTIONS="--defaults-file=/etc/mysql/mysql-proxy.cnf"
Installation Procedure :-
Download mysql-proxy 0.8.x
Untar in /usr/local
Update PATH environment with /usr/local/mysql-proxy-0.8.5-linux-debian6.0-x86-64bit/bin
vim /etc/environment (to update environment path)
cd /usr/local/mysql-proxy-0.8.5-linux-debian6.0-x86-64bit/bin
Run command sudo ./mysql-proxy --defaults-file=/etc/mysql/mysql-proxy.cnf
Sample mysql-proxy.cnf file
[mysql-proxy]
log-level=debug
log-file=/var/log/mysql-proxy.log
pid-file = /var/run/mysql-proxy.pid
daemon = true
--no-proxy = false
admin-username=ADMIN
admin-password=ADMIN
proxy-backend-addresses=RDS-ENDPOINT:RDS-PORT
admin-lua-script=/usr/lib/mysql-proxy/lua/admin.lua
proxy-address=0.0.0.0:4040
admin-address=localhost:4041
change host ip and port of RDS or mysql
connect to Mysql server via proxy with
mysql -h{proxy-host-ip} -P 4040 -u{mysql_username} -p