I am newbie on using Ubuntu, I am trying to install apache-superset and successfully installed it by using Docker by directly sudo docker pull apache/superset, but I am stuck at adding Database Connector to the running superset
As for my local database I am using MySQL and I happen to use SSH Tunnel in localhost to access it in server. So I think at Docker container perspective this must be an "external" databases
What I have tried:
I installed mysqlclient from pip3
By following this references: https://devopsheaven.com/docker/devops/add-host/link/2017/10/04/connect-external-services-from-docker-container.html
I tried to type: sudo docker run -it mysql -h 192.168.100.1 -P 33063 -u czjovan --password=mypw cz_payment_merged but then i get:
2021-03-04 11:34:53+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.23-1debian10 started.
2021-03-04 11:34:53+00:00 [ERROR] [Entrypoint]: mysqld failed while attempting to check config
command was: mysqld -h 192.168.100.1 -P 33063 -u czjovan --password=mypw cz_payment_merged --verbose --help --log-bin-index=/tmp/tmp.EV6L0jrspQ
2021-03-04T11:34:53.402148Z 0 [ERROR] [MY-010124] [Server] Fatal error: Can't change to run as user 'czjovan' ; Please check that the user exists!
2021-03-04T11:34:53.403355Z 0 [ERROR] [MY-010119] [Server] Aborting
By Following this also: From inside of a Docker container, how do I connect to the localhost of the machine?
I tried to type: sudo docker run --rm it --network=host mysql mysql -h 127.0.0.1 but got error 2003 (hy000) can't connect to mysql server on 127.0.0.1
I tried to add mysqlconnector to sqlalchemy uri, but the driver not found
I am not an expert by setting these, I lack of Docker mechanism.. appreciate it if anyone willingly to direct me how to step by step resolving this..
UPDATE-------------------------------------------------:
-> Following Mustafa Guler to add -p 3306:3306, the mysql container now starts..
but I still cannot add database in Superset, what should I do next?
You need to provide the IP of the docker host machine. If you're using default networking, use the static IP 172.17.0.1 for the host.
Alternatively you can try host.docker.internal as the hostname.
You need to define user and pass as environment to create new user during start of mysql container. Maybe problem is user and pass MYSQL_USER, MYSQL_PASSWORD Also you do not expose 3306 port on docker run. Please use -p 3306:3306
I found out that installing Superset manually from scratch than using Superset from Docker Container are more a solution to me, since configuring Superset from docker manually can be a little cumbersome, what I did:
#-- Install Superset and MySQL Locally from Scratch ---#
sudo pip install apache-superset (
sudo apt install mysql-client-core-8.0
sudo apt install mysql-server
sudo pip install mysqlclient (for Superset Database Connectors)
#NOTE: there is also a package called 'superset' in pip, in my case i uninstalled this to ensure that only apache-superset is used
service mysql start
when mysql start, try to set password for the first time:
sudo mysqladmin -u root password
then to test it: mysql -u root -p, enter a created password
if it can enter mysql normally, the password set is successful,
CREATE DATABASE superset (this will save all config that superset progress will be saved)
Editing config.py in apache superset, which usually located on /usr/local/lib/python3.8/dist-packages/superset/config.py (this depends on pip installation from no 1),
8a) Edit the sqlalchemy uri part in config.py, so it can connect to a local installed mysql in ubuntu, to something like 'mysql://root:#localhost:3306/superset'
8b) Ensure that the database part in config.py is superset, or the same name with database name created in MySQL
After all database, sqlalchemy uri, and database name is prepared, its good to go to follow with Superset configs stated from: https://superset.apache.org/docs/installation/installing-superset-from-scratch
A) sudo superset db upgrade
B) sudo superset fab create-admin
C) sudo superset load examples
D) sudo superset init
E) sudo superset run -p 8088 --with-threads --reload --debugger (run this to start superset daily)
Run a Private SSH Tunnel Connecting to Database (Optional, in my case I use SSH Tunnel to access database)
And then I am able to add connection locally in Superset defining sqlalchemy uri
A) towards the SSH Tunnel mysql://(server_user):(server_password)#127.0.0.1:33063/(database_name)
(this is outside from local MySQL)
B) towards Local Installed MySQL 'mysql://root:rootpwd#127.0.0.1:3306/database_name'
I will recommend to use docker than manual installation by pip. Package dependencies especially flask are real problems.
Do not follow docker compose as beginner.
I had a problems using docker compose. Port is closed was annoying problem due to networking. Host.docker.internal doesn’t worked for me on Ubuntu 22 on Google cloud. I switched to manual installation and it was package versions hell on python 3.8. I would like to recommend to not follow official doc and use better approach with single docker image to start. Instead of running 5 containers by compose, run everything in one. Use official docker image, here image. Than modify docker file as follows to install custom db driver:
FROM apache/superset
USER root
RUN pip install mysqlclient
RUN pip install sqlalchemy-redshift
USER superset
Second step is to build new image based on docker file description. To avoid networking problems start both containers on same network (superset, your db) easier is to use host network. I used this on Google cloud, example as follow:
docker run -d --network host --name superset supers
The same command to start container with your database. —network host. This solved my problems. More about in whole step to step tutorial: medium or here blog
I was having mysql 5.1.x installed on my windows machine. Beacuse of some reason I needed 5.5.x or above. I uninstalled mysql 5.1.x and downloaded the 5.7.x version. I extracted and tried to install the service using below command and I got the response saying Service already exists. How to fix the issue?
C:\mysql-5.7.10\bin>mysqld --install
The service already exists!
The current server installed: "C:\Program Files (x86)\MySQL\MySQL Server 5.1\bin\mysqld" --defaults-file="C:\Program Files (x86)\MySQL\MySQL Server 5.1\my.ini" MySQL
You must run the command sc delete {{MySQL}} where {{MySQL}} is the name of the service.
I installed a MySQL server with installer and it started. After reboot I tried to start it again and get the error:
D:\Program Files\MySQL\MySQL Server 5.7\bin>mysqld -u root -p
mysqld: Can't change dir to 'D:\Program Files\MySQL\MySQL Server 5.7\data\' (Errcode: 2 - No such file or directory)
2015-11-17T08:30:18.822962Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2015-11-17T08:30:18.822962Z 0 [Warning] Insecure configuration for --secure- file -priv: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path.
2015-11-17T08:30:18.822962Z 0 [Note] mysqld (mysqld 5.7.9) starting as process 1108 ...
2015-11-17T08:30:18.838586Z 0 [Warning] Can't create test file D:\Program Files\MySQL\MySQL Server 5.7\data\DESKTOP-RNBR3E8.lower-test
2015-11-17T08:30:18.838586Z 0 [Warning] Can't create test file D:\Program Files\MySQL\MySQL Server 5.7\data\DESKTOP-RNBR3E8.lower-test
2015-11-17T08:30:18.838586Z 0 [ERROR] failed to set datadir to D:\Program Files\MySQL\MySQL Server 5.7\data\
2015-11-17T08:30:18.838586Z 0 [ERROR] Aborting
2015-11-17T08:30:18.838586Z 0 [Note] Binlog end
2015-11-17T08:30:18.838586Z 0 [Note] mysqld: Shutdown complete
I tried to reinstall MySQL.
UPDATE:
When I run mysqld -u root -p as administrator, nothing happens.
I have met same problem. In my case I had no ..\data dir in my C:\mysql\ so I just executed mysqld --initialize command from c:\mysql\bin\ directory and I got the data directory in c:\mysql\data. Afterwards I could use mysqld.exe --console command to test the server startup.
Since you used the Windows installer, everything is set up for you to run MySQL 5.7 as a Windows service, which is a great option in most cases.
Instead of running mysqld.exe from the command line,
Win + R
Run services.msc
Right-click on MySQL57
Start the service.
In my case, I had installed the data directory to a different location. So the data directory really wasn't in the default location. Therefore, when I ran the mysqld command from the command prompt, I had to specify the data directory manually:
mysqld --datadir=D:/MySQLData/Data
Here's the documentation for mysqld command-line arguments.
First run
mysqld -u root --initialize-insecure
It will create data folder with root as user without password. Then run
mysqld.exe -u root --console
What I did (Windows 10) for a new installation:
Start cmd in admin mode (run as administrator by hitting windows key, typing cmd, right clicking on it and selecting "Run as Administrator"
Change into "MySQL Server X.Y" directory (for me the full path is C:\Program Files\MySQL\MySQL Server 5.7")
using notepad create a my.ini with a mysqld section that points at your data directory
[mysqld]
datadir="X:\Your Directory Path and Name"
created the directory identified in my.ini above.
change into bin Directory under server directory and execute:
mysqld --initialize
Once complete, started the service and it came up fine.
Check that the director exists.
If it exists make sure mysql has access rights on it (read/write). Can be a good idea to run MySQL as a system process, but not mandatory.
If this is stil not working try having the following notation with double slashes: C:\WebSerer\MySQL\data or C:/WebServer/MySQL/data
If you are having this issue and can't find the MySQL service either, just launch "MySQL Installer" from the Start menu (or download it), then choose "Reconfigure" on MySQL Server. Follow the steps to set a root user password and then the service will be created and started automatically (and set to auto-start on boot).
In mysql 8.0.13 zip package initializing.
Verify that data folder is empty.
Under the mysql bin path run
mysqld.exe --initialize-insecure
Add to my.ini native mysql
[mysqld]
default_authentication_plugin=mysql_native_password
When I encountered this same error, I noticed MySQL Configuration file in "C:\Program Files\MySQL\MySQL Server X.Y\" has changed to my-default.ini
I solved it by
Copy my.ini from "C:\ProgramData\MySQL\MySQL Server X.Y\my.ini"
Paste it in "C:\Program Files\MySQL\MySQL Server X.Y\my.ini"
Restart MySQL Server from services.msc
In the .ini file, their is part that reads:
# On Windows you should keep this file in the installation directory
# of your server (e.g. C:\Program Files\MySQL\MySQL Server X.Y). To
# make sure the server reads the config file use the startup option
# "--defaults-file".
Check your real my.ini file location and set --defaults-file="location" with this command
mysql --defaults-file="C:\MYSQL\my.ini" -u root -p
This solution is permanently for your cmd Screen.
remove all files in "{path-to-mysql}\data" directory and run:
mysqld --initialize-insecure --basedir={path-to-mysql}\mysql --datadir={path-to-mysql}\data --console
I don't know if it's your problem but try this...
For me, the version of my.ini which is created under "ProgramData\MySQL\MySQL Server [version]" wasn't read. I had to move it to "Program Files\MySQL\MySQL Server [version]".
RTS.
If you installed MySQL Server using the Windows installer and as a Window's service, then you can start MySQL Server using PowerShell and experience the convenience of not leaving the command line.
Open PowerShell and run the following command:
Get-Service *sql*
A list of MySQL services will be retrieved and displayed. Choose the one that you want and run the following command while replacing service-name with the actual service name:
Start-Service -Name service-name
Done. You can check that the service is running by running the command Get-Service *sql* again and checking the status of the service.
mariofertc completely solved this for me here are his steps:
Verify mysql's data directory is empty (before you delete it though, save the err file for your records).
Under the mysql bin path run:
mysqld.exe --initialize-insecure
add to my.ini (mysql's configuration file) the following:
[mysqld]
default_authentication_plugin=mysql_native_password
Then check services (via task manager) to make sure MySql is running, if not - right click MySql and start it.
I'll also note, if you don't have your mysql configuration file in the mysql bin and can't find it via the windows search, you will want to look for it in C:\Program Data\Mysql\ Note that it might be a different name other than my.ini, like a template, as Heesu mentions here: Can't find my.ini (mysql 5.7)
Just find the template that matches the version of your mysql via the command mysql --version
This solution uses the windows mysql installer.
I have tried every other way mentioned here and other related posts, but it did not solve my problem, the service just wont start, but the below approach with the mysql-installer did.
If you still have your installer or atleast remember the version then follow below steps:
Start your windows mysql installer. For me it was "mysql-installer-community-8.0.20.0"
Then remove/uninstall the SQL Server and remove all configurations
Manually delete the SQL Server folder from "C:\Program Files\MySQL\MySQL Server 8.0."
Start your mysql installer again and install the SQL Server again
You can check from the window's services that the MySqL Server has started.
Hope it helps someone.
Check for missing folders that are required by the server, in my case it was UPLOADS folder in programData which was deleted by empty folder cleaner utility that I used earlier.
How did I find out:
run the server config file my.ini(in my case it was in programData) as the defaults-file param for mysqld (don't forget to use --console option to view error log on screen)
'mysqld --defaults-file="C:\ProgramData\MySQL\MySQL Server 8.0\my.ini" --console'
Error:
mysqld: Can't read dir of 'C:\ProgramData\MySQL\MySQL Server 8.0\Uploads\' (OS errno 2 - No such file or directory)
Solution:
Once I manually created the Uploads folder the server started successfully.
In my case, I'm MySQL5.6, install by workbrench install tools in Win10 pro.
there are two configuration file need to modify,
only edit the line: datadir=E:\MySQL
stop MySQL56 service using services.msc
edit my.ini under C:\ProgramData\MySQL\MySQL Server 5.6
edit my-default.ini under C:\Program Files\MySQL\MySQL Server 5.6
copy C:\ProgramData\MySQL\MySQL Server 5.6*.* to E:\MySQL
start MySQL56 service using services.msc
test: create database 'testdb';
It's successed if found testdb folder exist in E:\MySQL.
i am trying to setup mysql-proxy on ubuntu on amazon ec2
i have done following:
sudo apt-get install mysql-proxy --yes
vi /etc/default/mysql-proxy
i put following content on "/etc/default/mysql-proxy"
ENABLED="true"
OPTIONS="--proxy-lua-script=/usr/share/mysql-proxy/rw-splitting.lua
--proxy-address=127.0.0.1:3306
--proxy-backend-addresses=private_ip_of_another_ec2_db_server:3306,private_ip_of_another_ec2_db_server:3306"
also tied with "--proxy-address=private_ip_or_public_ip_of_proxy-server:3306 or 4040"
and "--proxy-backend-addresses=public_ip_of_another_ec2_db_server:3306,public_ip_of_another_ec2_db_server:3306"
after that i tried to connect proxy server from another pc using mysql like:
mysql -u some_user -pxxxxx -h proxy_server_ip
or
mysql -u some_user -pxxxxx -h proxy_server_ip -P 4040
but its not working
its showing error:
ERROR 2003 (HY000): Can't connect to MySQL server on 'ip' (10061)
i want to tell you can connect the db server remotely where i allowed remote connection to any host
i also tried /etc/init.d/mysql-proxy start or /etc/init.d/mysql-proxy restart but no result
just to inform you that /etc/init.d/mysql-proxy stop is showing failed
can anyone please help me to setup and configure mysql-proxy on ubuntu
===
Edit
i found some help from other question of stackoverflow and also according to a suggestion in the comments, have done following procedure. and it seems its working now.
i installed mysql-client and mysql-server locally(on proxy server)
then i tried to run mysql-proxy using following command:
mysql-proxy --proxy-backend-addresses=10.73.151.244:3306 --proxy-backend-addresses=10.73.198.7:3306 --proxy-address=:4040 --admin-username=root --admin-password=root --admin-lua-script=>/usr/lib/mysql-proxy/lua/admin.lua
then i tried to connect remotely to the proxy server and its working.
but it seems i need to run this command under screen because when i close the terminal proxy stops working.
Can you please tell me that do i need to run this command under screen or is there any other way to make it alive all time?
There is no need to install Mysql client or Mysql Server on your mysql-proxy.
Installing mysql-proxy does have "full daemon capabilities" compiled into it.
If your are running Ubuntu Server, you may wish to use an UPSTART service script.
This script can be copied into /etc/init/mysql-proxy.conf
# mysql-proxy.conf (Ubuntu 14.04.1) Upstart proxy configuration file for AWS RDS
# mysql-proxy - mysql-proxy job file
description "mysql-proxy upstart script"
author "shadowbq <shadowbq#gmail.com>"
# Stanzas
#
# Stanzas control when and how a process is started and stopped
# See a list of stanzas here: http://upstart.ubuntu.com/wiki/Stanzas#respawn
# When to start the service
start on runlevel [2345]
# When to stop the service
stop on runlevel [016]
# Automatically restart process if crashed
respawn
# Essentially lets upstart know the process will detach itself to the background
expect daemon
# Run before process
pre-start script
[ -d /var/run/mysql-proxy ] || mkdir -p /var/run/mysql-proxy
echo "starting mysql-proxy"
end script
# Start the process
exec /usr/bin/mysql-proxy --plugins=proxy --proxy-lua-script=/usr/share/mysql-proxy/rw-splitting.lua --log-level=debug --proxy-backend-addresses=private_ip_of_another_ec2_db_server:3306,private_ip_of_another_ec2_db_server:3306 --daemon --log-use-syslog --pid-file=/var/run/mysql-proxy/mysql-proxy.pid
In the above example I hard coded the AWS RDS server into script, instead of fiddling with defaults and config file
Install Upgraded version 0.8.5
Note:
apt repo does not have 0.8.5 so we need to download tar from mysql official site
Prerequisite :-
Create file /etc/default/mysql-proxy with following content
ENABLED="true"
OPTIONS="--defaults-file=/etc/mysql/mysql-proxy.cnf"
Installation Procedure :-
Download mysql-proxy 0.8.x
Untar in /usr/local
Update PATH environment with /usr/local/mysql-proxy-0.8.5-linux-debian6.0-x86-64bit/bin
vim /etc/environment (to update environment path)
cd /usr/local/mysql-proxy-0.8.5-linux-debian6.0-x86-64bit/bin
Run command sudo ./mysql-proxy --defaults-file=/etc/mysql/mysql-proxy.cnf
Sample mysql-proxy.cnf file
[mysql-proxy]
log-level=debug
log-file=/var/log/mysql-proxy.log
pid-file = /var/run/mysql-proxy.pid
daemon = true
--no-proxy = false
admin-username=ADMIN
admin-password=ADMIN
proxy-backend-addresses=RDS-ENDPOINT:RDS-PORT
admin-lua-script=/usr/lib/mysql-proxy/lua/admin.lua
proxy-address=0.0.0.0:4040
admin-address=localhost:4041
change host ip and port of RDS or mysql
connect to Mysql server via proxy with
mysql -h{proxy-host-ip} -P 4040 -u{mysql_username} -p
Also the error message in xampp control panel states:
Found Path: "C:\Program Files\MySQL\MySQL Server 5.5\bin\mysqld" --defaults-file="C:\Program Files\MySQL\MySQL Server 5.5\my.ini" MySQL
20:08:20 [mysql]
Expected Path: c:\xampp\mysql\bin\mysqld.exe --defaults-file=c:\xampp\mysql\bin\my.ini mysql
Can you help as this is a fresh install of xampp v.3.2.1
I followed the instructions on install I don't know why it's not working.
Do I need to change the path in the httpd.conf file in mysql for it to find the path?
It looks mysql is already installed in your system in the above mentioned "Found Path". If already installed mysql is not needed, uninstall it. Now try to run our new Xampp mysql server, it will run. Otherwise you can run the xampp mysql using "mysql_start" batch file which is under XAMPP folder.