I am comfortable in using SQL query languages and need to develop a web application.
I am learning python -sqlachemy and stuck in get connected to postgressql database.
Please let me know:
what are the libraries i need to install ?
How do i Configure DATABASE_URL in flask.[enter image description here][1]
I am stuck with code line no-6.
I need to know how to get DATABASE_URL configured? I am very new potgressql.
[1]: https://i.stack.imgur.com/7oGRC.png
You can set it up in bash or you can use dotenv package from pypi:
$ export DATABASE_URL=postgresql://username:password#host:port/dbname
If your username=blue, password=pass, host=localhost, and port=5432, dbname=testDB then:
$ export DATABASE_URL=postgresql://blue:pass#localhost:5432/testDB
To check if the postgres db is running-- or check the port on which it is running:
$ sudo lsof -i -n -P | grep LISTEN
Related
I am newbie on using Ubuntu, I am trying to install apache-superset and successfully installed it by using Docker by directly sudo docker pull apache/superset, but I am stuck at adding Database Connector to the running superset
As for my local database I am using MySQL and I happen to use SSH Tunnel in localhost to access it in server. So I think at Docker container perspective this must be an "external" databases
What I have tried:
I installed mysqlclient from pip3
By following this references: https://devopsheaven.com/docker/devops/add-host/link/2017/10/04/connect-external-services-from-docker-container.html
I tried to type: sudo docker run -it mysql -h 192.168.100.1 -P 33063 -u czjovan --password=mypw cz_payment_merged but then i get:
2021-03-04 11:34:53+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.23-1debian10 started.
2021-03-04 11:34:53+00:00 [ERROR] [Entrypoint]: mysqld failed while attempting to check config
command was: mysqld -h 192.168.100.1 -P 33063 -u czjovan --password=mypw cz_payment_merged --verbose --help --log-bin-index=/tmp/tmp.EV6L0jrspQ
2021-03-04T11:34:53.402148Z 0 [ERROR] [MY-010124] [Server] Fatal error: Can't change to run as user 'czjovan' ; Please check that the user exists!
2021-03-04T11:34:53.403355Z 0 [ERROR] [MY-010119] [Server] Aborting
By Following this also: From inside of a Docker container, how do I connect to the localhost of the machine?
I tried to type: sudo docker run --rm it --network=host mysql mysql -h 127.0.0.1 but got error 2003 (hy000) can't connect to mysql server on 127.0.0.1
I tried to add mysqlconnector to sqlalchemy uri, but the driver not found
I am not an expert by setting these, I lack of Docker mechanism.. appreciate it if anyone willingly to direct me how to step by step resolving this..
UPDATE-------------------------------------------------:
-> Following Mustafa Guler to add -p 3306:3306, the mysql container now starts..
but I still cannot add database in Superset, what should I do next?
You need to provide the IP of the docker host machine. If you're using default networking, use the static IP 172.17.0.1 for the host.
Alternatively you can try host.docker.internal as the hostname.
You need to define user and pass as environment to create new user during start of mysql container. Maybe problem is user and pass MYSQL_USER, MYSQL_PASSWORD Also you do not expose 3306 port on docker run. Please use -p 3306:3306
I found out that installing Superset manually from scratch than using Superset from Docker Container are more a solution to me, since configuring Superset from docker manually can be a little cumbersome, what I did:
#-- Install Superset and MySQL Locally from Scratch ---#
sudo pip install apache-superset (
sudo apt install mysql-client-core-8.0
sudo apt install mysql-server
sudo pip install mysqlclient (for Superset Database Connectors)
#NOTE: there is also a package called 'superset' in pip, in my case i uninstalled this to ensure that only apache-superset is used
service mysql start
when mysql start, try to set password for the first time:
sudo mysqladmin -u root password
then to test it: mysql -u root -p, enter a created password
if it can enter mysql normally, the password set is successful,
CREATE DATABASE superset (this will save all config that superset progress will be saved)
Editing config.py in apache superset, which usually located on /usr/local/lib/python3.8/dist-packages/superset/config.py (this depends on pip installation from no 1),
8a) Edit the sqlalchemy uri part in config.py, so it can connect to a local installed mysql in ubuntu, to something like 'mysql://root:#localhost:3306/superset'
8b) Ensure that the database part in config.py is superset, or the same name with database name created in MySQL
After all database, sqlalchemy uri, and database name is prepared, its good to go to follow with Superset configs stated from: https://superset.apache.org/docs/installation/installing-superset-from-scratch
A) sudo superset db upgrade
B) sudo superset fab create-admin
C) sudo superset load examples
D) sudo superset init
E) sudo superset run -p 8088 --with-threads --reload --debugger (run this to start superset daily)
Run a Private SSH Tunnel Connecting to Database (Optional, in my case I use SSH Tunnel to access database)
And then I am able to add connection locally in Superset defining sqlalchemy uri
A) towards the SSH Tunnel mysql://(server_user):(server_password)#127.0.0.1:33063/(database_name)
(this is outside from local MySQL)
B) towards Local Installed MySQL 'mysql://root:rootpwd#127.0.0.1:3306/database_name'
I will recommend to use docker than manual installation by pip. Package dependencies especially flask are real problems.
Do not follow docker compose as beginner.
I had a problems using docker compose. Port is closed was annoying problem due to networking. Host.docker.internal doesn’t worked for me on Ubuntu 22 on Google cloud. I switched to manual installation and it was package versions hell on python 3.8. I would like to recommend to not follow official doc and use better approach with single docker image to start. Instead of running 5 containers by compose, run everything in one. Use official docker image, here image. Than modify docker file as follows to install custom db driver:
FROM apache/superset
USER root
RUN pip install mysqlclient
RUN pip install sqlalchemy-redshift
USER superset
Second step is to build new image based on docker file description. To avoid networking problems start both containers on same network (superset, your db) easier is to use host network. I used this on Google cloud, example as follow:
docker run -d --network host --name superset supers
The same command to start container with your database. —network host. This solved my problems. More about in whole step to step tutorial: medium or here blog
Performing database migrations using flyway. Using the dockerized version and using conf files for mirgations configurations.
Below is my config file
flyway.url = jdbc:mysql://${MYSQLHOST}:3306/myschema
flyway.user = myusername
flyway.password = mypassword
flyway.schemas = myschema
flyway.cleanDisabled = true
Am running the below command to perform migration
sudo docker run -e "MYSQLHOST=myhostip" --rm -it -v `pwd`/path/to/confi/:/flyway/conf/ -v `pwd`/path/to/migrations:/flyway/sql boxfuse/flyway:5.1.4 -configFiles=/flyway/conf/flyway.conf migrate
Am getting the below error
ERROR:
Unable to obtain connection from database (jdbc:mysql://${MYSQLHOST}:3306/myschema) for user 'myuser': Could not connect to address=(host=${MYSQLHOST})(port=3306)(type=master) : ${MYSQLHOST}
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL State : 08
Error Code : -1
Message : Could not connect to address=(host=${MYSQLHOST})(port=3306)(type=master) : ${MYSQLHOST}
If I change the config file with my host ip details the migrations are successful without any errors. What am I doing wrongly?
Flyway allows you to pass in configuration parameters via a specific set of environment variables; it doesn't perform a complete variable substitution in the config file. See https://flywaydb.org/documentation/envvars.
What you could do is delete the first line of the config file, then
sudo docker run -e "FLYWAY_URL=jdbc:mysql://myhostip:3306/myschema" ...
I found it not very helpful to only see a page explaining that there are envVars -- but no mention of the exact env variable names to actually use.
Here is the definition of the env names / especially for the gradle plugin but I suspect that they are generally valid:
https://github.com/flyway/flyway/blob/master/flyway-core/src/main/java/org/flywaydb/core/internal/configuration/ConfigUtils.java#L139
I'm trying to install gogs on minishift with persistentVolumes.
I did this:
c:\> oc login -u system -p admin
c:\> oc new-project cicd
c:\> oc create -f https://raw.githubusercontent.com/OpenShiftDemos/gogs-openshift-docker/master/openshift/gogs-persistent-template.yaml
from the webconsole I load and execute the imported yaml template and I have gogs and postgresql up and running as you can see in the image below
Unfortunately, when I try to do this first installation of gogs I stuck in this error:
I know the issue is that gogs docker can access in write mode on /etc/gogs/conf/ dir.
How can I solve this on minishift?
thx
fabrizio
What is the optimal way to load in a sql dump when using docker-compose + docker automated builds?
Have been ignoring docker-compose for a moment and trying to understand docker and it's automated builds at first but have come to realize that i will probably need docker-compose if i want to accomplish my project goal that is to use one 1 command and from that have a fully working 3 site Docker cluster
1xHAProxy
3xUbuntu/wp
3xMysqld
In my Dockerfile i can just include the db.sql from my Github repo like
ADD db.sql /tmp/db.sql
Failing to find a best practise how i should load my DB without writing any commands outside of build.
Want to know your solution to this using Dockerfile or Compose
By just executing one of the commands below a mysql FROM mysql with ADD db.sql db.sql should be build / run while loading db.sql in to mysql db wp
Dockerfile
$docker run -d user/repo:tag
docker-compose.yml
$docker-compose up
If am totally on the wrong path here please give me some references. Could also mention that am planning to use CoreOS once i feel OK with Docker. So if best practices on a CoreOS > Docker setup is something else, let me know!
There are two options for initializing a SQL file during build or run time:
The first would be to just base your MySQL image on the official image and place your SQL file in /docker-entrypoint-initdb.d (using something like ADD my.sql /docker-entrypoint-initdb.d/ in the Dockerfile). The official image has a fairly complex entrypoint script (https://github.com/docker-library/mysql/blob/master/5.7/docker-entrypoint.sh) which starts MySQL, initializes a username and password, and scripts from the /docker-entrypoint-initdb.d folder.
The other option would be to do something like the answer at https://stackoverflow.com/a/25920875/684908 and just add a command such as:
COPY dump.sql /tmp/
RUN /bin/bash -c "/usr/bin/mysqld_safe &" && \
sleep 5 && \
mysql -u root -e "CREATE DATABASE mydb" && \
mysql -u root mydb < /tmp/dump.sql
I need a generic way to install MySQL 5.5 in almost any Linux OS from non-root User. Hence I thought to install MySQL from its source and install it where ever I need.
Is it really possible to install MySQL in non-root user home?
Anybody have any idea for this? Please do share your expertise for the same.
Major constraint here is that, I need to install MySQL 5.5 from any non-root User in a Generic way And possibly for almost any Linux OS.
Any suggestion would be appreciated.
Thanks.
CONCLUSION
I've tried with Ubuntu-11.10, finally I was able to install MySQL-5.5 from non-root user with the constraint that MySQL is not accessible from console/command prompt. As mysqld is up and running fine hence MySQL was easily accessible via any GUI tool which connects to MySQL via JDBC connectors. If you try to access mysql from command prompt using
mysql -u root -p
command it was giving segmentation fault problem. One more thing I tried for Fedora Linux also from Non-Root user, in that mysqld was failing and can't access mysql anyway :( .
You should customize this three variables:
MYSQL_DATADIR
SYSCONFDIR
CMAKE_INSTALL_PREFIX
Example:
$ cd <mysql_src_dir>
$ cmake -i .
Would you like to see advanced options? [No]:Yes
Please wait while cmake processes CMakeLists.txt files....
...
Variable Name: CMAKE_INSTALL_PREFIX
Description: install prefix
Current Value: /usr/local/mysql
New Value (Enter to keep current value): /home/user/mysql
...
Variable Name: MYSQL_DATADIR
Description: default MySQL data directory
Current Value: /usr/local/mysql/data
New Value (Enter to keep current value): /home/user/mysql/data
...
Variable Name: SYSCONFDIR
Description: config directory (for my.cnf)
Current Value: /usr/local/mysql/etc
New Value (Enter to keep current value): /home/user/mysql/etc
$ make
$ make install
Instead of cmake -i . you can use cmake -D MYSQL_DATADIR=/home/user/mysql/data -D SYSCONFDIR=/home/user/mysql/etc -D CMAKE_INSTALL_PREFIX=/home/user/mysql .
I imagine this should be possible but not necessarily easy to do. You would need to build from source and change the Makefile so that the install target points to the user's local directory; additionally, I would think that you'd need to mess around with other default configuration options in MySQL, which can also be changed from configuration files. In the end, you should be able to launch mysqld as long as you don't bind to any port below 1000 (which is the case anyway since MySQL runs by default on port 3306).