Mocking sqlalchemy.create_engine fails to mock the function - sqlalchemy

I want to test my DBConnect class, which manages and returns connections made with sqlalchemy.
I want to specifically test that the constructor of DBConnect calls sqlalchemy.create_engine and stores the engine in the connection.
My test function is this:
from unittest import mock
from catcom.db_connect import DBConnect
import os
class MockEngine:
def connect(self):
return "test_connection"
#mock.patch("sqlalchemy.create_engine", return_value=MockEngine())
def test_from_key_uses_correct_key(mock):
print(mock)
db = DBConnect("user", "pw", "127.0.0.1", "1234", "db")
assert db.connection == "test_connection"
However, I see a response of:
E sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) connection to server at "127.0.0.1", port 1234 failed: Connection refused
E Is the server running on that host and accepting TCP/IP connections?
How can I mock the create_engine method without it being actually called?
UPDATE
Adding the DB Connect class and relevant methods.
from sqlalchemy import create_engine
import os
class DBConnect:
"""Manages database connection using SQLalchemy"""
def __init__(self, user: str, password: str, host: str, port: str, database: str):
self.user = user
self.password = password
self.host = host
self.port = port
self.database = database
self.connection = self.connect()
def connect(self):
"""creates a connection object that is used to query database using SQLalchemy core
automatically created in __init__"""
engine = create_engine(
f"postgresql+psycopg2://{self.user}:{self.password}#{self.host}:{self.port}/{self.database}"
)
return engine.connect()

You are importing create_engine out of the sqlalchemy namespace with from sqlalchemy import create_engine, which puts it in the module's namespace.
That's what you need to patch, see where to patch.
#mock.patch('catcom.db_connect.create_engine', return_value='test_success')
Although, patching catcom.db_connect.create_engine to return 'test_success' will cause a problem when the DBConnect.connect method tries to call the connect method of the engine (which will be a string).

Related

FLASK: How to establish connection with mysql server?

I already have a mysql connection from Flask like this:
app.config['MYSQL_HOST'] = 'one.hostname.net'
app.config['MYSQL_USER'] = 'my_username'
app.config['MYSQL_PASSWORD'] = 'my_password'
app.config['MYSQL_DB'] = 'user_mydb'
mysql = MySQL(app)
with this setup I am able to use mysql database connection in flask. but when it comes to celery task, which is inside the same python file as flask.
#mycelery.task(bind=True, name='mytask')
def mytask(self, userid, port):
cursor = mysql.connection.cursor(MySQLdb.cursors.DictCursor)
cursor.execute('SELECT * FROM mytable WHERE id = %s', (userid,))
It throws me an error saying
cursor = mysql.connection.cursor(MySQLdb.cursors.DictCursor)
AttributeError: 'NoneType' object has no attribute 'cursor'
I understand it is because the celery has no MySQL connection made. But How can I establish the connection? so that I don't have to connect with the MySQL server whenever I create a task just like how it is don't for flask where we already established the connection by MySQL = MySQL(app)??
Here is my celery setup, if it is helpful to add something to this code
mycelery = Celery(app.name)
mycelery.conf.update({
'broker_url': 'filesystem://',
'broker_transport_options': {
'data_folder_in': 'app/broker/out',
'data_folder_out': 'app/broker/out',
'data_folder_processed': 'app/broker/processed'
},
'result_persistent': False,
'task_serializer': 'json',
'result_serializer': 'json',
'accept_content': ['json']})
Firstly if you want to do dabatabase operation you can do in celery task, but you dont have to connect db with celery.
You can connect flask with db, and install celery in your project and make db operation in your celery task.
Sample:
app.py
from flask import Flask
from flaskext.mysql import MySQL
app = Flask(__name__)
mysql = MySQL()
app.config['MYSQL_DATABASE_USER'] = ''
app.config['MYSQL_DATABASE_PASSWORD'] = ''
app.config['MYSQL_DATABASE_DB'] = ''
app.config['MYSQL_DATABASE_HOST'] = ''
mysql.init_app(app)
tasks.py
#celery.task
def db_connect_things():
conn = mysql.connect()
cursor =conn.cursor()
sql_query = """select from where """
cursor.execute(sql_query)
...
celery_config.py
from celery import Celery
celery = Celery(__name__)
celery = Celery('tasks', broker=) # rabbit,redis, ..
celery.conf.update({'CELERY_ACCEPT_CONTENT': ['pickle', 'json', 'msgpack', 'yaml']})
celery.conf.add_defaults(...)
celery.conf.update(CELERYBEAT_SCHEDULE={
'db_connect_things': {
'task': 'application.lib.tasks.db_connect_things',
'schedule': crontab(minute=0, hour='*/12'),
}})
class ContextTask(celery.Task):
...

Why do I get TCP/IP error when trying to create DB in my Lambda?

So I'm trying to deploy my Django project using lambda, with zappa. I'm using MySQL for DB engine. Now after doing some research, I realized that I needed to create a custom Django command to create DB, since I'm using MySQL. So I created crate_db command, zappa updated, then ran zappa manage dev create_db. Then I got this error: 2004 (HY000): Can't create TCP/IP socket (97)
below is my create_db.py file, for your information.
import sys
import logging
import mysql.connector
import os
from django.core.management.base import BaseCommand, CommandError
from django.conf import settings
rds_host = os.environ.get("MY HOST")
db_name = os.environ.get("")
user_name = os.environ.get("MY USERNAME")
password = os.environ.get("MY PASSWORD")
port = os.environ.get("3306")
logger = logging.getLogger()
logger.setLevel(logging.INFO)
class Command(BaseCommand):
help = 'Creates the initial database'
def handle(self, *args, **options):
print('Starting db creation')
try:
db = mysql.connector.connect(host=rds_host, user=user_name,
password=password, db="mysql", connect_timeout=10)
c = db.cursor()
print("connected to db server")
c.execute("""CREATE DATABASE bookcake_db;""")
c.execute("""GRANT ALL ON bookcake_db.* TO 'ryan'#'%'""")
c.close()
print("closed db connection")
except mysql.connector.Error as err:
logger.error("Something went wrong: {}".format(err))
sys.exit()
Any ideas? Thanks.

pymysql.err.InternalError: (1109, "Unknown table 'ALL_PLUGINS' in information_schema")

I have a Python Flask Server setup in an Ubuntu Machine and a MySQL from XAMPP as backend for the same.
How ever when I try to access the database tables from my python program it shows as
pymysql.err.InternalError: (1109, "Unknown table 'ALL_PLUGINS' in information_schema")
but i can access the database directly in MySQL admin page
the sample program I used to access the data.
from flaskext.mysql import MySQL
from flask import (Flask, request, session, g, redirect, url_for, abort, render_template, flash, Response)
import os
from werkzeug.utils import secure_filename
mysql = MySQL()
app = Flask(__name__)
app.config['MYSQL_DATABASE_USER'] = 'root'
app.config['MYSQL_DATABASE_PASSWORD'] = 'root'
app.config['MYSQL_DATABASE_DB'] = 'information_schema'
app.config['MYSQL_DATABASE_HOST'] = 'localhost'
mysql.init_app(app)
#app.route('/')
def insert_student():
qry = "SELECT * FROM ALL_PLUGINS "
conn = mysql.connect()
cursor = conn.cursor()
cursor.execute(qry)
data = cursor.fetchall()
print(data)
conn.commit()
return "Sucess"
if __name__ == '__main__':
app.secret_key = 'super secret key'
app.debug = True
app.run()
instead of normal running I ge the following
the screen shot
Mysql does not have an all_plugins table in information schema. The plugins table (well, view) is called plugins.
So, your query should be:
SELECT * FROM PLUGINS
Based on the comment from #snakecharmerb:
Mariadb, on the other hand, does have all_plugins table, which presumably is the cause of the confusion.

Docker + MYSQL: Can't access mysql outside the container

I have 2 docker container which both contain mysql database and I open the port 3305, 3306 for each.
for the 3306:3306's container I can access it outside the container by
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, Integer, Numeric, String
import os
cwd = os.getcwd()
import sys
sys.path.insert(0, cwd + '/vatic-docker/vatic')
#from models import *
engine = create_engine('mysql://root:1111' '#172.17.0.2:3306/vatic')
Session = sessionmaker (bind = engine)
session = Session()
Base = declarative_base()
However I can't access the 3305:3306 container's mysql database by
engine = create_engine('mysql://root:1111' '#172.17.0.3:3306/vatic')
which return me the error messages:
sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '172.17.0.3' (111)")
What should I do now?
The port in the connection string is wrong
So
engine = create_engine('mysql://root:1111' '#172.17.0.3:3306/vatic')
should be
engine = create_engine('mysql://root:1111#<host IP>:3305/vatic')
since 3305 is the port that you are exposing on the host machine
Specifically if the Docker host machine has an IP address 192.168.2.100 then the connection should be
engine = create_engine('mysql://root:1111#192.168.2.100:3305/vatic')
format for -p is hostPort:containerPort
https://docs.docker.com/engine/reference/commandline/run/#publish-or-expose-port--p---expose

Flask-MySQLdb raises _mysql_exceptions.ProgrammingError: closing a closed connection

I'm trying to use Flask-MySQLdb. Every time I go to a page I get the error _mysql_exceptions.ProgrammingError: closing a closed connection. Why isn't this working?
from flask import Flask, render_template, request, json
from flask_mysqldb import MySQL
application = Flask(__name__)
mysql = MySQL(application)
application.config['MYSQL_USER'] = 'root'
application.config['MYSQL_PASSWORD'] = 'password'
application.config['MYSQL_DB'] = 'db name'
application.config['MYSQL_HOST'] = 'localhost'
mysql.init_app(application)
#application.route('/')
def index():
cur = mysql.connection.cursor()
cur.execute("SHOW TABLES")
rv= cur.fetchall()
return str(rv)
if __name__ == "__main__":
application.run()
You've initialize the extension on the same application twice. Each time it's initialized, it registers a function that closes the connection. So the second function closes a connection that was already closed by the first registered function. Either remove application from MySQL (preferable) or remove the explicit call to mysql.init_app.
mysql = MySQL()
mysql.init_app(application)