I have one database that I specifically using for Django. I ran into a simple problem. For the user authentication in the login I need to use data from another database(mysql) and table. Both the tables are in the same server. How can I do that? Thanks
You need to define both the databases in your settings.py. You should read through setting up multiple databases documentation. It's pretty straight forward. Once you have databases configured and synchronized, you can manually select a database for a QuerySet as follows:
Model.objects.using('database').all()
Update:
To specify a connection for a model you can define a Router as follows.
from django.db import connections
class MyRouter(object):
"""
This router object will take care of the database
operations for models.
"""
def db_for_read(self, model, **hints):
if hasattr(model,'connection_name'):
return model.connection_name
return None
def db_for_write(self, model, **hints):
if hasattr(model,'connection_name'):
return model.connection_name
return None
def allow_syncdb(self, db, model):
if hasattr(model,'connection_name'):
return model.connection_name == db
return db == default
and then for your models in models.py you can do something like this:
class MyModel(models.Model):
connection_name="filename"
Update#2
you can setup the router anywhere you like. All you need to do is pass the Routers path to the DATABASE_ROUTERS variable in the settings.py file. For the sake of cleanliness of code and being organised i generally do this in a separate routers.py file under my application folder and then set the DATABASE_ROUTERS
example:
DATABASE_ROUTERS = ['myapp.routers.MyRouter',]
Related
I was trying to use fastapi users package to quickly Add a registration and authentication system to my FastAPI project which uses the PostgreSQL database. I am using asyncio to be able to create asynchronous functions.
In the beginning, I used only sqlAlchemy and I have tried their example here. And I added those line of codes to my app/app.py to create the database at the starting of the server. and everything worked like a charm. the table users was created on my database.
#app.on_event("startup")
async def on_startup():
await create_db_and_tables()
Since I am using SQLModel I added FastAPI Users - Database adapter for SQLModel to my virtual en packages. And I added those lines to fastapi_users/db/__init__.py to be able to use the SQL model database.
try:
from fastapi_users_db_sqlmodel import ( # noqa: F401
SQLModelBaseOAuthAccount,
SQLModelBaseUserDB,
SQLModelUserDatabase,
)
except ImportError: # pragma: no cover
pass
I have also modified app/users.py, to use SQLModelUserDatabase instead of sqlAchemy one.
async def get_user_manager(user_db: SQLModelUserDatabase = Depends(get_user_db)):
yield UserManager(user_db)
and the app/dp.py to use SQLModelUserDatabase, SQLModelBaseUserDB, here is the full code of app/db.py
import os
from typing import AsyncGenerator
from fastapi import Depends
from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine
from sqlalchemy.orm import sessionmaker
from fastapi_users.db import SQLModelUserDatabase, SQLModelBaseUserDB
from sqlmodel import SQLModel
from app.models import UserDB
DATABASE_URL = os.environ.get("DATABASE_URL")
engine = create_async_engine(DATABASE_URL)
async_session_maker = sessionmaker(
engine, class_=AsyncSession, expire_on_commit=False)
async def create_db_and_tables():
async with engine.begin() as conn:
await conn.run_sync(SQLModel.metadata.create_all)
async def get_async_session() -> AsyncSession:
async_session = sessionmaker(
engine, class_=AsyncSession, expire_on_commit=False
)
async with async_session() as session:
yield session
async def get_user_db(session: AsyncSession = Depends(get_async_session)):
yield SQLModelUserDatabase(UserDB, session, SQLModelBaseUserDB)
Once I run the code, the table is not created at all. I wonder what could be the issue. I could not understand. Any idea?
I had the same problem, but managed to make it work by making a couple changes
The changes that I needed to make (code is based on the full example in the documentation):
In models.py, make UserDB inherit from SQLModelBaseUserDB, User, and add table=True for sqlmodel to create the table:
class UserDB(SQLModelBaseUserDB, User, table=True):
pass
It's important that SQLModelBaseUserDB is inherited from first, because otherwise User.id trumps SQLModelBaseUserDB.id and sqlmodel cannot find primary_key column
Use SQLModelUserDatabaseAsync in get_user_db, like this (as far as I understand, you don't need to pass in SQLModelBaseUserDB in SQLModelUserDatabase. The third argument is for oauth account model):
async def get_user_db(session: AsyncSession = Depends(get_async_session)):
yield SQLModelUserDatabaseAsync(UserDB, session)
By the time I posted this question that was the answer I received from one of the maintainer of fastapi-users that made me switch to sqlAlchemy that time, actually I do not know if they officially released sqlModel DB adapter or not
My guess is that you didn't change the UserDB model so that it inherits from the SQLModelBaseUserDB one. It's necessary in order to let SQLModel detect all your models and create them.
You can have an idea of what it should look like in fastapi-users-db-sqlmodel tests: https://github.com/fastapi-users/fastapi-users-db-sqlmodel/blob/3a46b80399f129aa07a834a1b40bf49d08c37be1/tests/conftest.py#L25-L27
Bear in mind though that we didn't officially release this DB adapter; as they are some problems with SQLModel regarding UUID (tiangolo/sqlmodel#25). So you'll probably run into issues.
and here is the GitHub link of the issue: https://github.com/fastapi-users/fastapi-users/discussions/861
I'm building a python app around an existing (mysql) database and am using automap to infer tables and relationships:
base = automap_base()
self.engine = create_engine(
'mysql://%s:%s#%s/%s?charset=utf8mb4' % (
config.DB_USER, config.DB_PASSWD, config.DB_HOST, config.DB_NAME
), echo=False
)
# reflect the tables
base.prepare(self.engine, reflect=True)
self.TableName = base.classes.table_name
Using this I can do things like session.query(TableName) etc...
However, I'm worried about performance, because every time the app runs it will do the whole inference again.
Is this a legitimate concern?
If so, is there a possibility to 'cache' the output of Automap?
I think that "reflecting" the structure of your database is not the way to go. Unless your app tries to "infer" things from the structure, like static code analysis would for source files, then it is unnecessary. The other reason for reflecting it at run-time would be the reduced time to begin "using" the database using SQLAlchemy. However:
Another option would be to use something like SQLACodegen (https://pypi.python.org/pypi/sqlacodegen):
It will "reflect" your database once and create a 99.5% accurate set of declarative SQLAlchemy models for you to work with. However, this does require that you keep the model subsequently in-sync with the structure of the database. I would assume that this is not a big concern seeing as the tables you're already-working with are stable-enough such that run-time reflection of their structure does not impact your program much.
Generating the declarative models is essentially a "cache" of the reflection. It's just that SQLACodegen saved it into a very readable set of classes + fields instead of data in-memory. Even with a changing structure, and my own "changes" to the generated declarative models, I still use SQLACodegen later-on in a project whenever I make database changes. It means that my models are generally consistent amongst one-another and that I don't have things such as typos and data-mismatches due to copy-pasting.
Performance can be a legitimate concern. If the database schema is not changing, it can be time consuming to reflect the database every time a script is run. This is more of an issue during development, not starting up a long running application. It's also a significant time saver if your database is on a remote server (again, particularly during development).
I use code that is similar to the answer here (as noted by #ACV). The general plan is to perform the reflection the first time, then pickle the metadata object. The next time the script is run, it will look for the pickle file and use that. The file can be anywhere, but I place mine in ~/.sqlalchemy_cache. This is an example based on your code.
import os
from sqlalchemy.ext.declarative import declarative_base
self.engine = create_engine(
'mysql://%s:%s#%s/%s?charset=utf8mb4' % (
config.DB_USER, config.DB_PASSWD, config.DB_HOST, config.DB_NAME
), echo=False
)
metadata_pickle_filename = "mydb_metadata"
cache_path = os.path.join(os.path.expanduser("~"), ".sqlalchemy_cache")
cached_metadata = None
if os.path.exists(cache_path):
try:
with open(os.path.join(cache_path, metadata_pickle_filename), 'rb') as cache_file:
cached_metadata = pickle.load(file=cache_file)
except IOError:
# cache file not found - no problem, reflect as usual
pass
if cached_metadata:
base = declarative_base(bind=self.engine, metadata=cached_metadata)
else:
base = automap_base()
base.prepare(self.engine, reflect=True) # reflect the tables
# save the metadata for future runs
try:
if not os.path.exists(cache_path):
os.makedirs(cache_path)
# make sure to open in binary mode - we're writing bytes, not str
with open(os.path.join(cache_path, metadata_pickle_filename), 'wb') as cache_file:
pickle.dump(Base.metadata, cache_file)
except:
# couldn't write the file for some reason
pass
self.TableName = base.classes.table_name
For anyone using declarative table class definitions, assuming a Base object defined as e.g.
Base = declarative_base(bind=engine)
metadata_pickle_filename = "ModelClasses_trilliandb_trillian.pickle"
# ------------------------------------------
# Load the cached metadata if it's available
# ------------------------------------------
# NOTE: delete the cached file if the database schema changes!!
cache_path = os.path.join(os.path.expanduser("~"), ".sqlalchemy_cache")
cached_metadata = None
if os.path.exists(cache_path):
try:
with open(os.path.join(cache_path, metadata_pickle_filename), 'rb') as cache_file:
cached_metadata = pickle.load(file=cache_file)
except IOError:
# cache file not found - no problem
pass
# ------------------------------------------
# define all tables
#
class MyTable(Base):
if cached_metadata:
__table__ = cached_metadata.tables['my_schema.my_table']
else:
__tablename__ = 'my_table'
__table_args__ = {'autoload':True, 'schema':'my_schema'}
...
# ----------------------------------------
# If no cached metadata was found, save it
# ----------------------------------------
if cached_metadata is None:
# cache the metadata for future loading
# - MUST DELETE IF THE DATABASE SCHEMA HAS CHANGED
try:
if not os.path.exists(cache_path):
os.makedirs(cache_path)
# make sure to open in binary mode - we're writing bytes, not str
with open(os.path.join(cache_path, metadata_pickle_filename), 'wb') as cache_file:
pickle.dump(Base.metadata, cache_file)
except:
# couldn't write the file for some reason
pass
Important Note!! If the database schema changes, you must delete the cached file to force the code to autoload and create a new cache. If you don't, the changes will be be reflected in the code. It's an easy thing to forget.
The answer to your first question is largely subjective. You are adding database queries to fetch the reflection metadata to the application load time. Whether or not that overhead is significant depends on your project requirements.
For reference, I have an internal tool at work that uses a reflection pattern because the the load-time is acceptable for our team. That might not be the case if it were an externally-facing product. My hunch is that for most applications the reflection overhead will not dominate the total application load time.
If you decide it is significant for your purposes, this question has an interesting answer where the user pickles the database metadata in order to locally cache it.
Adding to this, what #Demitri answered is close to correct but (at least in sqlalchemy 1.4.29), the example will fail on the last line self.TableName = base.classes.table_name when generating from the cached file. In this case declarative_base() has no attribute classes.
To fix is as simple as altering:
if cached_metadata:
base = declarative_base(bind=self.engine, metadata=cached_metadata)
else:
base = automap_base()
base.prepare(self.engine, reflect=True) # reflect the tables
to
if cached_metadata:
base = automap_base(declarative_base(bind=self.engine, metadata=cached_metadata))
base.prepare()
else:
base = automap_base()
base.prepare(self.engine, reflect=True) # reflect the tables
This will create your automap object with the proper attributes.
I am trying to deploy an application to a production server, but testing has revealed a strange inconsistency in django_rest-framework (first with version 2.3.14, now upgraded to 3.0.1).
On my development machine the json response comes wrapped with some metadata:
{u'count': 2, u'previous': None, u'results': [json objects here]}
Whereas on the production machine only the 'results' array is returned.
Is there a setting or to change this one way or the other?
Serializers are as follows :
class SampleSerializer(serializers.ModelSerializer):
class Meta:
model = Sample
class LibrarySerializer(serializers.ModelSerializer):
sample = SampleSerializer()
class Meta:
model = Library
views.py
class PullLibraryView(generics.ListAPIView):
serializer_class = LibrarySerializer
filter_backends = (filters.DjangoFilterBackend,)
def get_queryset(self, *args, **kwargs):
slug = self.kwargs.get('submission_slug', '')
return Library.objects.filter(sample__submission__submission_slug=slug)
This metadata is added to the response through the pagination serializer, which is a part of the pagination that is built-in. The metadata will only be added if pagination is enabled, so you need to check your settings to make sure that pagination is enabled on your production machine.
Pagination is determined by your settings and paginate_by property on your views. Make sure your requests do include the page_size parameter, which should force pagination on your views.
Is there a way to mark tables as read-only when using Pyramid with SQLAlchemy?
Some brief background: I am working on an app with an existing database which is very complicated. My app adds a few new tables to this database, but updating (update, insert, delete) other tables is unsafe because the schema is not well understood (and isn't documented).
My basic aim is to fail-fast before any unsafe db updates so the offending code can be found and removed, so I and future developers don't cause any data issues with this app.
My naive approach: add a pyramid_tm commit veto hook that checks objects to be updated (new, dirty, and deleted in the db session) and vetos if there are any read-only tables.
Is there an existing mechanism to handle this? Or a better approach?
If you're using the "declarative" approach, you can try making the classes themselves "read-only":
class MyBase(object):
"""
This class is a superclass of SA-generated Base class,
which in turn is the superclass of all db-aware classes
so we can define common functions here
"""
def __setattr__(self, name, value):
"""
Raise an exception if attempting to assign to an atribute of a "read-only" object
Transient attributes need to be prefixed with "_t_"
"""
if (getattr(self, '__read_only__', False)
and name != "_sa_instance_state"
and not name.startswith("_t_")):
raise ValueError("Trying to assign to %s of a read-only object %s" % (name, self))
super(MyBase, self).__setattr__(name, value)
Base = declarative_base(cls=MyBase)
Similarly you can try to override __init__ method to prevent the objects from being instantiated.
However, solving this problem at the application level feels a bit fragile - I would just look at creating a special user in the database for your app to connect with, which would save limited permissions on the tables you want to keep intact.
In a Django application, I'm trying to access an existing MySQL database created with Hibernate (a Java ORM). I reverse engineered the model using:
$ manage.py inspectdb > models.py
This created a nice models file from the Database and many things were quite fine. But I can't find how to properly access boolean fields, which were mapped by Hibernate as columns of type BIT(1).
The inspectdb script by default creates these fields in the model as TextField and adds a comment saying that it couldn't reliably obtain the field type. I changed these to BooleanField but and opened my model objects using the admin, but it doesn't work (the model objects always fetch a value of true for these fields). Using IntegerField won't work as well (e.g. in the admin these fields show strange non-ascii characters).
Any hints of doing this without changing the database? (I need the existing Hibernate mappings and Java application to still work with the database).
Further info: I left these fields as BooleanField and used the interactive shell to look at the fetched values. They are returned as '\x00' (when Java/Hibernate value is false) and '\x01' (when true), instead of Python boolean values "True" and "False".
>>> u = AppUser.objects.all()[0]
>>> u.account_expired
'\x00'
>>> u.account_enabled
'\x01'
Where the model includes:
class AppUser(models.Model):
account_expired = models.BooleanField()
account_enabled = models.BooleanField(blank=True)
# etc...
This is the detailed solution from Dmitry's suggestion:
My derived field class:
class MySQLBooleanField(models.BooleanField):
__metaclass__ = models.SubfieldBase
def to_python(self, value):
if isinstance(value, bool):
return value
return bytearray(value)[0]
def get_db_prep_value(self, value):
return '\x01' if value else '\x00'
Fields in my model:
account_enabled = MySQLBooleanField()
account_expired = MySQLBooleanField()
I had to deal the same problem, but rather than subclassing the field, I extended the MySQL backend to understand the Hibernate way. It's only a few lines of code and has the advantage that the DB introspection can be made to work correctly, as well.
See it here.
hibernateboolsbackend / backends / mysql / base.py
# We want to import everything since we are basically subclassing the module.
from django.db.backends.mysql.base import *
django_conversions.update({
FIELD_TYPE.BIT: lambda x: x != '\x00',
})
DatabaseIntrospection.data_types_reverse.update({
FIELD_TYPE.BIT: 'BooleanField',
})
The django-mysql package provides a BooleanField subclass called Bit1BooleanField that solves this:
from django.db import Model
from django_mysql.models import Bit1BooleanField
class AppUser(Model):
bit1bool = Bit1BooleanField()
Easier than rolling your own, and tested on several Django and Python versions.
I guess that only way is to subclass, say, BooleanField, and override to_python/get_prep_value functions, so the field works seamlessly with django and
your db.
To make it work on django 1.7.1 i've to change the "to_python" function because it was not working to read correctly the data from the db:
def to_python(self, value):
if value in (True, False): return value
if value in ('t', 'True', '1', '\x01'): return True
if value in ('f', 'False', '0', '\x00'): return False
In Python Boolean type is a subclass of integer But whereas Java it is of type Bit.
So in DB for Python Boolean field datatype should be Tinyint instead of BIT type.
Due to the above reason the application behaving unexpectedly returning \x00 and \x01 value.
If we add a boolean field in Django model and run the migration then migrate, It will add a column of type Tinyint instead of bit.
So updating the column type to Tinyint should resolve the issue.