I have also asked this question in sqlalchemy discussion group.
I am using alembic to apply a migration for a postgresql view using an async engine. This is successfully applied to the database.
I have the following declarative mapped class to the view, defined as:
class MailingListView(Base):
"""View for mailing labels.
After metata reflection from db -> model expecting columns to
be available on this class.
"""
__tablename__ = "mailing_list_view"
# Specify the column override from the underlying view that is the primary key
id = Column(UUID(as_uuid=True), primary_key=True)
# Expecting these columns below to be mapped in this class after
# metadata reflection. Currently have to uncomment these
# to manually synchronise with view!
#
# addressee = Column(String)
# street = Column(String)
# town = Column(String)
# county = Column(String)
# postcode = Column(String)
# cursor = Column(String)
I am reflecting the views using the following:
def use_inspector(conn):
inspector = inspect(conn)
return inspector.get_view_names()
views = await connection.run_sync(use_inspector)
# I can see the table columns in __table__.c.keys()
# after the reflection below has run
await connection.run_sync(
target_metadata.reflect,
only=views,
views=True,
extend_existing=True,
)
After applying migrations and performing the above reflection I can see that my mapped model has the underlying table columns updated with those defined in the underlying view.
obj = MailingListView()
obj.__table__.c.keys()
However, the properties of my mapped class are not updated after reflection, raising an exception:
obj = MailingListView()
obj.town = "town" # this raises an exception with unavailable property
How is it possible for a postgresql db (asyncpg) + async sqlalchemy to:
Synchronise the columns of a declarative model with its underlying table after metadata reflection?
Currently, I have to manually specify the columns in the declarative model.
Related
I'm working on a project using Flask and a PostgreSQL database, with SQLAlchemy.
I have Group objects which have a list of User IDs who are members of the group. For some reason, when I try to add an ID to a group, it will not save properly.
If I try members.append(user_id), it doesn't seem to work at all. However, if I try members += [user_id], the id will show up in the view listing all the groups, but if I restart the server, the added value(s) is (are) not there. The initial values, however, are.
Related code:
Adding group to the database initially:
db = SQLAlchemy(app)
# ...
g = Group(request.form['name'], user_id)
db.session.add(g)
db.session.commit()
The Group class:
from flask.ext.sqlalchemy import SQLAlchemy
from sqlalchemy.dialects.postgresql import ARRAY
class Group(db.Model):
__tablename__ = "groups"
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(128))
leader = db.Column(db.Integer)
# list of the members in the group based on user id
members = db.Column(ARRAY(db.Integer))
def __init__(self, name, leader):
self.name = name
self.leader = leader
self.members = [leader]
def __repr__(self):
return "Name: {}, Leader: {}, Members: {}".format(self.name, self.leader, self.members)
def add_user(self, user_id):
self.members += [user_id]
My test function for updating the Group:
def add_2_to_group():
g = Group.query.all()[0]
g.add_user(2)
db.session.commit()
return redirect(url_for('show_groups'))
Thanks for any help!
As you have mentioned, the ARRAY datatype in sqlalchemy is immutable. This means it isn’t possible to add new data into array once it has been initialised.
To solve this, create class MutableList.
from sqlalchemy.ext.mutable import Mutable
class MutableList(Mutable, list):
def append(self, value):
list.append(self, value)
self.changed()
#classmethod
def coerce(cls, key, value):
if not isinstance(value, MutableList):
if isinstance(value, list):
return MutableList(value)
return Mutable.coerce(key, value)
else:
return value
This snippet allows you to extend a list to add mutability to it. So, now you can use the class above to create a mutable array type like:
class Group(db.Model):
...
members = db.Column(MutableList.as_mutable(ARRAY(db.Integer)))
...
You can use the flag_modified function to mark the property as having changed. In this example, you could change your add_user method to:
from sqlalchemy.orm.attributes import flag_modified
# ~~~
def add_user(self, user_id):
self.members += [user_id]
flag_modified(self, 'members')
To anyone in the future: so it turns out that arrays through SQLAlchemy are immutable. So, once they're initialized in the database, they can't change size. There's probably a way to do this, but there are better ways to do what we're trying to do.
This is a hacky solution, but what you can do is:
Store the existing array temporarily
Set the column value to None
Set the column value to the existing temporary array
For example:
g = Group.query.all()[0]
temp_array = g.members
g.members = None
db.session.commit()
db.session.refresh(g)
g.members = temp_array
db.session.commit()
In my case it was solved by using the new reference for storing a object variable and assiging that new created variable in object variable.so, Instead of updating the existing objects variable it will create a new reference address which reflect the changes.
Here in Model,
Table: question
optional_id = sa.Column(sa.ARRAY(sa.Integer), nullable=True)
In views,
option_list=list(question.optional_id if question.optional_id else [])
if option_list:
question.optional_id.clear()
option_list.append(obj.id)
question.optional_id=option_list
else:
question.optional_id=[obj.id]
I am working through SQL Alchemy but struggling with how to structure the information from the docs into my project. I have two databases, my first will be used to store all new information from the python application. Where as the second database (DB1 in this case) is an existing database that I need to access information from. What is the right way to create this structure using SQLAlchemy?
I used the suggested BINDS method for multiple databases. This seems to be working.
class BaseConfig(object):
SECRET_KEY = "SO_SECURE"
DEBUG = True
SQLALCHEMY_DATABASE_URI = 'mssql+pyodbc://sa:funpassword#(localdb)\MSSQLLocalDB/Testing?driver=SQL+Server+Native+Client+11.0'
SQLALCHEMY_BINDS = {
'DB1': 'mssql+pyodbc://sa:$funpassword#ProdDB/DB1?driver=SQL+Server+Native+Client+11.0'
}
SQLALCHEMY_TRACK_MODIFICATIONS = True
This configuration seems to work okay because I am able to create new models in both of these databases using the code below. (This was done to just confirm that I was able to connect to both). db is my SqlAlchemy(app) initialization in my index.py file.
from index import app, db
#Test Writing To Default DB
class Test(db.Model):
id = db.Column(db.Integer(), primary_key=True)
employeeNum = db.Column(db.String(255), unique=False)
job = db.Column(db.String(255))
def __init__(self, employeeNum, job):
self.employeeNum = employeeNum
self.job = job
# Test Writing To DB1
class Test(db.Model):
__bind_key__ = 'DB1'
id = db.Column(db.Integer(), primary_key=True)
employeeNum = db.Column(db.String(255), unique=False)
job = db.Column(db.String(255))
def __init__(self, employeeNum, job):
self.employeeNum = employeeNum
self.job = job
I have tried many combinations using the table and automap_base from SQLAlchemy docs but this does not seem to work. I am not sure how I can use the bind_key of DB1 when I am trying to map existing tables.
from index import app, db
def loadSession():
Table('employee', db.metadata, Column('emp_num', Text, primary_key=True))
Base = automap_base(metadata=metadata)
Base.prepare()
employee = Base.classes.employee
emp = db.session.query(employee).all()
for i in emp:
print(i.data)
Thanks for any help you can provide.
for your DB1 with existing schema & data you could use reflection to get the tables mapped into sqlalchemy.
For your example it would probably look something like this:
from sqlalchemy.ext.automap import automap_base
from sqlalchemy import MetaData
from index import app, db
def loadSession():
# get a db engine object for DB1 configuration
engine = db.get_engine(bind='DB1')
# create a empty db Metadata object and bind it to DB1
database = MetaData(bind=engine)
# load the DB1 table/column structure into the Metadata object. This is quite a heavy operation and should optimally be done once on app setup and then the created automap_base can be stored / reused on demand until app shutdown
database.reflect(
bind=engine, views=True, autoload_replace=False
)
# create a python class structure out of the db metadata structure
auto_base = automap_base(metadata=database)
auto_base.prepare()
# create a db session for DB1
db_session = scoped_session(sessionmaker(autocommit=False,
autoflush=False,
bind=engine))
# get the reflected employees class from DB1. The employee table must exist in the database
employee = auto_base.classes.employee
# query DB1 with the new session object for all employee records
emp = db_session.query(employee).all()
for i in emp:
print(i.data)
# the following could also be substituted for the query without creating a session object:
for entry in employee.query.all():
print(entry)
I'm using the SQLalchemy association-object pattern (http://docs.sqlalchemy.org/en/rel_1_1/orm/basic_relationships.html#association-object) for three model classes.
Basic relationship is on the left side one User can belong to multiple Organizations. I'm storing extra User-Organization relevant data in the association object class. Then, the association-object class maps a many-to-one to the Organization.
From SQLAlchemy point, the relationship works fine. The problem is testing this with factory boy has proven difficult and always results in error RecursionError: maximum recursion depth exceeded.
Below are the three models for the association object relationship, where User is parent and the Child is Organization:
class MemberOrgsAssoc(Model):
"""The left side of the relationship maps a User as a one-to-many to
Organizations. User-Organization relevant data is stored in
this association-object table. Then, there is a one-to-many from
this association-object table to the Organization table. """
__tablename__ = 'member_orgs'
member_id = Column(db.Integer, db.ForeignKey("users.id"), primary_key=True)
org_id = Column(db.Integer, db.ForeignKey("organizations.id"), primary_key=True)
manager_id = Column(db.Integer, db.ForeignKey("users.id"))
org_title = Column(db.Unicode(50))
organization = relationship("Organization", back_populates="members")
member = relationship("User", back_populates="organizations",
foreign_keys=[member_id])
manager = relationship("User", back_populates="subordinates",
foreign_keys=[manager_id])
class User(SurrogatePK, Model):
"""A user of the app."""
__tablename__ = 'users'
username = Column(db.Unicode(80), unique=True, nullable=False)
organizations = relationship("MemberOrgsAssoc", back_populates="member",
primaryjoin = "member_orgs.c.member_id == User.id",
lazy="dynamic")
subordinates = relationship("MemberOrgsAssoc", back_populates="manager",
primaryjoin = "member_orgs.c.manager_id == User.id",
lazy="dynamic")
class Organization(SurrogatePK, Model):
"""An organization that Users may belong to."""
__tablename__ = 'organizations'
name = Column(db.Unicode(128), nullable=False)
members = relationship("MemberOrgsAssoc", back_populates="organization")
So all the above SQLAlchemy model classes and relationships seem to work as intended for now.
Below are the three factory-boy classes I'm attempting to make work.
MemberOrgs association-object factory:
class MemberOrgsAssocFactory(BaseFactory):
"""Association-object table Factory"""
class Meta:
"""Factory config"""
model = MemberOrgsAssoc
member_id = factory.SubFactory('tests.factories.UserFactory')
org_id = factory.SubFactory('tests.factories.OrganizationFactory')
manager_id = factory.SubFactory('tests.factories.UserFactory')
org_title = Sequence(lambda n: 'CEO{0}'.format(n))
organization = factory.SubFactory('tests.factories.OrganizationFactory')
member = factory.SubFactory('tests.factories.UserFactory')
manager = factory.SubFactory('tests.factories.UserFactory')
class UserFactory(BaseFactory):
"""User factory."""
class Meta:
"""Factory configuration."""
model = User
username = Sequence(lambda n: 'user{0}'.format(n))
organizations = factory.List(
[factory.SubFactory('tests.factories.MemberOrgsAssocFactory')])
subordinates = factory.List(
[factory.SubFactory('tests.factories.MemberOrgsAssocFactory')])
class OrganizationFactory(BaseFactory):
"""Company factory"""
class Meta:
"""Factory config"""
model = Organization
id = Sequence(lambda n: '{0}'.format(n))
name = Sequence(lambda n: 'company{0}'.format(n))
members = factory.List(
[factory.SubFactory('tests.factories.MemberOrgsAssocFactory')])
Finally, need to make a user for the tests and so below is a pytest fixture to make a User. This is where the tests fail due to `RecursionError: maximum recursion depth exceeded".
#pytest.fixture(scope='function')
def user(db):
"""An user for the unit tests.
setup reference: https://github.com/FactoryBoy/factory_boy/issues/101
# how to handle self referential foreign key relation in factory boy
# https://github.com/FactoryBoy/factory_boy/issues/173
"""
user = UserFactory(
organizations__0=None,
subordinates__0=None,
)
a = MemberOrgsAssocFactory(
is_org_admin=True,
is_default_org=True,
is_active=True,
)
a.organization=OrganizationFactory()
user.organizations.append(a)
db.session.commit()
return user
Error message:
E RecursionError: maximum recursion depth exceeded
!!! Recursion detected (same locals & position)
More or less resolved this, though a bit fragile overall. Must follow required pattern carefully as laid out in the sqlalchemy docs:
""" EXAMPLE USE:
# create User object, append an Organization object via association
p = User()
a = MemberOrgsAssoc(extra_data="some data")
a.organization = Organization()
p.organizations.append(a)
# iterate through Organization objects via association, including association attributes:
for assoc in p.organizations:
print(assoc.extra_data)
print(assoc.child)
"""
Below changes to the pytest fixture resolved the RecursionError issue and got it working:
#pytest.fixture(scope='function')
def user(db):
"""An user for the tests."""
user = UserFactory(
organizations='',
subordinates=''
)
a = MemberOrgsAssocFactory(
member_id=None,
org_id=None,
manager_id=None,
is_org_admin=True,
is_default_org=True,
is_active=True,
organization=None,
member=None,
manager=None
)
a.organization = OrganizationFactory(members=[])
user.organizations.append(a)
db.session.commit()
# debugging
# thisuser = User.get_by_id(user.id)
# for assoc in thisuser.organizations:
# if assoc.is_default_org:
# print('The default organization of thisuser is -> {}'.format(assoc.organization.name))
return user
I have two databases (both Mysql) that have exactly the same tables, and I want to copy some data from one to another using Sqlalchemy.
I can copy simple objects following the answer given in this question:
Cannot move object from one database to another
The problem is when the object has dependencies from another table, and I want to copy the dependencies as well.
So to make it more clear, this is my model (the same for both databases but using a different bind_key that points to a different database):
db1 = SQLAlchemy()
Class Payment(db.Model):
__tablename__ = 'payments'
__bind_key__ = 'db1'
id = db1.Column(db.Integer, primary_key=True)
paymethod_id = db1.Column(db.Integer(), db1.ForeignKey(PaymentMethod.id))
payment_method = db1.relationship(PaymentMethod)
What I would like to do is the following:
from models1 import Payment as Payment1
from models2 import Payment as Payment2
# query from one database
payment1 = db1.session.query(Payment1).first()
# create and add it to the other database
payment2 = Payment2(**payment1.__dict__.copy())
db2.session.add(payment)
db2.session.commit()
But in this case the foreign key fails because I don't have the PaymentMethod stored yet.
Is there a different approach to do that or I would have to do this procedure for every dependency of my object and be sure that I store the children beforehand?
Any help is appreciated :)
I came up with a solution that remaps the object to the right model and stores all its children. You call the method save_obj and pass the object you want to map. It will then retrieve a table with the same name but then from the model you want to remap the object to and it will recursively do the same for all its children. You have to define the right model in the method get_model.
To run this is necessary to disable autoflush to prevent committing before the object is correctly formed and it is also necessary to commit after calling the method. I'm using flask-sqlalchemy.
Hope this can help or give some insight to someone that faces a similar problem :)
def save_obj(obj, checked=[]):
if obj in checked:
# if the object was already converted, retrieve the right object
model = get_model(obj.__mapper__.mapped_table.name)
return get_obj(obj, model)
checked.append(obj)
children = []
relations = obj.__mapper__.relationships.items()
# get all the relationships of this model (foreign keys)
for relation in relations:
model = get_model(relation[1].table.name)
if model:
# remove the cascade option for this object, so the children are not stored automatically in the session
relation[1]._cascade = CascadeOptions('')
child = getattr(obj, relation[0])
if not child:
continue
# if the child is a set of children
if isinstance(child, list):
new_children = []
for ch in copy(child):
# convert the child
new_child = save_obj(ch, checked)
new_children.append(new_child)
children.append((relation[0], new_children))
else:
new_child = save_obj(child, checked)
children.append((relation[0], new_child))
# get the model of the object passed
model = get_model(obj.__mapper__.mapped_table.name)
new_obj = get_obj(obj, model)
# set all the children in this object
for child in children:
if child[1]:
setattr(new_obj, child[0], child[1])
checked.append(new_obj)
session.add(new_obj)
return new_obj
def get_model(table_name):
# get the right model for this object
for table in db.Model._decl_class_registry.values():
if hasattr(table, '__tablename__') and table.__tablename__ == table_name:
return table
return None
def create_new_obj(obj, model):
params = obj.__dict__.copy()
params.pop('_sa_instance_state')
return model(**params)
def get_obj(child, model):
# check if the object is already stored in the db
child_in_db = session.query(model).get(child.id)
if child_in_db:
return child_in_db
# check if the object is already in the session
for s in session.new:
if type(s) == model and s.id == child.id:
return s
return create_new_obj(child, model)
I need some models for instance following:
Work - e.g. works of literature.
Worker - e.g. composer, translator or something similar has contribution to work.
Thus, a 'type' field is required to distinguish workers by division of work. As SQLAlchemy's documentation, this case can benifit from association object like following:
class Work(base):
id = Column(Integer, primary_key=True)
name = Column(String(50))
description = Column(Text)
class Worker(base):
id = Column(Integer, primary_key=True)
name = Column(String(50))
description = Column(Text)
class Assignment(base):
work_id = Column(Integer, Foreignkey('work.id'), primary_key=True)
worker_id = Column(Integer, Foreignkey('worker.id'), primary_key=True)
type = Column(SmallInteger, nullable=True)
Nonetheless, how to take advantage of backref and alternatvie join condition for building relation immediately to implement that each Work object can retrieve and modify corresponding Worker(s) via different attributions for distinction. For example:
work = session.query(Work).get(1)
work.name
>>> 'A Dream of The Red Mansions'
work.composers
>>> [<Worker('Xueqin Cao')>]
work.translators
>>> [<Worker('Xianyi Yang')>, <Worker('Naidie Dai')>]
Vice versa:
worker = session.query(Worker).get(1)
worker.name
>>> 'Xueqin Cao'
worker.composed
>>> [<Work('A Dream of The Red Mansions')>]
worker.translated
>>> []
Adding secondaryjoin directly without secondary specified seems not feasible, besides, SQLAlchemy's docs notes that:
When using the association object pattern, it is advisable that the association-mapped table not be used as the secondary argument on a relationship() elsewhere, unless that relationship() contains the option viewonly=True. SQLAlchemy otherwise may attempt to emit redundant INSERT and DELETE statements on the same table, if similar state is detected on the related attribute as well as the associated object.
Then, is there some way to build these relations elegantly and readily ?
There's three general ways to go here.
One is, do a "vanilla" setup where you have "work"/"workers" set up without distinguishing on "type" - then, use relationship() for "composer", "composed", "translator", "translated" by using "secondary" to Assignment.__table__ along with custom join conditions, as well as viewonly=True. So you'd do writes via the vanilla properties only. A disadvantage here is that there's no immediate synchronization between the "vanilla" and "specific" collections.
Another is, same with the "vanilla" setup, but just use plain Python descriptors to give "composer", "composed", "translator", "translated" views in memory, that is, [obj.worker for obj in self.workers if obj.type == 'composer']. This is the simplest way to go. Whatever you put in the "vanilla" collections shows right up in the "filtered" collection, the SQL is simple, and there's fewer SELECT statements in play (one per Worker/Work instead of N per Worker/Work).
Finally, the approach that's closest to what you're asking, with primary joins and backrefs, but note with the association object, the backrefs are between Work/Assignment and Assignment/Worker, but not between Work/Worker directly. This approach probably winds up using more SQL to get at the results but is the most complete, and also has the nifty feature that the "type" is written automatically. We're also using a "one way backref", as Assignment doesn't have a simple way of relating back outwards (there's ways to do it but it would be tedious). Using a Python function to automate creation of the relationships reduces the boilerplate, and note here I'm using a string for "type", this can be an integer if you add more arguments to the system:
from sqlalchemy import *
from sqlalchemy.orm import *
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.ext.associationproxy import association_proxy
Base = declarative_base()
def _work_assignment(name):
assign_ = relationship("Assignment",
primaryjoin="and_(Assignment.work_id==Work.id, "
"Assignment.type=='%s')" % name,
back_populates="work", cascade="all, delete-orphan")
assoc = association_proxy("%s_assign" % name, "worker",
creator=lambda worker: Assignment(worker=worker, type=name))
return assoc, assign_
def _worker_assignment(name):
assign_ = relationship("Assignment",
primaryjoin="and_(Assignment.worker_id==Worker.id, "
"Assignment.type=='%s')" % name,
back_populates="worker", cascade="all, delete-orphan")
assoc = association_proxy("%s_assign" % name, "work",
creator=lambda work: Assignment(work=work, type=name))
return assoc, assign_
class Work(Base):
__tablename__ = 'work'
id = Column(Integer, primary_key=True)
name = Column(String(50))
description = Column(Text)
composers, composer_assign = _work_assignment("composer")
translators, translator_assign = _work_assignment("translator")
class Worker(Base):
__tablename__ = 'worker'
id = Column(Integer, primary_key=True)
name = Column(String(50))
description = Column(Text)
composed, composer_assign = _worker_assignment("composer")
translated, translator_assign = _worker_assignment("translator")
class Assignment(Base):
__tablename__ = 'assignment'
work_id = Column(Integer, ForeignKey('work.id'), primary_key=True)
worker_id = Column(Integer, ForeignKey('worker.id'), primary_key=True)
type = Column(String, nullable=False)
worker = relationship("Worker")
work = relationship("Work")
e = create_engine("sqlite://", echo=True)
Base.metadata.create_all(e)
session = Session(e)
ww1, ww2, ww3 = Worker(name='Xueqin Cao'), Worker(name='Xianyi Yang'), Worker(name='Naidie Dai')
w1 = Work(name='A Dream of The Red Mansions')
w1.composers.append(ww1)
w1.translators.extend([ww2, ww3])
session.add(w1)
session.commit()
work = session.query(Work).get(1)
assert work.name == 'A Dream of The Red Mansions'
assert work.composers == [ww1]
assert work.translators == [ww2, ww3]
worker = session.query(Worker).get(ww1.id)
assert worker.name == 'Xueqin Cao'
assert worker.composed == [work]
assert worker.translated == []
worker.composed[:] = []
# either do this...
session.expire(work, ['composer_assign'])
# or this....basically need composer_assign to reload
# session.commit()
assert work.composers == []