before I start contriving a minimal example with a lot of sqlalchemy boilerplate stuff, maybe I can explain the concept theoretically.
I have a package "foo" that defines some tables in a database schema "foo_db" in the standard ORM manner:
class FooTable(Base):
__tablename__ = 'foo_data'
id = Column(Integer, primary_key=True)
The package is typically used stand-alone with its own database schema, "foo_db."
A separate package "bar" has its own schema "bar_db" with its own tables, but it also needs to use the tables of "foo" on "foo_db," and it has foreign keys into foo_db's tables (both schemas obviously are on the same server):
from foo.models import Base, FooTable
class BarTable(Base):
__tablename__ = 'bar_data'
id = Column(Integer, primary_key=True)
foo_id = Column(ForeignKey('foo_db.foo_data.id'))
foo = relationship(FooTable)
When I try to use this code I get errors like these:
(MySQLdb._exceptions.ProgrammingError) (1146, "Table 'bar_db.foo_data' doesn't exist")
The only way I found to get around this is to literally re-define FooTable in package bar:
class FooTable(Base):
__tablename__ = 'foo_data'
__table_args__ = {'schema': 'foo_db'}
id = Column(Integer, primary_key=True)
This is very obviously not how it should be done. Any suggestions?
Related
I use sqlalchemy's ORM library to interface with one of our application's databases. On one host, the database backend of an application instance is PostgreSQL, while on another, we use MySQL for historical reasons.
The schemas (and relationships) are identical on both backends, but the table and column names in PostgreSQL are lowercase, while on MySQL they are uppercase. Is there any way I can adapt my schema in sqlalchemy such that I don't need to duplicate code?
The following is an example of my schema. Hint, the application is a Confluence wiki ;)
from sqlalchemy import (
BigInteger,
Column,
DateTime,
String,
)
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class Space(Base):
__tablename__ = 'spaces'
spaceid = Column(BigInteger, primary_key=True)
spacename = Column(String(255))
spacekey = Column(String(255), nullable=False, unique=True)
lowerspacekey = Column(String(255), nullable=False, index=True)
spacedescid = Column(BigInteger, index=True)
homepage = Column(BigInteger, index=True)
creator = Column(String(255), index=True)
creationdate = Column(DateTime, index=True)
lastmodifier = Column(String(255), index=True)
lastmoddate = Column(DateTime)
spacetype = Column(String(255))
spacestatus = Column(String(255), index=True)
Thanks to #rfkortekaas' suggestion, I managed to solve the issue by automapping the schema and subsequently overriding the naming scheme for table and column names.
The documentation of sqlalchemy has two very useful pages:
Overriding Naming Schemes
Intercepting Column Definitions
For posterity's sake:
from sqlalchemy import event
from sqlalchemy.ext.automap import automap_base
from .utilities import to_pascal_case, to_snake_case
AutomapBase = automap_base()
#event.listens_for(AutomapBase.metadata, "column_reflect")
def column_reflect(inspector, table, column_info):
column_info['key'] = to_snake_case(column_info['name'])
def classname_for_table(base, table_name, table):
return to_pascal_case(table_name)
if __name__ == '__main__':
dbcs = 'postgresql://username:password#127.0.0.1:3306/database'
automap_engine = create_engine(dbcs, future=True, echo=verbose > 2)
AutomapBase.prepare(
autoload_with=automap_engine,
classname_for_table=classname_for_table,
)
# Access the automapped classes with AutomapBase.classes.<class_name>
I am trying to save a list of VLAN IDs per network port and also per network circuit. The list itself is something like this:
class ListOfVlanIds(Base):
__tablename__ = 'listofvlanids'
id = Column(Integer, primary_key=True)
listofvlanids_name = Column('listofvlanids_name', String, nullable = True)
And I then have a Port
class Port(Base):
__tablename__ = 'ports'
id = Column(Integer, primary_key=True)
listofvlanids_id = Column('listofvlanids_id', ForeignKey('ListOfVlanIds.id'), nullable = True)
and a Circuit:
class Circuit(Base):
__tablename__ = 'circuits'
id = Column(Integer, primary_key=True)
listofvlanids_id = Column('listofvlanids_id', ForeignKey('ListOfVlanIds.id'), nullable = True)
Running code like this results (for me) in a sqlalchemy.exc.NoReferencedTableError error on the ForeignKey.
Looking for the error I read I should add a relationship back from the list. I haven't found a way (or an example) where I can build this from both Port and Circuit. What am I missing?
Creating a list table for Ports and Circuits just moves the problem downstream, since a VLAN ID is it's own table... I'd love to be able to use ORM, instead of having to write (a lot of) SQL by hand.
ForeignKey expects a table and column name, not model and attribute name, so it should be ForeignKey('listofvlanids.id').
I'm using alembic to maintain my tables. At the same time, I update my models using the declarative way.
This is one the alembic's table:
op.create_table(
'groups',
Column('id', Integer, Sequence('group_id_seq'), primary_key=True),
Column('name', Unicode(50)),
Column('description', Unicode(250)),
)
And the model is like the following:
class Group(Base):
__tablename__ = 'groups'
id = Column(Integer, Sequence('group_id_seq'), primary_key=True)
name = Column(Unicode(50))
description = Column(Unicode(250))
def __init__(self, name, description):
self.description = description
self.name = name
You can see, I'm using the Sequence in both the alembic migration and in the declarative model.
But I have noticed that when using PostgreSQL (v9.1) no sequences are created by alembic, and so the models fail to create instances since they will use the nextval(<sequence name>) clause.
So, how can I create my alembic migrations so that the sequences are truly generated in postgresql?
Just add following to your model:
field_seq = Sequence('groups_field_seq')
field = Column(Integer, field_seq, server_default=field_seq.next_value())
And add following to your migration file (before creating table):
from sqlalchemy.schema import Sequence, CreateSequence
op.execute(CreateSequence(Sequence('groups_field_seq')))
Found a hint at https://bitbucket.org/zzzeek/alembic/issue/60/autogenerate-for-sequences-as-well-as#comment-4100402
Following the CreateSequence found in the previous link I still have to jump through several hoops to make my migrations works in SQLite and PostgreSQL. Currently I have:
def dialect_supports_sequences():
return op._proxy.migration_context.dialect.supports_sequences
def create_seq(name):
if dialect_supports_sequences():
op.execute(CreateSequence(Sequence(name)))
And then call the create_seq whenever I need it.
Is this the best practice?
Not sure if I got your question right but as nobody else chose to answer, here is how I get perfectly normal ids:
Alembic:
op.create_table('event',
sa.Column('id', sa.INTEGER(), autoincrement=True, nullable=False),
The class:
class Event(SQLADeclarativeBase):
__tablename__ = 'event'
id = Column(Integer, primary_key = True)
I ran into this same issue recently and here is how i solved it.
op.execute("create sequence SEQUENCE_NAME")
I ran the above command inside the upgrade function and for the downgrade run the below code inside the downgrade function respectively.
op.execute("drop sequence SEQUENCE_NAME")
I have two databases that I'm working with in with Python using SQLAlchemy, the databases share table names and therefore I'm getting an error message when running the code.
The error message is :
sqlalchemy.exc.InvalidRequestError: Table 'wo' is already defined for this MetaData instance. Specify 'extend_existing=True' to redefine options and columns on an existing Table object.
The simplified code is below:
from sqlalchemy import create_engine, Column, Integer, String, DateTime, ForeignKey
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker, relationship, backref
from mysql.connector.connection import MySQLConnection
Base = declarative_base()
def get_characterset_info(self):
return self.get_charset()
MySQLConnection.get_characterset_info = MySQLConnection.get_charset
mysqlengine = create_engine('mysql+mysqlconnector://......../mp2', echo=True)
MYSQLSession = sessionmaker(bind=mysqlengine)
mysqlsession= MYSQLSession()
MP2engine = create_engine('mssql+pyodbc://......../mp2', echo=True)
MP2Session = sessionmaker(bind=MP2engine)
mp2session= MP2Session()
class MYSQLWo(Base):
__tablename__= 'wo'
wonum = Column(String, primary_key=True)
taskdesc = Column(String)
comments = relationship("MYSQLWocom", order_by="MYSQLWocom.wonum", backref='wo')
class MYSQLWocom (Base):
__tablename__='wocom'
wonum = Column(String, ForeignKey('wo.wonum'), primary_key=True)
comments = Column(String, primary_key=True)
class MP2Wo(Base):
__tablename__= 'wo'
wonum = Column(String, primary_key=True)
taskdesc = Column(String)
comments = relationship("MP2Wocom", order_by="MP2Wocom.wonum", backref='wo')
class MP2Wocom (Base):
__tablename__='woc'
wonum = Column(String, ForeignKey('wo.wonum'), primary_key=True)
location = Column(String)
sublocation1 = Column(String)
texts = Column(String, primary_key=True)
How do I deal with databases having the same table structure? I'm guessing it has something to do with the MetaData instance, but the SQLAlchemy documentation gets a little confusing when talking about the difference in the class declarative and classical usage..
Since in reality the tables had different structures, the solution was to simply create a separate declarative base. If the tables indeed had the same structure, I would have only needed one class for both tables.
Base = declarative_base()
Base2 = declarative_base() #this is all I needed
class MYSQLWo(Base):
....
class MYSQLWocom(Base):
....
class MP2Wo(Base2):
....
class MP2Wocom(Base2)
http://groups.google.com/group/sqlalchemy/browse_thread/thread/afe09d6387a4dc69?hl=en
You can use one db instance with two Model to bypass this problem.
And this can also be used to implement a master/slave use case in Flask-SQLAlchemy.
Just like this:
app = Flask(__name__)
app.config['SQLALCHEMY_BINDS'] = {'rw': 'rw', 'r': 'r'}
db = SQLAlchemy(app)
db.Model_RW = db.make_declarative_base()
class A(db.Model):
__tablename__ = 'common'
class B(db.Model_RW):
__tablename__ = 'common'
I have some code here. I recently added this root_id parameter. The goal of that is to let me determine whether a File belongs to a particular Project without having to add a project_id FK into File (which would result in a model cycle.) Thus, I want to be able to compare Project.directory to File.root. If that is true, File belongs to Project.
However, the File.root attribute is not being autogenerated for File. My understanding is that defining a FK foo_id into table Foo implicit creates a foo attribute to which you can assign a Foo object. Then, upon session flush, foo_id is properly set to the id of the assigned object. In the snippet below that is clearly being done for Project.directory, but why not for File.root?
It definitely seems like it has to do with either 1) the fact that root_id is a self-referential FK or 2) the fact that there are several self-referential FKs in File and SQLAlchemy gets confused.
Things I've tried.
Trying to define a 'root' relationship() - I think this is wrong, this should not be represented by a join.
Trying to define a 'root' column_property() - allows read access to an already set root_id property, but when assigning to it, does not get reflected back to root_id
How can I do what I'm trying to do? Thanks!
from sqlalchemy import create_engine, Column, ForeignKey, Integer, String
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import backref, relationship, scoped_session, sessionmaker, column_property
Base = declarative_base()
engine = create_engine('sqlite:///:memory:', echo=True)
Session = scoped_session(sessionmaker(bind=engine))
class Project(Base):
__tablename__ = 'projects'
id = Column(Integer, primary_key=True)
directory_id = Column(Integer, ForeignKey('files.id'))
class File(Base):
__tablename__ = 'files'
id = Column(Integer, primary_key=True)
path = Column(String)
parent_id = Column(Integer, ForeignKey('files.id'))
root_id = Column(Integer, ForeignKey('files.id'))
children = relationship('File', primaryjoin=id==parent_id, backref=backref('parent', remote_side=id), cascade='all')
Base.metadata.create_all(engine)
p = Project()
root = File()
root.path = ''
p.directory = root
f1 = File()
f1.path = 'test.txt'
f1.parent = root
f1.root = root
Session.add(f1)
Session.add(root)
Session.flush()
# do this otherwise f1 will be returned when calculating rf1
Session.expunge(f1)
rf1 = Session.query(File).filter(File.path == 'test.txt').one()
# this property does not exist
print rf1.root
My understanding is that defining a FK foo_id into table Foo implicit creates a foo attribute to which you can assign a Foo object.
No, it doesn't. In the snippet, it just looks like it is being done for Project.directory, but if you look at the SQL statements being echo'ed, there is no INSERT at all for the projects table.
So, for it to work, you need to add these two relationships:
class Project(Base):
...
directory = relationship('File', backref='projects')
class File(Base):
...
root = relationship('File', primaryjoin='File.id == File.root_id', remote_side=id)