I'm considering porting my app to the SQLAlchemy as it's much more extensive than my own ORM implementation, but all the examples I could find show how to set the schema name at class declaration rather than dynamically at runtime.
I need to map my objects to Postgres tables from multiple schemas. Moreover, the application creates new schemas in runtime and I need to map new instances of the class to rows of the table from that new schema.
Currently, I use my own ORM module where I just provide the schema name as an argument when creating new instances of a class (I call class' method with the schema name as an argument and it returns an object(s) that holds the schema name). The class describes a table that can exist in many schemas. The class declaration doesn't contain information about schema, but instances of that class do contain it and include it when generating SQL statements.
This way, the application can work with many schemas simultaneously and even create foreign keys in tables from "other" schemas to the "main" table in the public schema. In such a way it is also possible to delete data in other schemas cascaded when deleting the row in the public schema.
The SQLAlchemy gives this example to set the schema for the table (documentation):
metadata_obj = MetaData(schema="remote_banks")
financial_info = Table(
"financial_info",
metadata_obj,
Column("id", Integer, primary_key=True),
Column("value", String(100), nullable=False),
)
But on ORM level, when I declare the class, I should pass an already constructed table (example from documentation):
metadata = MetaData()
group_users = Table(
"group_users",
metadata,
Column("user_id", String(40), nullable=False),
Column("group_id", String(40), nullable=False),
UniqueConstraint("user_id", "group_id"),
)
class Base(DeclarativeBase):
pass
class GroupUsers(Base):
__table__ = group_users
__mapper_args__ = {"primary_key": [group_users.c.user_id, group_users.c.group_id]}
So, the question is: is it possible to map class instances to tables/rows from dynamically created database schemas (in runtime) in SQLAlchemy? The way of altering the connection to set the current schema is not acceptable to me. I want to work with all schemas simultaneously.
I'm free to use the newest SQLAlchemy 2.0 (currently in BETA release).
You can set the schema per table so I think you have to make a table and class per schema. Here is a made up example. I have no idea what the ramifications are of changing the mapper registry during runtime. Especially as I have done below, mid-transaction and what would happen with threadsafety. You could probably use a master schema list table in public and lock it or lock the same row across connections to syncronize the schema list and provide threadsafety when adding a schema. I'm suprised it works. Kind of cool.
import sys
from sqlalchemy import (
create_engine,
Integer,
MetaData,
Float,
event,
)
from sqlalchemy.schema import (
Column,
CreateSchema,
Table,
)
from sqlalchemy.orm import Session
from sqlalchemy.orm import registry
username, password, db = sys.argv[1:4]
engine = create_engine(f"postgresql+psycopg2://{username}:{password}#/{db}", echo=True)
metadata = MetaData()
mapper_registry = registry()
def map_class_to_some_table(cls, table, entity_name, **mapper_kwargs):
newcls = type(entity_name, (cls,), {})
mapper_registry.map_imperatively(newcls, table, **mapper_kwargs)
return newcls
class Measurement(object):
pass
units = []
cls_for_unit = {}
tbl_for_unit = {}
def add_unit(unit, create_bind=None):
units.append(unit)
schema_name = f"unit_{unit}"
if create_bind:
create_bind.execute(CreateSchema(schema_name))
else:
event.listen(metadata, "before_create", CreateSchema(schema_name))
cols = [
Column("id", Integer, primary_key=True),
Column("value", Float, nullable=False),
]
# One table per schema.
tbl_for_unit[unit] = Table("measurement", metadata, *cols, schema=schema_name)
if create_bind:
tbl_for_unit[unit].create(create_bind)
# One class per schema.
cls_for_unit[unit] = map_class_to_some_table(
Measurement, tbl_for_unit[unit], Measurement.__name__ + f"_{unit}"
)
for unit in ["mm", "m"]:
add_unit(unit)
metadata.create_all(engine)
with Session(engine) as session, session.begin():
# Create a value for each unit (schema).
session.add_all([cls(value=i) for i, cls in enumerate(cls_for_unit.values())])
with Session(engine) as session, session.begin():
# Read back a value for each unit (schema).
print(
[
(unit, cls.__name__, cls, session.query(cls).first().value)
for (unit, cls) in cls_for_unit.items()
]
)
with Session(engine) as session, session.begin():
# Add another unit, add a value, flush and then read back.
add_unit("km", create_bind=session.bind)
session.add(cls_for_unit["km"](value=100.0))
session.flush()
print(session.query(cls_for_unit["km"]).first().value)
Output of last add_unit()
2022-12-16 08:16:13,446 INFO sqlalchemy.engine.Engine CREATE SCHEMA unit_km
2022-12-16 08:16:13,446 INFO sqlalchemy.engine.Engine [no key 0.00015s] {}
2022-12-16 08:16:13,447 INFO sqlalchemy.engine.Engine COMMIT
2022-12-16 08:16:13,469 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2022-12-16 08:16:13,469 INFO sqlalchemy.engine.Engine
CREATE TABLE unit_km.measurement (
id SERIAL NOT NULL,
value FLOAT NOT NULL,
PRIMARY KEY (id)
)
Ian Wilson posted a great answer to my question which I'm going to use.
About the same time I got an idea of how it can work and would like to post it here just as a very simple example. I think this is the same mechanism behind it as posted by Ian.
This example only "reads" an object from the schema that can be referenced at runtime.
from sqlalchemy import create_engine, Column, Integer, String, MetaData
from sqlalchemy.orm import DeclarativeBase
from sqlalchemy.orm import sessionmaker
import psycopg
engine = create_engine(f"postgresql+psycopg://user:password#localhost:5432/My_DB", echo=True)
Session = sessionmaker(bind=engine)
session = Session()
class Base(DeclarativeBase):
pass
class A(object):
__tablename__ = "my_table"
id = Column("id", Integer, primary_key=True)
name = Column("name", String)
def __repr__(self):
return f"A: {self.id}, {self.name}"
metadata_obj = MetaData(schema="my_schema") # here we create new mapping
A1 = type("A1", (A, Base), {"metadata": metadata_obj}) # here we make a new subclass with desired mapping
data = session.query(A1).all()
print(data)
This info helped me to come to this solution:
https://github.com/sqlalchemy/sqlalchemy/wiki/EntityName
"... SQLAlchemy mapping makes modifications to the mapped class, so it's not really feasible to have many mappers against the exact same class ..."
This means a separate class must be created in runtime for each schema
I am trying to use two tables inside one django query. But my query is resulting as "invalid JSON" format.
I want to filter data in Request table by (status="Aprv"). The Request table contains attributes 'from_id' and 'to_id'.
The uid is the id of the user who is currently logged in.
If the current user(uid) is having the 'from_id' of Requests table, the query should return data of 'to_id' from the 'RegUsers' table.
If the current user(uid) is having the 'to_id' of Requests table, the query should return data of 'from_id' from the 'RegUsers' table.
class frnds(APIView):
def post(self, request):
uid = request.data['uid']
ob = Requests.objects.filter(status="Aprv")
fid = ob.value('from_id')
tid = ob.value('to_id')
if fid == uid:
obj = RegUsers.objects.filter(u_id=tid)
else:
obj = RegUsers.objects.filter(u_id=fid)
ser = android_serialiser(obj, many=True)
return Response(ser.data)
I don't want to use foreign keys.
Please Do Help me Correct the syntax.
The error message
You may need to serialize your data to JSON using Django's serializer first:
serializers.serialize('json', obj)
note that you need to import the serializers first from django core
from django.core import serializers
To avoid using JSON serializing in every request I can recommend you to take a look on Parsers on official Django site
https://www.django-rest-framework.org/api-guide/parsers/
Using the code shown below I can obtain the vendor type that corresponds to the SQLAlchemy generic type. In this case it is "VARCHAR(10)". How can I get the vendor type without creating a table?
engine = create_engine(DB_URL)
metadata_obj = MetaData()
table = Table('Table', metadata_obj,
Column('Column', types.String(10))
)
metadata_obj.create_all(bind=engine)
metadata_obj = MetaData()
metadata_obj.reflect(bind=engine)
print(metadata_obj.tables['Table'].columns[0].type)
You can't obtain the type directly, but you could use a mock_engine to generate the DDL as a string which can be parsed. A mock_engine must be coupled with a callable that will process the SQL expression object that it generates.
This snippet is based on the example code from the SQLAlchemy docs.
import sqlalchemy as sa
tbl = sa.Table('drop_me', sa.MetaData(), sa.Column('col', sa.String(10)))
def dump(sql, *multiparams, **params):
print(sql.compile(dialect=engine.dialect))
mock_engine = sa.create_mock_engine('postgresql://', executor=dump)
tbl.create(mock_engine)
Outputs
CREATE TABLE "Table" (
"Column" VARCHAR(10)
)
sqlalchemy.schema.CreateTable, could also be used, but binding it to an engine is deprecated, to be removed in SQLAlchemy 2.0.
from sqlalchemy.schema import CreateTable
print(CreateTable(tbl, bind=some_engine)
I wrote a general dbhandler module that can entangle data containers and uploade them to a mySQL database and is independent of the DB structure. Now I want to add a default or the possibility to shove the data into a sqlite DB. Structure-wise this is related to this question. The package looks like this:
dbhandler\
dbhandler.py
models\
meta.py
default\
default_DB_map.py
default_DB.cfg
default.cfg is the config file that describes the database for the dbhandler script. default_DB_map.py contains a map for each table of the DB, which inherits from BASE:
from sqlalchemy import BigInteger, Column, Integer, String, Float, DateTime
from sqlalchemy import Date, Enum
from ..meta import BASE
class db_info(BASE):
__tablename__ = "info"
id = Column(Integer, primary_key=True)
name = Column(String)
project = Column(String)
manufacturer = Column(String)
...
class db_probe(BASE):
__tablename__ = "probe"
probeid = Column(Integer, primary_key=True)
id = Column(Integer)
paraX = Column(String)
...
In meta.py I initialize the declarative_base object:
from sqlalchemy.ext.declarative import declarative_base
BASE = declarative_base()
And eventually, I import BASE within the dbhandler.py and create the engine and session:
"DBHandler module"
...
import sqlalchemy
from sqlalchemy.orm import sessionmaker
from models import meta #pylint: disable=E0401
....
class DBHandler(object):
"""Database handling
Methods:
- get_dict: returns table row
- add_item: adds dict to DB table
- get_table_keys: gets list of all DB table keys
- get_values: returns all values of key in DB table
- check_for_value: checks if value is in DB table or not
- upload: uploads data container to DB
- get_dbt: returns DBTable object
"""
def __init__(self, db_cfg=None):
"""Load credentials, DB structure and name of DB map from cfg file,
create DB session. Create DBTable object to get table names of DB
from cfg file, import table classes and get name of primary keys.
Args:
- db_cfg (yaml) : contains infos about DB structure and location
of DB credentials.
Misc:
- cred = {"host" : "...",
"database" : "...",
"user" : "...",
"passwd" : "..."}
"""
...
db_cfg = self.load_cfg(db_cfg)
if db_cfg["engine"] == "sqlite":
engine = sqlalchemy.create_engine("sqlite:///mySQlite.db")
meta.BASE.metadata.create_all(engine)
session = sessionmaker(bind=engine)
self.session = session()
elif db_cfg["engine"] == "mysql+mysqlconnector":
cred = self.load_cred(db_cfg["credentials"])
engine = sqlalchemy.create_engine(db_cfg["engine"]
+ "://"
+ cred["user"] + ":"
+ cred["passwd"] + "#"
+ cred["host"] + ":"
+ "3306" + "/"
+ cred["database"])
session = sessionmaker(bind=engine)
self.session = session()
else:
self.log.warning("Unkown engine in DB cfg...")
# here I'm importing the table classes stated in the config file
self.dbt = DBTable(map_file=db_cfg["map"],
table_dict=db_cfg["tables"],
cr_dict=db_cfg["cross-reference"])
I'm obviously doing something wrong within the if db_cfg["engine"] == "sqlite": paragraph, but I can't figure out what.
The script is working just fine with the mySQL engine. When I initialize the handler object I'm getting an empty mySQLite.db file.
Adding something with that session yields:
(sqlite3.OperationalError) no such table: info....
I can however use something like ´sqlalchemy.inspect´ on a table object without any errors. So I have the correct table objects at hand, but they are somehow not connected to the base?
For SQLite, apperently the import of the table classes needs to happen before the DB is created.
# here I'm importing the table classes stated in the config file
self.dbt = DBTable(map_file=db_cfg["map"],
table_dict=db_cfg["tables"],
cr_dict=db_cfg["cross-reference"])
(which is done via pydoc.locate btw) has to be done before
engine = sqlalchemy.create_engine("sqlite:///mySQlite.db")
meta.BASE.metadata.create_all(engine)
session = sessionmaker(bind=engine)
self.session = session()
is called. I thought this was not important since I imported BASE at the beginning and since it works just fine when using a different engine.
Following on from this question SQLAlchemy: Modification of detached object.
This makes a copy of the object fine, but it loses any many-to-many relationships the original object had. Is there a way to copy the object and any many-to-many relationships as well?
Cheers!
I got this to work by walking the object graph and doing the expunge(), make_transient() and id = None steps on each object in the graph as described in SQLAlchemy: Modification of detached object.
Here is my sample code. The agent has at most one campaign.
from sqlalchemy.orm.session import make_transient
def clone_agent(id):
s = app.db.session
agent = s.query(Agent).get(id)
c = None
# You need to get child before expunge agent, otherwise the children will be empty
if agent.campaigns:
c = agent.campaigns[0]
s.expunge(c)
make_transient(c)
c.id = None
s.expunge(agent)
agent.id = None
# I have unique constraint on the following column.
agent.name = agent.name + '_clone'
agent.externalId = - agent.externalId # Find a number that is not in db.
make_transient(agent)
s.add(agent)
s.commit() # Commit so the agent will save to database and get an id
if c:
assert agent.id
c.agent_id = agent.id # Attach child to parent (agent_id is a foreign key)
s.add(c)
s.commit()