I'm setting up alembic for our project, which is already really big, and has a lot of tables. The thing is that our project's DB has been managed via SQL for a long time, and our Alchemy models are almost all reflected like so (I obscured the name, but the rest is all directly from our code):
class SomeModel(Base, BaseModelMixin):
"""
Model docstring
"""
"""Reflect Table"""
__table__ = Table('some_models', metadata, autoload=True)
What's happening is that when I create an automatic migration, a lot of drop table (and a lot of create table) operations are created for some reason. I assumed it's because the model class doesn't explicitly define the tables, but I don't see why that would drop the tables as well.
I'm making sure all model definitions are processed before setting the target_metadata variable in env.py:
# this imports every model in our app
from shiphero_app.utils.sql_dependencies import import_dependencies
import_dependencies()
from shiphero_app.core.database import Base
target_metadata = Base.metadata
Any ideas of what might I be missing here?
This is probably what you are looking for - this makes Alembic ignore predefined tables:
https://alembic.sqlalchemy.org/en/latest/cookbook.html#don-t-generate-any-drop-table-directives-with-autogenerate
Unfortunately this also prevents Alembic from dropping tables within scope as well
Related
Using SQLAlchemy with flask_sqlalchemy and alembic for PostgreSQL. I have the following definition for a field:
date_updated = db.Column(db.DateTime, server_default=db.func.now(), server_onupdate=db.func.now())
However the field never updates when the record is modified. It is set on create and never updates. This is what is generated by alembic to create the table:
sa.Column('date_updated', sa.DateTime(), server_default=sa.text('now()'), nullable=True),
So it's no wonder that it's not being updated, since the server_onupdate param doesn't seem to be getting past alembic.
I'm not sure of the right way to do this. The SQLAlchemy documentation is frustratingly complex and unclear where this is concerned.
Edit: From looking at how to do this in PostgreSQL directly, it looks like it requires the use of triggers. I would prefer to do it at the DB level rather than at the application level if possible, but I don't know if I can add a trigger through SQLAlchemy. I can add it directly at the DB but that could break when migrations are applied.
The way you say "I'm not sure of the right way to do this", I'm not sure if you mean specifically updating the date on the server side, or just updating it in general.
If you just want to update it and it doesn't matter how, the cleanest way in my opinion is to use event listeners.
Here's an example using the normal sqlalchemy, it will probably be the same (or at least very similar) in flask-sqlalchemy:
from datetime import datetime
from sqlalchemy import event
#event.listens_for(YourModel, 'before_insert')
#event.listens_for(YourModel, 'before_update')
def date_insert(mapper, connection, target):
target.date_updated = datetime.utcnow()
My application works with huge database contents for which we can't always migrate to the latest database schema version we designed at software upgrade time. And yes, we're using database migration tools (Alembic), but this doesn't yet allow us to have Python application code that can handle multiple schema versions. At some point in time when the system downtime is accepted, a migration to the latest version will be performed, but in the meantime the application code is required to be able to handle both (multiple) versions.
So, for example, we can offer Feature X only if the database migration has been performed. It should also be able to function if the migration hasn't been performed yet, but then doesn't offer Feature X with a warning printed in the log. I see several ways of doing this with SQLAlchemy, but they all feel hackish ugly. I'd like to get some advice on how to handle this properly.
Example:
Base = declarative_base()
class MyTable(Base):
__tablename__ = 'my_table'
id = Column(MyCustomType, nullable=False, primary_key=True)
column_a = Column(Integer, nullable=True)
column_b = Column(String(32)) # old schema
column_b_new = Column(String(64)) # new schema
New schema version has a new column replacing the old one. Note that both the column name and column data specification change.
Another requirement is that the use of this class from other parts of the code must be transparent to support backwards compatibility. Other components of the product will get awareness of the new column/datatype later. This means that if initialized with the new schema, the old attribute still has to be functional. So, if a new object is created with Mytable(column_a=123, column_b="abc"), it should work with both the new and old schema.
What would be the best way to move from here? Options to support two schemas I see:
Define two MyTable classes for both schema versions, then determine the schema version (how?) and based on the result, use either version. With this approach I think this requires the logic for which schema to use in every place the MyTable class is used, and therefore breaks easily. Link the class attributes to each other (column_b = column_b_new) for backward compatibility (does that actually work?).
Initialize the database normally and alter the MyTable class object based on the schema version detected. I'm not sure whether SQLAlchemy support changing the attributes (columns) of a declarative base class after initialization.
Create a custom Mapper configuration as desribed here: http://docs.sqlalchemy.org/en/rel_1_1/orm/extensions/declarative/basic_use.html#mapper-configuration I'm not sure how to get from this SQLAlchemy feature to my desired solution. Perhaps a custom attribute set dynamically can be checked in a custom mapper function?
I am a newbie to wewb2py,is it possible to create at runtime a model of a legacy database, for using DAL with it? I saw that there are some scripts that create the model file, but I do not know whether it is correct to put this file in the model directory of my application, I think not, I did some experiments, I can connect to the database with DAL querying its tables and for every table I can get the definition of the fields, the I tried to define the table with define_table,it works but try to create the table on the database and return an error because the table already exists; this is the relevant part of my code:
conn_string = "mysql://{0}:{1}#{2}/{3}".format(user,pwd,host,db_name)
db = DAL(conn_string)
db.define_table('test1',Field('prova','string'))
it works only the first time, when the table test1 does not exist yet on the database, I do not need to create the tables only work with their data, can you put me on the right way?
db = DAL(conn_string, migrate_enabled=False)
The above will prevent web2py from doing any migrations, including attempting to create any tables.
In my MVC application I imported a stored procedure as a function import (in EDMX File)
The stored procedure changed (new parameter) but I don't know how to update it.
For now I just deleted and re-add it manually, but I would like to know what's the best way to achieve this.
UPDATE:
I found an option in the update model from database wizar, there is a refresh tab there, but when attempting to refresh, It does not create the new parameter
First, to understand your problem, what you need to know is that the EDMX file is just an XML file that contains 3 different sections:
CSDL: Conceptual schema definition language
SSDL: Store schema definition language
MSL: Mapping specification language
The CSDL contains the entities and relationships that make up your conceptual model. The SSDL describes your DB model and the MSL is the mapping between the 2.
The “Update Model From DB” process will update the SSDL (change everything that is inconsistent with the current DB schema), it will only modify the CSDL in case you’ve added new things to your DB schema.
This is quite a normal behavior since your Conceptual schema may/should differ from your DB schema (unless you want your Domain model to look exactly like a DB model which obviously do not sound as OOP/DDD best practices).
The Function Import mechanism works the same way. As soon as you import a new Stored Procedure, a new FunctionImport Element will be added in the CSDL. This new element will describe the SP including its parameters. As I said, only new things will be added in the CSDL if you run the Update Wizard, that's why if you change any SP parameter in your DB, it won't be changed in the conceptual model.
To force the conceptual model to change, open your EDMX, go in your Model Browser, expand the Function Import entry:
If you want everything to be refreshed, simply remove the function and import it again
If you want to change input parameters, expand the right function, remove the parameters and update the function
If you want to update only the return type, right click on the right function, select update and click on Update
I would like to implement a custom database initialization strategy so that I can:
generate the database if not exists
if model change create only new tables
if model change create only new fields without dropping the table and losing the data.
Thanks in advance
You need to implement IDatabaseInitializer interface.
Eg
public class MyInitializer : IDatabaseInitializer<MyDbContext>
{
public void InitializeDatabase(MyDbContext context)
{
//your logic here
}
}
And then set your initializer at your application startup
Database.SetInitializer<ProductCatalog>(new MyInitializer());
Here's an example
You will have to manually execute commands to alter the database.
context.ObjectContext.ExecuteStoreCommand("ALTER TABLE dbo.MyTable ADD NewColumn VARCHAR(20) NULL");
You can use a tool like SQL Compare to script changes.
There is a reason why this doesn't exist yet. It is very complex and moreover IDatabaseInitializer interface is not very prepared for such that (there is no way to make such initialization database agnostic). Your question is "too broad" to be answered to your satisfaction. With your reaction to #Eranga's correct answer you simply expect that somebody will tell you step by step how to do that but we will not - that would mean we will write the initializer for you.
What you need to do what you want?
You must have very good knowledge of SQL Server. You must know how does SQL server store information about database, tables, columns and relations = you must understand sys views and you must know how to query them to get data about current database structure.
You must have very good knowledge of EF. You must know how does EF store mapping information. You must be able to explore metadata get information about expected tables, columns and relations.
Once you have old database description and new database description you must be able to write a code which will correctly explore changes and create SQL DDL commands for changing your database. Even this look like the simplest part of the whole process this is actually the hardest one because there are many other internal rules in SQL server which cannot be violated by your commands. Sometimes you really need to drop table to make your changes and if you don't want to lose data you must first push them to temporary table and after recreating table you must push them back. Sometimes you are doing changes in constraints which can require temporarily turning constrains off, etc. There is good reason why tools which do this on SQL level (comparing two databases) are probably all commercial.
Even ADO.NET team doesn't implemented this and they will not implement it in the future. Instead they are working on something called migrations.
Edit:
That is true that ObjectContext can return you script for database creation - that is exactly what default initializers are using. But how it could help you? Are you going to parse that script to see what changed? Are you going to execute that script in another connection to use the same code as for current database to see its structure?
Yes you can create a new database, move data from the old database to a new one, delete the old one and rename a new one but that is the most stupid solution you can ever imagine and no database administrator will ever allow that. Even this solution still requires analysis of changes to create correct data transfer scripts.
Automatic upgrade is a wrong way. You should always prepare upgrade script manually with help of some tools, test it and after that execute it manually or as part of some installation script / package. You must also backup your database before you are going to do any changes.
The best way to achieve this is probably with migrations:
http://nuget.org/List/Packages/EntityFramework.SqlMigrations
Good blog posts here and here.