I am trying to do a migration to update the value of the column has_bubble_in_countries based on the has_bubble_v1 s column value.
I created before the upgrade() the table:
subscription_old_table = sa.Table(
'Subscription',
sa.MetaData(),
sa.Column('id', sa.Unicode(255), primary_key=True, unique=True, nullable=False),
sa.Column('has_bubble_v1', sa.Boolean, nullable=False, default=False),
sa.Column('has_bubble_in_countries', MutableList.as_mutable(ARRAY(sa.Enum(Country))), nullable=False, default=[], server_default='{}')
)
And then the upgrade() method looks like:
def upgrade():
connection = op.get_bind()
for subscription in connection.execute(subscription_old_table.select()):
if subscription.has_bubble_v1:
connection.execute(
subscription_old_table.update().where(
subscription_old_table.c.id == subscription.id
).values(
has_bubble_in_countries=subscription.has_bubble_in_countries.append(Country.NL),
)
)
# Then drop the column after the data has been migrated
op.drop_column('Subscription', 'has_bubble_v1')
All the rows in the database of has_bubble_in_countries column have this value {} when I check the database using pgadmin's interface.
When the upgrade() function gets to the update method it throws this error:
sqlalchemy.exc.IntegrityError: (psycopg2.errors.NotNullViolation) null value in column "has_bubble_in_countries" of relation "Subscription" violates not-null constraint
DETAIL: Failing row contains (keydsakwlkad, null, 2027-08-14 00:00:00+00,groot abonnement, big, {nl}, null, null, 2022-08-08 08:45:52.875931+00, 3482992, {}, f, null, null, null, t, 2011-05-23 08:55:20.538451+00, 2022-08-08 09:10:15.577283+00, ***null***).
[SQL: UPDATE "Subscription" SET has_bubble_in_countries=%(has_bubble_in_countries)s::country[] WHERE "Subscription".id = %(id_1)s]
[parameters: {'has_bubble_in_countries': None, 'id_1': '1pwohLmjftAZdIaJ'}]
The bolded value from the error is the value that is retrieved for the has_bubble_in_countries column even if it has a server_default='{}' and nullable=False.
Is there any possibility to add a configuration to alembic to recognize the server default s value when it is retrieved from the database? Or how can this be fixed?
I think the problem is actually that are you passing in the result of .append() which is None. Unlike other languages where it is common to return the altered list, append changes the list in place. I'm not sure that is a great idea for a core query result here but it seems to work. Also as far as I know, if you pass in NULL it doesn't trigger the default. The default is used when you pass in no value at all either when inserting or updating.
with Session(engine) as session, session.begin():
for subscription in session.execute(subscription_old_table.select()):
if subscription.has_bubble_v1:
# Append here.
subscription.has_bubble_in_countries.append(Country.NL)
# Then set values:
session.execute(
subscription_old_table.update().where(
subscription_old_table.c.id == subscription.id
).values(has_bubble_in_countries=subscription.has_bubble_in_countries,
)
)
Maybe cloning the list and then adding the element like this would be safer and clearer:
has_bubble_in_countries=subscription.has_bubble_in_countries[:] + [Country.NL]
Related
I need to add a new UUID column to an existing SQLAlchemy / MySQL table where.
For doing so I added in my database model:
class MyTable(db.Model):
uid = db.Column(db.BINARY(16), nullable=False, unique=True, default=uuid.uuid4)
Doing so generates the following alembic upgrade code which of course does not work as the default value if the new column is null:
op.add_column('my_table', sa.Column('uid', sa.BINARY(length=16), nullable=True))
op.create_unique_constraint(None, 'my_table', ['uid'])
I tried to extend the db.Column(db.BINARY(16), nullable=False, unique=True, default=uuid.uuid4) definition with an appropriate server_default=... parameter but wasn't able to find a parameter that generates for each row a new random UUID.
How to add a new column and generate for all existing rows a new random and unique UUID value?
A solution that uses sa.String instead of sa.BINARY would also be acceptable.
In the end I manually created the necessary UPDATE statements in the alembic update file so existing rows are assigned a unique UID.
For new entries default=uuid.uuid4 in the SQLAlchemy column definition is sufficient.
Note that the MySQL UUID() function generates timestamp based UUID values for the existing records, and new records created via Python/SQLAlchemy use uuid.uuid4 which generates a v4 (random) UUID. So both are UUIDs but you will see which UUIDs were generated by UUID() as they only differ in the first block when generating them using the UPDATE statement.
Using binary column type
class MyTable(db.Model):
uid = db.Column(db.BINARY(16), nullable=False, unique=True, default=uuid.uuid4)
def upgrade():
op.add_column('my_table', sa.Column('uid', sa.BINARY(length=16), nullable=False))
op.execute("UPDATE my_table SET uid = (SELECT(UUID_TO_BIN(UUID())))")
op.alter_column('my_table', 'uid', existing_type=sa.BINARY(length=16), nullable=False)
op.create_unique_constraint(None, 'my_table', ['uid'])
Using String/varchar column type
class MyTable(db.Model):
uid = db.Column(db.String(36), nullable=False, unique=True, default=uuid.uuid4)
def upgrade():
op.add_column('my_table', sa.Column('uid', sa.String(length=36), nullable=False))
op.execute("UPDATE my_table SET uid = (SELECT(UUID()))")
op.alter_column('my_table', 'uid', existing_type=sa.String(length=36), nullable=False)
op.create_unique_constraint(None, 'my_table', ['uid'])
As per mysqlalchemy's documentation on server_default:
A text() expression will be rendered as-is, without quotes:
Column('y', DateTime, server_default = text('NOW()'))
y DATETIME DEFAULT NOW()
Based on this, your server_default definition should look like this:
server_default=text('(UUID_TO_BIN(UUID())))')
However, if your mysql version is earlier than v8.0.12, then you cannot use the server side default like this, you need to use either the default with setting uuid from python or you need a trigger as specified in the following SO question: MySQL set default id UUID
I have this kind of model:
class A(Base):
id = Column(UUID(as_uuid=True), primary_key=True, server_default=text("uuid_generate_v4()"))
name = Column(String, nullable=False, unique=True)
property = Column(String)
parent_id = Column(UUID(as_uuid=True), ForeignKey(id, ondelete="CASCADE"))
children = relationship(
"A", cascade="all,delete", backref=backref("parent", remote_side=[id])
)
An id is created automatically by the server
I have a relationship from a model to itself (parent and children).
In the background I run a task that periodically receives a message with id of parent and list of pairs (name, property) of children. I would like to update the parent's children in table (Defined by name). Is there a way to do so without reading all children, see which one is missing (name not present is message), need to be updated (name exists but property has changed) or new (name not present in db)?
Do I need to set name to be my primary key and get rid of the UUID?
Thanks
I'd do a single query and compare the result against the message you receive. That way it's easier to handle both additions, removals and updates.
msg_parent_id = 5
msg_children = [('name', 'property'), ('name2', 'property2')]
stmt = select(A).where(A.parent_id == msg_parent_id)
children = session.execute(stmt).scalars()
# Example of determining what to change
name_map = {row.name: row for row in children}
for child_name, child_prop in msg_children:
# Child exists
if child_name in name_map:
# Edit child
if name_map[child_name].property != child_prop:
print(child_name, 'has changed to', property)
del name_map[child_name]
# Add child
else:
print(child_name, 'was added')
# Remove child
for child in name_map.values():
print(child, 'was removed')
Do I need to set name to be my primary key and get rid of the UUID?
Personally I'd add a unique constraint on the name, but still have a separate ID column for the sake of relationships.
Edit for a more ORM orientated way. I believe you can already use A.children = [val1, val2], which is really what you need.
In the past I have used this answer on how to intercept the call, parse the input data, and fetch the existing record from the database if it exists. As part of the that call you could update the property of that record.
Finally use a cascade on the relationship to automatically delete records when parent_id is set to None.
How do I get a constraint's default name as generated using MetaData's naming convention? I'm dynamically creating a constraint, and then looking it up, but to look it up I need its name. I need to identify the constraint uniquely, without giving it a user-specified name.
I tried appending it to an SQLALchemy Table with Table.append_constraint, but even so its name field is None. I expect it to have a name generated according to the naming convention. Example:
naming_convention = {
"uq": '%(table_name)s_%(column_0_name)s_uq'
}
metadata = MetaData(naming_convention = naming_convention)
table = Table("table1", metadata, Column('col1', Integer))
constraint = UniqueConstraint('col1', name = None)
table.append_constraint(constraint)
default_constraint_name = constraint.name
default_constraint_name == 'table1_col1s_uq'
# False
default_constraint_name == None
# True
Preface:
My task is storing files on a disk, the part of a file name is a timestamp. Path to these files is storing in DB. Multiple files may have a single owner entity (one message can contain multiple attachments).
To make things easier I want to have the same timestamp for file paths in DB (it's set to default now()) and files on the disk.
Question:
So after insertion, I need to get back default inserted values (in most cases primary_key_id and created_datetime).
I tried:
db_session # Just for clarity
<sqlalchemy.orm.session.AsyncSession object at 0x7f836691db20>
str(statement) # Just for clarity. Don't know how to get back the original python (not an SQL) statement
'INSERT INTO users (phone_number, login, full_name, hashed_password, role) VALUES (:phone_number, :login, :full_name, :hashed_password, :role)'
query_result = await db_session.execute(statement=statement)
query_result.returned_defaults_rows # Primary_key, but no datetime
[(243,)]
query_result.returned_defaults # Primary_key, but no datetime
(243,)
query_result.fetchall()
[]
My tables:
Base = declarative_base() # Main class of ORM; Put in config by suggestion https://t.me/ru_python/1450665
claims = Table( # TODO set constraints for status
"claims",
Base.metadata,
Column("id", Integer, primary_key=True),
My queries
async def create_item(statement: Insert, db_session: AsyncSession, detail: str = '') -> dict:
try: # return default created values
statement = statement.returning(statement.table.c.id, statement.table.c.created_datetime)
return (await db_session.execute(statement=statement)).fetchone()._asdict()
except sqlalchemy.exc.IntegrityError as error:
# if psycopg2_errors.lookup(code=error.orig.pgcode) in (psycopg2_errors.UniqueViolation, psycopg2_errors.lookup):
detail = error.orig.args[0].split('Key ')[-1].replace('(', '').replace(')', '').replace('"', '')
raise HTTPException(status_code=422, detail=detail)
P.S. Sqlalchemy v 1.4
I was able to do this with session.bulk_save_objects(objects, return_defaults=True)
Docs on this method are here
I'm using Fanbatis framework to access a MySQL database. The documentation here: http://www.talesframework.org/fanbatis/ says that I can use the #Column annotation to map a class attribute to a column with a different name:
#Column{name="xx"} - By default column name is assumed to be the field name,
to change this use this annotation on a field
I have this class...
using fanbatis
#Table {name = "APPS"}
class App
{
#Primary
Str? id
Str? name
#Column{name="description"}
Str? descr
}
And the APPS table created with this SQL:
create table APPS (
ID varchar(36) not null,
NAME varchar(100) not null,
DESCRIPTION varchar(400) ,
primary key (ID)
);
And I'm trying to retrieve a record from the database using this DAO
class AppDao
{
static App? findById(Str id)
{
S<|
select * from apps where id = #{id}
|>
}
}
My code compiles fine, but when I run it, passing in the ID of an existing record, it retrieves the record from the database and set the values of the attributes that match the column names, but the app.descr remains null. However, if I just remove the #Column annotation from the "descr" attribute and rename it to match the column ("description"), then the code runs fine and returns the expected values.
-- Run: auth::TestAppDao.testFindById...
Test setup!
app.id: 0615a6cb-7bda-cc40-a274-00130281bd51
app.name: MyApp
app.descr: null
TEST FAILED
sys::TestErr: Test failed: null != "MyApp descr" [sys::Str]
fan.sys.Test.err (Test.java:206)
fan.sys.Test.fail (Test.java:198)
fan.sys.Test.verifyEq (Test.java:121)
fan.sys.Test.verifyEq (Test.java:90)
auth::TestAppDao.testFindById (TestAppDao.fan:36)
java.lang.reflect.Method.invoke (Unknown)
fan.sys.Method.invoke (Method.java:559)
fan.sys.Method$MethodFunc.callList (Method.java:204)
fan.sys.Method.callList (Method.java:138)
fanx.tools.Fant.runTest (Fant.java:191)
fanx.tools.Fant.test (Fant.java:110)
fanx.tools.Fant.test (Fant.java:32)
fanx.tools.Fant.run (Fant.java:284)
fanx.tools.Fant.main (Fant.java:327)
Test teardown!
Am I doing something wrong or is this a bug in Fanbatis?
It could be due to case sensitivity - see MySQL 9.2.2. Identifier Case Sensitivity
Although database and table names are not case sensitive on some
platforms, you should not refer to a given database or table using
different cases within the same statement. The following statement
would not work because it refers to a table both as my_table and as
MY_TABLE:
mysql> SELECT * FROM my_table WHERE MY_TABLE.col=1;
Try:
#Column{name="DESCRIPTION"}
It's also mentioned in this question.