How to differentiate causes of SQLAlchemy's IntegrityError? - sqlalchemy

Question

SQLAlchemy appears to just throw a general IntegrityError when there is a data integrity problem with a transaction. Of course, the exact query and error message are contained in the exception, which is sufficient for a human debugging the program. However, when writing error handling code for the exception, there doesn't seem to be a good way so far as I can tell to check which constraint on which table was responsible for the error. Also, the exception is raised by the session.commit() line rather than the line actually responsible for producing the error, so I can't differentiate using multiple try/except blocks either.
Is there a way, short of trying to programmatically parse the error message and/or query, that I can for example distinguish a duplicate primary key error from a foreign key error or a failed CHECK constraint and so forth? Or even just a way to tell which column of which table is in violation of the data integrity? Or just a way to raise the exception immediately on the line that caused the error rather than waiting for the transaction to be committed?

Answer 1

The IntegrityError instance has orig and statement attributes which can be inspected to obtain the error message and the failing SQL statement, respectively.
Given this model:
class Foo(Base):
__tablename__ = 'foo20201209'
id = sa.Column(sa.Integer, primary_key=True)
bar = sa.Column(sa.String(2), unique=True)
baz = sa.Column(sa.Integer, sa.CheckConstraint('baz >= 0'), default=0)
this code:
conn_strings = ['postgresql+psycopg2:///test',
'mysql+mysqlconnector:///test',
'sqlite://']
for cs in conn_strings:
engine = sa.create_engine(cs)
Base.metadata.drop_all(bind=engine)
Base.metadata.create_all(bind=engine)
session = orm.Session(bind=engine)
for kwds in [{'bar': 'a'}, {'bar': 'a'}, {'bar': 'b', 'baz': -11}]:
session.add(Foo(**kwds))
try:
session.commit()
except sa.exc.IntegrityError as ex:
print(ex.orig)
print(ex.statement)
print()
session.rollback()
session.close()
engine.dispose()
will produce this output:
duplicate key value violates unique constraint "foo20201209_bar_key"
DETAIL: Key (bar)=(a) already exists.
INSERT INTO foo20201209 (bar, baz) VALUES (%(bar)s, %(baz)s) RETURNING foo20201209.id
new row for relation "foo20201209" violates check constraint "foo20201209_baz_check"
DETAIL: Failing row contains (3, b, -11).
INSERT INTO foo20201209 (bar, baz) VALUES (%(bar)s, %(baz)s) RETURNING foo20201209.id
1062 (23000): Duplicate entry 'a' for key 'bar'
INSERT INTO foo20201209 (bar, baz) VALUES (%(bar)s, %(baz)s)
4025 (23000): CONSTRAINT `foo20201209.baz` failed for `test`.`foo20201209`
INSERT INTO foo20201209 (bar, baz) VALUES (%(bar)s, %(baz)s)
UNIQUE constraint failed: foo20201209.bar
INSERT INTO foo20201209 (bar, baz) VALUES (?, ?)
CHECK constraint failed: foo20201209
INSERT INTO foo20201209 (bar, baz) VALUES (?, ?)

Answer 2

As DMJ commented below my original answer, different database engines emit different error messages in the event of an integrity error, and SQLAlchemy does not attempt present these messages in a consistent way, so the short answer to the question is: you can't get useful information from the exception without parsing the error messages.
That said, parsing the error messages may not be so difficult, if the constraints are named in a consistent fashion, and these names appear in the error messages. Here SQLAlchemy provides some assistance: we can create a Metadata object with a naming convention for constraints defined (docs). With this convention in place any constraints we create will follow it, we can match constraint names in error messages and use the matches to lookup the table and constraint objects in the metadata.
Here's an example of how you might parse error messages using the conventions. I haven't covered all possible constraint types, nor handled Sqlite's unique key violation message, which omits the constraint name. These are left as exercises for the reader ;-)
import re
import sqlalchemy as sa
convention = {
"ix": 'ix_%(column_0_label)s',
"uq": "uq_%(table_name)s_%(column_0_name)s",
"ck": "ck_%(table_name)s_%(constraint_name)s",
"fk": "fk_%(table_name)s_%(column_0_name)s_%(referred_table_name)s",
"pk": "pk_%(table_name)s"
}
metadata = sa.MetaData(naming_convention=convention)
tbl = sa.Table(
't65189213',
metadata,
sa.Column('id', sa.Integer, primary_key=True),
sa.Column('foo', sa.String(2), unique=True),
sa.Column('bar', sa.Integer, default=0),
sa.CheckConstraint('bar >= 0', name='positive_bar'),
)
# Basic pattern to match constraint names (not exhaustive).
pattern = r'(?P<type_key>uq|ck)_(?P<table>[a-z][a-z0-9]+)_[a-z_]+'
regex = re.compile(pattern)
def parse_exception(ex):
# Returns an informative message, or the original error message.
# We could return the table, constraint object etc. instead.
types = {'ck': 'Check constraint', 'uq': 'Unique key'}
m = regex.search(str(ex.orig))
if m:
type_ = types[m.groupdict()['type_key']]
table_name = m.groupdict()['table']
constraint_name = m.group(0)
table = metadata.tables[table_name]
constraint = next(c for c in table.constraints if c.name == constraint_name)
columns = ','.join(constraint.columns.keys())
return f'{type_} {constraint_name} has been violated on table {table_name!r} columns: {columns}'
return f'{ex.orig}'
conn_strings = ['postgresql+psycopg2:///test',
'mysql+mysqlconnector://root:root#localhost/test',
'sqlite://']
for cs in conn_strings:
engine = sa.create_engine(cs, future=True)
tbl.drop(engine, checkfirst=True)
tbl.create(engine)
with engine.connect() as conn:
print(engine.dialect.name)
for kwds in [{'foo': 'a'}, {'foo': 'a'}, {'foo': 'b', 'bar': -11}]:
try:
conn.execute(tbl.insert(), kwds)
conn.commit()
except sa.exc.IntegrityError as ex:
print(parse_exception(ex))
conn.rollback()
print('-'*10)
engine.dispose()
Output:
postgresql
Unique key uq_t65189213_foo has been violated on table 't65189213' columns: foo
Check constraint ck_t65189213_positive_bar has been violated on table 't65189213' columns:
----------
mysql
Unique key uq_t65189213_foo has been violated on table 't65189213' columns: foo
Check constraint ck_t65189213_positive_bar has been violated on table 't65189213' columns:
----------
sqlite
UNIQUE constraint failed: t65189213.foo
Check constraint ck_t65189213_positive_bar has been violated on table 't65189213' columns:
----------

Answer 3

I ended up using session.flush() to trigger the exceptions earlier. I call it once before the line(s) in question (so I know for 100% certain that the exception wasn't triggered by previous lines) and again inside a try/catch block to see if the line(s) in question caused an error.
I admin I'm not completely happy with this solution, but I haven't been able to find anything else. I'd still love to hear if there is a better solution, ideally one that will tell me exactly which constraint of which table caused the error. But, this is a workaround that might help someone.

Answer 4

I have faced the same trouble, and it looks like, for instance, for constraints it's a better solution to give them names, parsing those exception messages then. Anyway, for my SQLAlchemy==1.4.28, a UNIQUE constraint name is still not presented anywhere in the Exception thrown.
Consider the next example:
class M(Model):
a = Column(String)
i = Column(Integer)
UniqueConstraint(M.i, name="my unique")
CheckConstraint(0 <= M.i, name="my check")
def test_test():
db = create_test_database(Model)
try:
with db.session() as s:
# Here we break our UNIQUE constraint
try:
with s.begin_nested():
s.add_all(
[
M(a="Alice", i=1),
M(a="Bob", i=1),
]
)
except IntegrityError as err:
rich.inspect(err)
# Here we break our CHECK constraint
s.add(M(a="Alice", i=-1))
except IntegrityError as err:
rich.inspect(err)
And here are the results:
┌────────────────────── <class 'sqlalchemy.exc.IntegrityError'> ───────────────────────┐
│ Wraps a DB-API IntegrityError. │
│ │
│ ┌──────────────────────────────────────────────────────────────────────────────────┐ │
│ │ IntegrityError('(sqlite3.IntegrityError) UNIQUE constraint failed: M.i') │ │
│ └──────────────────────────────────────────────────────────────────────────────────┘ │
│ │
│ args = ('(sqlite3.IntegrityError) UNIQUE constraint failed: M.i',) │
│ code = 'gkpj' │
│ connection_invalidated = False │
│ detail = [] │
│ hide_parameters = False │
│ ismulti = False │
│ orig = IntegrityError('UNIQUE constraint failed: M.i') │
│ params = ('Bob', 1) │
│ statement = 'INSERT INTO "M" (a, i) VALUES (?, ?)' │
└──────────────────────────────────────────────────────────────────────────────────────┘
┌──────────────────────── <class 'sqlalchemy.exc.IntegrityError'> ─────────────────────────┐
│ Wraps a DB-API IntegrityError. │
│ │
│ ┌──────────────────────────────────────────────────────────────────────────────────────┐ │
│ │ IntegrityError('(sqlite3.IntegrityError) CHECK constraint failed: my check') │ │
│ └──────────────────────────────────────────────────────────────────────────────────────┘ │
│ │
│ args = ('(sqlite3.IntegrityError) CHECK constraint failed: my check',) │
│ code = 'gkpj' │
│ connection_invalidated = False │
│ detail = [] │
│ hide_parameters = False │
│ ismulti = False │
│ orig = IntegrityError('CHECK constraint failed: my check') │
│ params = ('Alice', -1) │
│ statement = 'INSERT INTO "M" (a, i) VALUES (?, ?)' │
└──────────────────────────────────────────────────────────────────────────────────────────┘
So, it looks like this CHECK constraint name will always be somewhere in the exception's string, and you can write a code around this. And it's clear why this is extra info for a UNIQUE constraint, cause you have already had the word UNIQUE and a field's name (A.i in my example). I don't think they will ever change these strings formats, but it'd be interesting to ask them about.
WARNING:
The problem is that my code relates to SQLite, while yours may relate to another DB and those messages will differ cause they are originally from the underlying DB engine, not from SQLAlchemy itself. So you have to take care about abstracting your code for those strings.