With SQLAlchemy, I'm finding that sometimes I mis-type a name of an attribute which is mapped to a column, which results in rather difficult to catch errors:
class Thing(Base):
foo = Column(String)
thing = Thing()
thing.bar = "Hello" # a typo, I actually meant thing.foo
assert thing.bar == "Hello" # works here, as thing.bar is a transient attribute created by the assignment above
session.add(thing)
session.commit() # thing.bar is not saved in the database, obviously
...
# much later
thing = session.query(Thing)...one()
assert thing.foo == "Hello" # fails
assert thing.bar == "Hello" # fails, there's no even such attribute
Is there a way to configure the mapped class so assigning to anything which is not mapped to an SQLAlchemy column would raise an exception?
Ok, the solution seems to be to override __setattr__ method of the base class, which allows us to check if the atribute already exists before setting it.
class BaseBase(object):
"""
This class is a superclass of SA-generated Base class,
which in turn is the superclass of all db-aware classes
so we can define common functions here
"""
def __setattr__(self, name, value):
"""
Raise an exception if attempting to assign to an atribute which does not exist in the model.
We're not checking if the attribute is an SQLAlchemy-mapped column because we also want it to work with properties etc.
See http://stackoverflow.com/questions/12032260/ for more details.
"""
if name != "_sa_instance_state" and not hasattr(self, name):
raise AttributeError("Attribute %s is not a mapped column of object %s" % (name, self))
super(BaseBase, self).__setattr__(name, value)
Base = declarative_base(cls=BaseBase)
Sort of "strict mode" for SQLAlchemy...
Override the __get__ method of objects, and check to see if it is in the column (by storing it with the class definition or runtime search)
More information here from SO.
Related
In SqlAlchemy I use:
price = Column(Numeric(18, 5))
in various placed throught my app. When I get a number formatted in swedish, with a comma instead of a dot (0,34 instead of 0.34) and try to change the price column the number gets set to 0.00000.
To solve this I have this code:
obj.price = price.replace(',','.')
But having this all over the code makes it pretty ugly and the risk is that I forget one place. Would it be possible to have some kind of generic converter function which gets called before a value is converted from a string to a Numeric? And that I have that in one place only.
Check the validates decorator of SQLAlchemy: http://docs.sqlalchemy.org/en/rel_1_0/orm/mapped_attributes.html
A quick way to add a “validation” routine to an attribute is to use
the validates() decorator. An attribute validator can raise an
exception, halting the process of mutating the attribute’s value, or
can change the given value into something different.
In your case the code could look similar to:
from sqlalchemy.orm import validates
class Obj(Base):
__tablename__ = 'obj'
id = Column(Integer, primary_key=True)
price = Column(Numeric(18, 5))
#validates('price')
def validate_price(self, key, price):
if ',' in price:
return float(price.replace(',','.'))
else:
return float(price)
I've created a class modeling a group of files within a software product build, on a Django server using the Django-REST package. The design is that the group of files (a Depot instance) should be able to be assigned to multiple Build instances (e.g. both the "alpha" and "beta" builds using the same exact audio file depot). However, at the time that the depot is created, it is being created as part of the creation of single Build on the client; it is only later that a utility script will allow an existing Depot to be added to other Builds.
It seemed natural to me that the Depot class should represent this relationship with a ManyToManyField. The problem is that the serializer does not seem to know what to do with this ManyToManyField. I've tried several workarounds, but each has its own error. I've tried having my DepotSerializer be either a rest_framework.serializers.Serializer or a rest_framework.serializers.ModelSerializer, but that seems largely unrelated to this problem.
Models.py:
class Depot(models.Model):
name = models.CharField(max_length=64)
builds = models.ManyToManyField(Build)
TYPE_EXECUTABLE = 0
TYPE_CORE = 1
TYPE_STREAMING = 2
depot_type = models.IntegerField(choices = (
(TYPE_EXECUTABLE, 'Executable'),
(TYPE_CORE, 'Core'),
(TYPE_STREAMING, 'Streaming'),
))
def __str__(self):
return self.name
Views.py:
class DepotCreate(mixins.CreateModelMixin,
generics.GenericAPIView):
serializer_class = DepotSerializer
queryset = Depot.objects.all()
def post(self, request, *args, **kwargs):
return self.create(request, *args, **kwargs)
Serializers.py version 1:
class DepotSerializer(serializers.ModelSerializer):
builds = serializers.PrimaryKeyRelatedField()
class Meta:
model = Depot
fields = ('id', 'name', 'builds', 'depot_type')
read_only_fields = ('id',)
def validate(self, attrs):
build = attrs['builds']
if build == None:
raise serializers.ValidationError("Build could not be found")
for depot in build.depot_set.all():
if depot.name == attrs['name']:
raise serializers.ValidationError("Build already contains a depot \"{}\"".format(depot.name))
return attrs
def restore_object(self, attrs, instance=None):
# existence of the build has already been validated
return Depot(**attrs)
This version results in the following error during the Depot init call:
Exception Type: TypeError
Exception Value:
'builds' is an invalid keyword argument for this function
Exception Location: /webapps/cdp_admin_django/lib/python3.4/site-packages/django/db/models/base.py in __init__, line 417
That appears to indicate that the Depot model cannot handle the 'builds' parameter despite the fact that it has a 'builds' ManyToManyField member.
Serializers.py 'restore_object' ver 2:
def restore_object(self, attrs, instance=None):
# existence of the build has already been validated
build = attrs['builds']
depotObj = Depot(name=attrs['name'], depot_type=attrs['depot_type'])
depotObj.builds.add(build)
return depotObj
This gave me the error:
Exception Type: ValueError
Exception Value:
"<Depot: depot_test4>" needs to have a value for field "depot" before this many-to-many relationship can be used.
Exception Location: /webapps/cdp_admin_django/lib/python3.4/site-packages/django/db/models/fields/related.py in __init__, line 524
After quite a bit of investigation, I found that ManyToMany relationships can give you trouble if you don't save the MYSQL entry before attempting to manipulate that field. Hence, restore_object ver 3:
def restore_object(self, attrs, instance=None):
# existence of the build has already been validated
build = attrs['builds']
depotObj = Depot(name=attrs['name'], depot_type=attrs['depot_type'])
depotObj.save()
depotObj.builds.add(build)
return depotObj
This does successfully create the table entry for this instance, but ends up throwing the following error:
Exception Type: IntegrityError
Exception Value:
(1062, "Duplicate entry '5' for key 'PRIMARY'")
Exception Location: /webapps/cdp_admin_django/lib/python3.4/site-packages/MySQLdb/connections.py in defaulterrorhandler, line 38
This error takes place during rest_framework/mixins.py call to serializer.save(force_insert=True). Which looks like it is supposed to force the creation of a new table entry, presumably disagreeing with my earlier call to Model.save.
Does anyone know the correct approach for a design like this? I feel like this can't be that unusual of a table structure.
EDIT 10/20/2014:
After the suggestion below, I experimented with writing a new ModelSerializer for one of my models; for the most part because of these types of order-of-operations problems, I'd backed off from using ModelSerializer and did all of my data-to-object field processing in views.py by reading serializer.data.
Having a PrimaryKeyRelatedField(many=True) in the ModelSerializer DID help. Notably, I was able to create a serializer instance with existent models and get the correct serializer.data. However, I still have the problem where restore_object can do everything except create a new model instance and pass down the ManyToManyField value. I still get "TypeError: '[PrimaryKeyRelatedField name]' is an invalid keyword argument for this function" if I pass the field to the model's init func. I still cannot save the model before the REST library does it itself. In addition, in this mode, the serializer populates serializer.data with the values of the Model, not the values provided in the data input. So if you do not use the PrimaryKeyRelatedField's attrs value in restore_object, it is discarded.
It appears that I need to override ModelSerializer.save to some kind of a pre-save, apply ManyToMany input, and a post-save, but I would need the attrs values so I can apply and modify the ManyToManyField at that time. I realize that the serializer does have the init_data field to see the original inputs, but in the case where the serializer is being used to deserialize a list of data into a list of new objects, I don't think there's a way to trace which serializer.init_data corresponds with which serializer.object.
In your serializer version 1, you do not have to add
builds = serializers.PrimaryKeyRelatedField()
as the model serializer will create this for you. In fact if you look at the exemple of the documentation (http://www.django-rest-framework.org/api-guide/relations/) you'll see that the PrimaryKeyRelatedField is applied when there is a FK 'to' the current model (not a M2M relation).
I would remove this from the serializer and see then what's going on.
I have a declarative mapping:
class User(base):
username = Column(Unicode(30), unique=True)
How can I tell sqlalchemy that this attribute may not be modified?
The workaround I came up with is kindof hacky:
from werkzeug.utils import cached_property
# regular #property works, too
class User(base):
_username = Column('username', Unicode(30), unique=True)
#cached_property
def username(self):
return self._username
def __init__(self, username, **kw):
super(User,self).__init__(**kw)
self._username=username
Doing this on the database column permission level will not work because not all databases support that.
You can use the validates SQLAlchemy feature.
from sqlalchemy.orm import validates
...
class User(base):
...
#validates('username')
def validates_username(self, key, value):
if self.username: # Field already exists
raise ValueError('Username cannot be modified.')
return value
reference: https://docs.sqlalchemy.org/en/13/orm/mapped_attributes.html#simple-validators
I can suggest the following ways do protect column from modification:
First is using hook when any attribute is being set:
In case above all column in all tables of Base declarative will be hooked, so you need somehow to store information about whether column can be modified or not. For example you could inherit sqlalchemy.Column class to add some attribute to it and then check attribute in the hook.
class Column(sqlalchemy.Column):
def __init__(self, *args, **kwargs):
self.readonly = kwargs.pop("readonly", False)
super(Column, self).__init__(*args, **kwargs)
# noinspection PyUnusedLocal
#event.listens_for(Base, 'attribute_instrument')
def configure_listener(class_, key, inst):
"""This event is called whenever an attribute on a class is instrumented"""
if not hasattr(inst.property, 'columns'):
return
# noinspection PyUnusedLocal
#event.listens_for(inst, "set", retval=True)
def set_column_value(instance, value, oldvalue, initiator):
"""This event is called whenever a "set" occurs on that instrumented attribute"""
logging.info("%s: %s -> %s" % (inst.property.columns[0], oldvalue, value))
column = inst.property.columns[0]
# CHECK HERE ON CAN COLUMN BE MODIFIED IF NO RAISE ERROR
if not column.readonly:
raise RuntimeError("Column %s can't be changed!" % column.name)
return value
To hook concrete attributes you can do the next way (adding attribute to column not required):
# standard decorator style
#event.listens_for(SomeClass.some_attribute, 'set')
def receive_set(target, value, oldvalue, initiator):
"listen for the 'set' event"
# ... (event handling logic) ...
Here is guide about SQLAlchemy events.
Second way that I can suggest is using standard Python property or SQLAlchemy hybrid_property as you have shown in your question, but using this approach result in code growing.
P.S. I suppose that compact way is add attribute to column and hook all set event.
Slight correction to #AlexQueue answer.
#validates('username')
def validates_username(self, key, value):
if self.username and self.username != value: # Field already exists
raise ValueError('Username cannot be modified.')
return value
How do I update an HSTORE field with Flask-Admin?
The regular ModelView doesn't show the HSTORE field in Edit view. It shows nothing. No control at all. In list view, it shows a column with data in JSON notation. That's fine with me.
Using a custom ModelView, I can change the HSTORE field into a TextAreaField. This will show me the HSTORE field in JSON notation when in edit view. But I cannot edit/update it. In list view, it still shows me the object in JSON notation. Looks fine to me.
class MyView(ModelView):
form_overrides = dict(attributes=fields.TextAreaField)
When I attempt to save/edit the JSON, I receive this error:
sqlalchemy.exc.InternalError
InternalError: (InternalError) Unexpected end of string
LINE 1: UPDATE mytable SET attributes='{}' WHERE mytable.id = ...
^
'UPDATE mytable SET attributes=%(attributes)s WHERE mytable.id = %(mytable_id)s' {'attributes': u'{}', 'mytable_id': 14L}
Now -- using code, I can get something to save into the HSTORE field:
class MyView(ModelView):
form_overrides = dict(attributes=fields.TextAreaField)
def on_model_change(self, form, model, is_created):
model.attributes = {"a": "1"}
return
This basically overrides the model and put this object into it. I can then see the object in the List view and the Edit view. Still not good enough -- I want to save/edit the object that the user typed in.
I tried to parse and save the content from the form into JSON and back out. This doesn't work:
class MyView(ModelView):
form_overrides = dict(attributes=fields.TextAreaField)
def on_model_change(self, form, model, is_created):
x = form.data['attributes']
y = json.loads(x)
model.attributes = y
return
json.loads(x) says this:
ValueError ValueError: Expecting property name: line 1 column 1 (char
1)
and here are some sample inputs that fail:
{u's': u'ff'}
{'s':'ff'}
However, this input works:
{}
Blank also works
This is my SQL Table:
CREATE TABLE mytable (
id BIGSERIAL UNIQUE PRIMARY KEY,
attributes hstore
);
This is my SQA Model:
class MyTable(Base):
__tablename__ = u'mytable'
id = Column(BigInteger, primary_key=True)
attributes = Column(HSTORE)
Here is how I added the view's to the admin object
admin.add_view(ModelView(models.MyTable, db.session))
Add the view using a custom Model View
admin.add_view(MyView(models.MyTable, db.session))
But I don't do those views at the same time -- I get a Blueprint name collision error -- separate issue)
I also attempted to use a form field converter. I couldn't get it to actually hit the code.
class MyModelConverter(AdminModelConverter):
def post_process(self, form_class, info):
raise Exception('here I am') #but it never hits this
return form_class
class MyView(ModelView):
form_overrides = dict(attributes=fields.TextAreaField)
The answer gives you a bit more then asked
Fist of all it "extends" hstore to be able to store actually JSON, not just key-value
So this structure is also OK:
{"key":{"inner_object_key":{"Another_key":"Done!","list":["no","problem"]}}}
So, first of all your ModelView should use custom converter
class ExtendedModelView(ModelView):
model_form_converter=CustomAdminConverter
Converter itself should know how to use hstore dialect:
class CustomAdminConverter(AdminModelConverter):
#converts('sqlalchemy.dialects.postgresql.hstore.HSTORE')
def conv_HSTORE(self, field_args, **extra):
return DictToHstoreField(**field_args)
This one as you can see uses custom WTForms field which converts data in both directions:
class DictToHstoreField(TextAreaField):
def process_data(self, value):
if value is None:
value = {}
else:
for key,obj in value.iteritems():
if (obj.startswith("{") and obj.endswith("}")) or (obj.startswith("[") and obj.endswith("]")):
try:
value[key]=json.loads(obj)
except:
pass #
self.data=json.dumps(value)
def process_formdata(self, valuelist):
if valuelist:
self.data = json.loads(valuelist[0])
for key,obj in self.data.iteritems():
if isinstance(obj,dict) or isinstance(obj,list):
self.data[key]=json.dumps(obj)
if isinstance(obj,int):
self.data[key]=str(obj)
The final step will be to actual use this data in application
I did not make it in common nice way for SQLalchemy, since was used with flask-restful, so I have only adoption for flask-restful in one direction, but I think it's easy to get the idea from here and do the rest.
And if your case is simple key-value storage so nothing additionaly should be done, just use it as is.
But if you want to unwrap JSON somewhere in code, it's simple like this whenever you use it, just wrap in function
if (value.startswith("{") and value.endswith("}")) or (value.startswith("[") and value.endswith("]")):
value=json.loads(value)
Creating dynamical field for actual nice non-JSON way for editing of data also possible by extending FormField and adding some javascript for adding/removing fields, but this is whole different story, in my case I needed actual json storage, with blackjack and lists :)
Was working on postgres JSON datatype. The above solution worked great with a minor modifications.
Tried
'sqlalchemy.dialects.postgresql.json.JSON',
'sqlalchemy.dialects.postgresql.JSON',
'dialects.postgresql.json.JSON',
'dialects.postgresql.JSON'
The above versions did not work.
Finally the following change worked
#converts('JSON')
And changed class DictToHstoreField to the following:
class DictToJSONField(fields.TextAreaField):
def process_data(self, value):
if value is None:
value = {}
self.data = json.dumps(value)
def process_formdata(self, valuelist):
if valuelist:
self.data = json.loads(valuelist[0])
else:
self.data = '{}'
Although, this is might not be the answer to your question, but by default SQLAlchemy's ORM doesn't detect in-place changes to HSTORE field values. But fortunately there's a solution: SQLAlchemy's MutableDict type:
from sqlalchemy.ext.mutable import MutableDict
class MyClass(Base):
__tablename__ = 'mytable'
id = Column(Integer, primary_key=True)
attributes = Column(MutableDict.as_mutable(HSTORE))
Now when you change something in-place:
my_object.attributes.['some_key'] = 'some value'
The hstore field will be updated after session.commit().
Good day everyone,
I have a file of strings corresponding to the fields of my SQLAlchemy object. Some fields are floats, some are ints, and some are strings.
I'd like to be able to coerce my string into the proper type by interrogating the column definition. Is this possible?
For instance:
class MyClass(Base):
...
my_field = Column(Float)
It feels like one should be able to say something like MyClass.my_field.column.type and either ask the type to coerce the string directly or write some conditions and int(x), float(x) as needed.
I wondered whether this would happen automatically if all the values were strings, but I received Oracle errors because the type was incorrect.
Currently I naively coerce -- if it's float()able, that's my value, else it's a string, and I trust that integral floats will become integers upon inserting because they are represented exactly. But the runtime value is wrong (e.g. 1.0 vs 1) and it just seems sloppy.
Thanks for your input!
SQLAlchemy 0.7.4
You can iterate over columns of the mapped Table:
for col in MyClass.__table__.columns:
print col, repr(col.type)
... so you can check the type of each field by its name like this:
def get_col_type(cls_, fld_):
for col in cls_.__table__.columns:
if col.name == fld_:
return col.type # this contains the instance of SA type
assert Float == type(get_col_type(MyClass, 'my_field'))
I would cache the results though if your file is large in order to save the for-loop on every row imported from the file.
Type coercion for sqlalchemy prior to committing to some database.
How can I verify Column data types in the SQLAlchemy ORM?
from sqlalchemy import (
Column,
Integer,
String,
DateTime,
)
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import event
import datetime
Base = declarative_base()
type_coercion = {
Integer: int,
String: str,
DateTime: datetime.datetime,
}
# this event is called whenever an attribute
# on a class is instrumented
#event.listens_for(Base, 'attribute_instrument')
def configure_listener(class_, key, inst):
if not hasattr(inst.property, 'columns'):
return
# this event is called whenever a "set"
# occurs on that instrumented attribute
#event.listens_for(inst, "set", retval=True)
def set_(instance, value, oldvalue, initiator):
desired_type = type_coercion.get(inst.property.columns[0].type.__class__)
coerced_value = desired_type(value)
return coerced_value
class MyObject(Base):
__tablename__ = 'mytable'
id = Column(Integer, primary_key=True)
svalue = Column(String)
ivalue = Column(Integer)
dvalue = Column(DateTime)
x = MyObject(svalue=50)
assert isinstance(x.svalue, str)
I'm not sure if I'm reading this question correctly, but I would do something like:
class MyClass(Base):
some_float = Column(Float)
some_string = Column(String)
some_int = Column(Int)
...
def __init__(self, some_float, some_string, some_int, ...):
if isinstance(some_float, float):
self.some_float = somefloat
else:
try:
self.some_float = float(somefloat)
except:
# do something intelligent
if isinstance(some_string, string):
...
And I would repeat the checking process for each column. I would trust anything to do it "automatically". I also expect your file of strings to be well structured, otherwise something more complicated would have to be done.
Assuming your file is a CSV (I'm not good with file reads in python, so read this as pseudocode):
while not EOF:
thisline = readline('thisfile.csv', separator=',') # this line is an ordered list of strings
thisthing = MyClass(some_float=thisline[0], some_string=thisline[1]...)
DBSession.add(thisthing)