I have two models in consideration. RV_Offers and RV_Details. Each offer can have multiple details i.e. I have a foreignkey relationship field in RV_Details table.
Here is my view:
rv_offers_queryset = RV_Offers.objects.all().select_related('vendor').prefetch_related('details')
details_queryset = RV_Details.objects.all().select_related('rv_offer')
title = Subquery(details_queryset.filter(
rv_offer=OuterRef("id"),
).order_by("-created_at").values("original_title")[:1])
offers_queryset = rv_offers_queryset.annotate(
title=title).filter(django_query)
offers = RVOffersSerializer(offers_queryset, many=True).data
return Response({'result': offers}, status=HTTP_200_OK)
As can be seen, I am passing the offers queryset to the serializer.
Now, here is my serializer:
class RVOffersSerializer(serializers.ModelSerializer):
details = serializers.SerializerMethodField()
vendor = VendorSerializer()
def get_details(self, obj):
queryset = RV_Details.objects.all().select_related('rv_offer')
queryset = queryset.filter(rv_offer=obj.id).latest('created_at')
return RVDetailsSerializer(queryset).data
class Meta:
model = RV_Offers
fields = '__all__'
If you look at the get_details method, I am trying to fetch the latest detail that belongs to an offer. My problem is, even though I am using select_related to optimize the query, the results are still extremely slow, In fact, I am using django debug toolbar to inspect the query and apparently select_related seems to have no effect.
What am I doing wrong or how else can I approach this problem?
This is what I did to reduce the number of queries being hit on the db:
def get_details(self, obj):
details = obj.details.last()
return RVDetailsSerializer(details).data
I was able to reduce the number of queries from 45 to 4 using this.
This is because in the view, I had already used select_related to make the queryset, which in turn is being used here using obj.
Related
I have a Django app, where each user can add a product with multiple possible metrics (width, height and length combination). A user must also specify in which city this product is located.
Users can also search within the database all products matching specific metrics.
I use Django 1.11 and am seaching for a solution to display on an interactive map all the products matching a queryset.
I am trying to do it with django-leaflet and django-geojson (as my db is not gis-oriented and I don't need heavy geo-computations), but I am facing some difficulties because my "PointField" is not in my product Model but in the Location Model and on the map I need to display Product properties, so I must serialize all these data together.
If you prefer code rather than words, here is a simplified version of my relevant files.
#models.py
class Product(models.Model):
name = models.CharField()
owner = models.ForeignKey(User)
photo = models.ImageField(...)
dimensions = models.ManyToManyField(Metrics)
location = models.ForeignKey(Location, related_name='products', related_query_name='product')
class Metrics(models.Model):
width = models.PositiveIntegerField()
height = models.PositiveIntegerField()
length = models.PositiveIntegerField()
class Location(models.Model):
zip_code = models.PositiveIntegerField()
city_name = models.CharField()
slug = models.SlugField(max_length=500, blank=True)
geom = PointField(default={'type': 'Point', 'coordinates': [0, 0]})
#views.py
class SearchResultListView(ListView):
model = models.Product
template_name='my_app/searchresult_list.html'
context_object_name = 'product_list'
def get_context_data(self, **kwargs):
context = super().get_context_data(**kwargs)
query_width = self.request.GET['width']
query_height = self.request.GET['height']
query_length = self.request.GET['length']
context['product_list'] = context['product_list'].filter(metrics__width=query_width,
metrics__length=query_length, metrics__height=query_height)
return context
#urls.py
????
#template.html
????
I saw in the django-geojson documentation multiple ways to hit the db (GeoJSON layer view,
Tiled GeoJSON layer view, GeoJSON template filter, low-level serialization). But I struggle to find the way to match my needs as my properties are in the Product Model, my coordinates are in the Location Model and my queryset in a non-related class-based view.
Any idea on the best way to perform my task? Should I continue with django-geojson or are there better apps for my purpose?
Your Product and Location look OK, but it is not clear what you are trying to do with Metrics. To select products near some place you want something like:
queryset = Product.objects.filter(location__geom__distance_lt=(someLocation, D(m=50)))
https://docs.djangoproject.com/en/3.0/ref/contrib/gis/geoquerysets/#distance-lt
I have two databases (both Mysql) that have exactly the same tables, and I want to copy some data from one to another using Sqlalchemy.
I can copy simple objects following the answer given in this question:
Cannot move object from one database to another
The problem is when the object has dependencies from another table, and I want to copy the dependencies as well.
So to make it more clear, this is my model (the same for both databases but using a different bind_key that points to a different database):
db1 = SQLAlchemy()
Class Payment(db.Model):
__tablename__ = 'payments'
__bind_key__ = 'db1'
id = db1.Column(db.Integer, primary_key=True)
paymethod_id = db1.Column(db.Integer(), db1.ForeignKey(PaymentMethod.id))
payment_method = db1.relationship(PaymentMethod)
What I would like to do is the following:
from models1 import Payment as Payment1
from models2 import Payment as Payment2
# query from one database
payment1 = db1.session.query(Payment1).first()
# create and add it to the other database
payment2 = Payment2(**payment1.__dict__.copy())
db2.session.add(payment)
db2.session.commit()
But in this case the foreign key fails because I don't have the PaymentMethod stored yet.
Is there a different approach to do that or I would have to do this procedure for every dependency of my object and be sure that I store the children beforehand?
Any help is appreciated :)
I came up with a solution that remaps the object to the right model and stores all its children. You call the method save_obj and pass the object you want to map. It will then retrieve a table with the same name but then from the model you want to remap the object to and it will recursively do the same for all its children. You have to define the right model in the method get_model.
To run this is necessary to disable autoflush to prevent committing before the object is correctly formed and it is also necessary to commit after calling the method. I'm using flask-sqlalchemy.
Hope this can help or give some insight to someone that faces a similar problem :)
def save_obj(obj, checked=[]):
if obj in checked:
# if the object was already converted, retrieve the right object
model = get_model(obj.__mapper__.mapped_table.name)
return get_obj(obj, model)
checked.append(obj)
children = []
relations = obj.__mapper__.relationships.items()
# get all the relationships of this model (foreign keys)
for relation in relations:
model = get_model(relation[1].table.name)
if model:
# remove the cascade option for this object, so the children are not stored automatically in the session
relation[1]._cascade = CascadeOptions('')
child = getattr(obj, relation[0])
if not child:
continue
# if the child is a set of children
if isinstance(child, list):
new_children = []
for ch in copy(child):
# convert the child
new_child = save_obj(ch, checked)
new_children.append(new_child)
children.append((relation[0], new_children))
else:
new_child = save_obj(child, checked)
children.append((relation[0], new_child))
# get the model of the object passed
model = get_model(obj.__mapper__.mapped_table.name)
new_obj = get_obj(obj, model)
# set all the children in this object
for child in children:
if child[1]:
setattr(new_obj, child[0], child[1])
checked.append(new_obj)
session.add(new_obj)
return new_obj
def get_model(table_name):
# get the right model for this object
for table in db.Model._decl_class_registry.values():
if hasattr(table, '__tablename__') and table.__tablename__ == table_name:
return table
return None
def create_new_obj(obj, model):
params = obj.__dict__.copy()
params.pop('_sa_instance_state')
return model(**params)
def get_obj(child, model):
# check if the object is already stored in the db
child_in_db = session.query(model).get(child.id)
if child_in_db:
return child_in_db
# check if the object is already in the session
for s in session.new:
if type(s) == model and s.id == child.id:
return s
return create_new_obj(child, model)
I'm using both django-taggit and django-filter in my web application, which stores legal decisions. My main view (below) inherits from the stock django-filter FilterView and allows people to filter the decisions by both statutes and parts of statutes.
class DecisionListView(FilterView):
context_object_name = "decision_list"
filterset_class = DecisionFilter
queryset = Decision.objects.select_related().all()
def get_context_data(self, **kwargs):
# Call the base implementation to get a context
context = super(DecisionListView, self).get_context_data(**kwargs)
# Add in querysets for all the statutes
context['statutes'] = Statute.objects.select_related().all()
context['tags'] = Decision.tags.most_common().distinct()
return context
I also tag decisions by topic when they're added and I'd like people to be able to filter on that too. I currently have the following in models.py:
class Decision(models.Model):
citation = models.CharField(max_length = 100)
decision_making_body = models.ForeignKey(DecisionMakingBody)
statute = models.ForeignKey(Statute)
paragraph = models.ForeignKey(Paragraph)
...
tags = TaggableManager()
class DecisionFilter(django_filters.FilterSet):
class Meta:
model = Decision
fields = ['statute', 'paragraph']
I tried adding 'tags' to the fields list in DecisionFilter but that had no effect, presumably because a TaggableManager is a Manager rather than a field in the database. I haven't found anything in the docs for either app that covers this. Is it possible to filter on taggit tags?
You should be able to use 'tags__name' as the search/filter field. Check out the Filtering section on http://django-taggit.readthedocs.org/en/latest/api.html#filtering
Imagine, I have News models with many text fields
class News(models.Model):
title = models.CharField(max_length=255)
subtitle = models.CharField(max_length=255, blank=True)
lead = models.TextField(max_length=4096)
content = models.TextField()
...
last_visited = models.DateTimeField()
Every time my News object outputs, I update last_visited field:
news.last_visited = datetime.datetime.now()
news.save()
This code makes Django override all model fields:
UPDATE news SET title='...', subtitle='...', last_visited = '...' WHERE id = '...';
Instead of just one:
UPDATE news SET last_visited = '...' WHERE id = '...';
I worried how bad it is and is it worth of thinking about.
Django documentation offers queryset update but it looks not very elegant:
def upd(obj, **kwargs):
obj.__class__._default_manager.filter(pk=obj.pk).update(**kwargs)
upd(news, last_visited=datetime.datetime.now())
I use mysql backend.
Using update but with a cleaner approach:
class News(models.Model):
def update_visited(self):
News.objects.filter(pk=self.pk).update(
last_visited=datetime.datetime.now())
I think using queryset update is good. It removes the possibility that you overwrite changes to other fields by accident.
I know you're worried that it looks inelegant, but you only have to use it once in your upd function, then use upd in your views.
Supposing you want to use this on more than one model (guessing this because you pass obj to your upd function) it would probably make sense to have some base class that implements the last_visited field and your News class inherits from this class... Then you could do the update just on your base class.... Another possibilty would be putting the last_visited information into a seperate model and referencing the News model either through a ForeignKey or a GenericForeignKey (in the case you want to keep a 'history' for different models).
I defined some WTForms forms in an application that uses SQLALchemy to manage database operations.
For example, a form for managing Categories:
class CategoryForm(Form):
name = TextField(u'name', [validators.Required()])
And here's the corresponding SQLAlchemy model:
class Category(Base):
__tablename__= 'category'
id = Column(Integer, primary_key=True)
name = Column(Unicode(255))
def __repr__(self):
return '<Category %i>'% self.id
def __unicode__(self):
return self.name
I would like to add a unique constraint on the form validation (not on the model itself).
Reading the WTForms documentation, I found a way to do it with a simple class:
class Unique(object):
""" validator that checks field uniqueness """
def __init__(self, model, field, message=None):
self.model = model
self.field = field
if not message:
message = u'this element already exists'
self.message = message
def __call__(self, form, field):
check = self.model.query.filter(self.field == field.data).first()
if check:
raise ValidationError(self.message)
Now I can add that validator to the CategoryForm like this:
name = TextField(u'name', [validators.Required(), Unique(Category, Category.name)])
This check works great when the user tries to add a category that already exists \o/
BUT it won't work when the user tries to update an existing category (without changing the name attribute).
When you want to update an existing category : you'll instantiate the form with the category attribute to edit:
def category_update(category_id):
""" update the given category """
category = Category.query.get(category_id)
form = CategoryForm(request.form, category)
The main problem is I don't know how to access the existing category object in the validator which would let me exclude the edited object from the query.
Is there a way to do it? Thanks.
In the validation phase, you will have access to all the fields. So the trick here is to pass in the primary key into your edit form, e.g.
class CategoryEditForm(CategoryForm):
id = IntegerField(widget=HiddenInput())
Then, in the Unique validator, change the if-condition to:
check = self.model.query.filter(self.field == field.data).first()
if 'id' in form:
id = form.id.data
else:
id = None
if check and (id is None or id != check.id):
Although this is not a direct answer I am adding it because this question is flirting with being an XY Problem. WTForms primary job is to validate that the content of a form submission. While a decent case could be made that verifying that a field's uniqueness could be considered the responsibility of the form validator, a better case could be made that this is the responsibility of the storage engine.
In cases where I have be presented with this problem I have treated uniqueness as an optimistic case, allowed it to pass form submission and fail on a database constraint. I then catch the failure and add the error to the form.
The advantages are several. First it greatly simplifies your WTForms code because you do not have to write complex validation schemes. Secondly, it could improve your application's performance. This is because you do not have to dispatch a SELECT before you attempt to INSERT effectively doubling your database traffic.
The unique validator needs to use the new and the old data to compare first before checking if the data is unique.
class Unique(object):
...
def __call__(self, form, field):
if field.object_data == field.data:
return
check = DBSession.query(model).filter(field == data).first()
if check:
raise ValidationError(self.message)
Additionally, you may want to squash nulls too. Depending on if your truly unique or unique but allow nulls.
I use WTForms 1.0.5 and SQLAlchemy 0.9.1.
Declaration
from wtforms.validators import ValidationError
class Unique(object):
def __init__(self, model=None, pk="id", get_session=None, message=None,ignoreif=None):
self.pk = pk
self.model = model
self.message = message
self.get_session = get_session
self.ignoreif = ignoreif
if not self.ignoreif:
self.ignoreif = lambda field: not field.data
#property
def query(self):
self._check_for_session(self.model)
if self.get_session:
return self.get_session().query(self.model)
elif hasattr(self.model, 'query'):
return getattr(self.model, 'query')
else:
raise Exception(
'Validator requires either get_session or Flask-SQLAlchemy'
' styled query parameter'
)
def _check_for_session(self, model):
if not hasattr(model, 'query') and not self.get_session:
raise Exception('Could not obtain SQLAlchemy session.')
def __call__(self, form, field):
if self.ignoreif(field):
return True
query = self.query
query = query.filter(getattr(self.model,field.id)== form[field.id].data)
if form[self.pk].data:
query = query.filter(getattr(self.model,self.pk)!=form[self.pk].data)
obj = query.first()
if obj:
if self.message is None:
self.message = field.gettext(u'Already exists.')
raise ValidationError(self.message)
To use it
class ProductForm(Form):
id = HiddenField()
code = TextField("Code",validators=[DataRequired()],render_kw={"required": "required"})
name = TextField("Name",validators=[DataRequired()],render_kw={"required": "required"})
barcode = TextField("Barcode",
validators=[Unique(model= Product, get_session=lambda : db)],
render_kw={})
Looks like what you are looking for can easily be achieved with ModelForm which is built to handle forms that are strongly coupled with models (the category model in your case).
To use it:
...
from wtforms_components import Unique
from wtforms_alchemy import ModelForm
class CategoryForm(ModelForm):
name = TextField(u'name', [validators.Required(), Unique(Category, Category.name)])
It will verify unique values while considering the current value in the model. You can use the original Unique validator with it.
This worked for me, simple and easy:
Make sure that every time when a new row created in DB it must have unique name in colomn_name_in_db otherwise it will not work.
class SomeForm(FlaskForm):
id = IntegerField(widget=HiddenInput())
fieldname = StringField('Field name', validators=[DataRequired()])
...
def validate_fieldname(self, fieldname):
names_in_db = dict(Model.query.with_entities(Model.id,
Model.colomn_name_in_db).filter_by(some_filtes_if_needed).all())
if fieldname.data in names_in_db.values() and names_in_db[int(self.id)] != fieldname.data:
raise ValidationError('Name must be unique')