I'm running my own smart house project, using django backend with MySql on raspberry Pi.I've got SensorsData table in DB with thousands records with data from sensors. In my REST API I'm using view which looks like this:
#api_view(['GET'])
#permission_classes([IsAuthenticated])
def list_of_sensors_data(request, format=None):
"""
Get list of all sensors data, only for authenticated users
:param request: GET
:return: list of all sensors data if ok http 200 response
"""
sensors_data = SensorsData.objects.all()
serializer = SensorsDataSerializer(sensors_data, many=True)
return Response(serializer.data, status=status.HTTP_200_OK)
I've run perfomance test with locust simulating 10 users trying to use my endpoints. After some time, Django keeps returning 504 Timeout using this particular endpoint. My quest is, how can I optimize this queryset? I need to make it faster.
EDIT SensorsData model:
class SensorsData(models.Model):
sensor = models.ForeignKey(Sensors, on_delete=models.CASCADE)
delivery_time = models.DateTimeField(auto_now_add=True)
sensor_data = models.CharField(max_length=20)
class Meta:
verbose_name = "Sensor data"
verbose_name_plural = "Sensors data"
def __str__(self):
return f"{self.sensor.id}: {self.sensor.name}"
SensorsData Serializer:
class SensorsDataSerializer(serializers.ModelSerializer):
sensor = serializers.SlugRelatedField(read_only=False, many=False, slug_field='name', queryset=Sensors.objects.all())
class Meta:
model = SensorsData
fields = ("sensor", "delivery_time", "sensor_data")
This will introduce an N+1 problem, for each SensorsData object, you will make an additional query to fetch the related Sensor object. The good news is that you can use .select_related(…) [Django-doc] to let Django retrieve all related sensors in the same query:
#api_view(['GET'])
#permission_classes([IsAuthenticated])
def list_of_sensors_data(request, format=None):
sensors_data = SensorsData.objects.select_related('sensor')
serializer = SensorsDataSerializer(sensors_data, many=True)
return Response(serializer.data, status=status.HTTP_200_OK)
Related
Updated
I changed my simplified question into a real example.
I've created a working post response of data from the model using ModelSerialzer, which I call from a post method in a view class. I would like to add additional data to the response. This is the pertinent code from my CBV:
def post(self, request, format=None):
user_profile = UserProfiles.objects.get(user=request.user.id)
service_id = user_profile.service_id
rec_filter = Recommendations.objects.values_list('resource')
if service_id > 0:
service_name = Services.objects.get(pk=service_id)
programs = Programs.objects.filter(services=service_id)
resources_filtered = Resources.objects.filter(program__in=programs).exclude(id__in=rec_filter)
else:
service_name = 'All Services'
resources_filtered = Resources.objects.exclude(id__in=rec_filter)
serializer = ResourceSerializer(resources_filtered, many=True)
#serializer.data["service"] = service_name
return Response(serializer.data)
The commented out line was my attempt to add data base on a similar post here. I get a 500 response in my API call. What is the correct way to do it? The response data is JSON if that's necessary to mention.
This is the ModelSerializer:
class ResourceSerializer(serializers.ModelSerializer):
organization = OrganizationSerializer(read_only=True)
program = ProgramSerializer(read_only=True)
class Meta:
model = Resources
fields = [
'organization',
'program',
'link',
'contact',
'general_contact',
'eligibility',
'service_detail'
]
Test of the answer
Heres the updated code based on the answer with a correction to fix and error:
class ResourceSerializer(serializers.ModelSerializer):
organization = OrganizationSerializer(read_only=True)
program = ProgramSerializer(read_only=True)
service = serializers.SerializerMethodField()
def get_service(self, obj):
return "All Services"
class Meta:
model = Resources
fields = [
'organization',
'program',
'link',
'contact',
'general_contact',
'eligibility',
'service_detail',
'service'
]
The problem with this approach is that the value "All Services" is repeated in every row serialized. It's only needed once. I'd also like to keep the data transmitted minimized.
The problem with the original attempt is that serializer.data is immutable. It's necessary to make a copy and add to it.
serializer = ResourceSerializer(resources_filtered, many=True)
augmented_serializer_data = list(serializer.data)
augmented_serializer_data.append({'service': 'All Services'})
return Response(augmented_serializer_data)
This answer is based on one given by #andre-machado in this question.
This code here is an example to coincide with the other answer given.
You can do it in serializer itself. Define the new field required and add it in fields. Mark all the fields in serializer from resource model.
class ResourceSerializer(serializers.ModelSerializer):
service = serializers.SerializerMethodField()
def get_service(self):
return "All Services"
class Meta :
model = Resources
fields = ('service') #Mark all the fields required here from resource model
You can do it from the serilaizer. In this case i was adding the field isOpen to the response and this is how i did it .timeDifference is the name of the function that was to generate data for the extra field . I hope it helps
class ShopSearchSerializer(serializers.ModelSerializer):
isOpen = serializers.SerializerMethodField('timeDifference')
def timeDifference(self,*args):
requestTime = datetime.now()
return requestTime
class Meta:
model = Shop
fields =['name','city','street','house','opening_time','closing_time','isOpen']
There are lot of answers related to custom fields on stackoverflow but when trying with them, I am getting different error, so posting a separate question.
I wanted to return a JSON Response for the following url
urls.py
path('cards/<int:pk>/smarts', smarts.as_view(), name="smarts"),
I will be using below api.py file to return aggregate fields using Transaction model, the query is working fine, I only have to return appropriate response. Here I have one of the fields as Decimal hence tried with DjangoJSONEncoder but got an error.
api.py
class smarts(generics.ListAPIView):
serializer_class = TransactionSerializer
permission_classes = [permissions.IsAuthenticated, TransactionIsOwnerOrNot]
def get_queryset(self):
card = get_object_or_404(self.request.user.cards, pk=self.kwargs['pk'])
qs=card.transactions.values('vendor').annotate(a=Count('pk'),b=Sum('amount')).order_by('-b')
....CODE REQUIRED
return ....
models.py
class Transactions(models.Model):
amount = models.DecimalField(max_digits=19, decimal_places=2)
vendor = models.CharField(max_length=200)
category = models.CharField(max_length=200)
owner = models.ForeignKey(Cards, on_delete=models.CASCADE, related_name="transactions",null=True)
serializer.py
class TransactionSerializer(serializers.ModelSerializer):
class Meta:
model = Transactions
fields = '__all__'
I found the answer after hit and trial, I used custom serializer to return required field.
serializer.py
class SmartSerializer(serializers.Serializer):
vendor = serializers.CharField(max_length=200)
tot = serializers.IntegerField()
tot_amt = serializers.DecimalField(max_digits=19, decimal_places=2)
api.py
class smartstatements(generics.ListAPIView):
permission_classes = [permissions.IsAuthenticated, TransactionIsOwnerOrNot]
serializer_class = SmartSerializer
def get_queryset(self):
card = get_object_or_404(self.request.user.cards, pk=self.kwargs['pk'])
queryset=card.transactions.values('vendor')
.annotate(tot=Count('pk'),tot_amt=Sum('amount'))
.order_by('-tot_amt')
return queryset
im trying to post the recruitment date of an employee , but i keep having the 400 bad request error and recruitment_date: ["This field is required."] ,after searching i added the Date Formats on the settings but i still have the same error
views.py :
class CreateEmployee(generics.CreateAPIView):
lookup_field= 'pk'
serializer_class = EmployeeSerializer
def get_queryset(self):
return Employee.objects.all()
serializers.py :
class EmployeeSerializer(serializers.ModelSerializer):
url = serializers.SerializerMethodField(read_only=True)
recruitment_date =fields.DateField(format="%Y-%m-%d",input_formats=['%Y-%m-%d'])
class Meta:
model = Employee
fields =['url','id','f_name','l_name','job','telephone','recruitment_date','salary']
def get_url(self,obj):
request = self.context.get("request")
return obj.get_api_url(request=request)
settings.py :
"DATE_INPUT_FORMATS": ["iso-8601","%Y-%m-%d"],
I've got a model method that calculates the score of votes using the HN ranking algorithm in my model that also caches each score. calculate_score() is the main thing to focus on here.
class Submission(models.Model):
submission_type = models.CharField(_('Submission Type'),
max_length=255,
choices=tuple([(media.name.lower(), media.value) for media in MediaTypes]),
blank=False)
title = models.CharField(_('Title'), max_length=100, blank=False)
url = models.URLField(_('Link'), blank=False)
description = models.TextField(_('Description'))
flagged = models.BooleanField(_('Is Flagged for Review'), default=False)
user = models.ForeignKey(User, related_name='user_submissions')
thumbnail = models.ImageField()
date_submitted = models.DateTimeField(default=timezone.now)
by_votes = VoteCountManager()
objects = models.Manager()
def __str__(self):
return self.submission_type + ': ' + self.title
def get_absolute_url(self):
return reverse('submit_detail', args=[
str(self.submission_type),
str(self.id),
])
def get_vote(self):
"""
Returns the number of votes associated with a particular submission
:return: int
"""
return self.submission_votes.count()
def calculate_score(self):
"""
This is a variation of the HN ranking algorithm
:return: score
"""
secs_in_hour = float(60 * 60)
g = 1.2
delta = timezone.now() - self.date_submitted
item_hour_age = delta.total_seconds() / secs_in_hour
votes = self.submission_votes.count() - 1
score = votes / pow((item_hour_age + 2), g)
cached = cache.get('s{0}'.format(self.pk))
if not cached:
cache.set('s{0}'.format(self.pk), score, 30)
return score
I'm using the djangorestframework that serializes my model:
class SubmissionSerializer(serializers.HyperlinkedModelSerializer):
score = serializers.SerializerMethodField('rank')
class Meta:
model = Submission
def rank(self, obj):
return obj.calculate_score()
I have two ways that I think I can solve my problem, but I don't know how to do either of them, and I'm not sure which one is best. Since I'm caching the score for each individual submission. I was trying to order the submissions in a ListView like this pulling from the cache:
class SubmissionList(ListView):
model = Submission
def get_queryset(self):
return super(SubmissionHotList, self).get_queryset().annotate(
score=cache.get('s{0}'.format(x) for x in Submission.pk)
votes=models.Count('submission_votes'),
).order_by(
'-score', '-votes',
)
But I found that order_by only works on the database level, and I would have to create a database field for the calculated score, which I would like to avoid if possible. My other possibility is using the serialized data in my API for Submission in this ListView, but I'm not sure if APIs are only there for external applications, or if I could use them in the same application that the API is generated.
I guess my question is. Would it be better to get the cached objects to list each submission in a particular order, or could I use the API to accomplish this? And if I used the API, how could I parse and order the JSON data in the ListView, while keeping votes as a secondary ordering mechanism?
if you want to display all items without pagination, you will not lose any performance if you do something like
import operator
class SubmissionList(TemplateView):
template_name = "submissions.html"
def get_context_data(self, **kwargs):
context = super(SubmissionList, self).get_context_data(**kwargs)
submissions = Submission.objects.all().annotate(
votes=models.Count('submission_votes')
)
sorted_submissions = sorted(submissions, key=lambda x:(cache.get('s{0}'.format(x.pk)), x.votes), reverse=True)
context['submissions'] = sorted_submissions
return context
I have worked on this for sometime now and I have decided to seek for your help. I have an ongoing project and I am using the django framework to build the site.
I have an html page that requests for user input, I have been able to extract the given data using the form.cleaned_data['']. A
#This is my views.py;
from django.shortcuts import render_to_response
from django.template import RequestContext
from forms import CarloanForm
def index(request):
form = CarloanForm()
if request.POST:
form = CarloanForm(request.POST)
if form.is_valid():
form.save()
else:
form = CarloanForm()
return render_to_response('carloan/index.html', {'form': form},
context_instance=RequestContext(request))
#I am extracting the user inputs using;
amount_of_vehicle = form.cleaned_data['cost_of_vehicle_Naira']
tenure = form.cleaned_data['loan_repayment_tenure_Months']
interest_rate = form.cleaned_data['interest_rate_Percentage']
equity = form.cleaned_data['equity_contrib_rate_Percentage']
depreciation_rate = form.cleaned_data['depreciation_rate_Percentage']
What I need to do and don't know how to go about it are;
Carry out some operations on the data (extracted using form.cleaned_data['']) and I have written some lines of code to that effect;
class LoanCalc:
def __init__(self,amount_of_vehicle,tenure,interest_rate,equity,depreciation_rate):
self.amount_of_vehicle = float(amount_of_vehicle)
self.tenure = float(tenure)
self.interest_rate = float(interest_rate)
self.equity = float(equity)
self.depreciation_rate = float(depreciation_rate)
def interest(self):
return (self.interest_rate/100) * self.amount_of_vehicle *(self.tenure/12)
def management_fee(self):
return 0.01 * (self.amount_of_vehicle + user.interest())
def processing_fee(self):
return 0.0025 *(self.amount_of_vehicle + user.interest())
def legal_fee(self):
return 0.0075 * (self.amount_of_vehicle + user.interest())
def residual_amount(self):
return 0.01 * (self.amount_of_vehicle - ((self.depreciation_rate/100) * self.amount_of_vehicle *(self.tenure/12)))
def equity_contribution(self):
return (self.equity/100) * self.amount_of_vehicle
def LoanPaymentPlan(self):
months = 1
total_amount = self.amount_of_vehicle+user.interest()+user.management_fee()+user.processing_fee()+user.legal_fee()+user.residual_amount()
upfront_payment = user.management_fee()+user.processing_fee()+user.legal_fee()+user.equity_contribution()+user.residual_amount()
opening_balance = total_amount - upfront_payment
balance = opening_balance
while months <= self.tenure:
if balance > 0:
monthly_instalment =(opening_balance/self.tenure)
monthly_interest = (((self.interest_rate/100) * balance)/ 12)
loan_payment = monthly_instalment - monthly_interest
closing_balance = balance - monthly_instalment
print' ',months,' ',round(balance,2),' ', round(monthly_instalment,2),' ',round(monthly_interest,2) \
, ' ',' ',round(loan_payment,2),' ',round(closing_balance,2)
balance = closing_balance
months += 1
return 'Thank you for using the Loan Calculator App'
and i want to carry out the operations in the code above on the extracted data.
I am thinking of doing it in such a way like this;
Create an empty dictionary;
user = {}
user = LoanCalc(amount_of_vehicle,tenure,interest_rate,equity,depreciation_rate)
result= user.interest()
result1 = user.management_fee()
. .
. .
. .
result10 = user.LoanPaymentPlan()
Pass the result(s) obtained from (question 2) to the same template that generated the form.
Please help me out guys, i am still very new to django. Thanks
This is the full stack of the referenced error am getting:
Environment:
Request Method: GET
Request URL: http://127.0.0.1:9000/result/
Django Version: 1.4.1
Python Version: 2.5.4
Installed Applications:
('django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.sites',
'django.contrib.messages',
'django.contrib.staticfiles',
'django.contrib.admin',
'carloan')
Installed Middleware:
('django.middleware.common.CommonMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware')
Traceback:
File "C:\Python25\Lib\site-packages\django\core\handlers\base.py" in get_response
111. response = callback(request, *callback_args, **callback_kwargs)
File "C:\Users\AJIBIKE\Documents\Circuit Atlantic\calculator\carloan\views.py" in result_page
226. 'carloan': instance,
def result_page(request):
if request.POST:
form = Carloan_formForm(request.POST)
if form.is_valid():
instance = form.save(commit=False)
instance.interest = (instance.interest_rate/100) * instance.amount_of_vehicle * (instance.tenure/12)
instance.save()
else:
form = Carloan_formForm()
return render_to_response('carloan/result.html', {'carloan': instance,'form': form},
context_instance=RequestContext(request))
Exception Type: UnboundLocalError at /result/
Exception Value: local variable 'instance' referenced before assignment
Latest
from django.shortcuts import render_to_response
from django.template import RequestContext
from django.http import HttpResponseRedirect, HttpResponse
from forms import Carloan_formForm
def index(request):
if request.POST:
form = Carloan_formForm(request.POST)
if form.is_valid():
instance = form.save()
return HttpResponseRedirect ('/result/')
form = Carloan_formForm()
kwvars = {
'form': form,
}
return render_to_response('carloan/index.html', kwvars,
context_instance=RequestContext(request))
def result_page(request):
instance = None
if request.POST:
form = Carloan_formForm(request.POST)
if form.is_valid():
instance = form.save(commit=False)
instance.interest = (instance.interest_rate_Percentage/100) * instance.cost_of_vehicle_Naira * (instance.loan_repayment_tenure_Months/12)
instance.management_fee = 0.01 * (instance.cost_of_vehicle_Naira + instance.interest)
instance.processing_fee = 0.0025 * (instance.cost_of_vehicle_Naira + instance.interest)
instance.legal_fee = 0.0075 * (instance.cost_of_vehicle_Naira + instance.interest)
#i get syntax error starting from this line, and when i comment it out and the lines below, there is no syntax error.
instance.residual_amount = 0.01 * ((instance.cost_of_vehicle_Naira - ((instance.depreciation_rate_Percentage/100) * instance.cost_of_vehicle_Naira * (instance.tenure/12)))
instance.equity_contribution = (instance.equity_contrib_rate_Percentage/100) * instance.cost_of_vehicle_Naira)
instance.save()
else:
form = Carloan_formForm()
return render_to_response('carloan/result.html', {'instance': instance, 'form': form},
context_instance=RequestContext(request))
forms.py
from django import forms
from django.forms import ModelForm
from models import Carloan_form
class Carloan_formForm(ModelForm):
class Meta:
model = Carloan_form
exclude = ('interest', 'management_fee', 'processing_fee', 'legal_fee', \
'residual_amount', 'equity_contribution')
1 Save the result of the operations carried out on the extracted data in the Django Administration. (N.B: I already know how to save the user input)
Create a model to save the user's entries. app/models.py
class Carloan(models.Model) :
# In addition to your model fields add the functions that are part of your LoanCalc
...
Then create a ModelForm: app/forms.py
class CarLoanForm(forms.ModelForm) :
class Meta:
model = Carloan
Use CarLoanForm the same way you normally would except: app/views.py
if request.POST:
form = CarloanForm(request.POST)
if form.is_valid():
carloan = form.save()
#pass carloan (the instance) to the template and
#call its functions(from LoanCalc) instead of
#passing all of the values separately
Lastly, register this Model so it appears in the admin section app/admin.py
from django.contrib import admin
from app.models import Carloan
admin.sites.register(Carloan)
2 I have divided the page into two(one for the form and the other for
the result) and i want to pass the result to one-half of the page to
enable the user to see it.
Using the steps above, carloan will be passed to the view. go ahead and render its values.
Additionally:
Refactor index
your index definition should be refactored slightly, or your POST bound form will never make it to the template:
def index(request):
if request.POST:
form = CarloanForm(request.POST)
if form.is_valid():
instance = form.save()
else :
form = CarloanForm()
return render_to_response('carloan/index.html', {'form': form},
context_instance=RequestContext(request))
Refactor your ModelForm
Carry out some operations on the data (extracted using form.cleaned_data['']) and I have written some lines of code to that effect; - using a ModelForm instead of a regular django form you will get an instance of the model with the user's values already filled in, you reference them with instance.field.
and i want to carry out the operations in the code above on the extracted data. - the fields that you plan to calculate values for are listed in exclude they will not factor into form validation.
class CarLoanForm(forms.ModelForm) :
class Meta:
model = Carloan
# exclude fields that are calculated from user input
# NOTE: these fields must be in your model
exclude = ('interest', 'management_fee'...)
Refactor result page
Carry out some operations on the data (extracted using form.cleaned_data['']) and I have written some lines of code to that effect;
in results_page under the form.is_valid check, ther is a line: instance = form.save(commit=False) this gets all of the values that the user submitted in a instance of the Carloan Model that has NOT been saved yet.
further down: instance.interest = (instance.interest_rate/100) * instance.amount_of_vehicle *(instance.tenure/12) in this line I am calculating one of the excluded fields values (this calculation is a copy of your code).
lastly (after all of the operations on the data have been completed) I save the instance of the model.
then pass the instance to the template for display.
code:
def result_page(request):
if request.POST:
form = CarloanForm(request.POST)
if form.is_valid():
# get an instance from the form but don't save it
instance = form.save(commit=False)
# calculate excluded field values
instance.interest = (instance.interest_rate/100) * instance.amount_of_vehicle *(instance.tenure/12)
...
# after you have calculated all of the excluded fields save the instance
instance.save()
else :
form = CarloanForm()
return render_to_response('carloan/result.html', {'carloan' : instance, 'form' : form},
context_instance=RequestContext(request))
Error:
reference before error assignment - What does it say was referenced before assignment? Paste the entire stack trace please.
One of the simpler ways would be to store the data in the session in one view, retrieve it in the next. The docs will help.1
Having said that - there are a few other ways to approach the problem. It has been discussed more than once on SO:
Django Passing data between views
How do you pass or share variables between django views?
how to pass a list between views in django