Expense of django query - mysql

I have two tables: Actor and Images. The Images table is linked to the Actor table through a Foreign Key. I might make a query on Actor that pulls a list of all the actors with first name beginning with the letter A. Let's say I store this query result in actor_list and pass it to the template. I can get a list of all the Images through the template by iterating. Something like:
{% for actor in actor_list %}
{% for image in actor.img_set %}
{{ image }}
{% endfor %}
{% endfor %}
Is this costly to the database? Does the associated img_set come through with the initial actor object list, or does a new query hit the database every time I call for the actor.img_set?

You should install the Django debug toolbar and see what SQL queries are happening under the covers.
If you are getting too many queries, you can try using select_related() to pull in the related data in the initial query.

Better than using the django toolbar is using the connections library. With Debug set to True:
>>> from django.db import connection
>>> for actor in actor_list:
>>> for image in actor.img_set:
>>> image
>>> connection.queries()
This will show all the queries that were executed and the amount of time that each took.
The debug toolbar is cool, though, for many uses. :-)

Related

Comparing lists in DBT jinja macro inside for loop

I am storing output of each column in a list for the below query
select table,
columns,
flag
from {{ model }}
for each table I need to compare the flag. how do i get flag in list for the corresponding table
{% for n in table %}
{%- if flag[n] == 'False' %}
I tried as above its not working its returning null value
It's important to understand that jinja is a templating language. In dbt, we use jinja to automate the creation of sql that then gets executed in our data warehouse. Accordingly, data from your data warehouse does not (typically) flow through your jinja code.
flag is a reference to a column in your database, so to evaluate the contents of flag, you need to write sql to do that job.
The only exception is if you are using run_query or a similar macro to pull the data from your database into your jinja context at model-compile time. But I don't see any reference to that in your code (and frankly that's a complicating detail that's only relevant if you are trying to write SQL dynamically). If you are just trying to compare the value from a record in your database as part of your model, just do it in sql.

Queries in query set group in to one query using Django

I have a query set like the below.
<QuerySet[{'user':'xyz','id':12,'home':'qwe','mobile':1234},
{'user':'xyz','id':12,'home':'qwe','mobile':4321},
{'user':'abc','id':13,'home':'def','mobile':1233},
{'user':'abc','id':13,'home':'def','mobile':1555},]>
This QuerySet is returned by django using
users.objects.all()
For Each query in the query set, I'm drawing a user table in the front-end which shows the details of users. If the same 'user' registers with two 'mobile' numbers, it is showed as two rows in the table instead of one. My goal is to display two numbers in the same row instead of creating two rows.
I have thought of two possible solutions which are below:
Solution 1: I have thought of merging the two queries into one if the 'user' value matches in both the queries. For this we need to make lot of checks using conditional statements which works slowly when lot of users are there.
Solution 2: I have searched on Google and came up with Group By Django, but it is not working. I have tried below
query = users.objects.all().query
query.group_by = ['mobile']
results = QuerySet(query=query, model=users)
Please provide a way so that two queries can be clubbed into one based on 'user' and after clubbing 'mobile' should contain two values.
Edit: I will pass this query set to a template via view where the table is being drawn. At present, for each query, one row is the in table. Since the above query set has four queries, we will have four rows. But I want to display the information in just two rows since the 'id' and 'user' are same. Code for template is below:
{% if users %}
{% for user in users %}
{% for k,v in required_keys.items %}
<td>{{ user | get_item:k }}
{% endfor %}
{% endfor %}
{% endif %}
Required key is an dictionary which contains the keys which are used to display in table. Example:
Required_keys = {
'User Name':user,
'Contact':mobile,
'Address':home,
}
The get item is a function which gets the value when key is passed.
def get_item(dictionery,key):
return dictionery.get(key)
Edit 2: I have tried for a solution and it's not a full solution but it partially solves the problem. The solution perfectly works with one user. However if there are many users, the solution doesn't work. See below for example:
# input query set to the function which partially solves the problem
<QuerySet[{'user':'xyz','id':12,'home':'qwe','mobile':1234},
{'user':'xyz','id':12,'home':'qwe','mobile':4321},]>
# Now our team has written a function which gives the following output
<QuerySet[{'user':'xyz','id':12,'home':'qwe','mobile':1234,4321},]>
But if there are more than one user, the output is same as input, it doesn't club the queries. See the function below:
def merge_values(cls,values):
import itertools
grouped_results = itertools.groupby(values,key=lambda x:x['id'])
merged_values = []
for k,g in grouped_results:
groups=list(g)
merged_value = {}
for group in groups:
for key, val in group.iteritems():
if not merged_value.get(key):
merged_value[key] = val
elif val != merged_value[key]:
if isinstance(merged_value[key], list):
if val not in merged_value[key]:
merged_value[key].append(val)
else:
old_val = merged_value[key]
merged_value[key] = [old_value, val]
merged_values.append(merged_value)
return merged_values
The values parameter for the function is the whole query set. But this function
works if there is only one user in query set as mentioned above. But if there
are multiple users, it fails.
Edit 3: I have found why above function won't work for multiple users(don't know whether this is correct). The input query set to the function (for one user) is
<QuerySet[{'user':'xyz','id':12,'home':'qwe','mobile':1234},
{'user':'xyz','id':12,'home':'qwe','mobile':4321},]>
Since the two user queries are one after another, they are clubbed. But if there are multiple users, the query set passed to the function is
<QuerySet[{'user':'xyz','id':12,'home':'qwe','mobile':1234},
{'user':'xyz','id':13,'home':'qwe','mobile':4321},
{'user':'abc','id':12,'home':'def','mobile':1233},
{'user':'abc','id':13,'home':'def','mobile':1555},]>
In the above query set, the user with same id are not one after another, so the function failed to get desired output.
You can use the function which you have written, only change is required is need to order the values based on id.
So before taking the values from the objects, order_by('id'), then take your required values and pass it to your function to merge the values.
Eg.
query_set = Mymodel.objects.filter(name='Required').order_by('id')
values = query_set.values('id', 'field1', 'field2')
merged_values = merge_values(values)
As a start, what would be helpful is to organise your models better. You have user information duplicated in your user model. If a you know that a user has multiple mobile numbers, it may be better to create a separate model for that:
class Mobile(models.Model)
number = models.IntegerField()
user = models.ForeignKey(User)
Then, when you are make a query and you want to get all the mobile numbers, you would first get all the users using users = User.objects.all() in your view, then in your template:
{% for user in users %}
{% for mobile in user.mobile_set.all %}
<td>{{ user }}</td>
<td>{{ mobile.number }}</td>
{% endfor %}
{% endfor %}
As a side note, I feel it's better practice to name your models as a singular (User instead of Users)

Fetch currently logged in user relation with a given model instance

Assume we have a model called Author as such:
class Author(models.Model):
name = models.CharField(max_length=250)
We also have a feature which enables users to follow a certain author:
class UserFollow(models.Model):
author = models.ForeignKey(Author)
user = models.ForeignKey(User)
If you have a button in your UI that enables logged in users to follow / unfollow these authors. One way to check if this user is already following a certain author is to make a query on the existence of a record in UserFollow. However, when you're fetching a list of 10 / 20 .. etc authors, how would you check each author for the currently logged in user?
The usual approach would result in +X number of extra queries depending on how many items you're loading per page. Is there a more efficient way to achieve the same effect?
Thanks in advance
you can fetch all instances of UserFollow in view, pass it as a variable to the template and then you can check if given author is in this list
def my_view(request):
following = list(UserFollow.objects.filter(user=request.user))
template_context = {
'following': following
}
template:
{% for author in authors %}
{% if author in following %}
{# do some magic #}
{% endif %}
{% endfor %}

How to avoid a lot of queries when counting associated entities with Doctrine?

Imagine an User entity with an association with Group (many-to-many). I need to display user details (along with - say - group names or groups count). In my controller (Symfony 2) I can get all users:
$users = $this->em->getRepository('My\Entity\User')
->getAll();
// Assign $users to whatever the view engine
return $this->render('users.html.twig', array('users' => $users));
Of course , being the association LAZY by default, no JOIN operations with Group are performed. This is good most of the time.
But what happens when I try to get group count in my view?
{% for user in users %}
{{ user.groups|length }}
{% endfor %}
The result is: one query performed for each raw. That is, the number of the query equals the number of the user. What happens with 1000 users (and no pagination)?
How can fetch all associated entity for the User class (i.e. issue just one query)?
You do that by creating custom repository method where you join your entities.
More info -> http://symfony.com/doc/master/book/doctrine.html#joining-related-records

Flask SQLAlchemy does not commit until App ends

I've started my session like so:
from flask.ext.sqlalchemy import SQLAlchemy
app.config['SQLALCHEMY_DATABASE_URI'] = 'mysql://mydatabase'
db = SQLAlchemy(app)
And I'm Adding entries to my database like so:
db.session.add(user(name, password))
db.session.commit()
Immediately after I add it, I render a new template. When the template's rendered, it should include the data I've just added, but it doesn't. However, when I end my app with a ctrl-c, then restart it, the data I've added now shows.
The data is submitted using a GET. The arguments are obtained using flask.request.args.get and it gets the name and password. Then the data is added using the db.session.add thing above.
After the db.session.add is done, the data is obtained from the entire database like so:
users = []
for userentry in user.query.all():
users.append(userentry.name)
In the template:
{% for name in users %}
<p>{{name}}</p>
{% endfor %}
Then template is rendered using: return render_template('my_page.html', users=users)
It seems I've been using an older design pattern that I found on the Flask Mega Tutorial. I solved my problem by opening up individual sessions for adding data. http://docs.sqlalchemy.org/en/rel_0_7/orm/session.html