Using GROUP_CONCAT with other annotations in Django - mysql

I use an annotation which counts upvotes/downvotes while returning a list of articles:
queryset = queryset.annotate(
upvotes_count=models.Sum(
models.Case(
models.When(likes__like_state=1, then=1),
default=0,
output_field=models.IntegerField()
)
)
).annotate(
downvotes_count=models.Sum(
models.Case(
models.When(likes__like_state=-1, then=1),
default=0,
output_field=models.IntegerField()
))
)
But each article also has a few categories as ManyToMany related field and I needed to return those categories comma-separated, so I wrote this function:
class GroupConcat(models.Aggregate):
function = 'GROUP_CONCAT'
template = "%(function)s(%(distinct)s %(expressions)s %(separator)s)"
def __init__(self, expression, distinct=False, separator=', ', **extra):
super(GroupConcat, self).__init__(
expression,
distinct='DISTINCT' if distinct else '',
separator="SEPARATOR '%s'" % separator,
output_field=models.CharField(),
**extra
)
And added it to my annotation:
queryset = queryset.annotate(category=GroupConcat('categories__name'))
It works fine but upvotes_count and downvotes_count went crazy and started to multiply(!) results by amount of categories.
So the question is: "Is there a way to use GROUP_CONCAT in Django without breaking down SUM annotations?"

Very nice solution.
But to operate with group by field you should use order_by statement.
for example:
Store.objects.all().values('value').order_by('value').annotate(stores=GroupConcat('id'))
would generate sql statement
SELECT store.value, GROUP_CONCAT(store.id SEPARATOR ",") AS
stores FROM store WHERE store.value > 0 GROUP BY
store.value ORDER BY store.value ASC
and result would be
value, stores
1 "16,27"
Without order_by it would be like this:
SELECT store.value, GROUP_CONCAT(store.id SEPARATOR ",") AS
stores FROM store WHERE store.value > 0 GROUP BY store.id
ORDER BY store.value ASC
and result would be
value, stores
1 16
2 27

Related

How to create union of two different django-models?

I have two django-models
class ModelA(models.Model):
title = models.CharField(..., db_column='title')
text_a = models.CharField(..., db_column='text_a')
other_column = models.CharField(/*...*/ db_column='other_column_a')
class ModelB(models.Model):
title = models.CharField(..., db_column='title')
text_a = models.CharField(..., db_column='text_b')
other_column = None
Then I want to merge the two querysets of this models using union
ModelA.objects.all().union(ModelB.objects.all())
But in query I see
(SELECT
`model_a`.`title`,
`model_a`.`text_a`,
`model_a`.`other_column`
FROM `model_a`)
UNION
(SELECT
`model_b`.`title`,
`model_b`.`text_b`
FROM `model_b`)
Of course I got the exception The used SELECT statements have a different number of columns.
How to create the aliases and fake columns to use union-query?
You can annotate your last column to make up for column number mismatch.
a = ModelA.objects.values_list('text_a', 'title', 'other_column')
b = ModelB.objects.values_list('text_a', 'title')
.annotate(other_column=Value("Placeholder", CharField()))
# for a list of tuples
a.union(b)
# or if you want list of dict
# (this has to be the values of the base query, in this case a)
a.union(b).values('text_a', 'title', 'other_column')
In SQL query, we can use NULL to define the remaining columns/aliases
(SELECT
`model_a`.`title`,
`model_a`.`text_a`,
`model_a`.`other_column`
FROM `model_a`)
UNION
(SELECT
`model_b`.`title`,
`model_b`.`text_b`,
NULL
FROM `model_b`)
In Django, union operations needs to have same columns, so with values_list you can use those specific columns only like this:
qsa = ModelA.objects.all().values('text_a', 'title')
qsb = ModelB.objects.all().values('text_a', 'title')
qsa.union(qsb)
But there is no way(that I know of) to mimic NULL in union in Django. So there are two ways you can proceed here.
First One, add an extra field in your Model with name other_column. You can put the values empty like this:
other_column = models.CharField(max_length=255, null=True, default=None)
and use the Django queryset union operations as described in here.
Last One, the approach is bit pythonic. Try like this:
a = ModelA.objects.values_list('text_a', 'title', 'other_column')
b = ModelB.objects.values_list('text_a', 'title')
union_list = list()
for i in range(0, len(a)):
if b[i] not in a[i]:
union_list.append(b[i])
union_list.append(a[i])
Hope it helps!!

Ordering a queryset by occurrences

I have a django model:
class Field:
choice = models.CharField(choices=choices)
value = models.CharField(max_length=255)
In my database I have some cases where there are 3 "fields" with the same choice, and some cases where there is 1 field of that choice
How can I order the queryset so it returns, sorted by choice, but with all ones in a set of 3 at the start?
For example
[1,1,1,3,3,3,4,4,4,2,5] where 1,2,3,4,5 are possible choices?
This is the best I can do using django's ORM. Basically, just like in SQL, you have to construct a custom order_by statement. In our case, we'll place it in the SELECT and then order by it:
1) Get a list of choices sorted by frequency: [1, 3, 4, 2, 5]
freq_list = (
Field.objects.values_list('choice', flat=True)
.annotate(c=Count('id')).order_by('-c', 'choice')
)
2) Add indexes with enumerate: [(0,1), (1,3), (2,4), (3,2), (4,5)]
enum_list = list(enumerate(freq_list))
3) Create a list of cases: ['CASE', 'WHEN choice=1 THEN 0', ..., 'END']
case_list = ['CASE']
case_list += ["WHEN choice={1} THEN {0}".format(*tup) for tup in enum_list]
case_list += ['END']
4) Combine the case list into one string: 'CASE WHEN choice=1 THEN 0 ...'
case_statement = ' '.join(case_list)
5) Finally, use the case statement to select an extra field 'o' which will be corresponding order, then just order by this field
Field.objects.extra(select={'o': case_statement}).order_by('o')
To simplify all this, you can put the above code into a Model Manager:
class FieldManager(models.Manager):
def get_query_set(self):
freq_list = (
Field.objects.values_list('choice', flat=True)
.annotate(c=Count('id')).order_by('-c', 'choice')
)
enum_list = list(enumerate(freq_list))
case_list = ['CASE']
case_list += ["WHEN choice={1} THEN {0}".format(*tup) for tup in enum_list]
case_list += ['END']
case_statement = ' '.join(case_list)
ordered = Field.objects.extra(select={'o': case_statement}).order_by('o')
return ordered
class Field(models.Model):
...
freq_sorted = FieldManager()
Now you can query:
Field.freq_sorted.all()
Which will get you a Field QuerySet sorted by frequency of choices
You should make a function and detect which is repeated to select unique, then calling from mysql as a function over mysql

SQLAlchemy MySQL IF Statement

I'm in the middle of converting an old legacy PHP system to Flask + SQLAlchemy and was wondering how I would construct the following:
I have a model:
class Invoice(db.Model):
paidtodate = db.Column(DECIMAL(10,2))
fullinvoiceamount = db.Column(DECIMAL(10,2))
invoiceamount = db.Column(DECIMAL(10,2))
invoicetype = db.Column(db.String(10))
acis_cost = db.Column(DECIMAL(10,2))
The query I need to run is:
SELECT COUNT(*) AS the_count, sum(if(paidtodate>0,paidtodate,if(invoicetype='CPCN' or invoicetype='CPON' or invoicetype='CBCN' or invoicetype='CBON' or invoicetype='CPUB' or invoicetype='CPGU' or invoicetype='CPSO',invoiceamount,
fullinvoiceamount))) AS amount,
SUM(acis_cost) AS cost, (SUM(if(paidtodate>0,paidtodate,invoiceamount))-SUM(acis_cost)) AS profit FROM tblclientinvoices
Is there an SQLAlchemyish way to construct this query? - I've tried googling for Mysql IF statments with SQlAlchemy but drew blanks.
Many thanks!
Use func(documentation) to generate SQL function expression:
qry = select([
func.count().label("the_count"),
func.sum(func.IF(
Invoice.paidtodate>0,
Invoice.paidtodate,
# #note: I prefer using IN instead of multiple OR statements
func.IF(Invoice.invoicetype.in_(
("CPCN", "CPON", "CBCN", "CBON", "CPUB", "CPGU", "CPSO",)
),
Invoice.invoiceamount,
Invoice.fullinvoiceamount)
)
).label("amount"),
func.sum(Invoice.acis_cost).label("Cost"),
(func.sum(func.IF(
Invoice.paidtodate>0,
Invoice.paidtodate,
Invoice.invoiceamount
))
- func.sum(Invoice.acis_cost)
).label("Profit"),
],
)
rows = session.query(qry).all()
for row in rows:
print row

zend select from two tables 3 sets of rows

I have this problem using Zend and I think its db related at all:
I have two tables, one contains:
id, ..., file, desc, date
and the second table contains:
id, ..., file_1, desc_1, file_2, desc_2, date
What I need as a result is:
id, ..., file, desc, date
From both tables, which means I need to have coresponding file, desc and file_1 ->file, desc_1->desc and file_2->file, desc_2->desc in this one table.
Any idea how to do this with Zend 1.12?
You need to use JOIN in Zend ORM
for exmaple
public function getPendingProjects($owner){
$data = $this ->getAdapter()
->select()
->from('campaign' , array('title', 'id'))
->joinLeft('job', 'campaign.id = job.campaign_id', array('count(user_id)'))
->where('campaign.employer_id = ' . (int)$owner . ' AND job.status = 3' );
return $data->query()->fetchAll();
}
taked from here http://zend-frameworks.com/en/articles/zend_db_zend_mysql.html

how to translate a very long mysql query with select and join to zend framework 1.11 model

I have this mysql query:
SELECT
freeAnswers.*,
(SELECT `districtCode`
FROM `geodatas`
WHERE `zipCode` = clients.zipCode
GROUP BY `zipCode`
LIMIT 0, 1) as districtCode,
clients.zipCode,
clients.gender,
clients.startAge,
clients.endAge,
clients.mail,
clients.facebook,
surveys.customerId,
surveys.activityId,
surveys.name as surveyName,
customers.companyName,
activities.name as activityName
FROM freeAnswers,
clients,
surveys,
customers,
activities
WHERE freeAnswers.surveyId = surveys.id
AND surveys.customerId = customers.id
AND activities.id = surveys.activityId
AND clients.id = freeAnswers.clientId
AND customers.id = 1
ORDER BY activityName asc
LIMIT 0, 10
the query is correct on my mysql server but when I try to use it in Zend Framework 1.11 model
I get this error: Mysqli prepare error: Operand should contain 1 column(s)
Please, could anyone help me to make it run well?
Best Regards,
Elaidon
Here is some code that should work. Zend_Db_Select doesn't really provide a way to select from multiple tables in the FROM clause without using a JOIN so this feels a bit hackish to me in regards to one small part of the query. Your best bet will probably be to rewrite the query using JOINs where appropriate.
$subselect = $db->select()
->from('geodatas', 'districtCode')
->where('zipCode = clients.zipCode')
->group('zipCode')
->limit(1, 0);
$from = $db->quoteIdentifier('freeAnswers') . ', ' .
$db->quoteIdentifier('clients') . ', ' .
$db->quoteIdentifier('surveys') . ', ' .
$db->quoteIdentifier('customers') . ', ' .
$db->quoteIdentifier('activities');
$select = $db->select()
->from(array('activities' => new Zend_Db_Expr($from)),
array('freeanswers.*',
'districtCode' =>
new Zend_Db_Expr('(' . $subselect . ')'),
'clients.zipCode', 'clients.gender', 'clients.startAge',
'clients.endAge', 'clients.mail', 'clients.facebook',
'clients.customerId', 'clients.activityId',
'surveyName' => 'surveys.name', 'customers.companyName',
'activityName' => 'activities.name'))
->where('freeAnswers.surveyId = surveys.id')
->where('surveys.customerId = customers.id')
->where('activities.id = surveys.activityId')
->where('clients.id = freeAnswers.clientId')
->where('customers.id = ?', 1)
->order('activityName ASC')
->limit(10, 0);
The only reason I say it is hackish is because of the line:
->from(array('activities' => new Zend_Db_Expr($from)),
Since from() really only works with one table, I create a Zend_Db_Expr and specify the correlation as the last table name in the expression. If you don't pass a Zend_Db_Expr, it will either quote your comma separated table name incorrectly, or if you pass an array of table names, it just uses the first. When you pass a Zend_Db_Expr with no name, it defaults to use AS t which also doesn't work in your case. That is why I put it as is.
That returns the exact SQL you provided except for the last thing mentioned. Here is actually what it returns:
SELECT
`freeanswers`.*,
(SELECT `geodatas`.`districtCode`
FROM `geodatas`
WHERE (zipCode = clients.zipCode)
GROUP BY `zipCode`
LIMIT 1) AS `districtCode`,
`clients`.`zipCode`,
`clients`.`gender`,
`clients`.`startAge`,
`clients`.`endAge`,
`clients`.`mail`,
`clients`.`facebook`,
`clients`.`customerId`,
`clients`.`activityId`,
`surveys`.`name` AS `surveyName`,
`customers`.`companyName`,
`activities`.`name` AS `activityName`
FROM `freeAnswers`,
`clients`,
`surveys`,
`customers`,
`activities` AS `activities`
WHERE (freeAnswers.surveyId = surveys.id)
AND (surveys.customerId = customers.id)
AND (activities.id = surveys.activityId)
AND (clients.id = freeAnswers.clientId)
AND (customers.id = 1)
ORDER BY `activityName` ASC
LIMIT 10
So that will work but eventually you will want to rewrite it using JOIN instead of specifying most of the WHERE clauses.
When dealing with subqueries and Zend_Db_Select, I find it easy to write each subquery as their own queries before writing the final query, and just insert the subqueries where they need to go and Zend_Db handles the rest.
Hope that helps.