i'm having a little issue with doctrine using symfony 1.4 (I think it's using doctrine 1.2). I have 2 queries, using raw sql in the mysql console, they produce the same resultset. The queries can be generated using this code :
$dates = Doctrine::getTable('Picture')
->createQuery('a')
->select('substr(a.created_at,1,10) as date')
->leftjoin('a.PictureTag pt ON a.id = pt.picture_id')
->leftjoin('pt.Tag t ON t.id = pt.tag_id')
->where('a.created_at <= ?', date('Y-m-d 23:59:59'))
->orderBy('date DESC')
->groupby('date')
->limit(ITEMS_PER_PAGE)
->offset(ITEMS_PER_PAGE * $this->page)
->execute();
If I remove the two joins, it changes the query, but the resultset it's the same.
But using doctrine execute(), one produces only one row.
Somebody have an idea on what's going on here?
PS : Picture table has id, title, file, created_at (format 'Y-m-d h:i:s'), the Tag table is id, name and PictureTag is an relationship table with id and the two foreign keys.
PS 2 : Here are the two sql queries produced (the first without joins)
SELECT substr(l.created_at, 1, 10) AS l__0 FROM lupa_picture l WHERE (l.created_at <= '2010-03-19 23:59:59') GROUP BY l__0 ORDER BY l__0 DESC LIMIT 4
SELECT substr(l.created_at, 1, 10) AS l__0 FROM lupa_picture l LEFT JOIN lupa_picture_tag l2 ON (l.id = l2.picture_id) LEFT JOIN lupa_tag l3 ON (l3.id = l2.tag_id) WHERE (l.created_at <= '2010-03-19 23:59:59') GROUP BY l__0 ORDER BY l__0 DESC LIMIT 4
I had something similar this week. Doctrine's generated SQL (from the Symfony debug toolbar) worked fine in phpMyAdmin, but failed when running the query as in your question. Try adding in the following into your query:
->setHydrationMode(Doctrine::HYDRATE_SCALAR)
and see if it gives you the expected result. If so, it's down to the Doctrine_Collection using the Picture primary key as the index in the collection. If you have more than 1 result with the same index, Doctrine will refuse to add it into the collection, so you only end up with 1 result. I ended up running the query using a different table rather than the one I wanted, which resulted in a unique primary key and then the results I wanted appeared.
Well, the solution was that...besides substr(), it needs another column of the table. Using select(substr(), a.created_at) made it work
Related
I want to write a query with two levels of group by in Flask-SQLAlchemy which is equivalent to the following SQL code
select right_team_id team_id
,sum(score)-sum(deductions) score from (
select left_team_id, right_team_id
,1.0*sum(case when right_win then 1 else 0 end)/count(*) score
,1.0*sum(right_deductions)/2 deductions
from races
group by left_team_id, right_team_id ) A
group by right_team_id
I started with the following for the first group by
query = Races.query.group_by(Races.left_team_id, Races.right_team_id)
.add_columns(func.sum(Races.left_deductions).label('deductions')
,func.sum(case([(Races.left_win, 1)], else_ = 0)).label('wins')
,func.count().label('races'))
But each record in query is the following (<flaskapp.races.models.Races object at 0x107c5f358>, 0, 0, 1). How can I run another group by query, including on the aggregate columns at the end?
Thanks
Not having the db, or what the underlying schema is, this is difficult to verify. Plus the original sql query and sqlalchemy query are rather different. But I think approaching this using subqueries will work well.
subq = db.session.query(Races.right_team_id.label('right_team_id'),
(func.sum(case([(Races.right_win, 1)], else_ = 0))/func.count(Races.id)).label('score'),
(func.sum(right_deductions)/2).label('deductions')).group_by(Races.left_team_id, Races.right_team_id).subquery()
q = db.session.query(Races.right_team_id, func.sum(subq.c.score)-func.sum(subq.c.deductions)).\
join(subq, subq.c.right_team_id==Races.right_team_id).group_by(Races.right_team_id)
I am needing some SQL help. I have a SELECT statement that references several tables and is hanging up in the MySQL database. I would like to know if there is a better way to write this statement so that it runs efficiently and does not hang up the DB? Any help/direction would be appreciated. Thanks.
Here is the code:
Select Max(b.BurID) As BurID
From My.AppTable a,
My.AddressTable c,
My.BurTable b
Where a.AppID = c.AppID
And c.AppID = b.AppID
And (a.Forename = 'Bugs'
And a.Surname = 'Bunny'
And a.DOB = '1936-01-16'
And c.PostcodeAnywhereBuildingNumber = '999'
And c.PostcodeAnywherePostcode = 'SK99 9Q9'
And c.isPrimary = 1
And b.ErrorInd <> 1
And DateDiff(CurDate(), a.ApplicationDate) <= 30)
There is NO mysql error in the log. Sorry.
Pro tip: use explicit JOINs rather than a comma-separated list of tables. It's easier to see the logic you're using to JOIN that way. Rewriting your query to do that gives us this.
select Max(b.BurID) As BurID
From My.AppTable AS a
JOIN My.AddressTable AS c ON a.AppID = c.AppID
JOIN My.BurTable AS b ON c.AppID = b.AppID
WHERE (a.Forename = 'Bugs'
And a.Surname = 'Bunny'
And a.DOB = '1936-01-16'
And c.PostcodeAnywhereBuildingNumber = '999'
And c.PostcodeAnywherePostcode = 'SK99 9Q9'
And c.isPrimary = 1
And b.ErrorInd <> 1
And DateDiff(CurDate(), a.ApplicationDate) <= 30)
Next pro tip: Don't use functions (like DateDiff()) in WHERE clauses, because they defeat using indexes to search. That means you should change the last line of your query to
AND a.ApplicationDate >= CurDate() - INTERVAL 30 DAY
This has the same logic as in your query, but it leaves a naked (and therefore index-searchable) column name in the search expression.
Next, we need to look at your columns to see how you are searching, and cook up appropriate indexes.
Let's start with AppTable. You're screening by specific values of Forename, Surname, and DOB. You're screening by a range of ApplicationDate values. Finally you need AppID to manage your join. So, this compound index should help. Its columns are in the correct order to use a range scan to satisfy your query, and contains the needed results.
CREATE INDEX search1 USING BTREE
ON AppTable
(Forename, Surname, DOB, ApplicationDate, AppID)
Next, we can look at your AddressTable. Similar logic applies. You'll enter this table via the JOINed AppID, and then screen by specific values of three columns. So, try this index
CREATE INDEX search2 USING BTREE
ON AddressTable
(AppID, PostcodeAnywherePostcode, PostcodeAnywhereBuildingNumber, isPrimary)
Finally, we're on to your BurTable. Use similar logic as the other two, and try this index.
CREATE INDEX search3 USING BTREE
ON BurTable
(AppID, ErrorInd, BurID)
This kind of index is called a compound covering index, and can vastly speed up the sort of summary query you have asked about.
i have two tables i am trying to get information from.
login table - which has the list of employees
projects table - which has the projects
in short, i am trying to write a query that will select the copywriters and perform a subquery on each that will return a field dubbed 'open_projects'. This, i can get to work with the below sql:
select web_login_id,
(select count(project_web_id) from project
where copywriter = web_login_id
and (`status` = 'open' or `status` = 'qual')) as open_projects from login
where roles like '%copywriter%'
and tierLevel like '%c1%'
order by open_projects asc
This returns something like:
1982983 3
1982690 22
2987398 5
The problem with this is that sometimes 5 or 6 of the projects will belong to the same client and are not actually being worked on as they are dealt with in a queue-ish fashion.
My question is how to modify the above sql so that the subquery will GROUP subset based on the client_login_id field.
This sql gives me an error of : subquery returns more than 1 row
select web_login_id,
(select count(project_web_id) from project
where copywriter = web_login_id
and (`status` = 'open' or `status` = 'qual') group by client_login_id) as open_projects from login
where roles like '%copywriter%'
and tierLevel like '%c1%'
order by open_projects asc
You need to rephrase this as an explicit join. I think the following does the trick:
select web_login_id, cw.open_projects
from login l left outer join
(select copywriter, count(project_web_id) as open_projects
from project
where `status` in ('open', 'qual')
group by copywriter
) cw
on l.web_login_id = cw.copywriter
where l.roles like '%copywriter%' and l.tierLevel like '%c1%'
order by open_projects asc
I'm not sure what the "group by client_login_id" is doing. It doesn't seem necessary.
Once you've done this, you can return as many columns as you like from the subquery.
I'm using this kind of queries with different parameters :
EXPLAIN SELECT SQL_NO_CACHE `ilan_genel`.`id` , `ilan_genel`.`durum` , `ilan_genel`.`kategori` , `ilan_genel`.`tip` , `ilan_genel`.`ozellik` , `ilan_genel`.`m2` , `ilan_genel`.`fiyat` , `ilan_genel`.`baslik` , `ilan_genel`.`ilce` , `ilan_genel`.`parabirimi` , `ilan_genel`.`tarih` , `kgsim_mahalleler`.`isim` AS mahalle, `kgsim_ilceler`.`isim` AS ilce, (
SELECT `ilanresimler`.`resimlink`
FROM `ilanresimler`
WHERE `ilanresimler`.`ilanid` = `ilan_genel`.`id`
LIMIT 1
) AS resim
FROM (
`ilan_genel`
)
LEFT JOIN `kgsim_ilceler` ON `kgsim_ilceler`.`id` = `ilan_genel`.`ilce`
LEFT JOIN `kgsim_mahalleler` ON `kgsim_mahalleler`.`id` = `ilan_genel`.`mahalle`
WHERE `ilan_genel`.`ilce` = '703'
AND `ilan_genel`.`durum` = '1'
AND `ilan_genel`.`kategori` = '1'
AND `ilan_genel`.`tip` = '9'
ORDER BY `ilan_genel`.`id` DESC
LIMIT 225 , 15
and this is what i get in explain section:
these are the indexes that i already tried to use:
any help will be deeply appreciated what kind of index will be the best option or should i use another table structure ?
You should first simplify your query to understand your problem better. As it appears your problem is constrained to the ilan_gen1 table, the following query would also show you the same symptoms.:
SELECT * from ilan_gene1 WHERE `ilan_genel`.`ilce` = '703'
AND `ilan_genel`.`durum` = '1'
AND `ilan_genel`.`kategori` = '1'
AND `ilan_genel`.`tip` = '9'
So the first thing to do is check that this is the case. If so, the simpler question is simply why does this query require a file sort on 3661 rows. Now the 'hepsi' index sort order is:
ilce->mahelle->durum->kategori->tip->ozelik
I've written it that way to emphasise that it is first sorted on 'ilce', then 'mahelle', then 'durum', etc. Note that your query does not specify the 'mahelle' value. So the best the index can do is lookup on 'ilce'. Now I don't know the heuristics of your data, but the next logical step in debugging this would be:
SELECT * from ilan_gene1 WHERE `ilan_genel`.`ilce` = '703'`
Does this return 3661 rows?
If so, you should be able to see what is happening. The database is using the hepsi index, to the best of it's ability, getting 3661 rows back then sorting those rows in order to eliminate values according to the other criteria (i.e. 'durum', 'kategori', 'tip').
The key point here is that if data is sorted by A, B, C in that order and B is not specified, then the best logical thing that can be done is: first a look up on A then a filter on the remaining values against C. In this case, that filter is performed via a file sort.
Possible solutions
Supply 'mahelle' (B) in your query.
Add a new index on 'ilan_gene1' that doesn't require 'mahelle', i.e. A->C->D...
Another tip
In case I have misdiagnosed your problem (easy to do when I don't have your system to test against), the important thing here is the approach to solving the problem. In particular, how to break a complicated query into a simpler query that produces the same behaviour, until you get to a very simple SELECT statement that demonstrates the problem. At this point, the answer is usually much clearer.
MySQL Server Version: Server version: 4.1.14
MySQL client version: 3.23.49
Tables under discussion: ads_list and ads_cate.
Table Relationship: ads_cate has many ads_list.
Keyed by: ads_cate.id = ads_list.Category.
I am not sure what is going on here, but I am trying to use COUNT() in a simple agreggate query, and I get blank output.
Here is a simple example, this returns expected results:
$queryCats = "SELECT id, cateName FROM ads_cate ORDER BY cateName";
But if I modify it to add the COUNT() and the other query data I get no array return w/ print_r() (no results)?
$queryCats = "SELECT ads_cate.cateName, ads_list.COUNT(ads_cate.id),
FROM ads_cate INNER JOIN ads_list
ON ads_cate.id = ads_list.category
GROUP BY cateName ORDER BY cateName";
Ultimately, I am trying to get a count of ad_list items in each category.
Is there a MySQL version conflict on what I am trying to do here?
NOTE: I spent some time breaking this down, item by item and the COUNT() seems to cause the array() to disappear. And the the JOIN seemed to do the same thing... It does not help I am developing this on a Yahoo server with no access to the php or mysql error settings.
I think your COUNT syntax is wrong. It should be:
COUNT(ads_cate.id)
or
COUNT(ads_list.id)
depending on what you are counting.
Count is an aggregate. means ever return result set at least one
here you be try count ads_list.id not null but that wrong. how say Myke Count(ads_cate.id) or Count(ads_list.id) is better approach
you have inner join ads_cate.id = ads_list.category so Count(ads_cate.id) or COUNT(ads_list.id) is not necessary just count(*)
now if you dont want null add having
only match
SELECT ads_cate.cateName, COUNT(*),
FROM ads_cate INNER JOIN ads_list
ON ads_cate.id = ads_list.category
GROUP BY cateName
having not count(*) is null
ORDER BY cateName
all
SELECT ads_cate.cateName, IFNULL(COUNT(*),0),
FROM ads_cate LEFT JOIN ads_list
ON ads_cate.id = ads_list.category
GROUP BY cateName
ORDER BY cateName
Did you try:
$queryCats = "SELECT ads_cate.cateName, COUNT(ads_cate.id)
FROM ads_cate
JOIN ads_list ON ads_cate.id = ads_list.category
GROUP BY ads_cate.cateName";
I am guessing that you need the category to be in the list, in that case the query here should work. Try it without the ORDER BY first.
You were probably getting errors. Check your server logs.
Also, see what happens when you try this:
SELECT COUNT(*), category
FROM ads_list
GROUP BY category
Your array is empty or disappear because your query has errors:
there should be no comma before the FROM
the "ads_list." prefix before COUNT is incorrect
Please try running that query directly in MySQL and you'll see the errors. Or try echoing the output using mysql_error().
Now, some other points related to your query:
there is no need to do ORDER BY because GROUP BY by default sorts on the grouped column
you are doing a count on the wrong column that will always give you 1
Perhaps you are trying to retrieve the count of ads_list per ads_cate? This might be your query then:
SELECT `ads_cate`.`cateName`, COUNT(`ads_list`.`category`) `cnt_ads_list`
FROM `ads_cate`
INNER JOIN `ads_list` ON `ads_cate`.`id` = `ads_list`.`category`
GROUP BY `cateName`;
Hope it helps?