Filter by compound fields in SQLAlchemy - sqlalchemy

How can I filter by multiple fields using SQLAlchemy's ORM?
Example:
SELECT p.* FROM product p
WHERE (p.barcode, p.partner) NOT IN (
SELECT barcode, partner
FROM unallowed_products
);

Lets define some aliases in order to reduce verbosity before looking at possible solutions:
P, U = Product, UnallowedProduct
version-1: OUTER JOIN + WHERE IS NULL
This is quite universal solution and should work with all backends (RDBMS)
q = (
session.query(P)
.outerjoin(U, and_(P.barcode == U.barcode, P.partner == U.partner))
.filter(U.id == None)
)
version-2: tuple comparison
This does not work for all backends, but should work for mySQL, postgresql
See tuple_ documentation
q = (
session.query(P)
.filter(~tuple_(P.barcode, P.partner).in_(
select([U.barcode, U.partner])
))
)
version-3: use multiple OR statements
See this answer.
There is no reason to use it as version-1 and version-2 are cleaner.
version-4: concatenation of barcode and partner into one column
See this question.
This is basically your own solution.
Again, there is not reason to use it as other versions are much cleaner, and do not require conversions to strings etc.

This query should solve the problem:
SELECT p.* FROM product p
WHERE (p.barcode || "_" || p.partner) NOT IN (
SELECT barcode || "_" || partner
FROM unallowed_products
);
which can be translated to SQLAlchemy ORM like this:
subquery = session.query(
cast(UnallowedProducts.barcode, type_=Text) +
'_' +
cast(UnallowedProducts.partner, type_=Text)
).subquery()
session.query(Product)\
.filter((
cast(Product.id, type_=Text) +
'_' +
cast(Product.partner, type_=Text)
).notin_(subquery))
This tricky solution should work for virtually every case.

Related

Double AND Where Clause with between Dates

I currently have a query like this
WHERE Category = 'Freezing' OR Category ='Ooredoo FTTH fault'OR Category ='Ooredoo TT fault ' OR Category ='No Picture'
but i want use this date search query with the above code,
im able to use any 1 of this code but im unable to use both query, i have used "AND" or "OR" But its Not working,
ScheduleDate BETWEEN '".$_POST["from_date"]."' AND '".$_POST["to_date"]."' ```
a or b or c and d is ambiguous. You need to use parenthesis to make it explicit how to group the ands and ors. If you want category to match ANY of the options AND the schedule is between certain dates. (a or b or c) and d.
where (
Category = 'Freezing' or
Category = 'Ooredoo FTTH fault' or
Category = 'Ooredoo TT fault ' or
Category = 'No Picture'
)
and ScheduleDate BETWEEN ? AND ?
(Note: Do not use string concatenation to put values into queries, especially not straight from user input. It's a security hole and can cause errors. Use bind parameters.)
(Note: You have an extra space in one of your options.)
You can also use in to make the query simpler.
where
Category in (
'Freezing', 'Ooredoo FTTH fault', 'Ooredoo TT fault', 'No Picture'
)
and ScheduleDate BETWEEN ? AND ?

JOIN on keys that don't have the same value

I am trying to do an INNER JOIN on two tables that have similar values, but not quite the same. One table has a fully qualified host name for its primary key, and the other the hosts short name, as well as the subdomain. It it safe to assume that the short name and the subdomain together are unique.
So I've tried:
SELECT table1.nisinfo.* FROM table1.nisinfo INNER JOIN table2.hosts ON (table1.nisinfo.shortname + '.' + table1.nisinfo.subdomainname + '.domain.com') = table2.hosts.fqhn WHERE table2.hosts.package = 'somepkg';
This doesn't return the results I expect, it returns the first result hundreds of times. I'd like to return distinct rows. It takes a long time to run as well.
What am I doing wrong? I was thinking of running a subquery to get the hostnames, but I don't know what the right path from here is.
Thank you!
You can use group by in your query so you can achieve the desired results you want
please see this two links
Group by with 2 distinct columns in SQL Server
http://www.sqlteam.com/article/how-to-use-group-by-with-distinct-aggregates-and-derived-tables
Try putting your results into a temp table and then view the table to make sure that the columns are as expected.
SELECT table1.nisinfo.*, table1.nisinfo.shortname + '.' + table1.nisinfo.subdomainname + '.domain.com' AS ColID
INTO #temp
FROM table1.nisinfo;
Select *
from #temp INNER JOIN table2.hosts ON ##temp.ColID = table2.hosts.fqhn
WHERE table2.hosts.package = 'somepkg'
;
Put a Group By clause at the end of the second statement
So in this case, I used a subquery to get the initial results, and then used a join.
SELECT table1.nisinfo.* FROM table1.nisinfo JOIN (SELECT distinct(fqhn) FROM table2.hosts WHERE package = 'bash') AS FQ ON ((SUBSTRING_INDEX(FQ.fqhn, '.', 1)) = table1.nisinfo.shortname);

Does mutliple functions in GROUP BY like concat or cast or dateformat affect the query time?

I have a query which is taking more time to execute . I am trying to optimize it. There are functions in the GROUP BY which may be adding to the query time . So i am trying to figure out how to optimize that part.
I am using Mysql 5.5.17 .
I cannot use force index as i am using Hibernate, so that's out of option .
I have some other options like :
1) use the alias in place of concat(...) in group by.
2) or replace concat() with the the columns inside it .
Ex: group by concat(1,2,3.) => group by 1,2,3
After trying both the above options i dont see any difference in "Query Cost"
The query looks like this(table names changed) :
select sum(this_.a + this_.b + this_.c + this_.d + this_.e +
this_.f + this_.g + this_.h + this_.i + this_.j) as sent,
sum(this_.a + this_.b + this_.c + this_.d + this_.e) as delivered,
d2_.name as y2_, d2_.id as y3_, concat(cast(b4_.id as char),
'.',c1_.name,'.',cs5_.name,'_',date_format(b4_.createddate,
'%Y-%b-%d %H:%i:%s')
) as allias_concat, b4_.id as y5_,
this_.year as y6_, this_.month as y7_, this_.day as y8_,
this_.weekday as y9_, this_.weekofmonth as y10_,
this_.bltid as y11_,
b4_.id as y12_, c1_.name as y13_,
cs5_.name as y14_, b4_.createddate as y15_,
cs5_.subject as y16_, d2_.name as y17_
from TABLE1 this_
inner join TABLE2 c1_ on this_.cgnid=c1_.id
inner join TABLE3 b4_ on this_.bltid=b4_.id
inner join TABLE4 csc6_ on b4_.schdlid=csc6_.id
inner join TABLE5 cs5_ on this_.constid=cs5_.id
left outer join TABLE6 f3_ on this_.fid=f3_.id
inner join TABLE7 d2_ on this_.dptid=d2_.id
where (f3_.name<>'TRASH'
or c1_.fid=0
)
and c1_.status<>'K'
and d2_.id in (1)
and ((b4_.createddate between '2015-02-01 10:30:00'
AND '2015-05-13 10:29:59'
and csc6_.isrealtime = 'N'
)
or (csc6_.isrealtime = 'Y'
and this_.bltdate between '2015-02-01 10:30:00'
AND '2015-05-13 10:29:59')
)
group by d2_.id,
concat(cast(b4_.id as char),'.',c1_.name,'.',cs5_.name,'_',
date_format(b4_.createddate,'%Y-%b-%d %H:%i:%s')
),
b4_.id,
this_.year, this_.month, this_.day,
this_.bltid
limit 20001
Please suggest ...
Thanks In Advance ...
Sorry, but this is going to be an "anti-answer"...
It smells like "over-normalization". For example, it feels like b4_ is a normalization of a datetime (createddate) and nothing else. Is this correct? If so, that is adding complexity and slowing down both the WHERE and the GROUP BY, plus obliterating some optimization possibilities.
The "functions" are not the problem. Since the items in the GROUP BY are coming from different tables, there is no way to optimize it. The GROUP BY must gather the information from the various tables, put them in a temp table, sort it, then scan through it. (Or something like that. The point is that there is no shortcut.)
"I cannot use ... as i am using Hibernate" -- Yet another case where a 3rd party software "gets in the way". If you can't change the query, we can't help you.

Alternatives to Multiple Case Statements for Same Column

To match up some new schema to old, I'm having to do some ugly contortions that I figure could be done in a better way. For reference, I asked another question about this match-up process here: Creating View from Related Child Tables
I've placed a simplified example in SQLFiddle but the gist of it is, that the only way I can see reconciling these two different schemas is to do two case statements on the same value, something like this:
SELECT
CASE
WHEN n.FooBarStatusId = 1 OR n.FooBarStatusId = 2
THEN 1
ELSE 0
END as [IsFoo],
CASE
WHEN n.FooBarStatusId = 2
THEN 1
ELSE 0
END as [IsBar]
from Parent p
left join OldStuff o on p.ParentId = o.ParentId
left join NewStuff n on p.ParentId = n.ParentId
Is there a better and/or more efficient way of accomplishing the same thing? These case statements could be hit hundreds of times in a given query and I'm concerned about this specific logic.
I've thought about extracting this specific logic out (it is part of a larger query to build a view) into a temp table or perhaps even a table-valued function, but even still I can't come up with a way around using multiple case statements.
Just corrected grammar...
I found another solution:
select
p.Name,
ISNULL(o.IsFoo, CONVERT(BIT, n.FooBarStatusId)) as [IsFoo],
ISNULL(o.IsBar, CONVERT(BIT, n.FooBarStatusId * (n.FooBarStatusId - 1))) as [IsBar],
from Parent p
left join OldStuff o on p.ParentId = o.ParentId
left join NewStuff n on p.ParentId = n.ParentId
The only one arithmetic solution could be slow:
select
p.Name,
ISNULL(o.IsFoo, CAST( (n.FooBarStatusId % 0.35) * 4 AS int)) AS [IsFoo],
ISNULL(o.IsBar, n.FooBarStatusId/2) [IsBar]
from Parent p
left join OldStuff o on p.ParentId = o.ParentId
left join NewStuff n on p.ParentId = n.ParentId
Personally, I do not like to use the division, because of could be involved floating point operations, that way, it would be very slow.
As you have two columns, you will need two expressions... but they might not have to be CASE expressions. Reading your question, I get the impressions that the only possible values in the column are 0,1,2, and that this is an int type? If that's correct, you can use arithmetic rather than boolean logic to get what you need. Try this:
CAST( (n.FooBarStatusId % .35) * 4 AS int) AS [IsFoo],
n.FooBarStatusId/2 [IsBar]

Hierarchical SQL Query

I'm working on a simple CMS system for which I have a database with the following tables:
Items
Contents
Langs
The Items table has the following structure:
itemId
name (for semantic reasons)
type
parent (foreign key to itemId)
An item can be either a document or a section type. A section is a piece of content on a document which is linked to it via the parent collumn. But also a document can have a parent which makes it a subpage.
Now I get stuck on making a query to fetch all the items from the database hierarchically. So something like this:
documentId => name
metaDescription => language => meta
sections => sectionId => language => title
content
uri
subPages => documentId => name
metaDescription
sections => etc...
Just to clarify, a website can have multiple languages which are in the Langs table and every language is linked to a piece of content in the Contents table which is also linked to an item in the Items table. The metaDescription is the linked content collumn linked to a item of type document.
Is there a way to do this with one query? This was my first attempt, but it doesnt work for subPages:
SELECT
documents.itemId AS id,
documents.name AS documentName,
documents.lastModified AS lastModified,
meta.content AS metaDescription,
meta.uri AS documentUri,
sections.itemId AS sectionId,
sections.name AS sectionName,
sections.lastModified AS sectionLastModified,
contents.name AS sectionTitle,
contents.content AS sectionContent,
contents.uri AS contentUri,
contents.lastModified AS contentLastModified,
langs.name AS contentLang
FROM
SITENAME_kw_items AS documents
INNER JOIN
SITENAME_kw_contents AS meta
ON documents.itemId = meta.itemId
INNER JOIN
SITENAME_kw_items AS sections
ON sections.parent = documents.itemId
INNER JOIN
SITENAME_kw_contents AS contents
ON sections.itemId = contents.itemId
INNER JOIN
SITENAME_kw_langs AS langs
ON langs.langId = contents.langId
Sorry for the long question. Hope you guys can help!
Below is how I do it in "our" DMS (recursive CTE), which is Adam Gent's suggestion expanded.
Note that I just see one could use COALESCE instead of nesting ISNULL.
The order by you would do according to the breadcrumbs (here Bez_Path or UID_Path).
A far better way would be to use a closure-table architecture.
See here:
http://dirtsimple.org/2010/11/simplest-way-to-do-tree-based-queries.html
and here:
http://www.mysqlperformanceblog.com/2011/02/14/moving-subtrees-in-closure-table/
The closure table also has the advantage that it works on MySQL, where CTE & recursion are not supported.
Also note that closure tables are much better (and simpler and faster to query) than recursion.
Also think about symlinks in such a structure.
The something_UID, something_parent_UID pattern (as shown below) is almost always an antipattern.
CREATE VIEW [dbo].[V_DMS_Navigation_Structure]
AS
SELECT
NAV_UID
,NAV_Typ
,NAV_Parent_UID
,NAV_Stufe
,NAV_ApertureKey
,NAV_Nr
--,NAV_Bemerkung
,NAV_Status
,NAV_Referenz
,ISNULL(PJ_Bezeichnung, ISNULL(FO_Bezeichnung, DOC_Bezeichnung + '.' + DOC_Dateiendung) ) AS NAV_Bezeichnung
,NAV_PJ_UID
,NAV_FO_UID
,NAV_DOC_UID
,ISNULL(NAV_PJ_UID, ISNULL(NAV_FO_UID,NAV_DOC_UID)) AS NAV_OBJ_UID
FROM T_DMS_Navigation
LEFT JOIN T_DMS_Projekt
ON T_DMS_Projekt.PJ_UID = T_DMS_Navigation.NAV_PJ_UID
LEFT JOIN T_DMS_Folder
ON T_DMS_Folder.FO_UID = T_DMS_Navigation.NAV_FO_UID
LEFT JOIN T_DMS_Dokument
ON T_DMS_Dokument.DOC_UID = T_DMS_Navigation.NAV_DOC_UID
CREATE VIEW [dbo].[V_DMS_Navigation_Structure_Path]
AS
WITH Tree
(
NAV_UID
,NAV_Bezeichnung
,NAV_Parent_UID
,Depth
,Sort
,Bez_Path
,UID_Path
,PJ_UID
,FO_UID
,DOC_UID
,OBJ_UID
)
AS
(
SELECT
NAV_UID
,NAV_Bezeichnung
,NAV_Parent_UID
,0 AS Depth
,CAST('0' AS varchar(10)) AS Sort
,CAST(NAV_Bezeichnung AS varchar(4000)) AS Bez_Path
,CAST(NAV_OBJ_UID AS varchar(4000)) AS UID_Path
,NAV_PJ_UID AS PJ_UID
,NAV_FO_UID AS FO_UID
,NAV_DOC_UID AS DOC_UID
,NAV_OBJ_UID AS OBJ_UID
FROM V_DMS_Navigation_Structure
WHERE NAV_Parent_UID IS NULL
UNION ALL
SELECT
CT.NAV_UID
,CT.NAV_Bezeichnung
,CT.NAV_Parent_UID
,Parent.Depth + 1 AS Depth
,CONVERT(varchar(10), Parent.Sort + '.' + CAST(Parent.Depth + 1 AS varchar(10))) AS Sort
,CONVERT(varchar(4000), Parent.Bez_Path + '\' + CAST(CT.NAV_Bezeichnung AS varchar(1000))) AS Bez_Path
,CONVERT(varchar(4000), Parent.UID_Path + '\' + CAST(CT.NAV_OBJ_UID AS varchar(1000))) AS UID_Path
,NAV_PJ_UID AS PJ_UID
,NAV_FO_UID AS FO_UID
,NAV_DOC_UID AS DOC_UID
,NAV_OBJ_UID AS OBJ_UID
FROM V_DMS_Navigation_Structure CT
INNER JOIN Tree AS Parent
ON Parent.NAV_UID = CT.NAV_Parent_UID
)
SELECT TOP 999999999999999 * FROM Tree
ORDER BY Depth
The short answer is that you can't really do this with RDBMS. The long answer is you can sort of do it either programmatically (N+1 select) or you can use common table expressions (CTE).
The other option is to cheat and use a depth column as a hint for an order by.