This query will be done in a cached autocomplete text box, possibly by thousands of users at the same time. What I have below works, bit I feel there may be a better way to do what I am doing.
Any advice?
UPDATED -- it can be 'something%':
SELECT a.`object_id`, a.`type`,
IF( b.`name` IS NOT NULL, b.`name`,
IF( c.`name` IS NOT NULL, c.`name`,
IF( d.`name` IS NOT NULL, d.`name`,
IF ( e.`name` IS NOT NULL, e.`name`, f.`name` )
)
)
) AS name
FROM `user_permissions` AS a
LEFT JOIN `divisions` AS b
ON ( a.`object_id` = b.`division_id`
AND a.`type` = 'division'
AND b.`status` = 1 )
LEFT JOIN `departments` AS c
ON ( a.`object_id` = c.`department_id`
AND a.`type` = 'department'
AND c.`status` = 1 )
LEFT JOIN `sections` AS d
ON ( a.`object_id` = d.`section_id`
AND a.`type` = 'section'
AND d.`status` = 1 )
LEFT JOIN `units` AS e
ON ( a.`object_id` = e.`unit_id`
AND a.`type` = 'unit'
AND e.`status` = 1 )
LEFT JOIN `positions` AS f
ON ( a.`object_id` = f.`position_id`
AND a.`type` = 'position'
AND f.`status` = 1 )
WHERE a.`user_id` = 1 AND (
b.`name` LIKE '?%' OR
c.`name` LIKE '?%' OR
d.`name` LIKE '?%' OR
e.`name` LIKE '?%' OR
f.`name` LIKE '?%'
)
Two simple, fast queries is often better than one huge, inefficient query.
Here's how I'd design it:
First, create a table for all your names, in MyISAM format with a FULLTEXT index. That's where your names are stored. Each of the respective object type (e.g. departments, divisions, etc.) are dependent tables whose primary key reference the primary key of the main named objects table.
Now you can search for names with this much simpler query, which runs blazingly fast:
SELECT a.`object_id`, a.`type`, n.name, n.object_type
FROM `user_permissions` AS a
JOIN `named_objects` AS n ON a.`object_id = n.`object_id`
WHERE MATCH(n.name) AGAINST ('name-to-be-searched')
Using the fulltext index will run hundreds of times faster than using LIKE in the way you're doing.
Once you have the object id and type, if you want any other attributes of the respective object type you can do a second SQL query joining to the table for the appropriate object type:
SELECT ... FROM {$object_type} WHERE object_id = ?
This will also go very fast.
Re your comment: Yes, I'd create the table with names even if it's redundant.
Other than changing the nested Ifs to use a Coalesce() function (MySql has Coalesce() doesn't it)? There is not much you can do as long as you need to filter on that input parameter with a like expresion. Putting a filter on a column using a Like expression, where the Like parameter has a wildcard at the begining, as you do, makes the query argument non-SARG-able, which means that the query processor must do a complete table scan of all the rows in the table to evaluate the filter predicate.
It cannot use an index, because an index is based on the column values, and with your Like parameter, it doesn't know which index entries to read from (since the parameter starts with a wild card)
if MySql has Coalesce, you can replace your Select with:
SELECT a.`object_id`, a.`type`,
Coalesce(n.name, c.name, d.Name, e.Name) name
If you can replace the search argument parameter so that it does not start with a wildcard, then just ensure that there is an index on the name column in each of the tables, and (if there are not indices on that column now), the query performance will increase enormously.
There are 500 things you can do. Optimize once you know where your bottlenecks are. Until then, work on getting those users onto your app. Its a much higher priority.
Related
I've studied and tried days worth of SQL queries to find "something" that will work. I have a table, apj32_facileforms_subrecords, that uses 7 columns. All the data I want to display is in 1 column - "value". The "record" displays the number of the entry. The "title" is what I would like to appear in the header row, but that's not as important as "value" to display in 1 row based upon "record" number.
I've tried a lot of CONCAT and various Pivot queries, but nothing seems to do more than "get close" to what I'd like as the end result.
Here's a screen shot of the table:
The output "should" be linear, so that 1 row contains 9 columns:
Project; Zipcode; First Name; Last Name; Address; City; Phone; E-mail; Trade (in that order). And the values in the 9 columns come from "value" as they relate to the "record" number.
I know there are LOT of examples that are similar, but nothing I've found covers taking all the values from "value" and CONCAT to 1 row.
This works to get all the data I want - SELECT record,value FROM apj32_facileforms_subrecords WHERE (record IN (record,value)) ORDER BY record
But the values are still in multiple rows. I can play with that query to get just the values, but I'm still at a loss to get them into 1 row. I'll keep playing with that query to see if I can figure it out before one of the experts here shows me how simple it is to do that.
Any help would be appreciated.
Using SQL to flatten an EAV model representation into a relational representation can be somewhat convoluted, and not very efficient.
Two commonly used approaches are conditional aggregation and correlated subqueries in the SELECT list. Both approaches call out for careful indexing for suitable performance with large sets.
correlated subqueries example
Here's an example of the correlated subquery approach, to get one value of the "zipcode" attribute for some records
SELECT r.id
, ( SELECT v1.value
FROM `apj32_facileforms_subrecords` v1
WHERE v1.record = r.id
AND v1.name = 'zipcode'
ORDER BY v1.value LIMIT 0,1
) AS `Zipcode`
FROM ( SELECT 1 AS id ) r
Extending that, we repeat the correlated subquery, changing the attribute identifier ('firstname' in place of 'zipcode'. looks like we we could also reference it by element, e.g. v2.element = 2
SELECT r.id
, ( SELECT v1.value
FROM `apj32_facileforms_subrecords` v1
WHERE v1.record = r.id
AND v1.name = 'zipcode'
ORDER BY v1.value LIMIT 0,1
) AS `Zipcode`
, ( SELECT v2.value
FROM `apj32_facileforms_subrecords` v2
WHERE v2.record = r.id
AND v2.name = 'firstname'
ORDER BY v2.value LIMIT 0,1
) AS `First Name`
, ( SELECT v3.value
FROM `apj32_facileforms_subrecords` v3
WHERE v3.record = r.id
AND v3.name = 'lastname'
ORDER BY v3.value LIMIT 0,1
) AS `Last Name`
FROM ( SELECT 1 AS id UNION ALL SELECT 2 ) r
returns something like
id Zipcode First Name Last Name
-- ------- ---------- ---------
1 98228 David Bacon
2 98228 David Bacon
conditional aggregation approach example
We can use GROUP BY to collapse multiple rows into one row per entity, and use conditional tests in expressions to "pick out" attribute values with aggregate functions.
SELECT r.id
, MIN(IF(v.name = 'zipcode' ,v.value,NULL)) AS `Zip Code`
, MIN(IF(v.name = 'firstname' ,v.value,NULL)) AS `First Name`
, MIN(IF(v.name = 'lastname' ,v.value,NULL)) AS `Last Name`
FROM ( SELECT 1 AS id UNION ALL SELECT 2 ) r
LEFT
JOIN `apj32_facileforms_subrecords` v
ON v.record = r.id
GROUP
BY r.id
For more portable syntax, we can replace MySQL IF() function with more ANSI standard CASE expression, e.g.
, MIN(CASE v.name WHEN 'zipcode' THEN v.value END) AS `Zip Code`
Note that MySQL does not support SQL Server PIVOT syntax, or Oracle MODEL syntax, or Postgres CROSSTAB or FILTER syntax.
To extend either of these approaches to be dynamic, to return a resultset with a variable number of columns, and variety of column names ... that is not possible in the context of a single SQL statement. We could separately execute SQL statements to retrieve information, that would allow us to dynamically construct a SQL statement of a form show above, with an explicit set of columns to be returned.
The approaches outline above return a more traditional relational model, (individual columns each with a value).
non-relational munge of attributes and values into a single string
If we have some special delimiters, we could munge together a representation of the data using GROUP_CONCAT function
As a rudimentary example:
SELECT r.id
, GROUP_CONCAT(v.title,'=',v.value ORDER BY v.name) AS vals
FROM ( SELECT 1 AS id ) r
LEFT
JOIN `apj32_facileforms_subrecords` v
ON v.record = r.id
AND v.name in ('zipcode','firstname','lastname')
GROUP
BY r.id
To return two columns, something like
id vals
-- ---------------------------------------------------
1 First Name=David,Last Name=Bacon,Zip Code=98228
We need to be aware that the return from GROUP_CONCAT is limited to group_concat_max_len bytes. And here we have just squeezed the balloon, moving the problem to some later processing, to parse the resulting string. If we have any equal signs or commas that appear in the values, it's going to make a mess of parsing the result string. So we will have to properly escape any delimiters that appear in the data, so that GROUP_CONCAT expression is going to get more involved.
I've database in which I'm storing japanese dictionary: words, readings, tags, types, meanings in other languages (english is the most important here, but there's also a few other) and so on.
Now, I want to create an interface using Datatables js plugin, so user could see table and use some filtering options (like, show only verbs, or find entries containing "dog"). I'm struggling, however, with query which can be pretty slow when using filtering... I already speed it up a lot, but it still not good.
This is my basic query:
select
v.id,
(
select group_concat(distinct vke.kanji_element separator '; ') from vocabulary_kanji_element as vke
where vke.vocabulary_id = v.id
) kanji_notation,
(
select group_concat(distinct vre.reading_element separator '; ') from vocabulary_reading_element as vre
where vre.vocabulary_id = v.id
) reading_notation,
(
select group_concat(distinct vsg.gloss separator '; ') from vocabulary_sense_gloss as vsg
join vocabulary_sense as vs on vsg.sense_id = vs.id
join language as l on vsg.language_id = l.id and l.language_code = 'eng'
where vs.vocabulary_id = v.id
) meanings,
(
select group_concat(distinct pos.name_code separator '; ') from vocabulary_sense as vs
join vocabulary_sense_has_pos as vshp on vshp.sense_id = vs.id
join part_of_speech as pos on pos.id = vshp.pos_id
where vs.vocabulary_id = v.id
) pos
from vocabulary as v
join vocabulary_sense as vs on vs.vocabulary_id = v.id
join vocabulary_sense_gloss as vsg on vsg.sense_id = vs.id
join vocabulary_kanji_element as vke on vke.vocabulary_id = v.id
join vocabulary_reading_element as vre on vre.vocabulary_id = v.id
join language as l on l.id = vsg.language_id and l.language_code = 'eng'
join vocabulary_sense_has_pos as vshp on vshp.sense_id = vs.id
join part_of_speech as pos on pos.id = vshp.pos_id
where
-- pos.name_code = 'n' and
(vsg.gloss like '%eat%' OR vke.kanji_element like '%eat%' OR vre.reading_element like '%eat%')
group by v.id
order by v.id desc
-- limit 3900, 25
Output is something like this:
|id | kanji_notation | reading_notation | meanings | pos |
---------------------------------------------------------------
|117312| お手; 御手 | おて | hand; arm |n; int|
Right now (working on my local machine), If there's no WHERE statement, but with limit, it works fast - about 0,140 sec. But when text filtering is on, execution time wents up to 6,5 sec, and often above. With filtering on part_of_speech first, its like 5,5 sec. 3 sec would be ok, but 6 is just way too long.
There's 1 155 897 records in table vocabulary_sense_gloss, so I think that's not a lot.
CREATE TABLE `vocabulary_sense_gloss` (
`id` MEDIUMINT(8) UNSIGNED NOT NULL AUTO_INCREMENT,
`sense_id` MEDIUMINT(8) UNSIGNED NOT NULL,
`gloss` VARCHAR(255) NOT NULL,
`language_id` MEDIUMINT(8) UNSIGNED NOT NULL,
PRIMARY KEY (`id`),
INDEX `vocabulary_sense_gloss_vocabulary_sense_id` (`sense_id`),
INDEX `vocabulary_sense_gloss_language_id` (`language_id`),
FULLTEXT INDEX `vocabulary_sense_gloss_gloss` (`gloss`),
CONSTRAINT `vocabulary_sense_gloss_language_id` FOREIGN KEY (`language_id`) REFERENCES `language` (`id`),
CONSTRAINT `vocabulary_sense_gloss_vocabulary_sense_id` FOREIGN KEY (`sense_id`) REFERENCES `vocabulary_sense` (`id`)
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
;
I wonder, is there some way to optimize it? Or maybe should I change my database? I was trying to use fulltext search, but it's not much faster, and seems to work only on full terms, so its no use. Similiar story with using 'eat%' instead of '%eat%': it won't return what I want.
I tried to divide vocabulary_sense_gloss in two tables - one with english only terms, and other with the rest. Since users would use usually english anyway, it would make things faster, but I'm not sure if that's a good approach.
Also, I was trying to change VARCHAR to CHAR. It seemed to speed up execution time, though table size went up a lot.
This WHERE clause has extremely poor performance.
(vsg.gloss like '%eat%' OR
vke.kanji_element like '%eat%' OR
vre.reading_element like '%eat%')
Why? First of all: column LIKE '%constant%' requires the query engine to examine every possible value of column. It can't possibly use an index because of the leading % in the constant search term.
Second: the OR clause means the query planner has to scan the results three different times.
What are you going to do to improve this? It won't be easy. You need to figure out how to use column LIKE 'constant%' search terms eliminating the leading % from the constants.
Once you do that, you may be able to beat the triple scan of your vast joined result set with a construct like this
...
WHERE v.id IN
(SELECT sense_id AS id
FROM vocabulary_sense_gloss
WHERE gloss LIKE 'eat%'
UNION
SELECT vocabulary_id AS id
FROM vocabulary_kanji_element
WHERE kanji_element LIKE 'eat%'
UNION
SELECT vocabulary_id AS id
FROM vocabulary_reading_element
WHERE reading_element LIKE 'eat%'
)
This will pull the id numbers of the relevant words directly, rather than from the result of a multiway JOIN. For this to be fast, your vocabulary_sense_gloss will need an index on (vocabulary_sense_gloss, sense_id). The other two tables will need similar indexes.
i have an SQL Requests:
SELECT DISTINCT id_tr
FROM planning_requests a
WHERE EXISTS(
SELECT 1 FROM planning_requests b
WHERE a.id_tr = b.id_tr
AND trainer IS NOT NULL
AND trainer != 'FREE'
)
AND EXISTS(
SELECT 1 FROM planning_requests c
WHERE a.id_tr = c.id_tr
AND trainer IS NULL
)
but this requests take 168.9490 sec to execute for returning 23162 rows of 2545088 rows
should i use LEFT JOIN or NOT IN ? and how can i rewrite it thx
You can speed this up by adding indexes. I would suggest: planning_requests(id_tr, trainer).
You can do this as:
create index planning_requests_id_trainer on planning_requests(id_tr, trainer);
Also, I think you are missing an = in the first subquery.
EDIT:
If you have a lot of duplicate values of id_tr, then in addition to the above indexes, it might make sense to phrase the query as:
select id_tr
from (select distinct id_tr
from planning_requests
) a
where . . .
The where conditions are being run on every row of the original table. The distinct is processed after the where.
I think your query can be simplified to this:
SELECT DISTINCT a.id_tr
FROM planning_requests a
JOIN planning_requests b
ON b.id_tr = a.id_tr
AND b.trainer IS NULL
WHERE a.trainer < 'FREE'
If you index planning_requests(trainer), then MySQL can utilize an index range to get all the rows that aren't FREE or NULL. All numeric strings will meet the < 'FREE' criteria, and it also won't return NULL values.
Then, use JOIN to make sure each record from that much smaller result set has a matching NULL record.
For the JOIN, index planning_requests(id_tr, trainer).
It might be simpler if you don't mix types in a column like FREE and 1.
I have a table
form (
int id )
webformsystemflags ( int id, int formid int sysflagid )
sysflag ( int id, name char(10) )
form table is a table which has all the forms
webform is a table which has the forms which have flags applied to it. It has a foreign key formid which is id to the form table and sysflagid which is foreign key to the sys flag table
sys flag is the table which contains the flags. Lets say I have flags defined as 1,2,3
I can have forms which don't have all the flags applied to it, some may have 1, some may have 2 or some may have 3 applied to it or some may have none.
How can I find all the forms which have either flag 1 or flag 2 or flag 3 applied to it ?
This is a common trick to find EXCLUSION. The value I have below of "FlagYouAreExpectingTo_NOT_Exist" is explicitly the one you expect NOT to be there. Here's how it works.
Get every form and LEFT JOIN to the Web System Flags table WITH FINDING the matching form, and flag setting you DO NOT want. If it finds a valid entry for the form and flag, the "formid" in the (wsf) table will exist. So, we want all that DON'T exist, hence the closing WHERE wsf.formid is null.
It will be NULL for those where it is NOT already flagged.
select
f.ID
from
forms f
left join webformsystemflags wsf
on f.id = wsf.formid
AND wsf.sysflagid = FlagYouAreExpectingTo__NOT__Exist
where
wsf.formid is null
You could use a subquery:
SELECT * FROM `form` WHERE `id` IN (SELECT `formid` FROM `webformsystemflags`)
Careful with subqueries on huge databases though. You could do the same thing with joins but this is an easy solution that will get you going.
Or for all results that DO NOT have a certain flag:
SELECT * FROM `form` WHERE `id` IN (SELECT `formid` FROM `webformsystemflags` WHERE `sysflagid` != 1 OR `sysflagid` != 2)
or a join method:
SELECT f.*, r.`sysflagid` FROM `form` f LEFT JOIN `webformsystemflags` r ON r.`formid` = f.`id` WHERE r.`sysflagid` != null
will get you the forms and the related flags. However, it will not get ALL flags in one row if the form has multiple flags on it. That one you may need to do a concat on the flags, but this answer is already growing unnecessarily complex.
*LAST EDIT *
Ok nutsandbolts - You need to update your question cause the two of us have overshot ourselves in a number of different queries and it isn't really helping to come back saying it doesnt give the right results. The right results can easily be reached by simply examining the queries we have provided and using the general logic behind them to compose the query that is right for you.
So my last suggestion - you say you want a query that will return a form IF it has a certain flag applied to it AND that is does NOT have other flags applied to it.
Here it is supposing you wanted all forms with a flag of 1 AND NOT 2 or 3 or none:
SELECT f.*, r.`sysflagid` FROM `form` f LEFT JOIN `webformsystemflags` r ON r.`formid` = f.`id` WHERE r.`sysflagid` =1 AND r.`formid` NOT IN (SELECT `formid` FROM `webformsystemflags` WHERE `sysflagid` = 2 OR `sysflagid` = 3)
Because your webformsystemflags is relational this query will NOT return any forms that do not exist in the webformsystemflags table - so you don't need to consider null.
If this is not what you're looking for I strongly suggest you rewrite your question with absolute and perfect clarity on your needs cause after this one I'm out of this conversation. Much luck to you though. Have fun.
You can use an exists clause to pull records like this:
select a.*
from form a
where exists (select 1
from webformsystemflags
where formid = a.id
and sysflagid IN (1,2,3))
This won't give you the associated flag. If you want that:
select a.*, b.sysflagid
from form a
join (select formid, sysflagid
from webformsystemflags
where sysflagid in (1,2,3)) b
on a.id = b.formid
There are many different ways to solve this.
EDIT: By reading a comment on the other answer it seems the question was unclear. You want the result forms that only have ONE flag? i.e. the form has flag 1 but not 2 or 3?
edit2: if you really just want a true/false query pulling only the true (has a flag):
select a.*, b.sysflagid
from form a
join webformsystemflags b on a.id = b.formid
If you want forms without flags:
select a.*
from form a
left join webformsystemflags b on a.id = b.formid
where b.formid is null
edit3: Based on comment, forms with one flag and not one of the others:
select a.*
from form a
where exists (select 1 from webformsystemflags where formid = a.id and sysflagid = 1)
and (
not exists (select 1 from webformsystemflags where formid = a.id and sysflagid = 2)
or
not exists (select 1 from webformsystemflags where formid = a.id and sysflagid = 3)
)
Im trying to build a complex (well...) query with Zend_Db_Table where I will both need to join the original table with an extra table and get some extra info from the original table with zend_db_expr.
However, things go wrong from the start. What I to is this:
$select = $this->getDbTable()->select(Zend_Db_Table::SELECT_WITH_FROM_PART)
->setIntegrityCheck(false);
$select->from( $this->getDbTable() , array(
'*' ,
new Zend_Db_Expr('`end` IS NULL as isnull') ,
new Zend_Db_Expr('`sold` IN (1,2,3) as issold') ,
) );
Zend_Debug::dump( $select->__toString() );exit;
What results in this:
SELECT `items`.*, `items_2`.*, `end` IS NULL as isnull, `sold` IN (1,2,3) as issold FROM `items`
INNER JOIN `items` AS `items_2`
What I need it to be though, at this point before I do the join with the other table, is
SELECT `items`.*, `end` IS NULL as isnull, `sold` IN (1,2,3) as issold FROM `items`
I don't need an inner join with itself, I just need to add those two Zend_Db_Expr to the things that should be selected, after which I'd continue building the query with the join and where's etc. like
$select->joinLeft( ... )
->where(...)
Any ideas? Cheers.
You should not redo a ->from() call, which means yu add a new table in the query.
Instead you should just use ->where()->columns() calls containing you Zend_Db_expr.
edit: sorry for the mistake.