I had created a query to get a list of staffs using this query. It is run after checked the permission level of the login user.
if (Auth::user()->hasPermissionTo('All Sections')) {
$itemregistrations = DB::table('itemregistrations')
->join('sections', 'itemregistrations.sectionid', '=', 'sections.sectionid')
->join('categories', 'itemregistrations.categoryid', '=', 'categories.categoryid')
->join('operasi', 'itemregistrations.operasiid', '=', 'operasi.operasiid')
->select('itemregistrations.ItemRegistrationID','itemregistrations.name', 'itemregistrations.Nobadan', 'sections.sectionname', 'categories.categoryname', 'operasi.operasiname')
->get();
}
However, the query gets loading quite long, about a minute to finish loading. The list displayed about 1115.
How to reduce the loading time?
I read about eager loading to decrease the loading time. But my trial not success.
section is the department of staffs.
categories is the staff level
operasi is the grade of staff, related to categories, each category has its own operasiname.
This is the indexing on itemregistrations table.
I had installed laravel debugger and produce this result:
-6 views
-4 queries
select * from `users` where `id` = 1 limit 1
select `permissions`.*, `model_has_permissions`.`model_id` as `pivot_model_id`, `model_has_permissions`.`permission_id` as
`pivot_permission_id` from `permissions`
inner join `model_has_permissions` on `permissions`.`id` = `model_has_permissions`.`permission_id`
where `model_has_permissions`.`model_id` = 1 and `model_has_permissions`.`model_type` = 'App\User'
select `roles`.*, `model_has_roles`.`model_id` as `pivot_model_id`, `model_has_roles`.`role_id` as `pivot_role_id` from `roles`
inner join `model_has_roles` on `roles`.`id` = `model_has_roles`.`role_id`
where `model_has_roles`.`model_id` = 1 and `model_has_roles`.`model_type` = 'App\User'
select `itemregistrations`.`ItemRegistrationID`, `itemregistrations`.`name`,
`itemregistrations`.`Nobadan`, `sections`.`sectionname`, `categories`.`categoryname`, `operasi`.`operasiname`
from `itemregistrations` inner join `sections` on `itemregistrations`.`sectionid` = `sections`.`sectionid`
inner join `categories` on `itemregistrations`.`categoryid` = `categories`.`categoryid`
inner join `operasi` on `itemregistrations`.`operasiid` = `operasi`.`operasiid`
-1116 gates
The above query is filtered according to few permissions.
these are indexes for table involved:
role table index
permission table index
model has role table
role has permission table
for the DB side be sure you have proper index on the columns involved in JOIN
sections sectionid
categories categoryid
operasi operasiid
expecially a composite index on
itemregistrations (sectionid , categoryid , operasiid )
anyway the load of 1256 si pretty unuseful in real app ..
for this you could reduce the loading time for the data show using pagination
based on you cardinality you should build an index
itemregistrations (operasiid, sectionid , categoryid )
could be you need remove the index on the same columns involved and leave only the composite ...
Related
I am stuck in 1 left join query in which I want to check multiple columns in on statement.
By default in the database, some column is null which I want to check in the on statement.
Now the issue is when I run a query using the OR operator it only runs the 1st condition and the rest are skipped.
If I use AND operator it throws an error.
So is there any way to get data from multiple conditions?
Here is my query:
$data = "SELECT
b.book_name, b.book_id,
b.cats_id, b.cats_id1,
b.cats_id2, b.cats_id3,
b.cats_id4, b.cats_id5,
b.cats_id6,
b.book_rating,
b.book_author,
b.book_stock,
b.book_publisher,
b.book_front_img,
b.book_status,
p.publisher_id,
p.publisher_name,
a.author_id,
a.author_name,
cat.cats_id,
cat.cats_name,
cat.cats_status
FROM
`books` AS b
LEFT JOIN `publisher` AS p
ON b.book_publisher = p.publisher_id
LEFT JOIN `author` AS a
ON b.book_author = a.author_id
LEFT JOIN categorys As cat
ON b.cats_id = cat.cats_id
OR b.cats_id1 = cat.cats_id
OR b.cats_id2 = cat.cats_id
OR b.cats_id3 = cat.cats_id
OR b.cats_id4 = cat.cats_id
OR b.cats_id5 = cat.cats_id
OR b.cats_id6 = cat.cats_id
GROUP BY
b.book_name
HAVING
cat.cats_name = '$search_data'
AND b.book_status = 1
ORDER BY
$sorting
LIMIT $offset, $page_limit"
You probably don't have more than one author displayed for your multi-author books either. You are misusing MySQL's notorious nonstandard extension to GROUP BY.
To troubleshoot this kind of query, disable that extension with SET sql_mode = CONCAT_WS(',',##sql_mode, 'ONLY_FULL_GROUP_BY'), then try your query again. You'll need more terms in your GROUP BY clause.
It looks like each books row has multiple category id columns. And it looks like you want to display information from your categorys table for each of them.
Use GROUP BY b.book_id, p.publisher_id, a.author_id, cats.cats_id to prevent MySQL's bizarro handling of GROUP BY from concealing your data.
I must add this: your multiple books.cats_id columns are not the SQLish way to handle your many-to-many relationship between books and categories. In the parlance of our trade, your books table is denormalized.
What you want is a new table called books_categorys with two columns, book_id and cats_id. It's called a join table. When a row is present in that table, it means a particular book is in a particular category. It's the SQLish way of handling a setup where each book can be in zero or more categorys. Here's an explanation. MySQL join many to many single row
Then you remove all the cats_id columns from books, and retrieve the categories like this.
Then you do something like this SELECT to get the categories.
SELECT books.id, books.name,
categorys.cats_id, categorys.cats_name, categorys.cats_status
FROM books
JOIN books_categorys ON books.book_id = books_categorys.book_id
JOIN categorys ON books_categorys.cats_id = categorys.cats_id
``
I have what seems like an easy many-to-many relationship query with pagination. It works fine, but the downside is the time it takes. On the prod server, it's more than 20 seconds. On my development environment, 13 seconds.
Here is the code:
$query = $this->excerpt->orderBy($sort, $order);
$excerpts = $query->with('source.authors')
->with('excerptType')
->with('tags')
->whereHas('tags', function($q) use ($tagId){
$q->where('tag_id', $tagId);
})
->paginate($this->paginateCount);
These two queries take the longest
select count(*) as aggregate
from `excerpt`
where (select count(*)
from `tags`
inner join `excerpt_tag`
on `tags`.`id` = `excerpt_tag`.`tag_id`
where `excerpt_tag`.`excerpt_id` = `excerpt`.`id`
and `tag_id` = '655') >= 1
2.02 secs
select *
from `excerpt`
where (select count(*) from `tags`
inner join `excerpt_tag`
on `tags`.`id` = `excerpt_tag`.`tag_id`
where `excerpt_tag`.`excerpt_id` = `excerpt`.`id`
and `tag_id` = '655') >= 1
order by `created_at` desc limit 15 offset 0
2.02 secs
I was thinking of changing this to a simple query with inner joins, like:
select *
from `excerpt`
inner join excerpt_tag on excerpt.id = excerpt_tag.excerpt_id
inner join tags on excerpt_tag.tag_id = tags.id
where tags.id = 655
limit 10 offset 0
But then I lose the advantage of eager loading and so on.
Does anyone have an idea on what the best way to speed this up would be?
Change
( SELECT COUNT(*) ... ) > 0
to
EXISTS ( SELECT 1 ... )
Follow the instructions here for index tips in many:many tables.
If a tag is just a short string, don't bother having a table (tags) for them. Instead, simply have the tag in the excerpt_tag and get rid of tag_id.
A LIMIT without an ORDER BY is somewhat meaningless -- which 10 rows you get will be unpredictable.
Well I have a solution that has led to a significant improvement and only added a few lines of code and only 1 or maybe 2 extra sql queries.
I decided to query the tags first and find out which excerpts were connected and then use a whereIn to then query all information from the excerpts, thus hoping to still make use of the with function and eager loading. At least keep the number of queries down to an absolute minimum.
Here is the code with the solution:
// workaround to make excerpt query faster
$excerptsWithTag = $this->tag->with(['excerpts' => function($query) {
$query->select('excerpt.id');
}])->find($tagId,['tags.id']);
// actual excrpt query
$excerptIds = array_column($excerptsWithTag->excerpts()->get()->toArray(), 'id');
$query = $this->excerpt->orderBy($sort, $order);
$excerpts = $query->with([
'source.authors',
'excerptType',
'tags'
])
->whereIn('excerpt.id', $excerptIds)
->paginate($this->paginateCount);
There is very likely a much more eloquent way to solve this problem, but this works and I'm happy.
In a current web project of mine, we are implementing a complex search function. As part of that search functionality, we are using the MySQL COUNT function to be able to return the number of matching results.
We are running into a performance hiccup as a result. When getting the actual list of results, MySQL properly uses the indexes we have setup and returns results very quickly. When using the COUNT query, however, the results are sometimes returned very slowly. When examining the execution plans for various search queries, we have discovered that sometimes the COUNT query is doing a full table scan. Other times, despite the query logic being practically identical, the query is using an index. We can't seem to notice any particular pattern that distinguishes the two.
Here is an example of a query that is NOT doing a full table scan:
select COUNT(DISTINCT text.name) AS count
from `text_epigraphy`
inner join `text` ON `text`.`uuid` = `text_epigraphy`.`text_uuid`
inner join `hierarchy` ON `hierarchy`.`uuid` = `text_epigraphy`.`text_uuid`
inner join `text_epigraphy` as `t1` ON `t1`.`text_uuid` = `text_epigraphy`.`text_uuid`
and `t1`.`reading_uuid` in ('01f1e805-1278-ec9b-9f69-fced97bc923e',
'07a120bc-02ec-c1ac-e0ba-532de39766ed', '126f978b-bd99-40f0-8f3b-d2bcec1ed3fe',
'44ec304e-71f4-4995-a30d-0ca6d3bec95a', '4a1d8673-9e30-2d1e-7b87-453dec2886db',
'bce40e36-d6eb-c44a-d114-8c7653a0e68c', 'c9083b77-6122-7933-ea21-63d3777749f3' )
and t1.char_on_tablet=text_epigraphy.char_on_tablet + 1
and t1.line=text_epigraphy.line
inner join `text_epigraphy` as `t2` ON `t2`.`text_uuid` = `text_epigraphy`.`text_uuid`
and `t2`.`reading_uuid` in ('3fc156dc-e831-493e-5dc1-84a547aeb4fa',
'70f9be19-62b6-3fe8-ddda-32bd50a8d36e' )
and t2.char_on_tablet=text_epigraphy.char_on_tablet + 2
and t2.line=text_epigraphy.line
inner join `text_epigraphy` as `t3` ON `t3`.`text_uuid` = `text_epigraphy`.`text_uuid`
and `t3`.`reading_uuid` in ('1ee91402-ebb0-3be9-cc38-9d4187816031',
'25a44259-fe7a-2b73-6e2c-02171c924805', 'a23fd531-c796-353e-4a53-54680248438a',
'd55fa6ad-c523-2e33-6378-b4f2e2a020f1' )
and t3.char_on_tablet=text_epigraphy.char_on_tablet + 3
and t3.line=text_epigraphy.line
where `text_epigraphy`.`reading_uuid` in ('6c0e47d0-00aa-26fb-e184-07038ca64323',
'd8904652-f049-11f9-3f7a-038f1e3b6055', 'eca27c41-d3ca-417c-15e0-db5353ddaefb' )
and 1 = 1
and (1 = 1
or 1 = 0)
limit 1
And yet this query IS doing a full table scan:
select COUNT(DISTINCT text.name) AS count
from `text_epigraphy`
inner join `text` ON `text`.`uuid` = `text_epigraphy`.`text_uuid`
inner join `hierarchy` ON `hierarchy`.`uuid` = `text_epigraphy`.`text_uuid`
inner join `text_epigraphy` as `t1` ON `t1`.`text_uuid` = `text_epigraphy`.`text_uuid`
and `t1`.`reading_uuid` in ('3fc156dc-e831-493e-5dc1-84a547aeb4fa')
and t1.char_on_tablet=text_epigraphy.char_on_tablet + 1
and t1.line=text_epigraphy.line
inner join `text_epigraphy` as `t2` ON `t2`.`text_uuid` = `text_epigraphy`.`text_uuid`
and `t2`.`reading_uuid` in ('1ee91402-ebb0-3be9-cc38-9d4187816031',
'25a44259-fe7a-2b73-6e2c-02171c924805', 'a23fd531-c796-353e-4a53-54680248438a',
'd55fa6ad-c523-2e33-6378-b4f2e2a020f1' )
and t2.char_on_tablet=text_epigraphy.char_on_tablet + 2
and t2.line=text_epigraphy.line
where `text_epigraphy`.`reading_uuid` in ('c9083b77-6122-7933-ea21-63d3777749f3')
and 1 = 1
and (1 = 1
or 1 = 0)
limit 1
Like I said, we can't quite figure out why some searches are doing a full table scan when using COUNT but it is resulting in significantly slower searches. If anyone could help us figure out what is causing the difference and how we might be able to avoid the full table scan or at least optimize the queries.
Can't you remove hierarchy?
What indexes exist on text_epigraphy? This looks useful:
INDEX(line, char_on_tablet, reading_uuid, text_uuid)
On text: INDEX(uuid, name)
After that, please provide EXPLAIN SELECT; then I will look at your question.
My MySQL query code likes as shown below, and there are about several thousands of records in the table, by now below SQL executes about 5 minutes and more. I am looking for ways to optimize it so that it takes less time to execute. Thank you!
SELECT `m`.`id`,
`m`.`id`,
`tr`.`name`,
`m`.`m_date`,
`t1`.`t_name` AS home,
`t2`.`t_name` AS away,
`m`.`score1`,
`m`.`score2`,
`cw1`.`tid` AS tid1,
`cw2`.`tid` AS tid2,
`o1`.`odds` AS odds1,
`o2`.`odds` AS odds2,
`m`.`m_time`
FROM `jos_bl_match` AS `m`
LEFT JOIN `jos_bl_matchday` AS `md` ON (`md`.`id` = `m`.`m_id`)
LEFT JOIN `jos_bl_seasons` AS `s` ON (`s`.`s_id` = `md`.`s_id`)
LEFT JOIN `jos_bl_tournament` AS `tr` ON (`tr`.`id` = `s`.`t_id`)
LEFT JOIN `jos_bl_teams` AS `t1` ON (`m`.`team1_id` = `t1`.`id`)
LEFT JOIN `jos_bl_teams` AS `t2` ON (`m`.`team2_id` = `t2`.`id`)
LEFT JOIN `jos_vuvuzelaodds_odds` AS `o1` ON (`o1`.`m_id` = `m`.`id`)
AND `o1`.`market_id` = 1
AND `o1`.`bookmaker_id` = 1
LEFT JOIN `jos_vuvuzelaodds_odds` AS `o2` ON (`o2`.`m_id` = `m`.`id`)
AND `o2`.`market_id` = 1
AND `o2`.`bookmaker_id` = 2
LEFT JOIN `jos_cwtags_tags` AS `cw1` ON (`cw1`.`item_id` = `o1`.`m_id`)
LEFT JOIN `jos_cwtags_tags` AS `cw2` ON (`cw2`.`item_id` = `o2`.`m_id`)
WHERE `m`.`published` = 1
AND `s`.`published` = '1'
AND `tr`.`published` = '1'
AND `s`.`s_id` = 869
AND `m`.`m_played` = '1'
AND `m`.`m_date` > 2013-01-01
AND `o1`.`odds` != ''
AND `o2`.`odds` != ''
AND `cw1`.`cat_id` = 19
AND `cw2`.`cat_id` = 21
ORDER BY `m`.`m_date`,
`md`.`id`,
`s`.`s_id`,
`tr`.`id` DESC LIMIT 0, 15
"Normalize, but don't over-normalize."
Some composite indexes you may be missing...
jos_bl_match: INDEX(m_played, published, m_date)
The columns need to be in that order. That will more quickly start the filtering.
The following should speed up their JOINs:
jos_vuvuzelaodds_odds: INDEX(market_id, bookmaker_id, m_id)
jos_cwtags_tags: INDEX(cat_id, item_id)
It seems like those last two indexes could (should) be the PRIMARY KEY. Are they?
Some (perhaps all) of the LEFT JOINs may as well be INNER JOINs; did you consider that?
Please provide EXPLAIN SELECT.
Without having access to the database it is a little hard to tell. This seems to be a lot of data to only pull 15 records. Are you sure you need to pull the data this way?
Probably the best route:
Optimize your database as below.
Make a flat database view of all the games that has the fields that you need. A static table will be much faster but you would need to set up updates with triggers which is beyond the scope of this answer but the process is similar
Write queries against the
view.
You are not selecting any fields from the seasons table in your select. I used the field from the jos_bl_matchday table.
This should get you started. You can also use conditionals in your select statements
(IF value = 1, table.field, null) as yadda
instead of joining one table over and over again but you would have to experiment.
CREATE VIEW allTheGames AS SELECT
`m`.`id` as id,
`md`.`s_id` as seasonId,
`tr`.`name` as name,
`m`.`m_date` as m_date,
`t1`.`t_name` AS home,
`t2`.`t_name` AS away,
`m`.`score1` as score1,
`m`.`score2` as score2,
`cw1`.`tid` AS tid1,
`cw2`.`tid` AS tid2,
`o1`.`odds` AS odds1,
`o2`.`odds` AS odds2,
`m`.`m_time` as m_time
FROM `jos_bl_match` AS `m`
LEFT JOIN `jos_bl_matchday` AS `md` ON (`md`.`id` = `m`.`m_id`)
LEFT JOIN `jos_bl_tournament` AS `tr` ON (`tr`.`id` = `s`.`t_id`)
LEFT JOIN `jos_bl_teams` AS `t1` ON (`m`.`team1_id` = `t1`.`id`)
LEFT JOIN `jos_bl_teams` AS `t2` ON (`m`.`team2_id` = `t2`.`id`)
LEFT JOIN `jos_vuvuzelaodds_odds` AS `o1` ON (`o1`.`m_id` = `m`.`id`) AND `o1`.`market_id` = 1 AND `o1`.`bookmaker_id` = 1
LEFT JOIN `jos_vuvuzelaodds_odds` AS `o2` ON (`o2`.`m_id` = `m`.`id`) AND `o2`.`market_id` = 1 AND `o2`.`bookmaker_id` = 2
LEFT JOIN `jos_cwtags_tags` AS `cw1` ON (`cw1`.`item_id` = `o1`.`m_id`)
LEFT JOIN `jos_cwtags_tags` AS `cw2` ON (`cw2`.`item_id` = `o2`.`m_id`)
WHERE `m`.`published` = 1 AND `s`.`published` = '1' AND `tr`.`published` = '1'
AND `m`.`m_played` = '1'
AND `o1`.`odds1` != '' AND `o2`.`odds2` != ''
Then query it with:
select * from allTheGames
WHERE season_id = 869 AND m_date > 2013-01-01 AND tid1 = 19 AND tid2 = 21
Steps to optimize:
Figure out exactly which data you want out of this query and why:
Is this a custom report? A web page? do you need to have all of this data at once or would it make more sense to have the user drill down?
How often is the query run? Once a minute? Once a day?
How many records are in each table? Your view should reflect this "game object"
Your database:
Run "explain" against this query http://dev.mysql.com/doc/refman/5.7/en/explain.html.
It will show all of the work the database is doing, the query execution plan, and how many records it is looking in to do it. This usually happens quickly: it does not actually execute the query.
See if you can put the database on SSD drives or even RAM
Your table structure:
Make sure that you have indexes on all of the fields that you are searching on. There are many ways to optimize this.
Use "explain" to be sure MySQL is able to use indexes.
If there are really only 2 markets in the jos_vuvuzelaodds_odds table consider making 2 fields.
Good luck!
i have two tables i am trying to get information from.
login table - which has the list of employees
projects table - which has the projects
in short, i am trying to write a query that will select the copywriters and perform a subquery on each that will return a field dubbed 'open_projects'. This, i can get to work with the below sql:
select web_login_id,
(select count(project_web_id) from project
where copywriter = web_login_id
and (`status` = 'open' or `status` = 'qual')) as open_projects from login
where roles like '%copywriter%'
and tierLevel like '%c1%'
order by open_projects asc
This returns something like:
1982983 3
1982690 22
2987398 5
The problem with this is that sometimes 5 or 6 of the projects will belong to the same client and are not actually being worked on as they are dealt with in a queue-ish fashion.
My question is how to modify the above sql so that the subquery will GROUP subset based on the client_login_id field.
This sql gives me an error of : subquery returns more than 1 row
select web_login_id,
(select count(project_web_id) from project
where copywriter = web_login_id
and (`status` = 'open' or `status` = 'qual') group by client_login_id) as open_projects from login
where roles like '%copywriter%'
and tierLevel like '%c1%'
order by open_projects asc
You need to rephrase this as an explicit join. I think the following does the trick:
select web_login_id, cw.open_projects
from login l left outer join
(select copywriter, count(project_web_id) as open_projects
from project
where `status` in ('open', 'qual')
group by copywriter
) cw
on l.web_login_id = cw.copywriter
where l.roles like '%copywriter%' and l.tierLevel like '%c1%'
order by open_projects asc
I'm not sure what the "group by client_login_id" is doing. It doesn't seem necessary.
Once you've done this, you can return as many columns as you like from the subquery.