I am working on project management tool and while generating reports through admin end the loading time for the data from the database is very much > 5 minutes. There are few points which I know would help me to increase the performance but right now I need to have the help in SELECT query
SELECT timesheet_client.organisation as client_name, timesheet_project.title as project_name,
timesheet_task.name as task, CONCAT(timesheet_user.first_name, ' ', timesheet_user.last_name) as resource_name,
timesheet_user.bill_factor, timesheet_client.client_type, sum(spent) as spent, sum(delivered_hours) as delivered_hours,
sum(billable_hours) as billable_hours, comments, color, lock_color, updated_by, updated_by_date, timesheet_user_psdb.grp_id,
timesheet_user_psdb.client_id, timesheet_user_psdb.proj_id, timesheet_user_psdb.task_id, timesheet_user_psdb.uid,
timesheet_user_grp.grp_name
FROM timesheet_user_psdb,timesheet_user, timesheet_client,
timesheet_project,timesheet_task,timesheet_user_grp
WHERE timesheet_user.username=timesheet_user_psdb.uid and
timesheet_client.client_id=timesheet_user_psdb.client_id and timesheet_project.proj_id=timesheet_user_psdb.proj_id and
timesheet_task.task_id = timesheet_user_psdb.task_id and timesheet_user_grp.grp_id=timesheet_user_psdb.grp_id and month =3
AND year = 2017 and month!='' and timesheet_user_psdb.client_id=326
GROUP BY timesheet_user_psdb.task_id,timesheet_user_psdb.uid
ORDER BY timesheet_client.client_type desc,timesheet_client.organisation,timesheet_user_psdb.proj_id,
timesheet_user_psdb.task_id,timesheet_user.uid,timesheet_user_psdb.task_id;
I have already used an index on all the primary keys.
EXPLAIN Output:
Help for this would be highly appreciable.
Give this a try:
SELECT
TC.ORGANISATION AS CLIENT_NAME,
TP.TITLE AS PROJECT_NAME,
TT.NAME AS TASK,
CONCAT(TU.FIRST_NAME, ' ', TU.LAST_NAME) AS RESOURCE_NAME,
TU.BILL_FACTOR,
TC.CLIENT_TYPE, SUM(SPENT) AS SPENT, -- You should specify which table this comes from
SUM(DELIVERED_HOURS) AS DELIVERED_HOURS, -- You should specify which table this comes from
SUM(BILLABLE_HOURS) AS BILLABLE_HOURS, -- You should specify which table this comes from
COMMENTS, -- You should specify which table this comes from
COLOR, -- You should specify which table this comes from
LOCK_COLOR, -- You should specify which table this comes from
UPDATED_BY, -- You should specify which table this comes from
UPDATED_BY_DATE, -- You should specify which table this comes from
TUP.GRP_ID,
TUP.CLIENT_ID,
TUP.PROJ_ID,
TUP.TASK_ID,
TUP.UID,
TUG.GRP_NAME
FROM
TIMESHEET_USER AS TU
LEFT OUTER JOIN
TIMESHEET_USER_PSDB AS TUP
ON TU.USERNAME = TUP.UID
LEFT OUTER JOIN
TIMESHEET_USER_GRP AS TUG
ON TUP.GRP_ID = TUG.GRP_ID
LEFT OUTER JOIN
TIMESHEET_CLIENT AS TC
ON TUP.CLIENT_ID = TC.CLIENT_ID
LEFT OUTER JOIN
TIMESHEET_PROJECT AS TP
ON TUP.PROJ_ID = TP.PROJ_ID
LEFT OUTER JOIN
TIMESHEET_TASK TT
ON TUP.TASK_ID = TT.TASK_ID
WHERE
MONTH = 3 AND -- You should specify which table this comes from
YEAR = 2017 AND -- You should specify which table this comes from
MONTH != '' AND -- You should specify which table this comes from
TUP.CLIENT_ID = 326
GROUP BY
TUP.TASK_ID,
TUP.UID
ORDER BY
TC.CLIENT_TYPE DESC,
TC.ORGANISATION,
TUP.PROJ_ID,
TUP.TASK_ID,
TU.UID,
TUP.TASK_ID;
Doing the EXPLAIN should shed some light on it.
You might see a performance gain by doing the table joins explicitly in the FROM clause instead of inside the WHERE (https://dev.mysql.com/doc/refman/5.7/en/join.html). By doing explicit joins (e.g. LEFT OUTER, etc...) you will not only improve the readability of the query, but may be able to use less expensive joins where needed. This also affects how the query is executed as each clause is executed in a specific order (MySQL query / clause execution order).
Related
We have two tables one is properties and another one is property meta when we are getting data from one table "properties" , query only take less then one second in execution but when we are use join to get the data using bellow query from both tables its taking more then 5 second to fetch the data although we have only 12000 record in the tables , i think there is an issue in the sql query any help or suggestion will be appreciated.
SELECT
u.id,
u.property_title,
u.description,
u.city,
u.area,
u.address,
u.slug,
u.latitude,
u.longitude,
u.sync_time,
u.add_date,
u.is_featured,
u.pre_construction,
u.move_in_date,
u.property_status,
u.sale_price,
u.mls_number,
u.bedrooms,
u.bathrooms,
u.kitchens,
u.sub_area,
u.property_type,
u.main_image,
u.area_size as land_area,
pm7.meta_value as company_name,
pm8.meta_value as virtual_tour,
u.year_built,
u.garages
FROM
tbl_properties u
LEFT JOIN tbl_property_meta pm7
ON u.id = pm7.property_id
LEFT JOIN tbl_property_meta pm8
ON u.id = pm8.property_id
WHERE
u.status = 1
AND (pm7.meta_key = 'company_name')
AND (pm8.meta_key = 'virtual_tour')
AND (
(
( u.city = 'Delta'
OR u.post_code LIKE '%Delta%'
OR u.sub_area LIKE '%Delta%'
OR u.state LIKE '%Delta%')
AND country = 'Canada'
)
OR (
( u.city = 'Metro Vancouver Regional District'
OR u.post_code LIKE '%Metro Vancouver Regional District%'
OR u.sub_area LIKE '%Metro Vancouver Regional District%'
OR u.state LIKE '%Metro Vancouver Regional District%' )
AND country = 'Canada'
)
)
AND
u.pre_construction ='0'
GROUP BY
u.id
ORDER BY
u.is_featured DESC,
u.add_date DESC
Try adding this compound index:
ALTER TABLE tbl_property_meta ADD INDEX id_key (property_id, meta_key);
If it doesn't help make things faster, try this one.
ALTER TABLE tbl_property_meta ADD INDEX key_id (meta_key, property_id);
And, you should know that column LIKE '%somevalue' (with a leading %) is a notorious performance antipattern, resistant to optimization via indexes. (There's a way to create indexes for that shape of filter in PostgreSQL, but not in MariaDB / MySQL.)
Add another column with the meta stuff; throw city, post_code, sub_area, and state and probably some other things into it. Then build a FULLTEXT index on that column. Then use MATCH(..) AGAINST("Delta Metro Vancouver Regional District") in the WHERE clause _instead of the LEFT JOINs (which are actually INNER JOINs) and the really messy part of the WHERE clause.
Also, the GROUP BY is probably unnecessary, thereby eliminating extra sort on the intermediate set of rows.
Good day,
I have a small issue with MySQL Distinct.
Trying the following query in my system :
SELECT DISTINCT `booking_id`, `booking_ticket`, `booking_price`, `bookingcomment_id`, `bookingcomment_message` FROM `mysystem_booking`
LEFT JOIN `mysystem_bookingcomment` ON `mysystem_booking`.`booking_id` = `mysystem_bookingcomment`.`bookingcomment_link`
WHERE `booking_id` = 29791
The point is that there are bookings like 29791 that have many comments added.
Let's say 10. Then when running the above query I see 10 results instead of one.
And that's not the way DISTINCT supposes to work.
I simply want to know if there are any comments. If the comment ID is not 0 then there is a comment. Of course I can add COUNT(blabla) as comment_number but that's a whole different story. For me now I'd like just to have this syntax right.
You may try aggregating here, to find which bookings have at least a single comment associated with them:
SELECT
b.booking_id,
b.booking_ticket,
b.booking_price
FROM mysystem_booking b
LEFT JOIN mysystem_bookingcomment bc
ON b.booking_id = bc.bookingcomment_link
WHERE
b.booking_id = 29791
GROUP BY
b.booking_id
HAVING
COUNT(bc.bookingcomment_link) > 0;
Note that depending on your MySQL server mode, you might have to also add the booking_ticket and booking_price columns to the GROUP BY clause to get the above query to run.
You can try below - using a case when expression
SELECT DISTINCT `booking_id`, `booking_ticket`, `booking_price`, `bookingcomment_id`,
case when `bookingcomment_message`<>'0' then 'No' else 'Yes' end as comments
FROM `mysystem_booking`
LEFT JOIN `mysystem_bookingcomment` ON `mysystem_booking`.`booking_id` = `mysystem_bookingcomment`.`bookingcomment_link`
WHERE `booking_id` = 29791
I have a SQL query that links 2 tables to provide the data if a horse is in both the tables:
SELECT ProformSystem.TheDate as racedate,
ProformSystem.Course as course,
ProformSystem.TheTime as thetime,
ProformSystem.Horse as horse,
ATRSpeedRatings.rank as rank
FROM ATRSpeedRatings
INNER JOIN ProformSystem ON (ATRSpeedRatings.Horse = trim(ProformSystem.Horse)) AND (ATRSpeedRatings.TheDate = ProformSystem.TheDate) order by ProformSystem.TheTime;"
Is it possible that if the horse in ProformSystem.Horse is NOT in ATRSpeedRatings.Horse then I just make rank = 0 as a default value, or would I need to run a separate query?
This way I can display all horses from ProformSystem even if they don't have a rank in ATRSpeedRatings.
I think you want a left join and coalesce():
SELECT ps.TheDate as racedate, ps.Course as course, ps.TheTime as thetime,
ps.Horse as horse, COALESCE(sr.rank, 0) as rank
FROM ProformSystem ps LEFT JOIN
ATRSpeedRatings sr
ON sr.Horse = TRIM(ps.Horse) AND sr.TheDate = ps.TheDate)
ORDER BY ps.TheTime;
Note that this query uses table aliases. These make the query easier to write and to read.
Also, the JOIN condition sr.Horse = trim(ps.Horse) is highly suspect. You should fix the data so there are no spaces in ProformSystem. Fixing the data is more efficient and it will prevent problems on future queries.
i am writing a mysql query as below
SELECT `user_master`.`first_name`,
`city_name`,
`user_master`.`last_name`,
`user_master`.`user_master_id`,
`account_management_master`.`account_name`,
`donation_receipt_info`.`receipt_temple_id`,
date(dt) AS dt,
SUM(`donation_receipt_info`.`amount`) AS amount
FROM (`donation_receipt_info`)
JOIN `donation_receipt_master` ON donation_receipt_master`.`receipt_id`=`donation_receipt_info`.`receipt_id`
JOIN `account_management_master` ON `account_management_master`.`account_id`=`donation_receipt_info`.`account_id`
JOIN `user_master` ON `user_master`.`user_master_id`=`donation_receipt_master`.`user_master_id`
JOIN `user_address_info` ON `user_address_info`.`user_master_id`=`user_master`.`user_master_id`
JOIN `city_master` ON `city_master`.`city_id`=`user_address_info`.`city_id`
WHERE `donation_receipt_info`.`temple_id` = '1'
GROUP BY `donation_receipt_info`.`receipt_id`,
`donation_receipt_info`.`account_id`
the table donation_receipt_info and master have approx 42k results the query is taking way to much time of about 5 to 6 minutes to execute in mysql itself.
can someone please help me optimize the query, any help or suggestion would be very helpful
Thanks.
First, your query is impossible to read. You should format it and learn to use table aliases:
SELECT um.first_name, city_name, um.last_name, um.user_master_id, amm.account_name,
dri.receipt_temple_id, date(dt) AS dt, SUM(dri.amount) AS amount
FROM donation_receipt_info dri JOIN
donation_receipt_master drm
ON drm.receipt_id = dri.receipt_id JOIN
account_management_master amm
ON amm.account_id = dri.account_id JOIN
user_master um
ON um.user_master_id = drm.user_master_id JOIN
user_address_info uai
ON uai.user_master_id = um.user_master_id JOIN
city_master cm
ON cm.city_id = uai.city_id
WHERE dri.temple_id = '1'
GROUP BY dri.receipt_id, dri.account_id;
Next. Do all the tables have the obvious indexes? That is, each table appears to have an id and these should be declared as keys (primary keys preferably). For instance, city_master(city_id).
Next, there should be an index on donation_receipt_info(temple_id, receipt_id, account_id). This should help with the where. Note: if temple_id is really an integer, the where clause should be expressed as WHERE dri.temple_id = 1 -- no quotes. You don't want MySQL to get confused and decide not to use the index.
These changes will probably help. 5-6 minutes seems like a long time for such a query.
I'm in over my head with a big mysql query (mysql 5.0), and i'm hoping somebody here can help.
Earlier I asked how to get distinct values from a joined query
mysql count only for distinct values in joined query
The response I got worked (using a subquery with join as)
select *
from media m
inner join
( select uid
from users_tbl
limit 0,30) map
on map.uid = m.uid
inner join users_tbl u
on u.uid = m.uid
unfortunately, my query has grown more unruly, and though I have it running, joining into a derived table is taking too long because there is no indexes available to the derived query.
my query now looks like this
SELECT mdate.bid, mdate.fid, mdate.date, mdate.time, mdate.title, mdate.name,
mdate.address, mdate.rank, mdate.city, mdate.state, mdate.lat, mdate.`long`,
ext.link,
ext.source, ext.pre, meta, mdate.img
FROM ext
RIGHT OUTER JOIN (
SELECT media.bid,
media.date, media.time, media.title, users.name, users.img, users.rank, media.address,
media.city, media.state, media.lat, media.`long`,
GROUP_CONCAT(tags.tagname SEPARATOR ' | ') AS meta
FROM media
JOIN users ON media.bid = users.bid
LEFT JOIN tags ON users.bid=tags.bid
WHERE `long` BETWEEN -122.52224684058 AND -121.79760915942
AND lat BETWEEN 37.07500915942 AND 37.79964684058
AND date = '2009-02-23'
GROUP BY media.bid, media.date
ORDER BY media.date, users.rank DESC
LIMIT 0, 30
) mdate ON (mdate.bid = ext.bid AND mdate.date = ext.date)
phew!
SO, as you can see, if I understand my problem correctly, i have two derivative tables without indexes (and i don't deny that I may have screwed up the Join statements somehow, but I kept messing with different types, is this ended up giving me the result I wanted).
What's the best way to create a query similar to this which will allow me to take advantage of the indexes?
Dare I say, I actually have one more table to add into the mix at a later date.
Currently, my query is taking .8 seconds to complete, but I'm sure if I could take advantage of the indexes, this could be significantly faster.
First, check for indices on ext(bid, date), users(bid) and tags(bid), you should really have them.
It seems, though, that it's LONG and LAT that cause you most problems. You should try keeping your LONG and LAT as a (coordinate POINT), create a SPATIAL INDEX on this column and query like that:
WHERE MBRContains(#MySquare, coordinate)
If you can't change your schema for some reason, you can try creating additional indices that include date as a first field:
CREATE INDEX ix_date_long ON media (date, `long`)
CREATE INDEX ix_date_lat ON media (date, lat)
These indices will be more efficient for you query, as you use exact search on date combined with a ranged search on axes.
Starting fresh:
Question - why are you grouping by both media.bid and media.date? Can a bid have records for more than one date?
Here's a simpler version to try:
SELECT
mdate.bid,
mdate.fid,
mdate.date,
mdate.time,
mdate.title,
mdate.name,
mdate.address,
mdate.rank,
mdate.city,
mdate.state,
mdate.lat,
mdate.`long`,
ext.link,
ext.source,
ext.pre,
meta,
mdate.img,
( SELECT GROUP_CONCAT(tags.tagname SEPARATOR ' | ')
FROM tags
WHERE ext.bid = tags.bid
ORDER BY tags.bid GROUP BY tags.bid
) AS meta
FROM
ext
LEFT JOIN
media ON ext.bid = media.bid AND ext.date = media.date
JOIN
users ON ext.bid = users.bid
WHERE
`long` BETWEEN -122.52224684058 AND -121.79760915942
AND lat BETWEEN 37.07500915942 AND 37.79964684058
AND ext.date = '2009-02-23'
AND users.userid IN
(
SELECT userid FROM users ORDER BY rank DESC LIMIT 30
)
ORDER BY
media.date,
users.rank DESC
LIMIT 0, 30
You might want to compare your perforamnces against using a temp table for each selection, and joining those tables together.
create table #whatever
create table #whatever2
insert into #whatever select...
insert into #whatever2 select...
select from #whatever join #whatever 2
....
drop table #whatever
drop table #whatever2
If your system has enough memory to hold full tables this might work out much faster. It depends on how big your database is.