I am creating a report in Crystal and the report is consistently wrong and we tracked it back to the results being returned from the server.
Environment is MySQL 5.6.19 + MySQL ODBC 5.3.6.
Tables are InnoDB.
The query that Crystal is generating is:
SELECT `Encounter`.`Id`, `Transact`.`Client_No`, `Transact`.`Amount`,
`Transact`.`Process_Date`, `Codelist`.`Type`, `Encounter`.`Co_Res`,
`Codelist`.`ProcType`, `Transact`.`Id`, `Client`.`Name`,
`Transact`.`Patient_No`, `Transact`.`Encounter_No`, `Transact`.`Quantity`,
`Encounter`.`Date_Of_Service`, `Encounter`.`Status`,
`Insuranceschedule`.`Ins1Order`, `Insuranceschedule`.`Ins2Order`,
`Insuranceschedule`.`Ins3Order`, `Insuranceschedule`.`Ins4Order`,
`Insuranceschedule`.`Ins5Order`, `Insuranceschedule`.`Ins6Order`,
`Ins1`.`Plan_Type`, `Ins2`.`Plan_Type`, `Ins5`.`Plan_Type`,
`Encounter`.`No`, `Ins3`.`Plan_Type`, `Ins4`.`Plan_Type`,
`Ins6`.`Plan_Type`
FROM {oj (((((((((`Production`.`transact` `Transact`
INNER JOIN `Production`.`client` `Client`
ON `Transact`.`Client_No`=`Client`.`Client_No`)
INNER JOIN `Production`.`codelist` `Codelist`
ON `Transact`.`Code`=`Codelist`.`Code`)
INNER JOIN `Production`.`Encounter` `Encounter`
ON (`Transact`.`Patient_No`=`Encounter`.`Patient_No`)
AND (`Transact`.`Encounter_No`=`Encounter`.`No`))
LEFT OUTER JOIN `Production`.`insuranceschedule` `Insuranceschedule`
ON `Encounter`.`InsuranceRecord`=`Insuranceschedule`.`Insurance_Id`)
LEFT OUTER JOIN `Production`.`Insco` `Ins1`
ON `Insuranceschedule`.`Ins1`=`Ins1`.`No`)
LEFT OUTER JOIN `Production`.`Insco` `Ins2`
ON `Insuranceschedule`.`Ins2`=`Ins2`.`No`)
LEFT OUTER JOIN `Production`.`Insco` `Ins3`
ON `Insuranceschedule`.`Ins3`=`Ins3`.`No`)
LEFT OUTER JOIN `Production`.`Insco` `Ins4`
ON `Insuranceschedule`.`Ins4`=`Ins4`.`No`)
LEFT OUTER JOIN `Production`.`Insco` `Ins5`
ON `Insuranceschedule`.`Ins5`=`Ins5`.`No`)
LEFT OUTER JOIN `Production`.`Insco` `Ins6`
ON `Insuranceschedule`.`Ins6`=`Ins6`.`No`}
WHERE (`Transact`.`Process_Date`>={d '2016-01-01'}
AND `Transact`.`Process_Date`<={d '2016-12-31'})
AND `Transact`.`Client_No`=763 ORDER BY `Encounter`.`No`;
So far we cannot understand why this result is different almost every time it's run.
First time we might get 57,868 records
Second time.... 30,102
Third time 77,875
and so forth...
At least 1 out of 10 times it does produce the correct result.
We have:
- Dumped the tables and reloaded them.
- Checked collation on the tables.
- Tried a different physical server with a clean import of the data.
- Tried with and without ordering the results
- Looked for anything amiss with EXPLAIN - which is always the same every time it's run.
But this query is consistently inconsistent and we can't spot the problem. We are running this query on the commandline so it can't be Crystal or ODBC - unless the query it's creating is wrong?
Thanks in advance for looking at this!
Related
Recently I wrote a PHP web app to gather a list of data and output it. Originally I thought the PHP code was running slow but I checked the amount of time this query takes to run and noticed it's MySQL and not PHP.
My conclusion is that I need to make indexes on these tables but I wanted to get feedback from.others before moving forward and doing that.
Here's my query:
SELECT *
FROM claims c
LEFT
JOIN claims_data d
ON c.claim_number = d.claim_number
LEFT
JOIN merchant_category_code m
ON c.procedure_code = m.code
LEFT JOIN claim_log l
ON c.claim_number = l.claim_number
WHERE c.social_security_num = :num
ORDER
BY c.start_date DESC
For your QUERY, indexes are needed for each line with
ON left and right side of = for each table, column
WHERE for each table, column
SORT BY each table, column
spend some time with MYSQL INDEX TUTORIALS
I've a MySQL database and a MS Access front end. MySQL database tables are linked via ODBC connection to MS Access.
ANY query with multiple joined tables will run extremely slow in case of having anything in "WHERE" (or "HAVING") clause.
For example:
SELECT tblGuests.GuestName, Sum(tblPayments.Payment) AS SumOfPayment, tblRooms.RoomName
FROM (tblGuests LEFT JOIN tblPayments ON tblGuests.GuestID = tblPayments.GuestNo) LEFT JOIN tblRooms ON tblGuests.RoomNo = tblRooms.RoomID
WHERE tblGuests.NoShow=False
GROUP BY tblGuests.GuestName, tblRooms.RoomName;
will take for ages (approx. 3 minutes for 20K records.) Exactly the same script takes for 1-1.5 seconds in case of Pass Through Query, so the problem shouldn't be related to indexes or settings on server side. (By the way, indexes are set up on the necessary columns and relations are set up, too.)
The problem happens ONLY if there are more than 2 tables involved in the query AND there is something in the "WHERE" clause or in "HAVING".
For example if you modify the code above like
SELECT tblGuests.GuestName, Sum(tblPayments.Payment) AS SumOfPayment
FROM tblGuests LEFT JOIN tblPayments ON tblGuests.GuestID = tblPayments.GuestNo
WHERE tblGuests.NoShow=False
GROUP BY tblGuests.GuestName;
then it will be very quick again. (Only 2 tables are involved to the query.) Also
SELECT tblGuests.GuestName, Sum(tblPayments.HUFpayment) AS SumOfPayment, tblGuests.NoShow, tblRooms.RoomName
FROM (tblGuests LEFT JOIN tblPayments ON tblGuests.GuestID = tblPayments.GuestNo) LEFT JOIN tblRooms ON tblGuests.RoomNo = tblRooms.RoomID
GROUP BY tblGuests.GuestName, tblGuests.NoShow, tblRooms.RoomName;
will have no problem at all because there is no "WHERE" clause. However the very similar code I mentioned in the beginning of the post will be very slow, unless I run it directly on the server (or via Pass Through Query).
Do you have any idea what can cause this problem and how to avoid it (except to run Pass Through Queries all the time)?
when I run this query it just never seems to do anything, the cpu usage goes up to ~30% and stays there, and eventually mysql workbench times out. is there a way to do it more efficiently?
insert ignore into tbl_sub_model_image
select sub_model_sk,tbl_image.image_id
from tbl_sub_model sm
inner join autotalk_identicar_old.ad_vehicle av on av.vehiclekey=sm.oldid
inner join autotalk_identicar_old.id_ad_link al on av.vehiclekey=al.rbvehiclekey
inner join autotalk_identicar_old.id_features f on f.vehiclekey=al.idvehiclekey
inner join autotalk_identicar_old.id_images i on i.id_vehiclekey=f.vehiclekey
inner join tbl_image im on i.id_images.image=im.img_lnk
where substring(id_images.image,-4) <> '.tif';
basically in the old db one record in ad_vehicle is linked to one(?) in id_ad_link. many in id_ad_link are linked to one in id_features and id_features and id_images are one - one.
in tbl_sub_model (which is in the new database) I have a field called oldid which is the primary key for ad_vehicle.
what I am trying to do is, I have a tbl_image but the images have been assigned new autonumbers, so I need to join the old db together to find out which images I need to assign to which sub model in the new one by using their file path.
none of the tables are more than 60,000 records but still it just never seems to be able to be executed, what makes it stranger is that for the last few months I have been executing it fine on another virtual machine, but now on a new one it just won't work. it's either stuck in a loop or it's just taking forever.
could somebody please help me fix/improve it?
seems like you can cut out some joins...
insert ignore into tbl_sub_model_image
select sub_model_sk,tbl_image.image_id
from tbl_sub_model sm
inner join autotalk_identicar_old.ad_vehicle av on av.vehiclekey=sm.oldid
inner join autotalk_identicar_old.id_images i on i.id_vehiclekey=av.vehiclekey
inner join tbl_image im on i.id_images.image=im.img_lnk
where substring(id_images.image,-4) <> '.tif';
figured out the problem. some of the columns used on the joins aren't indexed so that's why it's taking forever to do the query.
so the moral of the story is... indexes are your friends
I'd like to be able to join a lot of tables together (9-10 or so), but even with only 7, it times out. I tried indexing to the best of my knowledge (see attached EXPLAIN). Are joins just not made to handle 7+ tables, or am I doing something wrong?
(As you can see in the EXPLAIN, there are VERY few records in the database. Even if I can get this to work, what happens when there are 500,000+ records?
SELECT `Node`.`id`, `Node`.`name`, `Node`.`slug`, `Node`.`node_type_id`, `Node`.`site_id`, `Node`.`created`, `Node`.`modified`
FROM `mysite`.`nodes` AS `Node`
LEFT JOIN `mysite`.`data_date_times` AS `DataDateTime` ON (`DataDateTime`.`node_id` = `Node`.`id`)
LEFT JOIN `mysite`.`data_locations` AS `DataLocation` ON (`DataLocation`.`node_id` = `Node`.`id`)
LEFT JOIN `mysite`.`data_media` AS `DataMedia` ON (`DataMedia`.`node_id` = `Node`.`id`)
LEFT JOIN `mysite`.`data_metas` AS `DataMeta` ON (`DataMeta`.`node_id` = `Node`.`id`)
LEFT JOIN `mysite`.`data_profiles` AS `DataProfile` ON (`DataProfile`.`node_id` = `Node`.`id`)
LEFT JOIN `mysite`.`data_products` AS `DataProduct` ON (`DataProduct`.`node_id` = `Node`.`id`)
LEFT JOIN `mysite`.`data_texts` AS `DataText` ON (`DataText`.`node_id` = `Node`.`id`)
WHERE 1=1
GROUP BY `Node`.`id`
I'm using JOINS because I want to be able to order the main results by it's joined tables, and be able to query against specific fields within each contain (potentially) - and I don't know of a good way to do that with separate queries.
Any thoughts/suggestions VERY welcome.
Update:
When run on a local copy of the database, it says that "copying to temp table" took 117 seconds, then does complete and shows all 7 records.
I know the usage of joins, but sometimes I come across such a situation when I am not able to decide which join will be suitable, a left or right.
Here is the query where I am stuck.
SELECT count(ImageId) as [IndividualRemaining],
userMaster.empName AS ID#,
CONVERT(DATETIME, folderDetails.folderName, 101) AS FolderDate,
batchDetails.batchName AS Batch#,
Client=#ClientName,
TotalInloaded = IsNull(#TotalInloaded,0),
PendingUnassigned = #PendingUnassigned,
InloadedAssigned = IsNull(#TotalAssigned,0),
TotalProcessed = #TotalProcessed,
Remaining = #Remaining
FROM
batchDetails
Left JOIN folderDetails ON batchDetails.folderId = folderDetails.folderId
Left JOIN imageDetails ON batchDetails.batchId = imageDetails.batchId
Left JOIN userMaster ON imageDetails.assignedToUser = userMaster.userId
WHERE folderDetails.ClientId =#ClientID and verifyflag='n'
and folderDetails.FolderName IN (SELECT convert(VARCHAR,Value) FROM dbo.Split(#Output,','))
and userMaster.empName <> 'unused'
GROUP BY userMaster.empName, folderDetails.folderName, batchDetails.batchName
Order BY folderDetails.Foldername asc
Yes, it depends on the situation you are in.
Why use SQL JOIN?
Answer: Use the SQL JOIN whenever multiple tables must be accessed through an SQL SELECT statement and no results should be returned if there is not a match between the JOINed tables.
Reading this original article on The Code Project will help you a lot: Visual Representation of SQL Joins.
Also check this post: SQL SERVER – Better Performance – LEFT JOIN or NOT IN?.
Find original one at: Difference between JOIN and OUTER JOIN in MySQL.
In two sets:
Use a full outer join when you want all the results from both sets.
Use an inner join when you want only the results that appear in both
sets.
Use a left outer join when you want all the results from set a, but
if set b has data relevant to some of set a's records, then you also
want to use that data in the same query too.
Please refer to the following image:
I think what you're looking for is to do a LEFT JOIN starting from the main-table to return all records from the main table regardless if they have valid data in the joined ones (as indicated by the top left 2 circles in the graphic)
JOIN's happen in succession, so if you have 4 tables to join, and you always want all the records from your main table, you need to continue LEFT JOIN throughout, for example:
SELECT * FROM main_table
LEFT JOIN sub_table ON main_table.ID = sub_table.main_table_ID
LEFT JOIN sub_sub_table on main_table.ID = sub_sub_table.main_table_ID
If you INNER JOIN the sub_sub_table, it will immediately shrink your result set down even if you did a LEFT JOIN on the sub_table.
Remember, when doing LEFT JOIN, you need to account for NULL values being returned. Because if no record can be joined with the main_table, a LEFT JOIN forces that field to appear regardless and will contain a NULL. INNER JOIN will obviously just "throw away" the row instead because there's no valid link between the two (no corresponding record based on the ID's you've joined)
However, you mention you have a where statement that filters out the rows you're looking for, so your question on the JOIN's are null & void because that is not your real problem. (This is if I understand your comments correctly)