I am trying to make a table that includes join between 3 tables in the MSSS 2008. There is a fact table, a date table, and a course table. I should join them to make a base table. In date table there is a one parameter that name is Academic Year lookup, and the values in this parameter is like 2000/1, 2001/2. This parameter in the base table should separate to three parameter such as CensusYear, StartYear, and ApplicationYear. Therefore, I need the data table multiple times. I executed a inner join query, and already I have four inner join statement, but I am getting some extra years, and I'm losing some years. I believe, my query should be wrong somewhere.
The attached file is include the design view that created in the MS Access, it'll help to see the tables, and understand what I need to create.
[Design View in Ms Access][1]
SELECT
A.[EventCount],
B.[AcademicYearLookup] AS [CensusYear],
C.[AcademicYearLookup] AS [StartYear],
D.[AcademicYearLookup] AS [ApplicationYear],
B.[CurrentWeekComparisonFlag],
B.[AcademicWeekOfYear],
case
when A.[ApplicationCensusSK] = 1 then 'Same Year'
when A.[ApplicationCensusSK] = 2 then 'Next Year'
when A.[ApplicationCensusSK] = 5 then 'Last Year'
ELSE 'Other'
END as [CensusYearDescription],
B.[CurrentAcademicYear],
A.[StudentCodeBK],
A.[ApplicationSequenceNoBK],
A.[CourseSK],
A.[CourseGroupSK],
A.[CourseMoaSK],
A.[CboSK],
A.[CourseTaughtAbroadSK],
A.[ApplicationStatusSK],
A.[ApplicationFeeStatusSK],
A.[DecisionResponseSK],
A.[NationalityCountrySK],
A.[DomicileCountrySK],
A.[TargetRegionSK],
A.[InternationalSponsorSK] INTO dbo.[BaseTable3yrs]
FROM Student.FactApplicationSnapshot A
INNER JOIN Conformed.DimDate AS B ON A.[CensusDateSK] = B.[DateSK]
INNER JOIN Conformed.DimDate AS C ON A.[AcademicYearStartDateSK] = C.[DateSK]
INNER JOIN Conformed.DimDate AS D ON A.[ApplicationDateSK] = D.[DateSK]
INNER JOIN Student.DimCourse ON A.CourseSK = Student.DimCourse.CourseSK
WHERE (((B.CurrentAcademicYear) In (0,-1))
AND ((A.ApplicationCensusSK) In (1,2,5))
AND ((Student.DimCourse.DepartmentShortName)= 'TEACH ED'));
/* the query to check that the result it's correct or not, and I check it by academic week of year, and I found that I am lossing some data, and I have some extra data, means maybe join is wrong*/
select * from [BaseTable3yrs]
where [StudentCodeBK]= '26002423'
AND [ApplicationSequenceNoBK] = '0101'
order by [AcademicWeekOfYear]
When doing recursive joins like this, it's easy to get duplicate records. You could try gathering the Conformed data separately into a table variable and then joining to it. This would also make your query more readable.
You might also try a SELECT DISTINCT on your main query.
Here's my problem: I need to get the amount of test cases and issues associated to a project that meet certain conditions (test cases that are successful, and issues that are flaws of the application), but for some reason the amount doesn't add up. I have 10 test cases in a project, of which 6 are successful; and 8 issues, of which only 4 are flaws. However, the respective results for COUNT each show 24, which makes no sense. I did notice, though, that 24 happens to be 6 times 4, but I don't see how the query would multiply them.
Anyway... Can someone help me find which part of my query is wrong? How can I get the correct result? Thanks in advance.
Here's the query:
SELECT
p.codigo_proyecto,
p.nombre,
IFNULL(COUNT(iep.id_incidencia_etapa_proyecto), 0) AS cantidad_defectos,
IFNULL(COUNT(tc.id_test_case), 0) AS test_cases_exitosos,
CASE IFNULL(COUNT(tc.id_test_case), 0) WHEN 0 THEN 'No aplica'
ELSE CONCAT((IFNULL(COUNT(tc.id_test_case), 0) / IFNULL(COUNT(tc.id_test_case), 0)) * 100, '%') END AS tasa_defectos
FROM proyecto p
INNER JOIN etapa_proyecto ep ON p.codigo_proyecto = ep.codigo_proyecto
INNER JOIN incidencia_etapa_proyecto iep ON ep.id_etapa_proyecto = iep.id_etapa_proyecto
INNER JOIN incidencia i ON iep.id_incidencia = i.id_incidencia
INNER JOIN test_case tc ON ep.id_etapa_proyecto = tc.id_etapa_proyecto
INNER JOIN etapa_proyecto ep_ultima ON ep_ultima.id_etapa_proyecto =
(SELECT ep_ultima2.id_etapa_proyecto FROM etapa_proyecto ep_ultima2
WHERE p.codigo_proyecto = ep_ultima2.codigo_proyecto ORDER BY ep_ultima2.fecha_termino_real DESC LIMIT 1)
WHERE p.esta_cerrado = 1
AND i.es_defecto = 1
AND tc.resultado = 'Exitoso'
AND ep_ultima.fecha_termino_real BETWEEN '2015-01-01' AND '2016-12-31';
I would have thought it obvious that you're not going to get the expected output from an aggregate query without a GROUP BY (which suggests you're not really in a position to evaluate any advice given here effectively).
You've not said how the states of your data are represented in the database - so I'm having to make a lot of guesses based on SQL which is clearly very wrong. And I don't speak spanish/portugese or whatever your native language is.
It looks like you are inferring that a defect exists if the primary key of the defects table is null. Primary keys cannot be null. The only way this would make any sort of sense (BTW it still won't give you the answer you're looking for) is to do a LEFT JOIN rather than an INNER JOIN.
But even then a simple COUNT() will consider null cases (no record in source table) as 1 record in the output set.
Then you've got the problem that you will have the product of defects and test cases in your output - consider the case where you have no defects, but 2 tests cases (1,2) - the result of an outer joiun will be:
defect test
------ ----
null 1
null 2
If you just count the rows, you'll get 2 defects in your output.
Taking a simpler schema, this demonstrates the 2 methods for getting the values - note that they have very different performance characteristics.
SELECT project.id
, dilv.defects
, (SELECT COUNT(*)
FROM test_cases) AS tests
FROM project
LEFT JOIN ( SELECT project_id, COUNT(*) AS defects
FROM defect_table
GROUP BY project_id) AS dilv
ON project.id=dilv.project_id
I have a database with a table for details of ponies, another for details of contacts (owners and breeders), and then several other small tables for parameters (colours, counties, area codes, etc.). To give me a list of existing pony profiles, with their various details given, i use the following query:
SELECT *
FROM profiles
INNER JOIN prm_breedgender
ON profiles.ProfileGenderID = prm_breedgender.BreedGenderID
LEFT JOIN contacts
ON profiles.ProfileOwnerID = contacts.ContactID
INNER JOIN prm_breedcolour
ON profiles.ProfileAdultColourID = prm_breedcolour.BreedColourID
ORDER BY profiles.ProfileYearOfBirth ASC $limit
In the above sample, the 'profiles' table is my primary table (holding the Ponies info), 'contacts' is second in importance holding as it does the owner and breeder info. The lesser parameter tables can be identified by their prm_ prefix. The above query works fine, but i want to do more.
The first big issue is that I wish to GROUP the results by gender: Stallions, Mares, Geldings... I used << GROUP BY prm_breedgender.BreedGender >> or << GROUP BY ProfileBreedGenderID >> before my ORDER BY line, but than only returns two results from all my available profiles. I have read up on this, and apparantly need to reorganise my query to accomodate GROUP within my primary SELECT clause. How to do this however, gets me verrrrrrry confused. Step by step help here would be fantabulous.
As a further note on the above - You may have noticed the $limit var at the end of my query. This is for pagination, a feature I want to keep. I shouldn't think that's an issue however.
My secondary issue is more of an organisational one. You can see where I have pulled my Owner information from the contacts table here:
LEFT JOIN contacts
ON profiles.ProfileOwnerID = contacts.ContactID
I could add another stipulation:
AND profiles.ProfileBreederID = contacts.ContactID
with the intention of being able to list a pony's Owner and Breeder, where info on either is available. I'm not sure how to echo out this info though, as $row['ContactName'] could apply in either the capacity of owner OR breeder.
Is this a case of simply running two queries rather than one? Assigning a variable $foo to the first run of the query, then just run another separate query altogether and assign $bar to those results? Or is there a smarter way of doing it all in the one query (e.g. $row['ContactName']First-iteration, $row['ContactName']Second-iteration)? Advice here would be much appreciated.
And That's it! I've tried to be as clear as possible, and do really appreciate any help or advice at all you can give. Thanks in advance.
##########################################################################EDIT
My query currently stands as an amalgam of that provided by Cularis and Symcbean:
SELECT *
FROM (
profiles
INNER JOIN prm_breedgender
ON profiles.ProfileGenderID = prm_breedgender.BreedGenderID
LEFT JOIN contacts AS owners
ON profiles.ProfileOwnerID = owners.ContactID
INNER JOIN prm_breedcolour
ON profiles.ProfileAdultColourID = prm_breedcolour.BreedColourID
)
LEFT JOIN contacts AS breeders
ON profiles.ProfileBreederID = breeders.ContactID
ORDER BY prm_breedgender.BreedGender ASC, profiles.ProfileYearOfBirth ASC $limit
It works insofar as the results are being arranged as I had hoped: i.e. by age and gender. However, I cannot seem to get the alias' to work in relation to the contacts queries (breeder and owner). No error is displayed, and neither are any Owners or Breeders. Any further clarification on this would be hugely appreciated.
P.s. I dropped the alias given to the final LEFT JOIN by Symcbean's example, as I could not get the resulting ORDER BY statement to work for me - my own fault, I'm certain. Nonetheless, it works now although this may be what is causing the issue with the contacts query.
GROUP in SQL terms means using aggregate functions over a group of entries. I guess what you want is order by gender:
ORDER BY prm_breedgender.BreedGender ASC, profiles.ProfileYearOfBirth ASC $limit
This will output all Stallions, etc. next to each other.
To also get the breeders contact, you need to join with the contacts table again, using an alias:
LEFT JOIN contacts AS owners
ON profiles.ProfileOwnerID = owners.ContactID
LEFT JOIN contacts AS breeders
ON profiles.ProfileBreederID = breeders.ContactID
To further expand on what #cularis stated, group by is for aggregations down to the lowest level of "grouping" criteria. For example, and I'm not doing per your specific tables, but you'll see the impact. Say you want to show a page grouped by Breed. Then, a user picks a breed and they can see all entries of that breed.
PonyID ProfileGenderID Breeder
1 1 1
2 1 1
3 2 2
4 3 3
5 1 2
6 1 3
7 2 3
Assuming your Gender table is a lookup where ex:
BreedGenderID Description
1 Stallion
2 Mare
3 Geldings
SELECT *
FROM profiles
INNER JOIN prm_breedgender
ON profiles.ProfileGenderID = prm_breedgender.BreedGenderID
select
BG.Description,
count(*) as CountPerBreed
from
Profiles P
join prm_BreedGender BG
on p.ProfileGenderID = BG.BreedGenderID
group by
BG.Description
order by
BG.Description
would result in something like (counts are only coincidentally sequential)
Description CountPerBreed
Geldings 1
Mare 2
Stallion 4
change the "order by" clause to "order by CountsPerBreed Desc" (for descending) and you would get
Description CountPerBreed
Stallion 4
Mare 2
Geldings 1
To expand, if you wanted the aggregations to be broken down per breeder... It is a best practice to group by all things that are NOT AGGREGATES (such as MIN(), MAX(), AVG(), COUNT(), SUM(), etc)
select
BG.Description,
BR.BreaderName,
count(*) as CountPerBreed
from
Profiles P
join prm_BreedGender BG
on p.ProfileGenderID = BG.BreedGenderID
join Breeders BR
on p.Breeder = BR.BreaderID
group by
BG.Description,
BR.BreaderName
order by
BG.Description
would result in something like (counts are only coincidentally sequential)
Description BreaderName CountPerBreed
Geldings Bill 1
Mare John 1
Mare Sally 1
Stallion George 2
Stallion Tom 1
Stallion Wayne 1
As you can see, the more granularity you provide to the group by, the aggregation per that level is smaller.
Your join conditions otherwise are obviously understood from what you've provided. Hopefully this sample clearly provides what the querying process will do. Your group by does not have to be the same as the final order... its just common to see so someone looking at the results is not trying to guess how the data was organized.
In your sample, you had an order by the birth year. When doing an aggregation, you will never have the specific birth year of a single pony to so order by... UNLESS.... You included the YEAR( ProfileYearOfBirth ) as BirthYear as a column, and included that WITH your group by... Such as having 100 ponies 1 yr old and 37 at 2 yrs old of a given breed.
It would have been helpful if you'd provided details of the table structure and approximate numbers of rows. Also using '*' for a SELECT is a messy practice - and will cause you problems later (see below).
What version of MySQL is this?
apparantly need to reorganise my query to accomodate GROUP within my primary SELECT clause
Not necessarily since v4 (? IIRC), you could just wrap your query in a consolidating select (but move the limit into the outer select:
SELECT ProfileGenderID, COUNT(*)
FROM (
[your query without the LIMIT]
) ilv
GROUP BY ProfileGenderID
LIMIT $limit;
(note you can't ORDER BY ilv.ProfileYearOfBirth since it is not a selected column / group by expression)
How many records/columns do you have in prm_breedgender? Is it just Stallions, Mares, Geldings...? Do you think this list is likely to change? Do you have ponies with multiple genders? I suspect that this domain would be better represented by an enum in the profiles table.
with the intention of being able to list a pony's Owner and Breeder,
Using the code you suggest, you'll only get returned instances where the owner and breeder are the same! You need to add a second instance of the contacts table with a different alias to get them all, e.g.
SELECT *
FROM (
SELECT *
FROM profiles
INNER JOIN prm_breedgender
ON profiles.ProfileGenderID = prm_breedgender.BreedGenderID
LEFT JOIN contacts ownerContact
ON profiles.ProfileOwnerID = ownerContact.ContactID
INNER JOIN prm_breedcolour
ON profiles.ProfileAdultColourID = prm_breedcolour.BreedColourID
) ilv LEFT JOIN contacts breederContact
ON ilv.ProfileBreederID = breederContact.ContactID
ORDER BY ilv.ProfileYearOfBirth ASC $limit
Explain SQL (in phpmyadmin) of a query that is taking more than 5 seconds is giving me the above. I read that we can study the Explain SQL to optimize a query. Can anyone tell if this Explain SQL telling anything as such?
Thanks guys.
Edit:
The query itself:
SELECT
a.`depart` , a.user,
m.civ, m.prenom, m.nom,
CAST( GROUP_CONCAT( DISTINCT concat( c.id, '~', c.prenom, ' ', c.nom ) ) AS char ) AS coordinateur,
z.dr
FROM `0_activite` AS a
JOIN `0_member` AS m ON a.user = m.id
LEFT JOIN `0_depart` AS d ON ( m.depart = d.depart AND d.rank = 'mod' AND d.user_sec =2 )
LEFT JOIN `0_member` AS c ON d.user_id = c.id
LEFT JOIN `zone_base` AS z ON m.depart = z.deprt_num
GROUP BY a.user
Edit 2:
Structures of the two tables a and d. Top: a and bottom: d
Edit 3:
What I want in this query?
I first want to get the value of 'depart' and 'user' (which is an id) from the table 0_activite. Next, I want to get name of the person (civ, prenom and name) from 0_member whose id I am getting from 0_activite via 'user', by matching 0_activite.user with 0_member.id. Here depart is short of department which is also an id.
So at this point, I have depart, id, civ, nom and prenom of a person from two tables, 0_activite and 0_member.
Next, I want to know which dr is related with this depart, and this I get from zone_base. The value of depart is same in both 0_activite and 0_member.
Then comes the trickier part. A person from 0_member can be associated with multiple departs and this is stored in 0_depart. Also, every user has a level, one of what is 'mod', stands for moderator. Now I want to get all the people who are moderators in the depart from where the first user is, and then get those moderaor's name from 0_member again. I also have a variable user_sec, but this is probably less important in this context, though I cannot overlook it.
This is what makes the query a tricky one. 0_member is storing id, name of users, + one depart, 0_depart is storing all departs of users, one line for each depart, and 0_activite is storing some other stuffs and I want to relate those through userid of 0_activite and the rest.
Hope I have been clear. If I am not, please let me know and I will try again to edit this post.
Many many thanks again.
Aside from the few answers provided by the others here, it might help to better understand the "what do I want" from the query. As you've accepted a rather recent answer from me in another of your questions, you have filters applied by department information.
Your query is doing a LEFT join at the Department table by rank = 'mod' and user_sec = 2. Is your overall intent to show ALL records in the 0_activite table REGARDLESS of a valid join to the 0_Depart table... and if there IS a match to the 0_Depart table, you only care about the 'mod' and 2 values?
If you only care about those people specifically associated with the 0_depart with 'mod' and 2 conditions, I would reverse the query starting with THIS table first, then join to the rest.
Having keys on tables via relationship or criteria is always a performance benefit (vs not having the indexes).
Start your query with whatever would be your smallest set FIRST, then join to other tables.
From clarification in your question... I would start with the inner-most... Who it is and what departments are they associated with... THEN get the moderators (from department where condition)... Then get actual moderator's name info... and finally out to your zone_base for the dr based on the department of the MODERATOR...
select STRAIGHT_JOIN
DeptPerMember.*
Moderator.Civ as ModCiv,
Moderator.Prenom as ModPrenom,
Moderator.Nom as ModNom,
z.dr
from
( select
m.ID,
m.Depart,
m.Civ,
m.Prenom,
m.Nom
from
0_Activite as a
join 0_member m
on a.User = m.ID
join 0_Depart as d
on m.depart = d.depart ) DeptPerMember
join 0_Depart as DeptForMod
on DeptPerMember.Depart = DeptForMod.Depart
and DeptForMod.rank = 'mod'
and DeptForMod.user_sec = 2
join 0_Member as Moderator
on DeptForMod.user_id = Moderator.ID
join zone_base z
on Moderator.depart = z.deprt_num
Notice how I tier'd the query to get each part and joined to the next and next and next. I'm building the chain based on the results of the previous with clear "alias" references for clarification of content. Now, you can get whatever respective elements from any of the levels via their distinct "alias" references...
The output from EXPLAIN is showing us that the first and third tables listed (a & d) are not having any indexes utilised by the database engine in executing this query. The key column is NULL for both - which is a shame since both are 'large' tables (OK, they're not really large, but compared to the rest of the tables they're the big 'uns).
Judging from the query, an index on user on 0_activite and an index on (depart, rank, user_sec) on 0_depart would go some way to improving performance.
you can see that columns key and key_len are null this means its not using any key in the possible_keys column. So table a and d are both scanning all rows. (check larger numbers in rows column. you want this smaller).
To deal with 0_depart:
Make sure you have a key on (d.depart, d.rank,d.user_sec) which are part of the join of 0_depart.
To deal with 0_activite:
I'm not positive but a GROUP column should be indexed too so you need a key on a.user
I have a query that can be fast or slow depending on how many records I'm fetching. Here's a table showing the number in my LIMIT clause and the corresponding time it takes to execute the query and fetch the results:
LIMIT | Seconds (Duration/Fetch)
------+-------------------------
10 | 0.030/ 0.0
100 | 0.062/ 0.0
1000 | 1.700/ 0.8
10000 | 25.000/100.0
As you can see, it's fine up to at least 1,000 but 10,000 is really slow, mostly due to a high fetch time. I don't understand why the growth of the fetch time isn't linear but I am grabbing over 200 columns from over 70 tables, so the fact that the result set takes a long time to fetch is not a surprise.
What I'm fetching, by the way, is data on all the accounts at a certain bank. The bank I'm dealing with has about 160,000 accounts so I ultimately need to fetch 160,000 rows from the database.
It's obviously not going to be feasible to try to fetch 160,000 rows at once (at least not unless I can somehow dramatically optimize my query). It seems to me that the biggest chunk I can reasonably grab is 1,000 rows, so I wrote a script that would run the query over and over with a SELECT INTO OUTFILE, limit and offset. Then, at the end, I take all the CSV files I dumped and cat them together. It works but it's slow. It takes hours. I have the script running right now and it's only dumped 43,000 rows in about an hour.
Should I attack this problem at the query optimization level or does the long fetch time suggest I should focus elsewhere? What would you recommend I do?
If you want to see the query you can see it here.
The answer is going to greatly depend on what you're doing with the data. Querying 215 columns through 29 joins will never be quick for non-trivial record sizes.
If you're trying to display 160,000 records to the user, you should page the results and only fetch one page at a time. This will keep the result set small enough that even a relatively inefficient query will return quickly. In this case, you will also want to examine just how much data the user needs in order to select or manipulate the data. Chances are good that you can pare it down to a handful of fields and some aggregates (count, sum, etc) that will let the user make an informed decision about which records they want to work with. Use LIMIT with an offset to pull single pages of arbitrary size.
If you need to export the data for reporting purposes, ensure that you are only pulling the exact data that the report needs. Eliminate joins where possible and use subqueries where you need an aggregate of child data. You'll want to tune/add indexes for the frequently used joins and criteria. In the case of your provided query, ib.id and the myriad of foreign keys you're joining through. You can leave off boolean columns because there are not enough distinct values to form a meaningful index.
Regardless of what you're trying to accomplish, removing some of the joins and columns will inherently speed up your processing. The amount of heavy lifting that MySQL needs to do to fill that query is your main stumbling block.
I've restructured your query to hopefully offer significant performance improvement time. By using the STRAIGHT_JOIN tells MySQL to do in the order you've stated (or I've adjusted here). The inner-most, first query "PreQuery" alias STARTS at your criteria of the import bundle and generic import, to the account import to the account... By pre-applying the WHERE clause there (and as you would test, add your LIMIT CLAUSE HERE) you are pre-joining these tables and getting them right out of the way before wasting any time trying to get the customers, address, etc other information going. In the query, I've adjusted the join/left joins to better show the relationship of the underlying linked tables (primarily for anyone else reading in).
As another person noted, what I've done in the PREQUERY could be a basis of "Account.ID" records in a master pre-query list used to go through and page-available. I would be curious to the performance of this to your existing especially at the 10,000 limit range.
The PREQUERY gets unique elements (including the Account ID used downstream, bank, month, year and category), so those tables don't have to be rejoined in the rest of the joining process.
SELECT STRAIGHT_JOIN
PreQuery.*,
customer.customer_number,
customer.name,
customer.has_bad_address,
address.line1,
address.line2,
address.city,
state.name,
address.zip,
po_box.line1,
po_box.line2,
po_box.city,
po_state.name,
po_box.zip,
customer.date_of_birth,
northway_account.cffna,
northway_account.cfinsc,
customer.deceased,
customer.social_security_number,
customer.has_internet_banking,
customer.safe_deposit_box,
account.has_bill_pay,
account.has_e_statement,
branch.number,
northway_product.code,
macatawa_product.code,
account.account_number,
account.available_line,
view_macatawa_atm_card.number,
view_macatawa_debit_card.number,
uc.code use_class,
account.open_date,
account.balance,
account.affinion,
northway_account.ytdsc,
northway_account.ytdodf,
northway_account.ytdnsf,
northway_account.rtckcy,
northway_account.rtckwy,
northway_account.odwvey,
northway_account.ytdscw,
northway_account.feeytd,
customer.do_not_mail,
northway_account.aledq1,
northway_account.aledq2,
northway_account.aledq3,
northway_account.aledq4,
northway_account.acolq1,
northway_account.acolq2,
northway_account.acolq3,
northway_account.acolq4,
o.officer_number,
northway_account.avg_bal_1,
northway_account.avg_bal_2,
northway_account.avg_bal_3,
account.maturity_date,
account.interest_rate,
northway_account.asslc,
northway_account.paidlc,
northway_account.lnuchg,
northway_account.ytdlc,
northway_account.extfee,
northway_account.penamt,
northway_account.cdytdwaive,
northway_account.cdterm,
northway_account.cdtcod,
account.date_of_last_statement,
northway_account.statement_cycle,
northway_account.cfna1,
northway_account.cfna2,
northway_account.cfna3,
northway_account.cfna4,
northway_account.cfcity,
northway_account.cfstate,
northway_account.cfzip,
northway_account.actype,
northway_account.sccode,
macatawa_account.account_type_code,
macatawa_account.account_type_code_description,
macatawa_account.advance_code,
macatawa_account.amount_last_advance,
macatawa_account.amount_last_payment,
macatawa_account.available_credit,
macatawa_account.balance_last_statement,
macatawa_account.billing_day,
macatawa_account.birthday_3,
macatawa_account.birthday_name_2,
macatawa_account.ceiling_rate,
macatawa_account.class_code,
macatawa_account.classified_doubtful,
macatawa_account.classified_loss,
macatawa_account.classified_special,
macatawa_account.classified_substandard,
macatawa_account.closed_account_flag,
macatawa_account.closing_balance,
macatawa_account.compounding_code,
macatawa_account.cost_center_full,
macatawa_account.cytd_aggregate_balance,
macatawa_account.cytd_amount_of_advances,
macatawa_account.cytd_amount_of_payments,
macatawa_account.cytd_average_balance,
macatawa_account.cytd_average_principal_balance,
macatawa_account.cytd_interest_paid,
macatawa_account.cytd_number_items_nsf,
macatawa_account.cytd_number_of_advanes,
macatawa_account.cytd_number_of_payments,
macatawa_account.cytd_number_times_od,
macatawa_account.cytd_other_charges,
macatawa_account.cytd_other_charges_waived,
macatawa_account.cytd_reporting_points,
macatawa_account.cytd_service_charge,
macatawa_account.cytd_service_charge_waived,
macatawa_account.date_closed,
macatawa_account.date_last_activity,
macatawa_account.date_last_advance,
macatawa_account.date_last_payment,
macatawa_account.date_paid_off,
macatawa_account.ddl_code,
macatawa_account.deposit_rate_index,
macatawa_account.employee_officer_director_full_desc,
macatawa_account.floor_rate,
macatawa_account.handling_code,
macatawa_account.how_paid_code,
macatawa_account.interest_frequency,
macatawa_account.ira_plan,
macatawa_account.load_rate_code,
macatawa_account.loan_rate_code,
macatawa_account.loan_rating_code,
macatawa_account.loan_rating_code_1_full_desc,
macatawa_account.loan_rating_code_2_full_desc,
macatawa_account.loan_rating_code_3_full_desc,
macatawa_account.loan_to_value_ratio,
macatawa_account.maximum_credit,
macatawa_account.miscellaneous_code_full_desc,
macatawa_account.months_to_maturity,
macatawa_account.msa_code,
macatawa_account.mtd_agg_available_balance,
macatawa_account.naics_code,
macatawa_account.name_2,
macatawa_account.name_3,
macatawa_account.name_line,
macatawa_account.name_line_2,
macatawa_account.name_line_3,
macatawa_account.name_line_1,
macatawa_account.net_payoff,
macatawa_account.opened_by_responsibility_code_full,
macatawa_account.original_issue_date,
macatawa_account.original_maturity_date,
macatawa_account.original_note_amount,
macatawa_account.original_note_date,
macatawa_account.original_prepaid_fees,
macatawa_account.participation_placed_code,
macatawa_account.participation_priority_code,
macatawa_account.pay_to_account,
macatawa_account.payment_code,
macatawa_account.payoff_principal_balance,
macatawa_account.percent_participated_code,
macatawa_account.pmtd_number_deposit_type_1,
macatawa_account.pmtd_number_deposit_type_2,
macatawa_account.pmtd_number_deposit_type_3,
macatawa_account.pmtd_number_type_1,
macatawa_account.pmtd_number_type_2,
macatawa_account.pmtd_number_type_6,
macatawa_account.pmtd_number_type_8,
macatawa_account.pmtd_number_type_9,
macatawa_account.principal,
macatawa_account.purpose_code,
macatawa_account.purpose_code_full_desc,
macatawa_account.pytd_number_of_items_nsf,
macatawa_account.pytd_number_of_times_od,
macatawa_account.rate_adjuster,
macatawa_account.rate_over_split,
macatawa_account.rate_under_split,
macatawa_account.renewal_code,
macatawa_account.renewal_date,
macatawa_account.responsibility_code_full,
macatawa_account.secured_unsecured_code,
macatawa_account.short_first_name_1,
macatawa_account.short_first_name_2,
macatawa_account.short_first_name_3,
macatawa_account.short_last_name_1,
macatawa_account.short_last_name_2,
macatawa_account.short_last_name_3,
macatawa_account.statement_cycle,
macatawa_account.statement_rate,
macatawa_account.status_code,
macatawa_account.tax_id_number_name_2,
macatawa_account.tax_id_number_name_3,
macatawa_account.teller_alert_1,
macatawa_account.teller_alert_2,
macatawa_account.teller_alert_3,
macatawa_account.term,
macatawa_account.term_code,
macatawa_account.times_past_due_01_29,
macatawa_account.times_past_due_01_to_29_days,
macatawa_account.times_past_due_30_59,
macatawa_account.times_past_due_30_to_59_days,
macatawa_account.times_past_due_60_89,
macatawa_account.times_past_due_60_to_89_days,
macatawa_account.times_past_due_over_90,
macatawa_account.times_past_due_over_90_days,
macatawa_account.tin_code_name_1,
macatawa_account.tin_code_name,
macatawa_account.tin_code_name_2,
macatawa_account.tin_code_name_3,
macatawa_account.total_amount_past_due,
macatawa_account.waiver_od_charge,
macatawa_account.waiver_od_charge_description,
macatawa_account.waiver_service_charge_code,
macatawa_account.waiver_transfer_advance_fee,
macatawa_account.short_first_name,
macatawa_account.short_last_name
FROM
( SELECT STRAIGHT_JOIN DISTINCT
b.name bank,
ib.YEAR,
ib.MONTH,
ip.category,
Account.ID
FROM import_bundle ib
JOIN generic_import gi ON ib.id = gi.import_bundle_id
JOIN account_import AI ON gi.id = ai.generic_import_id
JOIN Account ON AI.ID = account.account_import_id
JOIN import_profile ip ON gi.import_profile_id = ip.id
JOIN bank b ib.Bank_ID = b.id
WHERE
IB.ID = 95
AND IB.Active = 1
AND GI.Active = 1
LIMIT 1000 ) PreQuery
JOIN Account on PreQuery.ID = Account.ID
JOIN Customer on Account.Customer_ID = Customer.ID
JOIN Officer on Account.Officer_ID = Officer.ID
LEFT JOIN branch ON Account.branch_id = branch.id
LEFT JOIN cd_type ON account.cd_type_id = cd_type.id
LEFT JOIN use_class uc ON account.use_class_id = uc.id
LEFT JOIN account_type at ON account.account_type_id = at.id
LEFT JOIN northway_account ON account.id = northway_account.account_id
LEFT JOIN macatawa_account ON account.id = macatawa_account.account_id
LEFT JOIN view_macatawa_debit_card ON account.id = view_macatawa_debit_card.account_id
LEFT JOIN view_macatawa_atm_card ON account.id = view_macatawa_atm_card.account_id
LEFT JOIN original_address OA ON Account.ID = OA.account_id
JOIN Account_Address AA ON Account.ID = AA.account_id
JOIN address ON AA.address_id = address.id
JOIN state ON address.state_id = state.id
LEFT JOIN Account_po_box APB ON Account.ID = APB.account_id
LEFT JOIN address po_box ON APB.address_id = po_box.id
LEFT JOIN state po_state ON po_box.state_id = po_state.id
LEFT JOIN Account_macatawa_product amp ON account.id = amp.account_id
LEFT JOIN macatawa_product ON amp.macatawa_product_id = macatawa_product.id
LEFT JOIN product_type pt ON macatawa_product.product_type_id = pt.id
LEFT JOIN harte_hanks_service_category hhsc ON macatawa_product.harte_hanks_service_category_id = hhsc.id
LEFT JOIN core_file_type cft ON macatawa_product.core_file_type_id = cft.id
LEFT JOIN Account_northway_product anp ON account.id = anp.account_id
LEFT JOIN northway_product ON anp.northway_product_id = northway_product.id
The non-linear increase in fetch time is likely the result of key buffers filling up, and probably other memory related issues as well. You should both optimize the query using EXPLAIN to maximize use of indexes, and tune your MySQL server settings.