We have recently migrated a PHP web application to a different hosting. Along with that, the database was also successfully copied over to the new mysql server. The earlier version of database server was mysql5.6 and the new hosting has mariadb 10 installed. Below is one example query which is on old hosting is taking around 0.5s and on the new hosting with mariadb, it is taking around 40s which is very long. This long time is chocking up the hosting server.
SELECT
IB.IndicatorId,
IB.IndicatorTitle,
IB.IndicatorCode,
IC.IndicatorCategoryName,
AA.Answer,
GROUP_CONCAT(AI.`Answer`) AS CombinedAnswers
FROM answer AS A
LEFT JOIN answers_answer AS AA ON AA.AnswerId = A.AnswerId
LEFT JOIN activity_sections_indicators AS ASI ON ASI.SectionIndicatorId = AA.SectionIndicatorId
LEFT JOIN indicators_bank AS IB ON IB.IndicatorId = ASI.IndicatorId
LEFT JOIN indicator_categories AS IC ON IC.IndicatorCategoryId = IB.IndicatorCategoryId
LEFT JOIN answers_items AS AI ON AI.AnswerAnswerId = AA.AnswerAnswerId
WHERE A.ProgramActivityId IN (103,104,105)
AND A.Code = 'Newsas263'
GROUP BY IB.IndicatorId
Further more the number of records in tables is as follows
answer : 5355
answers_answer : 845209
activity_sections_indicators : 2866
indicators_bank : 1175
indicator_categories : 17
answers_items : 934347
My understanding is that there is something missing in mariadb server configuration though the query can be optimized as well. I have VPS and can change configuration for mariadb but I am not sure what to do from here.
Let me know if some more detail is needed. I really appreciate your help in this regard.
MySQL and MariaDB diverged somewhat significantly with 5.6. It would be hard to spot the particular thing that is different.
The EXPLAIN SELECT ... output might help.
These composite and/or covering indexes may help on both servers:
IB: INDEX(IndicatorId, IndicatorTitle, IndicatorCode, IndicatorCategoryId)
IC: INDEX(IndicatorCategoryId, IndicatorCategoryName)
AA: INDEX(AnswerId, Answer, SectionIndicatorId, AnswerAnswerId)
AI: INDEX(AnswerAnswerId, Answer)
A: INDEX(Code, ProgramActivityId, AnswerId)
ASI: INDEX(SectionIndicatorId, IndicatorId)
When adding a composite index, DROP index(es) with the same leading columns.
That is, when you have both INDEX(a) and INDEX(a,b), toss the former.
The index on "A" may be the most important.
Are any of the tables "many-to-many"?
Which version of MariaDB? There is already 10.0 through 10.7.
I was able to resolve the issue.
Thanks for the recommendation/suggestion by "Ergest Basha", "Markus Zeller" and "Rick James".
I ran the "Explain Select ..." on the query and the "Execution Plan" has some differences. I was able to mitigate those differences and the query on new server now works like normal.
Though I was not clearly able to find out the differences but I just re copied the whole database and the issue was resolved.
Related
I have been tasked with converting some Crystal SQL queries into QlikView and am having trouble deciphering the SQL code as it has been a decade since I last played with this. I'm modelling the data in MySQL workbench prior to importing it into QlikView.
I have posted the code bellow for the sake of completeness although I realize a lot of it is surplus.
The issue I am having is I don't know and cant seem to work out how do duplicate these queries in MySQL workbench as I don't understand what these code segments are doing:
={?APS: ITEM1.ST_Prodcode} and ={?FKE: ITEM1_1.ST_Prodcode} etc.
The above mentioned code appears to me to be calling the next query. Am I right in thinking this?
Pricing
SELECT
`stock_management1`.`st_prodcode`,
`stock_management1`.`st_sdesc`,
`stock_management1`.`st_mstockist`,
`stock_management1`.`APS_rol`,
`stock_management1`.`APS_eoq`,
`stock_management1`.`APS_ms`
FROM
`pricing`.`stock_management`
`stock_management1`
WHERE
(`stock_management1`.`st_mstockist`='BRA'
OR
`stock_management1`.`st_mstockist`='FCS'
OR
`stock_management1`.`st_mstockist`='FKE')
AND
`stock_management1`.`APS_ms`>0
AND
(`stock_management1`.`st_prodcode`>='A'
AND
`stock_management1`.`st_prodcode`<='WZZZZZ999')
EXTERNAL JOIN
stock_management1.st_prodcode={?APS: ITEM1.ST_Prodcode}
AND
stock_management1.st_prodcode={?FCS: stocktake1.S_ProdCode}
AND
stock_management1.st_prodcode={?CENTRAL: Command.mv_PRODCODE}
APS
SELECT `ITEM1`.`ST_SOH`, `ITEM1`.`ST_Prodcode`
FROM `aps`.`ITEM` `ITEM1`
WHERE `ITEM1`.`ST_Prodcode`={?pricing: stock_management1.st_prodcode}
EXTERNAL JOIN ITEM1.ST_Prodcode={?FKE: ITEM1_1.ST_Prodcode}
FKE
SELECT `ITEM1_1`.`ST_SOH`, `ITEM1_1`.`ST_Prodcode`
FROM `iewkelvin`.`ITEM` `ITEM1_1`
WHERE `ITEM1_1`.`ST_Prodcode`={?APS: ITEM1.ST_Prodcode}
EXTERNAL JOIN ITEM1_1.ST_Prodcode={?FCS: ITEM1_2.ST_Prodcode}
Thanks for comments imran & ralfbecher.
I found that the {? implies WHERE followed by the Database: table.field
Simple as that really.
I have a query running fine on my current PC that is my developpement PC but not in production environment.
The two environement have the same material configuration.
The setting of the database are the same two.
On dev the query execute in less than 30 sec, but when I try it on production even after 2 hours I don't get result.
This is the only running query on the server at the moment of execution.
Do you have any idea of why this is happening ?
SELECT td.td_date,
td.td_number,
td.td_status,
td.td_open_datetime,
td.td_update_datetime,
Min(ack.ackup) ackUpdate
FROM t_ticketdossier_td td
INNER JOIN (SELECT a.td_number,
a.td_update_datetime ackUp
FROM t_ticketdossier_td a
WHERE a.td_status = 'Acknowledged') ack
ON td.td_number = ack.td_number
AND td.td_update_datetime <= ack.ackup
WHERE td.td_number IN (SELECT td_number
FROM t_ticketdossier_td base
WHERE Date(base.td_date) = #date
AND base.td_status = 'Acknowledged'
AND base.td_scope IS NULL
AND base.td_subcategory NOT LIKE '%RDV%'
AND base.td_product_type NOT LIKE '%RDV%'
GROUP BY td_number)
AND td.td_status = 'Affected'
GROUP BY td.td_date,
td.td_number,
td.td_status,
td.td_update_datetime
Explain from the dev server.
Explain from the prod server.
Sorry for the links, I don't have enougth reputation to post the image directly.
Cheers,
Maxime.
From what I see in the EXPLAIN statements, it seems that your production database misses quite a few indexes.
For example:
TD_STATUS_IDX,TD_NUMBER_IDX
Try adding these and run the EXPLAIN again.
I am using the RJDBC package to connect to a MySQL (Maria DB) database in R on a Windows 7 machine and I am trying a statement like
select a as b
from table
but the column will always continue to be named "a" in the data frame.
This works normally with RODBC and RMySQL but doesn't work with RJDBC. Unfortunately, I have to use RJDBC as this is the only package that has no problem with the encoding of chinese, hebrew and so on letters (set names and so on don't seem to work with RODBC and RMySQL).
Has anybody experienced this problem?
I have run into the same frustrating issue. Sometimes the AS keyword would have its intended effect, but other times it wouldn't. I was unable to identify the conditions to make it work correctly.
Short Answer: (Thanks to Simon Urbanek (package maintainer for RJDBC), Yev, and Sebastien! See the Long Answer.) One thing that you may try is to open your JDBC connection using ?useOldAliasMetadataBehavior=true in your connection string. Example:
drv <- JDBC("com.mysql.jdbc.Driver", "C:/JDBC/mysql-connector-java-5.1.18-bin.jar", identifier.quote="`")
conn <- dbConnect(drv, "jdbc:mysql://server/schema?useOldAliasMetadataBehavior=true", "username", "password")
query <- "SELECT `a` AS `b` FROM table"
result <- dbGetQuery(conn, query)
dbDisconnect(conn)
This ended up working for me! See more details, including caveats, in the Long Answer.
Long Answer: I tried all sorts of stuff, including making views, changing queries, using JOIN statements, NOT using JOIN statements, using ORDER BY and GROUP BY statements, etc. I was never able to figure out why some of my queries were able to rename columns and others weren't.
I contacted the package maintainer (Simon Urbanek.) Here is what he said:
In the vast majority of cases this is an issue in the JBDC driver, because there is really not much RJDBC can do other than to call the driver.
He then recommended that I make sure I had the most recent JDBC driver for MySQL. I did have the most recent version. However, it got me thinking "maybe it IS a bug with the JDBC driver." So, I searched Google for: mysql jdbc driver bug alias.
The top result for this query was an entry at bugs.mysql.com. Yev, using MySQL 5.1.22, says that when he upgraded from driver version 5.0.4 to 5.1.5, his column aliases stopped working. Asked if it was a bug.
Sebastien replied, "No, it's not a bug! It's a documented change of behavior in all subsequent versions of the driver." and suggested using ?useOldAliasMetadataBehavior=true, citing documentation for the JDBC driver.
Caveat Lector: The documentation for the JDBC driver states that
useColumnNamesInFindColumn is preferred over useOldAliasMetadataBehavior unless you need the specific behavior that it provides with respect to ResultSetMetadata.
I haven't had the time to fully research what this means. In other words, I don't know what all of the ramifications are of using useOldAliasMetadataBehavior=true are. Use at your own risk. Does someone else have more information?
I don't know RJDBC, but in some cases when it is necessary to give permanent aliases to columns without renaming them, you can use VIEWs
CREATE OR REPLACE VIEW v_table AS
SELECT a AS b
FROM table
... and then ...
SELECT b FROM v_table
There is a separate function in the ResultSetMetaData interface for retrieving the column label vs the column name:
String getColumnLabel(int column) throws SQLException;
Gets the designated column's suggested title for use in printouts and
displays. The suggested title is usually specified by the SQL AS
clause. If a SQL AS is not specified, the value returned
fromgetColumnLabel will be the same as the value returned by the
getColumnName method.
Using getColumnLabel should resolve this issue (if not, check that your JDBC driver is following this spec).
e.g.
ResultSetMetaData rsmd = rs.getMetaData();
int columnCount = rsmd.getColumnCount();
while(rs.next()) {
for (int i = 1; i < columnCount + 1; i++) {
String label = rsmd.getColumnLabel(i);
System.out.println(rs.getString(label));
}
}
This is the work around we use for R and SAP HANA via RJDBC:
names(result)[1]<-"b"
It's not the nicest work around, but since Aaron's solution does work for us, we went with this "solution".
I found that code in the bottom executes more than 10 minutes on my server. The server itself is quite good and I cannot find any suitable explanation for that.
I am using Joomla 1.6.3 the data is migrated from Joomla 1.5.23 using jUpgrade and MySQL client version is 5.1.45.
SELECT a.*,COUNT(DISTINCT m1.id) AS count_published,COUNT(DISTINCT m2.id) AS count_unpublished,COUNT(DISTINCT m3.id) AS count_trashed
FROM `j16_menu_types` AS a
LEFT JOIN `j16_menu` AS m1 ON m1.menutype = a.menutype AND m1.published = 1
LEFT JOIN `j16_menu` AS m2 ON m2.menutype = a.menutype AND m2.published = 0
LEFT JOIN `j16_menu` AS m3 ON m3.menutype = a.menutype AND m3.published = -2
GROUP BY a.id
ORDER BY a.id asc;
I would be very pleased if someone could help me since I am in big trouble :)
P.s. I have downloaded db and checked it on my computer - still the same, execution time is terrible. Is there any way to solve this problem? Or maybe to remove this sql part without significant changes in administration of joomla?
Well it was done quite faster on my PC but still the result is far from one which would satisfy my.
EDIT
Well, I've found a reported bug of this problem. http://joomlacode.org/gf/project/joomla/tracker/?action=TrackerItemEdit&tracker_item_id=24868
And the solution is:
CREATE INDEX idx_menu_published ON j16_menu (published);
However, I am not sure how this will affect administration. I there is anyone who could briefly tell how this part works and should I edit Joomla core code or just use the above code on mysql once. I am wondering if I should index table everytime when I edit menu.
A database / table Index is just a kind of yellow pages if you don't mind the comparison. It updates automatically. If this Index fixes your problem there is nothing else you need to do or do again. You can't break anything either - so just give it a try.
I am trying to troubleshoot a particular 'report' that gets generated on a PHP application running on a VOIP platform (billing related). The person that asked me to do this stated "it worked before" ;)
MySQL Error thrown:
1111: Invalid use of group function
Crazy thing is, it 'worked before' but stopped after awhile, they cannot place what event took place that might have created this bug. One thought is maybe MYSQL was upgraded? I googled the error, and can't seem to find a clear answer.
At first glance do you guys see anything wrong here in the SQL?
SELECT C.ResourceGroupID AS CustomerID, CS.CustomerName, SUM(C.IN_RndDuration/60) AS Minutes, SUM(CASE WHEN (C.OUT_Duration>0 AND C.IN_RndDuration>0) THEN 1 ELSE 0 END) AS Successfull, count(0) AS Attempts
FROM ws_call_current AS C
LEFT JOIN customer_rg AS V_RG ON
V_RG.RGId=C.OtherResourceGroupID AND V_RG.RateTableId IS NOT NULL AND V_RG.Direction='O'
LEFT JOIN customer AS V ON V.CustomerId=V_RG.CustomerId
LEFT JOIN customer_rg AS CS_RG ON CS_RG.RGId=C.ResourceGroupID AND CS_RG.RateTableId IS NOT NULL AND CS_RG.Direction='I'
LEFT JOIN customer AS CS ON CS.CustomerId=CS_RG.CustomerId
WHERE DATE_FORMAT(DATE_ADD(C.SeizeDate, INTERVAL TIME_TO_SEC(C.SeizeTime) SECOND), '%Y-%m-%d %H:%i:00') BETWEEN '2010-10-19 00:00:00' AND '2010-10-19 23:59:59'
AND C.SeizeDate BETWEEN '2010-10-19' AND '2010-10-19'
GROUP BY CS.CustomerName
ORDER BY SUM(C.IN_RndDuration/60) DESC
This is written for MYSQL 3/4 I believe, not yet sure. Just wanted to get some feedback.
From google results I find some people had success with fixing this by modifying the query to be a Having instead of a where?
Not sure. Anything look odd below?
Try including C.ResourceGroupID in your group by statement. You need to include all non-aggregate items in your select in your group by.
You shouldn't need a having in the code above.
I'm not sure how it could have worked before...
What I found after review is that the cause of this was simply a MySQL discrepancy. Initially the MySQL code was written for 5.0.51a, and the running MySQL version (it was somehow 'attempted' to be upgraded, was 5.0.21).
This version 'downgrade' was the cause of the failed SQL. I honestly don't know what the cause was other than we installed a new version of MySQL to just do a test (on a remote box) and it solved the issue (pointing from primary box to backup MySQL server.
Solution for this error: Version compatibility