When number of pIDs in query is very big - I don't receive expected results. More exactly - I receive nothing.
Note that the same query works OK on another server.
I think (but I'm not sure) that problem may be in MySQL configuration but don't know how to solve it.
Does anyone know how can I resolve this issue?
Query looks like this:
SELECT `filtID`,`pID` FROM (
(SELECT `filtID`,`pID` FROM `ftox_params_prod_links`
LEFT JOIN `ftox_params_values` USING(`fvID`)
WHERE (`filtID` IN (1,4,5,14,15,302,303,304,388,389,390)))
UNION (SELECT `filtID`,`pID` FROM `ftox_params_prod_values`
WHERE (`filtID` IN (1,4,5,14,15,302,303,304,388,389,390)))) AS `_T_`
WHERE (`pID` IN (173,174,175,176,177,178,179,180,181,182,183,
184,185,186,187,188,189,190,191,192,193,194,195,196,197,
198,199,200,201,202,203,204,205,206,207,208,209, ...
....................................................
...................Very much pIDs..................
....................................................
)) ORDER BY `filtID` ASC
MySQL has a limit for a query size, defined by max_allowed_packet configuration value - in my.conf.
Or the result of the query may be too big - try to increase memory_limit php variable in php.ini.
Example:
memory_limit = 256M
Related
I am facing a performance issue that has manifested itself after a migration from mysql 5.5 to mariadb 10.2.13 . The sizing of both machines including configs is the same. We have noticed that a certain select statement suddenly takes ~20 secs to execute instead of 0.5 sec. The data is purely read from memory as there is zero read IO to be measured during execution. Changing the optimizer_switch on the mariadb to the settings of mysql5.5 did not change the execution plan.
The explain for the select on the mariadb looks as follows:
The explain for the select on the mysql5.5 :
Select-Statement:
SELECT acmh.OBJ_VERSION,acmh.peannr,acmh.pzsrnr,xuts.ptitle,
acmh.pcategory,acmh.psalutatio,acmh.plastname,acmh.pfirstname,
acmh.pcompanyna,syoq.paddress1,syoq.pzip,syoq.pcity,syoq.pphone1,
syoq.pphone2,syoq.pphone3,syoq.pphone4,syoq.pemail,acmh.piscompany,
xuts.id
FROM catalog1.tr_table_acmh acmh,catalog1.tr_table_syoq syoq,
catalog1.tr_table_xuts xuts,catalog1.tr_table_wdvi link0,
catalog1.tr_table_wdvi acl0
WHERE acmh.id=syoq.id
AND acmh.id=xuts.id
AND syoq.id=xuts.id
AND xuts.OBJ_TYPE IN (1557)
AND ( (xuts.id=link0.pchild
AND link0.pparent='xkgrrslqkeaaaaendrxa'
AND link0.pname='folder')
and 1 = 1 )
AND xuts.pvcurrent=1
AND (acl0.pchild=xuts.pacl
AND acl0.pparent='xkgrrswxbjaaaaaaabip'
AND acl0.pvalue>=20
)
order by acmh.plastname asc,
acmh.pfirstname asc,
acmh.pcompanyna asc
LIMIT 10 OFFSET 0;
Is there any switch that im missing that could enforce a different behaviour on the mariadb?
Updating the statistics for the query optimizer (ANALYZE TABLE on the 3 tables involved) fixed the issue in this case.
The new execution plan for the SELECT now looks like this:
These may help performance:
link0: INDEX(pchild, pparent, pname)
link0: INDEX(pparent, pname, pchild)
xuts: INDEX(pvcurrent, OBJ_TYPE, id)
acl0: INDEX(pchild, pparent, pvalue)
You seem to have several tables that are 1:1; is this true? If so, why?
What is the datatype of id? It seems to be a strange length: 20.
I have the following query
SELECT t.res, IF(t.res=0, "zero", "more than zero")
FROM (
SELECT table.*, IF (RAND()<=0.2,1, IF (RAND()<=0.4,2, IF (RAND()<=0.6,3,0))) AS res
FROM table LIMIT 20) t
which returns something like this:
That's exactly what you would expect. However, as soon as I remove the LIMIT 20 I receive highly unexpected results (there are more rows returned than 20, I cut it off to make it easier to read):
SELECT t.res, IF(t.res=0, "zero", "more than zero")
FROM (
SELECT table.*, IF (RAND()<=0.2,1, IF (RAND()<=0.4,2, IF (RAND()<=0.6,3,0))) AS res
FROM table) t
Side notes:
I'm using MySQL 5.7.18-15-log and this is a highly abstracted example (real query is much more difficult).
I'm trying to understand what is happening. I do not need answers that offer work arounds without any explanations why the original version is not working. Thank you.
Update:
Instead of using LIMIT, GROUP BY id also works in the first case.
Update 2:
As requested by zerkms, I added t.res = 0 and t.res + 1 to the second example
The problem is caused by a change introduced in MySQL 5.7 on how derived tables in (sub)queries are treated.
Basically, in order to optimize performance, some subqueries are executed at different times and / or multiple times leading to unexpected results when your subquery returns non-deterministic results (like in my case with RAND()).
There are two easy (and likewise ugly) workarounds to get MySQL to "materialize" (aka return deterministic results) these subqueries: Use LIMIT <high number> or GROUP BY id both of which force MySQL to materialize the subquery and return the expected results.
The last option is turn off derived_merge in the optimizer_switch variable: derived_merge=off (make sure to leave all the other parameters as they are).
Further readings:
https://mysqlserverteam.com/derived-tables-in-mysql-5-7/
Subquery's rand() column re-evaluated for every repeated selection in MySQL 5.7/8.0 vs MySQL 5.6
I tried to run the following query in my Cloud VPS cPanel phpMyadmin
SELECT bankcode, bankname
FROM newbankdetails
WHERE
type='DB' AND
bankcode IN ( SELECT drawingbankname
FROM dd1
WHERE entrydate BETWEEN '01/04/2014' AND '30/04/2014'
AND paymentmode='DD'
AND state='TAMIL NADU' )
It takes very very very long time nearly 5 to 6 hours and says out of time or keeps running for days and don't show any error also don't show results.
Whereas it works perfectly in my local machine xampp
It happens only when working on large size tables like 12GB and around like that
How to speed it up and display the result as it displays instantly in localhost xampp
I think, replacing IN with a LEFT JOIN can make more sense, like this:
SELECT
bankcode, bankname
FROM
newbankdetails nd
LEFT JOIN
dd1 ON nd.bankcode = dd1.drawingbankname
WHERE
nd.type='DB'
AND
dd1.entrydate BETWEEN '01/04/2014' AND '30/04/2014'
AND
dd1.paymentmode = 'DD'
AND
dd1.state = 'TAMIL NADU'
I can't test it, But I think with dd1.paymentmode = 'DD' automatically removes rows of dd1.paymentmode IS NULL because of using LEFT JOIN.
First you should use EXPLAIN QUERY (https://dev.mysql.com/doc/refman/5.0/en/using-explain.html)
That will give you information about what is slow.
Your problem seems to be related to the size of the table. On your localhost you may have a small table, but in production it is very large. Also put an index on the first table on nd.bankcode.
You should try to remove the subselect, and use a JOIN instead.
Also, you should put indexes on the columns you use for search.
Put an index on drawingbankname, entrydate, paymentmode, state. Put also an index on nd.backcode .
Column drawingbankname is varchar, you should convert it to fixed char.
Remove the subselect and use JOIN.
Use EXPLAIN after the changes to see progress.
Following this post: POST ABOUT CONCAT
My problem is that i have many rows CONCAT into one row. For example if i have 10 rows with string around 50 chars, my query will show me only 6-7 of that rows or something like that.
I searech in stack and google and i found that i can change CONCAT max length by command: SET group_concat_max_len := ##max_allowed_packet. What i am doing wrong?
EDIT:
When i SHOW VARIABLES LIKE 'group_concat_max_len' it's shows me 1024.
Mysql version 5.0.96-log. Tables type: MyISAM. Looks like it dont have any limits, i try to select simple varchar with 2000 chars, and it looks fine.
I have 3 tables: 1st - Item with ItemID, 2nd - Descriptionpack with ItemID and DescriptionID, 3rd Description with DescriptionID.
Select
DISTINCT Item.ItemID as item
,GROUP_CONCAT(Description.DescriptionID) AS description
From Item
LEFT OUTER JOIN descriptionpack
on Item.ItemID=descriptionpack.ItemID
LEFT OUTER JOIN description
on descriptionpack.descriptionID=description.descriptionID
GROUP BY item
EDIT2: I think i found the problem, i said my problem to my provider and they answer me this:
I reviewed your question with our hosting team. You wouldn't be able
to change the global settings for that and other variables. However,
you should be able to set that variable on a per session basis by
setting it first, before other queries. Hope that helps.
So now the problem is, how to do that.
Presumably you're using GROUP_CONCAT(), not simple CONCAT().
The default value of the group_concat_max_len is 1024, which is a pretty small limit if you're building up big long concatenations.
To change it, use this command. I've set the length in this example to 100,000. You could set it to anything you need.
SET SESSION group_concat_max_len = 100000;
The usual value for max_allowed_packet is one megabyte, which is likely more than you need.
group_concat_max_len itself has an effectively unlimited size. It's limited only by the unsigned word length of the platform: 2^32-1 on a 32-bit platform and 2^64-1 on a 64-bit platform.
If that still isn't enough for your application, it's time to take #eggyal's suggestion and rethink your approach.
You need change group_concat_max_len default value in the bellow config file
**my.cnf file(Linux) and my.ini file(windows)**
[mysqld]//Add this line group_concat_max_len=15000 under mysqld section
group_concat_max_len=15000
Note: After change is done You need to restart your MySQL server.
my.cnf file path in linux :
1. /etc/my.cnf
2./etc/mysql/my.cnf
3.$MYSQL_HOME/my.cnf
4.[datadir]/my.cnf
5.~/.my.cnf
I have a query running fine on my current PC that is my developpement PC but not in production environment.
The two environement have the same material configuration.
The setting of the database are the same two.
On dev the query execute in less than 30 sec, but when I try it on production even after 2 hours I don't get result.
This is the only running query on the server at the moment of execution.
Do you have any idea of why this is happening ?
SELECT td.td_date,
td.td_number,
td.td_status,
td.td_open_datetime,
td.td_update_datetime,
Min(ack.ackup) ackUpdate
FROM t_ticketdossier_td td
INNER JOIN (SELECT a.td_number,
a.td_update_datetime ackUp
FROM t_ticketdossier_td a
WHERE a.td_status = 'Acknowledged') ack
ON td.td_number = ack.td_number
AND td.td_update_datetime <= ack.ackup
WHERE td.td_number IN (SELECT td_number
FROM t_ticketdossier_td base
WHERE Date(base.td_date) = #date
AND base.td_status = 'Acknowledged'
AND base.td_scope IS NULL
AND base.td_subcategory NOT LIKE '%RDV%'
AND base.td_product_type NOT LIKE '%RDV%'
GROUP BY td_number)
AND td.td_status = 'Affected'
GROUP BY td.td_date,
td.td_number,
td.td_status,
td.td_update_datetime
Explain from the dev server.
Explain from the prod server.
Sorry for the links, I don't have enougth reputation to post the image directly.
Cheers,
Maxime.
From what I see in the EXPLAIN statements, it seems that your production database misses quite a few indexes.
For example:
TD_STATUS_IDX,TD_NUMBER_IDX
Try adding these and run the EXPLAIN again.