When you process a SELECT through phpmyadmin, behind the scenes, it will sometimes add a LIMIT 0,30, and/or it'll throw a SQL_CALC_FOUND_ROWS into the SELECT so it can tell me how many results there would have been without the LIMIT.
Unfortunately, adding the SQL_CALC_FOUND_ROWS sometimes requires much more processing than I was expecting (i.e. more than if I had ran my original untainted query).
Is there a global config option to disable phpmyadmin's modification(s) of my queries?
What tricks can I use on a per-query basis to prevent phpmyadmin's modification(s)?
A quick check of the PHPMyAdmin source code says there isn't one.
However, if you look in the file sql.php, and find the else statement labelled // n o t " j u s t b r o w s i n g ". Replace the code between there and // end else "just browsing" with something like $unlim_num_rows = 1000000; you'll prevent it doing its counting query, while still being able to browse.
(you'll have to repeat this each time you update PMA, which you should be doing regularly since its security reputation is not great, to say the least)
Related
I'm using the Wordnet SQL database from here: http://wnsqlbuilder.sourceforge.net
It's all built fine and users with appropriate privileges have been set.
I'm trying to find synonyms of words and have tried to use the two example statements at the bottom of this page: http://wnsqlbuilder.sourceforge.net/sql-links.html
SELECT synsetid,dest.lemma,SUBSTRING(src.definition FROM 1 FOR 60) FROM wordsXsensesXsynsets AS src INNER JOIN wordsXsensesXsynsets AS dest USING(synsetid) WHERE src.lemma = 'option' AND dest.lemma <> 'option'
SELECT synsetid,lemma,SUBSTRING(definition FROM 1 FOR 60) FROM wordsXsensesXsynsets WHERE synsetid IN ( SELECT synsetid FROM wordsXsensesXsynsets WHERE lemma = 'option') AND lemma <> 'option' ORDER BY synsetid
However, they never complete. At least not in any reasonable amount of time and I have had to cancel all of the queries. All other queries seem to work find and when I break up the second SQL example, I can get the individual parts to work and complete in reasonable times (about 0.40 seconds)
When I try and run the full statement however, the MySQL command line client just hangs.
Is there a problem with this syntax? What is causing it to take so long?
EDIT:
Output of "EXPLAIN SELECT ..."
Output of "EXPLAIN EXTENDED ...; SHOW WARNINGS;"
I did more digging and looking into the various statements used and found the problem was in the IN command.
MySQL repeats the statement for every single row in the database. This is the cause of the hang, as it had to run through hundreds of thousands of records.
My remedy to this was to split the command into two separate database calls first getting the synsets, and then dynamically creating a bound SQL string to look for the words in the synsets.
I'm setting up a project in which I want to utilize CakePHP's Translate Behavior.
Everything seemed to work fine until I reached 10 fields that I wanted it to translate. The Translate Behavior creates an INNER JOIN for each field it's trying to retrieve - which I believe is what's causing this error (only happens with 10+):
SQLSTATE[42000]: Syntax error or access violation: 1104 The SELECT
would examine more than MAX_JOIN_SIZE rows; check your WHERE and use
SET SQL_BIG_SELECTS=1 or SET MAX_JOIN_SIZE=# if the SELECT is okay
Two questions:
1) I tried fixing it by running the two listed SQL commands, but still no luck - how can I get it to work?
2) Is it ideal/ok/acceptable to have 10-20+ translated fields if it's going to create an INNER JOIN for each one? Should I re-think using this behavior and maybe create something on my own?
Did You try SET OPTION SQL_BIG_SELECTS = 1 more on https://stackoverflow.com/a/950576/182823
1:
Its a mysql security option, what you can override. Use this code in before filter in your app controller to avoid this error.
function beforeFilter() {
$this->{$this->modelClass}->query('SET SQL_BIG_SELECTS=1');
...
}
2:
It's ok to join table multipled, but some advice:
always use indexes! without table cell indexes the query can be slow
if you have a lot of translatable datadata in lot of table use separated tables to translate, like if you have Contents and Posts use Content_I18n and Post_I18n
http://book.cakephp.org/2.0/en/core-libraries/behaviors/translate.html -> Multiple Translation Models
you can override Translate Behavior. I'm using SmoothTranslate Behavior
http://bakery.cakephp.org/articles/sky_l3ppard/2010/01/05/smoothtranslate-to-make-smooth-translations
I think the best way to use Translate behavior or extend it, do not write own, its good enough in CakePhp. And it will better in Cake 3. (i hope)
I have an application that is defining some SQL code:
mySql = "SELECT
sq.question,
qs.title,
sq.id as question_id,
sq.type,
qs.id as option_id,
sri.title as rankTitle,
sri.id as rankId,
sfi.title as formTitle,
sfi.id as formId,
sq.sub_type,
sq.sort_order
FROM survey_questions as sq
LEFT JOIN question_suboptions as qs
ON sq.id = qs.question_id
LEFT JOIN survey_rankingitems as sri
ON sq.id = sri.question_id
LEFT JOIN survey_formitems as sfi
ON sq.id = sfi.question_id
WHERE sq.survey_id = #{#surveyId}
ORDER BY sq.sort_order"
I would like to paste this code (everything between the double quotes) in the MySQL command line, change the one parameter and execute it, but I have run into an issue where for every line above MySQL will display:
Display all 1450 possibilities? (y or n)
And then 1450 different available commands. If I remove all linebreaks and tabs then I can paste in, but that is time consuming and a pain. Is there a way that I can simply paste in the above code, edit it and then execute it as a single unit?
This is the default mysql (CLI) behavior each time the user presses the Tab key (mysql uses the underlying readline or EditLine libraries (not on Windows)).
By default, when the user requests to use a database, mysql reads tables and fields definitions. Then, pressing the Tab key makes mysql conveniently offers completion of the current input with the known tables and fields.
However, pasting some text into mysql that contains TAB characters (\t or 0x09) triggers the same behavior - even though no Tab key was actually pressed from the keyboard. And this can be annoying.
Two options given to mysql can prevent that behavior, though. My favorite is --disable-auto-rehash. The other one is --quiet or -q.
--disable-auto-rehash to prevent database, table, and column name completion (which are not read from the database, use the rehash command if later on you need completion). Commands history is kept, though (retrieved via the ↑ and ↓ keys for instance). Which is convenient.
--quick or -q which makes mysql not using the history file and no completion (does not read the database definitions).
On Linux one may add an alias in .bashrc to use --disable-auto-rehash automatically
alias mysql2='mysql --disable-auto-rehash'
Perhaps you could save the statement to a text file myTest.sql, then use the MySQL command source myTest.sql to run it? You could then tweak the SQL in the file, save the changes, and run it again.
You need to remove the line breaks and tabs. The double tab is causing it to display the Display all 1450 possibilities? (y or n) and the line breaks are causing it to execute early.
If it's PHP, write a little script to strip it for you:
echo (preg_replace("/\s+/", " ", $string));
Or something similar for other languages.
Breaking not so bad's answer explained the cause of this problem really well.
From the question:
If I remove all linebreaks and tabs then I can paste in, but that is time consuming and a pain.
In my case, I just replaced the tabs with spaces and I was able to paste the query just fine. The MySQL console doesn't seem to care about the newlines, just the tabs.
As a way to prevent this, most editors have a setting that will insert tabs instead of spaces when you press the Tab key. I normally have my IDEs configured this way, but in this instance it was a query I'd copied from MySQL workbench. Conveniently, it also has a setting to use spaces instead of tabs:
Edit > Preferences > General Editors > check Tab key inserts spaces instead of tabs > OK
We have a lot of queries
select * from tbl_message
that get stuck on the state "Writing to net". The table has 98k rows.
The thing is... we aren't even executing any query like that from our application, so I guess the question is:
What might be generating the query?
...and why does it get stuck on the state "writing to net"
I feel stupid asking this question, but I'm 99,99% sure that our application is not executing a query like that to our database... we are however executing a couple of querys to that table using WHERE statement:
SELECT Count(*) as StrCount FROM tbl_message WHERE m_to=1960412 AND m_restid=948
SELECT Count(m_id) AS NrUnreadMail FROM tbl_message WHERE m_to=2019422 AND m_restid=440 AND m_read=1
SELECT * FROM tbl_message WHERE m_to=2036390 AND m_restid=994 ORDER BY m_id DESC
I have searched our application several times for select * from tbl_message but haven't found anything... But still our query-log on our mysql server is full of Select * from tbl_message queries
Since applications don't magically generate queries as they like, I think that it's rather likely that there's a misstake somewhere in your application that's causing this. Here's a few suggestions that you can use to track it down. I'm guessing that your using PHP, since your using MySQL, so I'll use that for my examples.
Try adding comments in front of all your queries in the application, like this:
$sqlSelect = "/* file.php, class::method() */";
$sqlSelect .= "SELECT * FROM foo ";
$sqlSelect .= "WHERE criteria";
The comment will show up in your query log. If you're using some kind database api wrapper, you could potentially add these messages automatically:
function query($sql)
{
$backtrace = debug_backtrace();
// The function that executed the query
$prev = $backtrace[1];
$newSql = sprintf("/* %s */ ", $prev["function"]);
$newSql .= $sql;
mysql_query($newSql) or handle_error();
}
In case you're not using a wrapper, but rather executing the queries directly, you could use the runkit extension and the function runkit_function_rename to rename mysql_query (or whatever you're using) and intercept the queries.
There are (at least) two data retrieval modes for mysql. With the c api you either call mysql_store_result() or mysql_use_result().
mysql_store_result() returns when all result data is transferred from the MySQL server to your process' memory, i.e. no data has to be transferred for further calls to mysql_fetch_row().
However, by using mysql_use_result() each record has to be fetched individually if and when mysql_fetch_row() is called. If your application does some computing that takes longer than the time period specified in net_write_timeout between two calls to mysql_fetch_row() the MySQL server considers your connection to be timed out.
Temporarily enable the query log by putting
log=
into your my.cnf file, restart mysql and watch the query log for those mystery queries (you don't have to give the log a name, it'll assume one from the host value).
Is it particularly bad to have a very, very large SQL query with lots of (potentially redundant) WHERE clauses?
For example, here's a query I've generated from my web application with everything turned off, which should be the largest possible query for this program to generate:
SELECT *
FROM 4e_magic_items
INNER JOIN 4e_magic_item_levels
ON 4e_magic_items.id = 4e_magic_item_levels.itemid
INNER JOIN 4e_monster_sources
ON 4e_magic_items.source = 4e_monster_sources.id
WHERE (itemlevel BETWEEN 1 AND 30)
AND source!=16 AND source!=2 AND source!=5
AND source!=13 AND source!=15 AND source!=3
AND source!=4 AND source!=12 AND source!=7
AND source!=14 AND source!=11 AND source!=10
AND source!=8 AND source!=1 AND source!=6
AND source!=9 AND type!='Arms' AND type!='Feet'
AND type!='Hands' AND type!='Head'
AND type!='Neck' AND type!='Orb'
AND type!='Potion' AND type!='Ring'
AND type!='Rod' AND type!='Staff'
AND type!='Symbol' AND type!='Waist'
AND type!='Wand' AND type!='Wondrous Item'
AND type!='Alchemical Item' AND type!='Elixir'
AND type!='Reagent' AND type!='Whetstone'
AND type!='Other Consumable' AND type!='Companion'
AND type!='Mount' AND (type!='Armor' OR (false ))
AND (type!='Weapon' OR (false ))
ORDER BY type ASC, itemlevel ASC, name ASC
It seems to work well enough, but it's also not particularly high traffic (a few hundred hits a day or so), and I wonder if it would be worth the effort to try and optimize the queries to remove redundancies and such.
Reading your query makes me want to play an RPG.
This is definitely not too long. As long as they are well formatted, I'd say a practical limit is about 100 lines. After that, you're better off breaking subqueries into views just to keep your eyes from crossing.
I've worked with some queries that are 1000+ lines, and that's hard to debug.
By the way, may I suggest a reformatted version? This is mostly to demonstrate the importance of formatting; I trust this will be easier to understand.
select *
from
4e_magic_items mi
,4e_magic_item_levels mil
,4e_monster_sources ms
where mi.id = mil.itemid
and mi.source = ms.id
and itemlevel between 1 and 30
and source not in(16,2,5,13,15,3,4,12,7,14,11,10,8,1,6,9)
and type not in(
'Arms' ,'Feet' ,'Hands' ,'Head' ,'Neck' ,'Orb' ,
'Potion' ,'Ring' ,'Rod' ,'Staff' ,'Symbol' ,'Waist' ,
'Wand' ,'Wondrous Item' ,'Alchemical Item' ,'Elixir' ,
'Reagent' ,'Whetstone' ,'Other Consumable' ,'Companion' ,
'Mount'
)
and ((type != 'Armor') or (false))
and ((type != 'Weapon') or (false))
order by
type asc
,itemlevel asc
,name asc
/*
Some thoughts:
==============
0 - Formatting really matters, in SQL even more than most languages.
1 - consider selecting only the columns you need, not "*"
2 - use of table aliases makes it short & clear ("MI", "MIL" in my example)
3 - joins in the WHERE clause will un-clutter your FROM clause
4 - use NOT IN for long lists
5 - logically, the last two lines can be added to the "type not in" section.
I'm not sure why you have the "or false", but I'll assume some good reason
and leave them here.
*/
Default MySQL 5.0 server limitation is "1MB", configurable up to 1GB.
This is configured via the max_allowed_packet setting on both client and server, and the effective limitation is the lessor of the two.
Caveats:
It's likely that this "packet" limitation does not map directly to characters in a SQL statement. Surely you want to take into account character encoding within the client, some packet metadata, etc.)
SELECT ##global.max_allowed_packet
this is the only real limit it's adjustable on a server so there is no real straight answer
From a practical perspective, I generally consider any SELECT that ends up taking more than 10 lines to write (putting each clause/condition on a separate line) to be too long to easily maintain. At this point, it should probably be done as a stored procedure of some sort, or I should try to find a better way to express the same concept--possibly by creating an intermediate table to capture some relationship I seem to be frequently querying.
Your mileage may vary, and there are some exceptionally long queries that have a good reason to be. But my rule of thumb is 10 lines.
Example (mildly improper SQL):
SELECT x, y, z
FROM a, b
WHERE fiz = 1
AND foo = 2
AND a.x = b.y
AND b.z IN (SELECT q, r, s, t
FROM c, d, e
WHERE c.q = d.r
AND d.s = e.t
AND c.gar IS NOT NULL)
ORDER BY b.gonk
This is probably too large; optimizing, however, would depend largely on context.
Just remember, the longer and more complex the query, the harder it's going to be to maintain.
Most databases support stored procedures to avoid this issue. If your code is fast enough to execute and easy to read, you don't want to have to change it in order to get the compile time down.
An alternative is to use prepared statements so you get the hit only once per client connection and then pass in only the parameters for each call
I'm assuming you mean by 'turned off' that a field doesn't have a value?
Instead of checking if something is not this, and it's also not that etc. can't you just check if the field is null? Or set the field to 'off', and check if type or whatever equals 'off'.