MySQL set variable hack - mysql

Consider the query:
SELECT #xxx As thevar FROM (SELECT #xxx:=1) as temp
Will the resulting query always yield 1, even if there are many other similar queries running at the same time (or as close to the same time as they can get) that are setting #xxx to a different integer.
Also, what exactly is the scope of #xxx?
I'm trying to determine if this is a safe (albeit hacky) way to insure a variable is set to the value I want for the life a specific query.
These queries are run against a single instance of MySQL 5.5 and may be done on the same connection or different connections to the database.

From what I have just tested, it seems the variable is seen only on the current connection. Or more precisely on the connection on which it is set.
I believe your SQL Statement has the same effect as the SET statement for User Defined Variables.
The documentation for which states that:
User-defined variables are connection-specific. That is, a user variable defined by one client cannot be seen or used by other clients. All variables for a given client connection are automatically freed when that client exits.
Thus if the other queries are using there own separate connections, than they will have their own copies of #xxx, and will not interfere with each other. And your hack will work as expected. But if same connection is used and the order of query execution cannot be guaranteed then, other queries can change #xxx and affect the subsequent queries.
Regarding if this always return 1:
SELECT #xxx As thevar FROM (SELECT #xxx:=1) as temp
It should, BUT if using the same connection there is an outlier chance that a Seperate Thread using the same connection executes SET #xxx=2 just after the execution of the subquery and before outer select. Please correct me if I'm wrong about ATOMICITY of SELECT here.

Related

Qt SQL `nextResult` function for MySQL Server 8.0: delayed execution per result set?

We are currently doing a lot of small queries. We execute a query, read the results, and then execute the next one. Since network requests cost a lot of time, this ping-ponging gets slow very fast.
This is why we want to do multiple queries at once, sending all data that the SQL server must know to it, and only retrieving one result (consisting of multiple result sets).
We found that Qt 5.14.1's QSqlQuery has the nextResult() function, but in the documentation (link) it says:
Some databases may execute all statements at once while others may delay the execution until the result set is actually accessed, [...].
MY QUESTION:
So, does MySql Server 8.0 delay the execution until the result set is actually accessed? If this is the case, then we still have a ping-pong for every query right? Which would be very slow still.
P.S. Our current solution to just have 1 ping-pong is to union different result sets (resulting in kind of a block diagonal matrix) with lots and lots of null values), and this question is meant to find a better way to do this.

How to prematurely finish mysql_use_result() / mysql_fetch_row()?

I am in the process of writing my first C client for MySQL 5.5 and have stumbled across the following page in the documentation. Nearly at the end, it states (bold emphasis mine, italic emphasis not mine):
An advantage of mysql_use_result() is [...]. Disadvantages are that
[...]. Furthermore, you must retrieve all the rows even if you
determine in mid-retrieval that you've found the information you were
looking for.
The last sentence is not clear to me.
1) What happens if I don't follow that line?
2) I think that there actually must be a way to prematurely end fetching rows if I decide that I have enough information (otherwise, this whole thing wouldn't make much sense in my eyes).
I understand that something bad could happen if I just stop fetching rows and then try to execute the next statement, but isn't there a function like mysql_finish_fetch() or something like that?
And what happens if I call mysql_free_result()? This should free the result even if I haven't fetched all rows yet, so it should be safe to call it in mid-retrieval and continue with whatever I'd like to do. Am I wrong here?
This sounds like an internal threading issue that MySQL exposes to the client. Chalk it up to the various MySQL gotchas. The short of it is that MySQL apparently has a finite number of "searchers" internally, and using mysql_use_result() apparently dedicates one of them to your API request. Further, MySQL apparently has no exposed API call to cancel such a request. The only option is to see the fetch through until the end.
The slightly longer version: internally, MySQL's cursors apparently have a single code path -- I imagine for performance in the common cases. That code path exits only when the cursor finds no more results. When you use the more common mysql_store_result(), MySQL has done this already before returning the result to the application. When you use mysql_use_result(), however, MySQL requires that you do "the dirty work" of iterating the rest of the result set so as to clear the cursor. Fun.
From the documentation:
mysql_use_result() initiates a result set retrieval but does not actually read the result set into the client like mysql_store_result() does. Instead, each row must be retrieved individually by making calls to mysql_fetch_row(). This reads the result of a query directly from the server without storing it in a temporary table or local buffer, which is somewhat faster and uses much less memory than mysql_store_result(). The client allocates memory only for the current row and a communication buffer that may grow up to max_allowed_packet bytes.
On the other hand, you should not use mysql_use_result() for locking reads if you are doing a lot of processing for each row on the client side, or if the output is sent to a screen on which the user may type a ^S (stop scroll). This ties up the server and prevent other threads from updating any tables from which the data is being fetched.
When using mysql_use_result(), you must execute mysql_fetch_row() until a NULL value is returned, otherwise, the unfetched rows are returned as part of the result set for your next query. The C API gives the error Commands out of sync; you can't run this command now if you forget to do this!
So, to actually answer your questions:
1) What happens if I don't follow that line?
The C API will return the error message: Commands out of sync; you can't run this command now
2) I think that there actually must be a way to prematurely end fetching rows if I decide that I have enough information (otherwise, this whole thing wouldn't make much sense in my eyes).
One would think, but no. You must iterate the result set completely.

I'm confused about MySQLdb server side cursor and client cursor

I'm using python-mysql(MySQLdb) to query Mysql server.
There are two cursor modules: one is client cursor, such as:
cursor = db.cursor(MySQLdb.cursors.DictCursor)
Another one is server side cursor,such as:
cursor = db.cursor(MySQLdb.cursors.SSDictCursor)
The doc says Server side cursor means that Mysql would cache some results in mysql server side and then send them out to the client. I'm so confused about this, let's say, if I wanna kill one mysql server I could just use multiple server side cursors and then mysql will be dead because of memory ran out. Furthermore, does server size cursor make any sense? By default Mysql mechanism is that when mysql retrieved one record it would send it out the client immediately. Does make any sense to cache the results and then send them out?
I really don't known which cursor I should use, client cursor or server side cursor?
I'm not the greatest Database Ninja around, but often times things get built into server software that aren't really useful in the general or common cases, but are really, really awesome in that one little corner case.
Nimdil gave you one, but this is another:
http://techualization.blogspot.com/2011/12/retrieving-million-of-rows-from-mysql.html
This person asserts that SScursor is more of an "unbuffered" cursor.
This sort of seems to contradict that:
http://dev.mysql.com/doc/refman/5.7/en/cursor-restrictions.html
Anyway, it sort of seems that the use for Server Side Cursors are when you're dealing with datasets such that your query could overwhelm the client.
I believe MySQL would rather kill your cursor than crash because of few oversized cursors.
You can think of several scenarios when server side cursor makes sense. For example if you have slow network connection and the cursor is big, you can work on some small part of the data you can get quicker, possibly pass it to other system and then fetch some more. This way the overall speed of solution would be greater.
Other scenario that I can think of is when you have quite powerful database server and rather crappy machine under the client - this way in case of big dataset it would be easier for database to hold the whole set for your client while the client can micromanage memory efficiently.
There are possibly many other scenarios. If you think it doesn't make sense, just don't use it. Not all options are for every setup.
A cursor consists out of three parts:
A query
A query result
A pointer to the place until where data has been retrieved.
Depending on the query, the result can either be cached, or be retrieved in parts by the engine:
For example a query result which is usually not cached:
SELECT * FROM sometable;
MySQL (and most other DBMS) will just retrieve a row from the table every time you request a row. It can however use a table lock if you are using InnoDB and ACID compliant transactions in read committed style.
The second scenario is a query from which the result has to be cached:
SELECT * FROM sometable ORDER BY a,b,c;
In this case the MySQL (and again most other DBMS) has to get all the data in the correct order first. For this a temporary table will be created on disk in the #tmp location. This can cause disk full (translated most of the time as out of memory errors) issues and loss of a connection. MySQL however keeps running.

Can you sneak an "insert" into a "select" statement?

Is it possible to sneak an "insert" statement (or anything else that changes the database) into a MySQL "select" statement?
I ask because I'm concerned I've found an injection vulnerability, but it's safeguarded from obvious mayhem like '; drop database; -- by virtue of only being able to run a single statement at a time, no matter how many statements the query has been corrupted to contain. But if the back end is executing something like select bar from foo where param = '$improperly_escaped_input', is there something I can input that will compromise my database?
The vulnerability needs to be corrected, regardless. But if I can show an example of how it can be exploited to screw with the data, fixing it goes way up in the priority queue.
Modification of data is only one aspect of a Sql Injection vulnerability. Even with just read permissions, an attacker can elevate their privileges, or use a Blind Sql Injection attack to scrape every last bit of data out of your database.
I can't think of a way off the top of my head that data would be modified inside a select statement... but, are you sure that you're only able to run a single command at a time?
Regardless, the other attack vectors should be enough of a threat to raise the priority on the issue.
EDIT: Data modification is allowed in MySql sub-queries:
MySQL permits a subquery to refer to a stored function that has
data-modifying side effects such as inserting rows into a table. For
example, if f() inserts rows, the following query can modify data:
SELECT ... WHERE x IN (SELECT f() ...);
This behavior is nonstandard
(not permitted by the SQL standard). In MySQL, it can produce
indeterminate results because f() might be executed a different number
of times for different executions of a given query depending on how
the optimizer chooses to handle it.

MySQL Under High Load, Race Condition?

I am experiencing what appears to be the effects of a race condition in an application I am involved with. The situation is as follows, generally, a page responsible for some heavy application logic is following this format:
Select from test and determine if there are rows already matching a clause.
If a matching row already exists, we terminate here, otherwise we proceed with the application logic
Insert into the test table with values that will match our initial select.
Normally, this works fine and limits the action to a single execution. However, under high load and user-abuse where many requests are intentionally sent simultaneously, MySQL allows many instances of the application logic to run, bypassing the restriction from the select clause.
It seems to actually run something like:
select from test
select from test
select from test
(all of which pass the check)
insert into test
insert into test
insert into test
I believe this is done for efficiency reasons, but it has serious ramifications in the context of my application. I have attempted to use Get_Lock() and Release_Lock() but this does not appear to suffice under high load as the race condition still appears to be present. Transactions are also not a possibility as the application logic is very heavy and all tables involved are not transaction-capable.
To anyone familiar with this behavior, is it possible to turn this type of handling off so that MySQL always processes queries in the order in which they are received? Is there another way to make such queries atomic? Any help with this matter would be appreciated, I can't find much documented about this behavior.
The problem here is that you have, as you surmised, a race condition.
The SELECT and the INSERT need to be one atomic unit.
The way you do this is via transactions. You cannot safely make the SELECT, return to PHP, and assume the SELECT's results will reflect the database state when you make the INSERT.
If well-designed transactions (the correct solution) are as you say not possible - and I still strongly recommend them - you're going to have to make the final INSERT atomically check if its assumptions are still true (such as via an INSERT IF NOT EXISTS, a stored procedure, or catching the INSERT's error in the application). If they aren't, it will abort back to your PHP code, which must start the logic over.
By the way, MySQL likely is executing requests in the order they were received. It's possible with multiple simultaneous connections to receive SELECT A,SELECT B,INSERT A,INSERT B. Thus, the only "solution" would be to only allow one connection at a time - and that will kill your scalability dead.
Personally, I would go about the check another way.
Attempt to insert the row. If it fails, then there was already a row there.
In this manner, you check or a duplicate and insert the new row in a single query, eliminating the possibility of races.