what to do with concurrency-risky tasks? - mysql

I'm looking for a solution for this:
function foo()
{
client 1 executes an update/delete
// client 2 calls foo() and breaks data integrity
client 1 executes an update/delete
}
how do I solve this with mysql? I'm using myisam tables but I'm interested to the innoDB solution as well

Note: This answer assumes that you are InnoDB which allows row level locking instead of MyISAM which requires table locks.
For cases like this, you would use transactions and READ/WRITE locks. The exact details of which you need vary from case to case, and I cannot answer that without knowing you schema and what data integrity you are worried about, so I will give you a general explanation.
A read lock can be acquired on rows which you do not intend to write to, but mustn't change for the duration of your transaction. A write lock can be acquired on rows which you intend to change at some point in the future. A transaction is a sequence of multiple actions which are applied in an all-or-nothing way to the database.
So as an example lets suppose the following:
you have 3 tables: table_A, table_B, table_C
The operation which client 1 is performing makes an update to table_A and then to
table_B.
Client 2 could be updating table any of the tables.
You require some data consistency between all 3 tables.
You would do something like this:
// This makes it so that each operation is not automatically commited (saved)
// It implicitly makes all sequences of operations into transactions
execute("set autocommit=0");
// This gets you some data from table_B and also gets a read lock to prevent that data from changing
result = execute("SELECT * FROM `table_B` WHERE `condition` = 1 LOCK IN SHARE MODE");
// This gets some data from table_C and gets a write lock to prevent the data from changing and allowing you to write to it in the future
result2 = execute("SELECT * FROM `table_C` WHERE `condition` = 1 FOR UPDATE");
// This performs your update to table_A
execute("UPDATE `table_A` SET `value` = 1234 WHERE `condition` = 1");
// This performs your update to table_C
execute("UPDATE `table_C` SET `value` = 4321 WHERE `condition` = 1");
// This saves all of the changes that you made during your transaction and releases all locks
// Note: autocommit is still turned off
execute("COMMIT");
So lets have a more concrete example involving purchasing something. I realize that this could all be done by a single update statement but I am doing it this way to illustrate how to use transactions.
My tables are:
items (id int not null primary key, user_id int not null, item_type int not null)
accounts (user_id int not null primary key, balance int not null)
prices (item_type int not null primary key, price int not null)
limits (item_type int not null primary key, max_count int not null)
Note I am going to skip input sanitation for brevity sake, DO NOT do that for real. (http://xkcd.com/327/)
function purchase(user_id, item_type) {
execute("set autocommit=0");
// I am assuming that price and max_count can be changed but they require consistency with each other hence the read locks
var price = execute("SELECT `price` FROM `prices` WHERE `item_type` = " + item_type + " LOCK IN SHARE MODE")[0].price;
var max_count = execute("SELECT `max_count` FROM `limits` WHERE `item_type` = " + item_type + " LOCK IN SHARE MODE")[0].max_count;
// I need the write lock to prevent double spending
var account = execute("SELECT * FROM `accounts` WHERE `user_id` = " + user_id + " FOR UPDATE")[0];
// I need to guarantee that the user is not over the limit
var count = execute("SELECT count(*) AS `count` FROM `items` WHERE `user_id` = " + user_id + " FOR UPDATE")[0].count;
var new_balance = account.balance - price;
if(count >= max_count || new_balance < 0) {
return false;
}
execute("INSERT INTO `items` (`user_id`, `item_type`) VALUES (" + user_id + ", " + item_type + ")");
execute("UPDATE `accounts` SET `balance` = " + new_balance + " WHERE `user_id` = " + user_id);
execute("COMMIT");
return true;
}
Also it should be noted that you now have to worry about deadlocks, but that is an entirely separate topic.

Related

MySQL: Getting current row ID in update

So I'm working on this trigger that updates an account balance as soon as one of the transaction types logged on it is changed. Here is the query:
BEGIN
IF (OLD.debit != NEW.debit) THEN
UPDATE account,
(SELECT accountid,
sum(accountentry.amountaccountentry) AS total
FROM accountentry
WHERE accountentry.typeid = NEW.idaccounttype
AND accountentry.accountid = account.idaccount
GROUP BY accountentry.accountid) AS s
SET account.balanceaccount = IF(NEW.debit = 1, IF(s.total IS NOT NULL, balanceaccount - s.total * 2, balanceaccount), IF(s.total IS NOT NULL, balanceaccount + s.total * 2, balanceaccount));
END IF;
END
Now, this query does work if I remove the "AND accountentry.accountid = account.idaccount" part. Since I'm in some kind of "sub query", how would I go about retrieving the id of the account I'm currently updating?
Thanks a lot :)
I agree with the goal of keeping redundant data out of the database but I also understand that this is not always practical. In this case, retrieving the balance by aggregating the transactions should be very low cost if appropriate indices are in place. I do not know the details of your specific scenario but, at the very least, you should be very cautious when introducing redundant data. It will almost certainly become inconsistent at some point.
Having this update in a trigger, as opposed to plainly visible in a transaction in your application's code, is a likely source of hard to track bugs in the future.
You have not included enough information (CREATE TABLE statements, sample data, desired outcome) in your question for me to answer with confidence but I will have a go.
Your update query appears to be attempting to handle the scenario where the transaction type for one entry is being changed from debit to credit or vice versa but it does not allow for the amountaccountentry changing at the same time. I do not understand why you are using GROUP BY and SUM when you are only dealing with a change to one accountentry. Surely you could use -
BEGIN
IF (OLD.debit != NEW.debit AND OLD.amountaccountentry = NEW.amountaccountentry) THEN
UPDATE account
SET account.balanceaccount = IF(NEW.debit = 1, IF(NEW.amountaccountentry IS NOT NULL, balanceaccount - (NEW.amountaccountentry * 2), balanceaccount), IF(NEW.amountaccountentry IS NOT NULL, balanceaccount + (NEW.amountaccountentry * 2), balanceaccount))
WHERE idaccount = NEW.accountid;
END IF;
END
or if amountaccountentry could change at the same time -
BEGIN
IF (OLD.debit != NEW.debit) THEN
UPDATE account
SET account.balanceaccount = IF(NEW.debit = 1, IF(NEW.amountaccountentry IS NOT NULL, balanceaccount - (OLD.amountaccountentry + NEW.amountaccountentry), balanceaccount), IF(NEW.amountaccountentry IS NOT NULL, balanceaccount + (OLD.amountaccountentry + NEW.amountaccountentry), balanceaccount))
WHERE idaccount = NEW.accountid;
END IF;
END
If I have completely misunderstood your issue, please update your question with the additional details mentioned above.

UPDATE multiple rows with different values in one query in MySQL

I am trying to understand how to UPDATE multiple rows with different values and I just don't get it. The solution is everywhere but to me it looks difficult to understand.
For instance, three updates into 1 query:
UPDATE table_users
SET cod_user = '622057'
, date = '12082014'
WHERE user_rol = 'student'
AND cod_office = '17389551';
UPDATE table_users
SET cod_user = '2913659'
, date = '12082014'
WHERE user_rol = 'assistant'
AND cod_office = '17389551';
UPDATE table_users
SET cod_user = '6160230'
, date = '12082014'
WHERE user_rol = 'admin'
AND cod_office = '17389551';
I read an example, but I really don't understand how to make the query. i.e:
UPDATE table_to_update
SET cod_user= IF(cod_office = '17389551','622057','2913659','6160230')
,date = IF(cod_office = '17389551','12082014')
WHERE ?? IN (??) ;
I'm not entirely clear how to do the query if there are multiple condition in the WHERE and in the IF condition..any ideas?
You can do it this way:
UPDATE table_users
SET cod_user = (case when user_role = 'student' then '622057'
when user_role = 'assistant' then '2913659'
when user_role = 'admin' then '6160230'
end),
date = '12082014'
WHERE user_role in ('student', 'assistant', 'admin') AND
cod_office = '17389551';
I don't understand your date format. Dates should be stored in the database using native date and time types.
MySQL allows a more readable way to combine multiple updates into a single query. This seems to better fit the scenario you describe, is much easier to read, and avoids those difficult-to-untangle multiple conditions.
INSERT INTO table_users (cod_user, date, user_rol, cod_office)
VALUES
('622057', '12082014', 'student', '17389551'),
('2913659', '12082014', 'assistant','17389551'),
('6160230', '12082014', 'admin', '17389551')
ON DUPLICATE KEY UPDATE
cod_user=VALUES(cod_user), date=VALUES(date)
This assumes that the user_rol, cod_office combination is a primary key. If only one of these is the primary key, then add the other field to the UPDATE list.
If neither of them is a primary key (that seems unlikely) then this approach will always create new records - probably not what is wanted.
However, this approach makes prepared statements easier to build and more concise.
UPDATE table_name
SET cod_user =
CASE
WHEN user_rol = 'student' THEN '622057'
WHEN user_rol = 'assistant' THEN '2913659'
WHEN user_rol = 'admin' THEN '6160230'
END, date = '12082014'
WHERE user_rol IN ('student','assistant','admin')
AND cod_office = '17389551';
You can use a CASE statement to handle multiple if/then scenarios:
UPDATE table_to_update
SET cod_user= CASE WHEN user_rol = 'student' THEN '622057'
WHEN user_rol = 'assistant' THEN '2913659'
WHEN user_rol = 'admin' THEN '6160230'
END
,date = '12082014'
WHERE user_rol IN ('student','assistant','admin')
AND cod_office = '17389551';
To Extend on #Trevedhek answer,
In case the update has to be done with non-unique keys, 4 queries will be need
NOTE: This is not transaction-safe
This can be done using a temp table.
Step 1: Create a temp table keys and the columns you want to update
CREATE TEMPORARY TABLE temp_table_users
(
cod_user varchar(50)
, date varchar(50)
, user_rol varchar(50)
, cod_office varchar(50)
) ENGINE=MEMORY
Step 2: Insert the values into the temp table
Step 3: Update the original table
UPDATE table_users t1
JOIN temp_table_users tt1 using(user_rol,cod_office)
SET
t1.cod_office = tt1.cod_office
t1.date = tt1.date
Step 4: Drop the temp table
In php, you use multi_query method of mysqli instance.
$sql = "SELECT COUNT(*) AS _num FROM test;
INSERT INTO test(id) VALUES (1);
SELECT COUNT(*) AS _num FROM test; ";
$mysqli->multi_query($sql);
comparing result to transaction, insert, case methods in update 30,000 raw.
Transaction: 5.5194580554962
Insert: 0.20669293403625
Case: 16.474853992462
Multi: 0.0412278175354
As you can see, multiple statements query is more efficient than the highest answer.
Just in case if you get error message like this:
PHP Warning: Error while sending SET_OPTION packet
You may need to increase the max_allowed_packet in mysql config file.
UPDATE Table1 SET col1= col2 FROM (SELECT col2, col3 FROM Table2) as newTbl WHERE col4= col3
Here col4 & col1 are in Table1. col2 & col3 are in Table2 I Am trying to update each col1 where col4 = col3 different value for each row
I did it this way:
<update id="updateSettings" parameterType="PushSettings">
<foreach collection="settings" item="setting">
UPDATE push_setting SET status = #{setting.status}
WHERE type = #{setting.type} AND user_id = #{userId};
</foreach>
</update>
where PushSettings is
public class PushSettings {
private List<PushSetting> settings;
private String userId;
}
it works fine

Using mysql_affected_row to prevent race condition

Is it safe to use the following code to prevent race conditions? (key and status fields and mysql_affected_rows are used to implement locking)
$mres = mysql_query("SELECT `values`, `key`, `status`
FROM `test`
WHERE `id` = 1");
$row = mysql_fetch_array($mres);
if($row['status'] != UPDATING) {
$mres = mysql_query("UPDATE `test` SET
`status` = UPDATING,
`key` = `key` + 1
WHERE `id` = 1 AND `key` = ".$row['key']);
if($mres && mysql_affected_rows()) {
//update here safely and then...
mysql_query("UPDATE `test` SET
`status` = NOT_UPDATING,
`key` = `key` + 1
WHERE `id` = 1");
}
}
My test shows that either it is not safe or I should search for a well-hidden mistake in my code.
Table is MyISAM
You should "acquire the lock" first before you retrieve values. Otherwise they may change before you get the lock.
$mres = mysql_query("UPDATE `test` SET
`status` = 'UPDATING'
WHERE `id` = 1 AND `status` = 'NOT_UPDATING'");
if ($mres && mysql_affected_rows()) {
// got the lock
// now select and update
}
id better be a unique field in the db or things may behave very weird
I couldn't see a reason to increment key
notice I quoted the strings 'UPDATING' and 'NOT_UPDATING' in sql
in your code, you should have also checked that $row['status'] had a meaningful value(what if it was false/null?) before comparing to the php constant UPDATING
hopefully you understand enough php to know that php strings should be quoted.
You can check for GET_LOCK and RELEASE_LOCK functions in MySql to simulate row locks.
http://dev.mysql.com/doc/refman/5.1/en/miscellaneous-functions.html#function_get-lock
With this approach you don't need to update rows. Also with mysql_affected_rows() if something goes wrong you may finish with always locked row (for example if you script crash before releasing row by updating it status to NOT_UPDATING). Locks granted with GET_LOCK are released automatically when connection is terminated.

SQL: Auto Increment value during insert

I have the an existing table that for some reason the designer decided to manually control the Primary Key value by storing the last used value in a seperate table (changing the table to use Identity is not an option right now).
I now need to do a mass update to this table as follows:
DECLARE #NeedFieldID int
SET #NeedFieldID = 62034
INSERT INTO T_L_NeedField (NeedID, NeedFieldID, FieldName, Sequence, DisplayAs, FieldPrompt, DataType, ValidOptions, IsRequiredForSale)
(
SELECT
DISTINCT n.NeedID,
#NeedFieldID + 1,
'DetailedOutcome',
999,
'Detailed Outcome',
'Select appropriate reason for a No Sale outcome',
'T',
'Pricing, Appointment Date / Time Not Available, Will Call Back, Declined',
0
FROM T_L_Need n
INNER JOIN T_L_NeedField nf
ON n.NeedID = nf.NeedID
WHERE (n.Need LIKE 'Schedule%' AND n.Disabled = 0)
)
Obviously '#NeedFieldID + 1' doesn't work (just using it to show what I want to do). How can I increment #NeedFieldID as SQL inserts the values for each of the distinct NeedId's? I am using SQL Server 2008.
You want row_number():
DECLARE #NeedFieldID int
SET #NeedFieldID = 62034
INSERT INTO T_L_NeedField (NeedID, NeedFieldID, FieldName, Sequence, DisplayAs, FieldPrompt, DataType, ValidOptions, IsRequiredForSale)
(
SELECT
DISTINCT n.NeedID,
#NeedFieldID + row_number() over (order by n.NeedID),
'DetailedOutcome',
999,
'Detailed Outcome',
'Select appropriate reason for a No Sale outcome',
'T',
'Pricing, Appointment Date / Time Not Available, Will Call Back, Declined',
0
FROM T_L_Need n
INNER JOIN T_L_NeedField nf
ON n.NeedID = nf.NeedID
WHERE (n.Need LIKE 'Schedule%' AND n.Disabled = 0)
)
However, your best bet is to make NeedFieldID an identity column and just let SQL Server do the work for you.

Simple mySQL subqueries

I have a simple problem, but I am new to SQL so please forgive my ignorance.
I have a accounting report that figures out balances oweing and adds up the balances to report to me what the total outstanding are within a specific period. The issue is for every JOB there are many invoices that provide a running total/balance, because of this when my current query adds up the balances it shows me outstanding amounts that are sky high, we have found that the current code is adding the balances of all the invoices.
Example-
If JOB ID 001 has four invoices-
I-001 balance 200,
I-002 balance 100,
I-003 balance 50,
I-004 balance 0.
It will show me that there is $350 outstanding when in fact it is zero.
The solution that I can think of(which I am not sure how to code) are to group the results by job ID and use the MAX feature to select only the higest ID for every JOBID
The problem I am having is that the balances are not saved to the table but recalculated every time they are needed. What can you suggest to show me only the balance from the highest Invoice ID for a particular JOBID
My invoice table has the following columns:
1 ID int(11)
2 ParentID int(11)
3 JOBID varchar(100)
4 DATE date
5 LENSES decimal(10,2)
6 FRAMES decimal(10,2)
7 TAXABLEGOODS decimal(10,2)
8 DISCOUNT decimal(10,2)
9 PREVIOUSBALANCE decimal(10,2)
10 PAYMENT decimal(10,2)
11 PAYMENTTYPE varchar(200)
12 NOTES varchar(255)
13 PMA decimal(10,2)
The current code looks like this:
$pieces = explode("-", $_REQUEST["STARTDATE"]);
$startDate=$pieces[2] . "-" . $pieces[1] . "-" . $pieces[0];
if($_REQUEST["ENDDATE"]==""){
$endDate=0;
}else{
$pieces = explode("-", $_REQUEST["ENDDATE"]);
$endDate = $pieces[2] . "-" . $pieces[1] . "-" . $pieces[0];
}
$result = mysql_query("SELECT * FROM INVOICES WHERE DATE BETWEEN '" . $startDate . "' AND '" . $endDate . "'");
$totalCount = 0;
$total = 0;
$allPayments= 0;
$pmtTypes = Array();
$totalHST = 0;
$outstanding=0;
$payments=0;
while($theRow=mysql_fetch_array($result)){
$allPayments += $theRow["PAYMENT"];
if($theRow["PAYMENTTYPE"] == "") $theRow["PAYMENTTYPE"] = "BLANK";
if(isset($pmtTypes[$theRow["PAYMENTTYPE"]])){
$pmtTypes[$theRow["PAYMENTTYPE"]] += $theRow["PAYMENT"];;
}else{
$pmtTypes[$theRow["PAYMENTTYPE"]] = $theRow["PAYMENT"];;
}
if($theRow["PREVIOUSBALANCE"] != 0) continue;
$subTotal = ( ( $theRow["LENSES"] + $theRow["FRAMES"] + $theRow["TAXABLEGOODS"] ) - $theRow["DISCOUNT"]);
$HST = ( $theRow["TAXABLEGOODS"] * 0.13 );
$totalHST+= $HST;
$total += ( $subTotal + $HST );
$payments+=$theRow["PAYMENT"];
}
$outstanding=$total-$payments;
Anyone have anything to contribute?
I would appreciate any help.
show me only the balance from the highest Invoice ID for a particular JOBID
For a single job ID:
SELECT lenses+frames+taxablegoods-discount+previousbalance AS balance
FROM invoices WHERE jobid=?
ORDER BY id DESC LIMIT 1
group the results by job ID and use the MAX feature to select only the higest ID for every JOBID
If you want to query the latest invoice for many jobs at once, you are talking about a per-group-maximum selection. SQL doesn't make this as easy to do as you'd hope. There are various approaches, including subqueries, but on MySQL I typically favour the null-self-join:
SELECT i0.jobid, i0.lenses+...etc... AS balance
FROM invoices AS i0 LEFT JOIN invoices AS i1 ON i1.jobib=i0.jobid AND i1.id>i0.id
WHERE i1.id IS NULL
That is: “give me rows where there is no row with the same job ID but a higher invoice ID”.
If doing this between two dates, you'd need to apply the condition to both sides of the join:
SELECT i0.jobid, i0.lenses+...etc... AS balance
FROM invoices AS i0 LEFT JOIN invoices AS i1 ON
i1.jobib=i0.jobid AND i1.id>i0.id AND
i1.date BETWEEN ? AND ?
WHERE
i0.date BETWEEN ? AND ?
i1.id IS NULL
Incidentally you have an SQL-injection vulnerability from putting strings into your query. Use mysql_real_escape_string() or, better, parameterised queries to avoid these problems.
The subquery would look something like this:
SELECT * FROM INVOICES I1
WHERE DATE BETWEEN ? AND ?
AND ID = (SELECT MAX(ID) FROM INVOICES I2
WHERE DATE BETWEEN ? AND ?
AND I2.JOBID = I1.JOBID)
You should look into using parameterised queries, instead of concatenating the string with user input. At a bare minimum use mysql_real_escape_string - see here: http://www.php.net/manual/en/function.mysql-real-escape-string.php