How to implement an efficient database driven ticket/protocol control system? - mysql

Scenario: WAMP server, InnoDB Table, auto-increment Unique ID field [INT(10)], 100+ concurrent SQL requests. VB.Net should also be used if needed.
My database has an auto-increment field wich is used to generate a unique ticket/protocol number for each new information stored (service order).
The issue is that this number must be reseted each year. I.e. it starts at 000001/12 on 01/01/2012 00:00:00 up to maximum 999999/12 and then at 01/01/2013 00:00:00 it must start over again to 000001/13.
Obviously it should easly acomplished using some type of algorithm, but, I'm trying to find a more efficient way to do that. Consider:
It must (?) use auto-increment since the database has some concurrency (+100).
Each ticket must be unique. So 000001 on 2012 is not equal to 000001 on 2013.
It must be automatic. (no human interaction needed to make the reset, or whatever)
It should be reasonably efficient. (a watch program should check the database daily (?) but it seems not the best solution since it will 'fail' 364 times to have success only once).
The 'best' approach I could think of is to store the ticket number using year, such as:
12000001 - 12999999 (it never should reach the 999.999, anyway)
and then an watch program should set the auto increment field to 13000000 at 01/01/2013.
Any Suggestions?
PS: Thanks for reading... ;)

So, for futher reference I've adopted the following sollution:
I do create n tables on the database (one for each year) with only one auto-increment field wich is responsible to generate the each year unique id.
So, new inserts are done into the corresponding table considering the event date. After that the algorithm takes the last_inseted_id() an store that value into the main table using the format 000001/12. (ticket/year)
That because each year must have it's own counter since an 2012 event would be inserted even when the current date is already 2013.
That way events should be retroactive, no reset is needed and it's simple to implement.
Sample code for insertion:
$eventdate="2012-11-30";
$eventyear="2012";
$sql = "INSERT INTO tbl$eventyear VALUES (NULL)";
mysql_query ($sql);
$sql = "LAST_INSERT_ID()";
$row = mysql_fetch_assoc(mysql_query($sql));
$eventID = $row(0)
$sql = "INSERT INTO tblMain VALUES ('$eventID/$eventYear', ... ";
mysql_query($sql)

MongoDB uses something very similar to this that encodes the date, process id and host that generated an id along with some random entropy to create UUIDs. Not something that fulfills your requirement of monotonic increase, but something interesting to look at for some ideas on approach.
If I were implementing it, I would create a simple ID broker server that would perform the logic processing on date and create a unique slug for the id like you described. As long as you know how it's constructed, you have native MySql equivalents to get your sorting/grouping working, and the representation serializes gracefully this will work. Something with a UTC datestring and then a monotonic serial appended as a string.
Twitter had some interesting insights into custom index design here as they implemented their custom ID server Snowflake.
The idea of a broker endpoint that generates UUIDs that are not just simple integers, but also contain some business logic is becoming more and more widespread.

You can set up a combined PRIMARY KEY for both of the two columns ID and YEAR; this way you would only have one ID per year.
MySQL:
CREATE TABLE IF NOT EXISTS `ticket` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`year` YEAR NOT NULL DEFAULT '2012',
`data` TEXT NOT NULL DEFAULT '',
PRIMARY KEY (`id`, `year`)
)
ENGINE = InnoDB DEFAULT CHARACTER SET = utf8 COLLATE = utf8_unicode_ci
UPDATE: (to the comment from #Paulo Bueno)
How to reset the auto-increment-value could be found in the MySQL documentation: mysql> ALTER TABLE ticket AUTO_INCREMENT = 1;.
If you also increase the default value of the year-column when resetting the auto-increment-value, you 'll have a continuous two-column primary key.
But I think you still need some sort of trigger-program to execute the reset. Maybe a yearly cron-job, which is launching a batch-script to do so on each first of January.
UPDATE 2:
OK, I've tested that right now and one can not set the auto-increment-value to a number lower than any existing ID in that specific column. My mistake – I thought it would work on combined primary keys…
INSERT INTO `ticket` (`id`, `year`, `data`) VALUES
(NULL , '2012', 'dtg htg het'),
-- some more rows in 2012
);
-- this works of course
ALTER TABLE `ticket` CHANGE `year` `year` YEAR( 4 ) NOT NULL DEFAULT '2013';
-- this does not reset the auto-increment
ALTER TABLE `ticket` AUTO_INCREMENT = 1;
INSERT INTO `ticket` (`id`, `year`, `data`) VALUES
(NULL , '2013', 'sadfadf asdf a'),
-- some more rows in 2013
);
-- this will result in continously counted ID's
UPDATE 3:
The MySQL-documentation page has a working example, which uses grouped primary keys on MyISAM table. They are using a table similar to the one above, but with reversed column-order, because one must not have auto-increment as first column. It seems this works only using MyISAM instead of InnoDB. If MyISAM still fits your needs, you don't need to reset the ID, but merely increase the year and still have a result as the one you've questioned for.
See: http://dev.mysql.com/doc/refman/5.0/en/example-auto-increment.html (second example, after "For MyISAM and BDB tables you can specify AUTO_INCREMENT on a secondary column in a multiple-column index.")

Related

MySQL not executing CURRENT_TIMESTAMP updates as expected

After lots of research and several similar questions asked here, I have reached some conclusions, but as always it is, there are more questions.
This concerns the explicit_defaults_for_timestamp
Assuming the explicit_defaults_for_timestamp is turned off, this will work:
Schema:
CREATE TABLE IF NOT EXISTS `updated_tables` (
`table_name` VARCHAR(50) NOT NULL,
`updated_at` TIMESTAMP(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6),
PRIMARY KEY (`table_name`),
UNIQUE INDEX `table_name_UNIQUE` (`table_name` ASC))
ENGINE = InnoDB;
And the query:
INSERT INTO `updated_tables` (`table_name`,`updated_at`) VALUES ('products',NULL) ON DUPLICATE KEY UPDATE `table_name`=VALUES(`table_name`), `updated_at`=VALUES(`updated_at`);
First time the query is sent, the table is populated with 'products' and with the current time stamp.
If I repeat the query, then the field 'updated_at' is updated. Per definition, when I send NULL value, even though it is not allowed, the MySQL will update the column.
All is fine, and works as expected.
Lets assume I turn on the explicit_defaults_for_timestamp
If I user the above query, it will complain the the NULL is not allowed, which complies with the rules.
Question is, how can I have the same functionality with the explicit_defaults_for_timestamp turned on?
There is the solution to introduce additional column (varchar) which holds for example timestamp in miliseconds. And when I update it, then the MySQL updates the updated_at accordingly.
But it looks like an overkill, I might as well update the updated_at manually. I would like to move that responsibility to MySQL level, not doing it programatically.
In short, how can I perform updates on the table_name, and have the updated_at being set properly. The trick here is I have many updates (cache table), but actually never changing the table_name value at all.
Is it possible? Or I must turn off explicit_defaults_for_timestamp?
Is it bad decision to turn it off? Looking at this AWS RDS post seems it is ok, but I am not sure.
Side question:
If I decide to perform updates on my own, what would be the way to construct it?
Currently the MySQL CURRENT_TIMESTAMP(6) has this construct:
2018-07-10 11:32:43.490100
How could I create same construct with Javascript? First thing coming to my mind is to get current Date, and append to it last 6 digits of current timestamp.
You can create a trigger on INSERT and always set the value for updated_at with the CURRENT_TIMESTAMP - the cleanest approach but this may slow down your updates. Programmatically setting the column value would be faster than firing a trigger.
If you are executing your queries from Node.js then you can use new Date().getTime() to get a Unix timestamp in milliseconds and then construct your query like this
UPDATE tbl SET col_1 = val_1, col_2 = val_2, updated_at = FROM_UNIXTIME(js_milliseconds / 1000)
WHERE id = desired_id

MySQL atomic insert-if-not-exists with stable autoincrement

In MySQL, I am using an InnoDB table that contains unique names, and IDs for those names. Clients need to atomically check for an existing name, insert a new one if it does not exist, and get the ID. The ID is an AUTO_INCREMENT value, and it must not increment out-of-control when checking for existing values regardless of the setting of "innodb_autoinc_lock_mode"; this is because very often the same name will be checked (e.g. "Alice"), and every now and then some new name will come along (e.g. "Bob").
The "INSERT...ON DUPLICATE KEY UPDATE" statement causes an AUTO_INCREMENT increase even in the duplicate-key case, depending on "innodb_autoinc_lock_mode", and is thus unacceptable. The ID will be used as the target of a Foreign-Key Constraint (in another table), and thus it is not okay to change existing IDs. Clients must not deadlock when they do this action concurrently, regardless of how the operations might be interleaved.
I would like the processing during the atomic operation (e.g. checking for the existing ID and deciding whether or not to do the insert) to be done on the server-side rather than the client-side, so that the delay for other sessions attempting to do the same thing simultaneously is minimal and does not need to wait for client-side processing.
My test table to demonstrate this is named FirstNames:
CREATE TABLE `FirstNames` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`FirstName` varchar(45) COLLATE utf8mb4_unicode_ci NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `FirstName_UNIQUE` (`FirstName`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
The best solution that I have come up with thus far is as follows:
COMMIT;
SET #myName='Alice';
SET #curId=NULL;
SET autocommit=0;
LOCK TABLES FirstNames WRITE;
SELECT Id INTO #curId FROM FirstNames WHERE FirstName = #myName;
INSERT INTO `FirstNames` (`FirstName`) SELECT #myName FROM DUAL WHERE #curId IS NULL;
COMMIT;
UNLOCK TABLES;
SET #curId=IF(#curId IS NULL, LAST_INSERT_ID(), #curId);
SELECT #curId;
This uses "LOCK TABLES...WRITE" following the instructions given in the MySQL "Interaction of Table Locking and Transactions" documentation for the correct way to lock InnoDB tables. This solution requires the user to have the "LOCK TABLES" privilege.
If I run the above query with #myName="Alice", I obtain a new ID and then continue to obtain the same ID no matter how many times I run it. If I then run with #myName="Bob", I get another ID with the next AUTO_INCREMENT value, and so on. Checking for a name that already exists does not increase the table's AUTO_INCREMENT value.
I am wondering if there is a better solution to accomplish this, perhaps one that does not require the "LOCK TABLES"/"UNLOCK TABLES" commands and combines more "rudimentary" commands (e.g. "INSERT" and "SELECT") in a more clever way? Or is this the best methodology that MySQL currently has to offer?
Edit
This is not a duplicate of "How to 'insert if not exists' in MySQL?". That question does not address all of the criteria that I stated. The issue of keeping the AUTO_INCREMENT value stable is not resolved there (it is only mentioned in passing).
Many of the answers do not address getting the ID of the existing/inserted record, some of the answers do not provide an atomic operation, and some of the answers have the logic being done on the client-side rather than the server-side. A number of the answers change an existing record, which is not what I'm looking for. I am asking for either a better method to meet all of the criteria stated, or confirmation that my solution is the optimal one with existing MySQL support.
The question is really about how to normalize data when you expect there to be duplicates. And then avoid "burning" ids.
http://mysql.rjweb.org/doc.php/staging_table#normalization discusses a 2-step process and is aimed at mass updates due to high-speed ingestion of rows. It degenerates to a single row, but still requires the 2 steps.
Step 1 INSERTs any new rows, creating new auto_inc ids.
Step 2 pulls back the ids en masse.
Note that the work is best done with autocommit=ON and outside the main transaction that is loading the data. This avoids an extra cause for burning ids, namely potential rollbacks.
You can use a conditional INSERT in a single statement:
INSERT INTO FirstNames (FirstName)
SELECT i.firstName
FROM (SELECT 'Alice' AS firstName) i
WHERE NOT EXISTS (SELECT * FROM FirstNames t WHERE t.FirstName = i.firstName);
The next AUTO_INCREMENT value stays untouched in case of existance. But I can't tell you that would be the case in any (future) version or for every configuration. However, it is not much different from what you did - Just in a single statement and without locking the table.
At this point you can be sure that the name exists and just select the corresponding Id:
SELECT Id FROM FirstNames WHERE FirstName = 'Alice';

mySQL valid bit - alternatives?

Currently, I have a mySQL table with columns that looks something like this:
run_date DATE
name VARCHAR(10)
load INTEGER
sys_time TIME
rec_time TIME
valid TINYINT
The column valid is essentially a valid bit, 1 if this row is the latest value for this (run_date,name) pair, and 0 if not. To make insertions simpler, I wrote a stored procedure that first runs an UPDATE table_name SET valid = 0 WHERE run_date = X AND name = Y command, then inserts the new row.
The table reads are in such a way that I usually use only the valid = 1 rows, but I can't discard the invalid rows. Obviously, this schema also has no primary key.
Is there a better way to structure this data or the valid bit, so that I can speed up both inserts and searches? A bunch of indexes on different orders of columns gets large.
In all of the suggestions below, get rid of valid and the UPDATE of it. That is not scalable.
Plan A: At SELECT time, use 'groupwise max' code to locate the latest run_date, hence the "valid" entry.
Plan B: Have two tables and change both when inserting: history, with PRIMARY KEY(name, run_date) and a simple INSERT statement; current, with PRIMARY KEY(name) and INSERT ... ON DUPLICATE KEY UPDATE. The "usual" SELECTs need only touch current.
Another issue: TIME is limited to 838:59:59 and is intended to mean 'time of day', not 'elapsed time'. For the latter, use INT UNSIGNED (or some variant of INT). For formatting, you can use sec_to_time(). For example sec_to_time(3601) -> 01:00:05.

Mysql Incrementing field that includes PK and Date stamp

I have an server side application where I need to sync down new and updated items to mobile devices. The requirement is to store one "key" that can be passed between the server and the devices to mark where we are in the syncing process. We have a millisecond timestamp field that would work for this except for the fact that when we allow customers to upload a csv file the insert process groups all those inserts together with the same timestamp, since now() sets a variable at the beginning of the transaction and uses it throughout the transaction.
The only possibilities I could come up with were doing some sort of numeric hash or CRC where I added the timestamp and pk (in this case 1045) together like CRC32(now()+1045) but that does not produce a sequential pattern over time...
The other option I was considering is modifying the batch upload process to add an incrementing millisecond value onto each item as the insert was built:
insert into items (description,datemodified) VALUES
('toaster',(now(3)+.001)),
('blender', (now(3)+.002),
('coffee maker', (now(3)+.003));
Then I would have a unique-enough and sequential id that I could pass back and forth to track the sync status.
Running MySQL 5.7.11, all tables are InnoDB.
Anyone solved this a better way?
I think an integer column with AUTO_INCREMENT might be suitable for your use case.
CREATE TABLE foo
( id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT
, description VARCHAR(80)
, datemodified DATETIME
, ...
, CONSTRAINT foo_UX1 UNIQUE KEY (id)
)
;
INSERT INTO foo (id, description, datemodified)
VALUES
(NULL,'toaster',NOW())
,(NULL,'blender',NOW())
,(NULL,'coffee maker',NOW())
;

How to swap values of two rows in MySQL without violating unique constraint?

I have a "tasks" table with a priority column, which has a unique constraint.
I'm trying to swap the priority value of two rows, but I keep violating the constraint. I saw this statement somewhere in a similar situation, but it wasn't with MySQL.
UPDATE tasks
SET priority =
CASE
WHEN priority=2 THEN 3
WHEN priority=3 THEN 2
END
WHERE priority IN (2,3);
This will lead to the error:
Error Code: 1062. Duplicate entry '3' for key 'priority_UNIQUE'
Is it possible to accomplish this in MySQL without using bogus values and multiple queries?
EDIT:
Here's the table structure:
CREATE TABLE `tasks` (
`id` int(11) NOT NULL,
`name` varchar(200) DEFAULT NULL,
`priority` varchar(45) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `priority_UNIQUE` (`priority`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
Is it possible to accomplish this in MySQL without using bogus values and multiple queries?
No. (none that I can think of).
The problem is how MySQL processes updates. MySQL (in difference with other DBMS that implement UPDATE properly), processes updates in a broken manner. It enforces checking of UNIQUE (and other) constraints after every single row update and not - as it should be doing - after the whole UPDATE statement completes. That's why you don't have this issue with (most) other DBMS.
For some updates (like increasing all or some ids, id=id+1), this can be solved by using - another non-standard feature - an ORDER BY in the update.
For swapping the values from two rows, that trick can't help. You'll have to use NULL or a bogus value (that doesn't exist but is allowed in your column) and 2 or 3 statements.
You could also temporarily remove the unique constraint but I don't think that's a good idea really.
So, if the unique column is a signed integer and there are no negative values, you can use 2 statements wrapped up in a transaction:
START TRANSACTION ;
UPDATE tasks
SET priority =
CASE
WHEN priority = 2 THEN -3
WHEN priority = 3 THEN -2
END
WHERE priority IN (2,3) ;
UPDATE tasks
SET priority = - priority
WHERE priority IN (-2,-3) ;
COMMIT ;
I bumped into the same issue. Had tried every possible single-statement query using CASE WHEN and TRANSACTION - no luck whatsoever. I came up with three alternative solutions. You need to decide which one makes more sense for your situation.
In my case, I'm processing a reorganized collection (array) of small objects returned from the front-end, new order is unpredictable (this is not a swap-two-items deal), and, on top of everything, change of order (usually made in English version) must propagate to 15 other languages.
1st method: Completely DELETE existing records and repopulate entire collection using the new data. Obviously this can work only if you're receiving from the front-end everything that you need to restore what you just deleted.
2st method: This solution is similar to using bogus values. In my situation, my reordered collection also includes original item position before it moved. Also, I had to preserve original index value in some way while UPDATEs are running. The trick was to manipulate bit-15 of the index column which is UNSIGNED SMALLINT in my case. If you have (signed) INT/SMALLINT data type you can just invert the value of the index instead of bitwise operations.
First UPDATE must run only once per call. This query raises 15th bit of the current index fields (I have unsigned smallint). Previous 14 bits still reflect original index value which is never going to come close to 32K range.
UPDATE *table* SET `index`=(`index` | 32768) WHERE *condition*;
Then iterate your collection extracting original and new index values, and UPDATE each record individually.
foreach( ... ) {
UPDATE *table* SET `index`=$newIndex WHERE *same_condition* AND `index`=($originalIndex | 32768);
}
This last UPDATE must also run only once per call. This query clears 15th bit of the index fields effectively restoring original index value for records where it hasn't changed, if any.
UPDATE *table* SET `index`=(`index` & 32767) WHERE *same_condition* AND `index` > 32767;
Third method would be to move relevant records into temporary table that doesn't have a primary key, UPDATE all indexes, then move all records back to first table.
Bogus value option:
Okay, so my query is similar and I've found a way to update in "one" query. My id column is PRIMARY and position is part of a UNIQUE group. This is my original query that doesn't work for swapping:
INSERT INTO `table` (`id`, `position`)
VALUES (1, 2), (2, 1)
ON DUPLICATE KEY UPDATE `position` = VALUES(`position`);
.. but position is an unsigned integer and it's never 0, so I changed the query to the following:
INSERT INTO `table` (`id`, `position`)
VALUES (2, 0), (1, 2), (2, 1)
ON DUPLICATE KEY UPDATE `position` = VALUES(`position`);
.. and now it works! Apparently, MYSQL processes the values groups in order.
Perhaps this would work for you (not tested and I know almost nothing about MYSQL):
UPDATE tasks
SET priority =
CASE
WHEN priority=3 THEN 0
WHEN priority=2 THEN 3
WHEN priority=0 THEN 2
END
WHERE priority IN (2,3,0);
Good luck.
Had a similar problem.
I wanted to swap 2 id's that were unique AND was a FK from an other table.
The fastest solution for me to swap two unique entries was:
Create a ghost entry in my FK table.
Go back to my table where I want to switch the id's.
Turned of the FK Check SET FOREIGN_KEY_CHECKS=0;
Set my first(A) id to the ghost(X) fk (free's A)
Set my second (B) id to A (free's B)
Set A to B (free's X)
Delete ghost record and turn checks back on. SET FOREIGN_KEY_CHECKS=1;
Not sure if this would violate the constraints, but I have been trying to do something similar and eventually came up with this query by combining a few of the answers I found:
UPDATE tasks as T1,tasks as T2 SET T1.priority=T2.priority,T2.priority=T1.priority WHERE (T1.task_id,T2.task_id)=($T1_id, $T2_id)
The column I was swapping did not use a unique, so I am unsure if this will help...
you can achieve swapping your values with your above mentioned update statement, with a slight change in your key indexes.
CREATE TABLE `tasks` ( `id` int(11) NOT NULL, `name` varchar(200) DEFAULT NULL, `priority` varchar(45) DEFAULT NULL, PRIMARY KEY (`id`,`priority`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
This will have a primary key index as a combination of id and priority. you cna then swap values.
UPDATE tasks
SET priority =
CASE
WHEN priority=2 THEN 3
WHEN priority=3 THEN 2
END
WHERE priority IN (2,3);
I dont see any need of user variables or temp variables here.
Hope this solves your issue :)