I have a table that has a primary key and for some reason i was advised not to use the AUTO_INCREMENT flag.
So every time i have to insert a new entry i search for the last value inserted (highest value) of the primary key.
Then i increment it by 1 to get a new ID.
Now the problem i face is, when inserting the first entry, there is no data in the table.
Can anyone suggest the optimal way to check
if data exists in table,
if not set id as 1 and insert new row,
else get the last id from table, increment it by 1 and then insert new row.
PS: New to mysql so having difficulty with its syntax.
Based on your statement 'i search for the last value inserted (highest value) of the primary key', I presume that you're currently doing something like this to get the maximum existing ID:
SELECT MAX(id_column) + 1 FROM my_table
If you have an empty table, this will of course return NULL. In that case, just handle the NULL using IFNULL to return 0 if there is no maximum value:
SELECT IFNULL(MAX(id_column), 0) + 1 FROM my_table
This will output 1 as the next identifier if the table has no rows.
I've given this more thought, and it turns out there is a way to generate a unique primary key without using Auto Increment or worrying about race conditions, so long as you are willing and able to use a 36 byte primary key (or, alternatively, a 128 bit binary).
The solution (at least as of MySQL 5.5) is the UUID, which stands for Universal Unique Identifier.
You would use it thus:
CREATE TABLE uu_table (
id VARCHAR(36) PRIMARY KEY,
name VARCHAR(50),
{other interesting columns}
)
Then insert new rows thus:
INSERT INTO uu_table VALUES (UUID(), 'Name of this Row', {other interesting values});
The UUID() function is guaranteed to generate a unique key 99.99{bunch more 9's}% of the time, even if generated on independent systems. That's its whole purpose, to be as unique as snowflake patterns, no matter where it is created.
There are pros and cons to this method. Best to read up on it here: https://dev.mysql.com/doc/refman/5.7/en/miscellaneous-functions.html#function_uuid
It is basically a 128 bit number, which you can save as a binary value after conversion from the 36 characters. I believe some versions of MySQL come with functions for that purpose. That would use less space in your database than 36 bytes, but I'll leave that as an exercise for the reader.
Related
I was developing a database in SQL Server where I was using an identity column as a seed for a primary key field. The intention was to reset the identity to 1 at the beginning of every year. This would allow us to create a PK of the Year - Identity Column.
Create Table Issues (
IssueID AS RIGHT(CONVERT(VARCHAR, Year(getdate()), 4),2) + '-' + RIGHT(REPLICATE('0', 2) +
CONVERT(VARCHAR, RecordID),3) NOT NULL PRIMARY KEY,
RecordID int Identity (1,1),.........)
The result would be
IssueID RecordID
20-001 1
20-002 2
20-003 3
21-001 1
etc....
Now I've been told we are going to use a MySQL database instead.
Can an Auto-Increment field in MySQL contain duplicate values like it can in SQL Server?
If Not, how can I do what I need to do in MySQL?
In MySQL, you can't use the default auto-increment feature for what you describe, a incrementing value that starts over per year.
This was a feature of the MyISAM storage engine years ago. An auto-increment that was the second column of a multi-column primary key would start counting from one for each distinct value in the first column of the PK. See the example under "MyISAM Notes" on this page: https://dev.mysql.com/doc/refman/8.0/en/example-auto-increment.html
But it's considered not a good idea to use MyISAM because it does not support ACID. In general, I would find another way of solving this task. I would not use MyISAM.
In InnoDB, there's no way the table will generate a value that is a duplicate of a value currently in the table, or even a value less than the maximum value previously generated for that table. In other words, there's no way to "fill in the gaps" using auto-increment.
You can use ALTER TABLE mytable AUTO_INCREMENT=1 to reset the counter, but the value you set it will automatically advance to the max value currently in the table + 1.
So you'll have to generate it using either another table, or else something other than the MySQL database. For example, I've seen some people use memcached, which supports an atomic "increment and return counter" operation.
Another thing to consider: If you need a row counter per year, this is actually different from using MySQL's auto-increment feature. It's not easy to use the latter as a row counter. Besides, what happens if you roll back a transaction or delete a row? You'd end up with non-consecutive RecordId values, with unexplained "gaps." It's also a fact about the auto-increment feature that it guarantees that subsequent id's will be greater, but it does not guarantee to generate all values consecutively. So you'll get gaps eventually anyway.
In MySQL a table can have only one auto_increment column and this column must be a part of the primary key. See details here.
Technical workaround for your task would be creating of a table with a single auto_increment column, and you can obtain auto_increment value by inserting a record into this table and immediately calling standard MySQL function last_inser_id(). When time comes you should truncate the table - in this case the auto_increment count will be reset.
Currently, I have a mySQL table with columns that looks something like this:
run_date DATE
name VARCHAR(10)
load INTEGER
sys_time TIME
rec_time TIME
valid TINYINT
The column valid is essentially a valid bit, 1 if this row is the latest value for this (run_date,name) pair, and 0 if not. To make insertions simpler, I wrote a stored procedure that first runs an UPDATE table_name SET valid = 0 WHERE run_date = X AND name = Y command, then inserts the new row.
The table reads are in such a way that I usually use only the valid = 1 rows, but I can't discard the invalid rows. Obviously, this schema also has no primary key.
Is there a better way to structure this data or the valid bit, so that I can speed up both inserts and searches? A bunch of indexes on different orders of columns gets large.
In all of the suggestions below, get rid of valid and the UPDATE of it. That is not scalable.
Plan A: At SELECT time, use 'groupwise max' code to locate the latest run_date, hence the "valid" entry.
Plan B: Have two tables and change both when inserting: history, with PRIMARY KEY(name, run_date) and a simple INSERT statement; current, with PRIMARY KEY(name) and INSERT ... ON DUPLICATE KEY UPDATE. The "usual" SELECTs need only touch current.
Another issue: TIME is limited to 838:59:59 and is intended to mean 'time of day', not 'elapsed time'. For the latter, use INT UNSIGNED (or some variant of INT). For formatting, you can use sec_to_time(). For example sec_to_time(3601) -> 01:00:05.
I have a lists table that has an order field.
When I insert a new record, is it possible to find the order of the previous row and increment the new row?
Or should I go about it myself in PHP by doing an OrderBy('order') query and getting the max() value of that?
When you declare a table with MySQL you can use an auto-increment id so you won't have to deal about its incrementation:
CREATE TABLE people (
id MEDIUMINT NOT NULL AUTO_INCREMENT,
name CHAR(30) NOT NULL,
PRIMARY KEY (id)
);
As explained in the documentation,
An integer or floating-point column can have the additional attribute
AUTO_INCREMENT. When you insert a value of NULL (recommended) or 0
into an indexed AUTO_INCREMENT column, the column is set to the next
sequence value. Typically this is value+1, where value is the largest
value for the column currently in the table. AUTO_INCREMENT sequences
begin with 1.
I suggest you to ommit the field completly when inserting new records.
You can then retrieve the last id inserted with LAST_INSERT_ID() SQL function (or the mysqli_insert_id function of PHP languagefor example).
But since it's not what you wanted, probably because of one of the reasons quoted from MarioZ's comment:
If you are already using auto-increment for the ID you can use it for
the order (that can be one reason). For auto-increment the column
must be set as primary and unique, can't be repeated values. The auto-increment is from the number in the record, if you inserted 10
rows and you delete 2, the next insert with auto-increment will be
11(if the last now is 8 you'd want it to be 9). Those are posible
reasons not to use it for what #Notflip wants :P
... You'll have to use PHP, with LOCK TABLE and UNLOCK TABLE SQL instructions before and after the retrieving of the last order then the updating of the new order, to avoid having simultaneous records with the same "order".
I had a table with 3 columns and 3600K rows. Using MySQL as a key-value store.
The first column id was VARCHAR(8) and set to primary key.The 2nd and 3rd columns were MEDIUMTEXT. When calling SELECT * FROM table WHERE id=00000 MySQL took like 54 sec ~ 3 minutes.
For testing I created a table containing VARCHAR(8)-VARCHAR(5)-VARCHAR(5) where data casually generated from numpy.random.randint. SELECT takes 3 sec without primary key. Same random data with VARCHAR(8)-MEDIUMTEXT-MEDIUMTEXT, the time cost by SELECT was 15 sec without primary key.(note: in second test, 2nd and 3rd column actually contained very short text like '65535', but created as MEDIUMTEXT)
My question is: how can I achieve similar performance on my real data? (or, is it impossible?)
If you use
SELECT * FROM `table` WHERE id=00000
instead of
SELECT * FROM `table` WHERE id='00000'
you are looking for all strings that are equal to an integer 0, so MySQL will have to check all rows, because '0', '0000' and even ' 0' will all be casted to integer 0. So your primary key on id will not help and you will end up with a slow full table. Even if you don't store values that way, MySQL doesn't know that.
The best option is, as all comments and answers pointed out, to change the datatype to int:
alter table `table` modify id int;
This will only work if your ids casted as integer are unique (so you don't have e.g. '0' and '00' in your table).
If you have any foreign keys that references id, you have to drop them first and, before recreating them, change the datatype in the other columns too.
If you have a known format you are storing your values (e.g. no zeros, or filled with 0s up to the length of 8), the second best option is to use this exact format to do your query, and include the ' to not cast it to integer. If you e.g. always fill 0 to 8 digits, use
SELECT * FROM `table` WHERE id='00000000';
If you never add any zeros, still add the ':
SELECT * FROM `table` WHERE id='0';
With both options, MySQL can use your primary key and you will get your result in milliseconds.
If your id column contains only numbers so define it as int , because int will give you better performance ( it is more faster)
Make the column in your table (the one defined as key) integer and retry. Check first performance by running a test within your DB (workbench or simple command line). You should get a better result.
Then, and only if needed (I doubt it though), modify your python to convert from integer to string (and/or vise-versa) when referencing the key column.
I have a unique PK 'id' which is set to auto_increment. I have a second field called 'label' which is an alphanumeric field (e.g. W1000) that is intended to increment on each insert using PHP logic.
The 'label' field may have one of many alpha prefixes, followed by the incrementing number. Each prefix should increment independently. For example, the table may have W1000 and F1123. The next W would be W1001, and the next F would be F1124.
The current method (PHP select maximum label, insert maximum label + 1) creates a race condition, and occasionally I get a duplicate 'label'. I need to resolve these duplicated 'labels' and ensure that this field is unique. I am willing to split the prefix and the numeric into two fields if it helps.
What is the best way to accomplish this?
One approach to avoid generating duplicate label values is to use a MyISAM table to generate unique sequence numbers. MyISAM supports the behavior of AUTO_INCREMENT you need.
See the "MyISAM Notes" section in MySQL Reference 3.6.9. Using AUTO_INCREMENT
For this approach, you would create a separate MyISAM table; the purpose of this table is to generate unique sequence numbers:e.g.
CREATE TABLE foo
( prefix VARHCAR(1) NOT NULL
, num INT UNSIGNED AUTO_INCREMENT
, PRIMARY KEY (prefix, num)
) Engine=MyISAM
Assuming that the label prefix is one character, and the remainder is a numeric:
INSERT INTO foo (prefix, num)
SELECT SUBSTR(t.label,1,1) AS prefix
, MAX(SUBSTR(t.label,2,8) AS num
FROM mytable
GROUP BY SUBSTR(t.label,1,1)
Go get a new sequence number, insert a row to the new table, providing a value for prefix, and a NULL for the num column, and retrieve the value inserted for the num column:
INSERT INTO foo (prefix,num) VALUES ('W',NULL);
SELECT LAST_INSERT_ID();
You can use that to construct the value to be used for the label column in your original table.
Note that it's only the MyISAM engine that has the behavior you want (separately incrementing AUTO_INCREMENT sequences for each prefix.) Your original table can be whatever engine.
This approach avoids the race condition, but does introduce a concurrency bottleneck, due to the exclusive locks taken on the MyISAM table for the insert.
Another way to avoid the race condition is to obtain an exclusive lock on the table, then do a SELECT MAX(), then do an insert, then release the lock. But that approach introduces even more of a concurrency bottleneck, serializing access to a single resource.
If your question is about identifying the existing duplicate label values, then this query gets you rows that have a "duplicate" label. (This is picking out just one row, for each duplicated label.)
SELECT t.label
, MAX(t.id)
FROM mytable t
GROUP BY t.label
HAVING COUNT(1) > 1
To update the labels to be unique, you would need to generate a new label for those rows.
Go get that done in a single SQL statement is a bit tricky. I tried to come up with a statement, but it was broken, and I don't have the time to fix it.