I have a MySQL table that auto increments an id column every time an entry is added. About once per year, I delete all of the data in this table and start new. Right now, the id column's next auto increment value will be 27721.
Questions...
For this type of situation, would it be advisable to reset the auto incrementing to 1 when I delete the table data?
Does it matter?
How high is "too high" for ids like this?
Generally it would not be advisable. You will break things referencing specific IDs, but mostly because...
It doesn't matter.
The auto_increment limit is the sizes of the column. If it's an unsigned int, that's in the billions so if you only have 27721 after a couple years it's probably not a concern.
Moreover I would recommend that you not delete the data and just archive it instead. It would probably make it easier to restore if you had to.
Related
I'm looking for an insight as I'm a bit stuck.
Right now I'm using auto increment ID's in all of my MySQL innoDB tables. Even for transaction tables (large volume tables).
In my application, someone enters a financial journal which contains of a few transaction lines which are saved in a transaction table.
Now I get requirement requests that the users want to be able to change those transactions rather then insert a total correction for that journal and insert new lines.
No problem for my procedures, but one question arises for me: what if the number of transaction lines for that journal particular changes? If they are the same I could just overwrite the lines with the current auto increment IDs that they already have.
But what if there are more or less journal lines than the first time they were inserted?
Sure, I can delete all previously inserted lines and insert new ones which the auto increment ID field handles fine. But it leaves a gap in the IDs.
I've read before about peoples opinions about the gap I'm talking about, where things go from 'don't worry about it, you'll never run out' to 'oh my, a problem'.
Is it a bad thing if i leave out the auto increment ID as I do not even need it in this trans-table? I always find the rows in the trans-table through the journal number, which is unique.
Are there any drawbacks with leaving out an auto increment ID field on this table?
All I can think if that it maybe may slow down queries on that table (and am not sure of that), but can't think of anything else.
For reference, my trans table looks somewhat like this:
ID --> auto increment bigint
Journal --> int(11)
JournalLineNumber --> int(3)
Description --> varchar (50)
Amount --> decimal
etc.
etc.
By the waym I still do not know how to insert things like tables for displaying mysql table info in my questions in stackoverflow, does anyone know? Googled it again just now but all kinds of confusing tools seems to be needed to simple enter a table of information....but hopefully I'm wrong.
The important thing is that you have a primary key, and that it is reasonably compact. UUID isn't too great because it's big; bigint is fine. int is great. If your have an int or bigint journal number, you can use that as the PK, and you don't need an auto_increment. As long as you have a PK and it isn't huge, performance will not be negatively impacted.
Note that signed (default) int(11) is only good for up to 2^31 (~2bn). Given that your auto_increment was bigint, will you need more than -2bn to +2bn range for this? Should it be unsigned for 0 to 4bn instead? Is 4bn going to be enough for the foreseeable future? ALTER TABLE on 4bn rows is going to take a while.
Paste the output of SHOW CREATE TABLE. To show it as a block, make sure each line is prefixed with at least 4 spaces. Or just use the formatting toolbar.
I'm setting up a table that needs two auto-incrementing fields, 'id' and 'member#'.
I'll use AUTO_INCREMENT = 1001 on the latter for new data, as there is old data with member numbers less than 1000.
I'll use 'MAX(id)+1' on the 'id' field to auto-increment it.
But I'm not sure if this will do the job whenever there's an INSERT, or even where to put that bit of code. All I'm trying to do here is auto-increment the field, not SELECTing anything.
And out of curiosity, why is there only one AUTO_INCREMENTing field per table?
Surely, it can't be difficult to code AUTO_INCREMENT_2, AUTO_INCREMENT_3 etc.
All answers and assistance appreciated.
================================
ADDITIONAL INFORMATION AND LINKS
Sorry for the delay in my response, I've been doing additional research.
Ok so to explain further, we have people joining our group via the net. As such we need to assign a unique membership number to each person. Two John Does? Two different membership numbers. For this I've set the member# column as AUTO_INCREMENT, and then AUTO_INCREMENT = 1001 as a table option. Old membership numbers have three digits, new memberships have four. So each time someone registers as a new member on the web, there's an insert command that automatically assigns the next four digit membership number in the series to the new member.
member# INT(6) UNSIGNED NOT NULL UNIQUE KEY AUTO_INCREMENT
And as a table option AUTO_INCREMENT = 1001
I hope this is clear. Other situations where someone might want to use a similar strategy could be assigning consecutive invoice numbers, receipt numbers, account numbers, etc. So how does one guarantee a +1 result, ie consecutive numbers?
Now we also need a table id column. Lots of tables need a table id. It too needs to be assigned an AUTO_INCREMENT value, in our case, beginning with 1, and incrementing by 1 (the default), to identify and distinguish one row from another. But unfortunately there can be only one AUTO_INCREMENT column per table in MySQL. :-/
So this situation belongs to a class of problems known as MAX+1 problems. (It may also be related to ROW_COUNT and LAST_INSERT_ID solutions.) The limit of a single AUTO_INCREMENT field per table requires a MAX+1 workaround and I am looking for advice on the best way to implement this. For example, is there a way to set this up inside the CREATE TABLE command itself, when defining the id field? Or something else of an equally simple nature, such as writing a function. It is indeed preferable to optimize for efficiency and use only needed features rather than implement a series of commands. Typically a suggested work around might be:
Lock tables membership write;
$max = SELECT MAX(id) FROM membership;
INSERT INTO membership ( id, firstname, lastname )
VALUES ($max+1 , 'jane', 'smith')
unlock tables;
Is there something better?
As whether AUTO_INCREMENT_2 /_3... features should exist. Well, I'd have to point out that there are a lot of features in MySQL that I'll never use, but obviously someone needs them. Nevertheless, it would be convenient to have this for those (rare) occasions when you might need it. Perhaps there is a distinction to be drawn between having a feature available and using it on any given table. I doubt an unused feature requires much in the way of additional memory or clicks (which are pretty cheap these days anyways).
Some links that may prove useful in understanding this situation:
https://duckduckgo.com/?q=mysql+max%2B1+problems&t=ffab&atb=v1-1&ia=web
Insert and set value with max()+1 problems
Problem with MySql INSERT MAX()+1
https://bugs.mysql.com/bug.php?id=3575
All answers, advice and assistance appreciated.
Each InnoDB table has at most one counter for its auto-increment. This is part of the implementation. If you could define N auto-increment columns, in the same table, it would need more storage space to store N counters. It would require the auto-increment lock to last longer while you incremented N counters.
As for why is there only one per table, sure, it is possible that they could implement it to support more than one, but why?
It would make the implementation a lot more complex, and hinder performance, for cases that 99.99% of apps don't need.
They were trying to solve the needs for the majority of cases. In nearly every case of a table with an auto-increment, one per table is sufficient.
In nearly every case where someone like you thinks they need more than one per table, you'd be wise to step back and reconsider your design.
In MySQL the table structure cannot contain more than one auto_increment field. When you try to create a table with 2 autoincremented fields or alter the table in attempt to create second autoincrement, the query fails.
Autoincrement guarantees that each next value generated in the field will be greater than previous one in current connection. But it do NOT guarantee, that each next value generated in the field will be greater than previous value by 1. The "delta" may be 2 or even 1000... it cannot be negative or zero only.
I am using MySql in phpMyadmin. I have a table which contains a primary key. This primary key is the 'userid' and it is also an "auto increment" field. The application also has a functionality of deleting a particular user with a 'userid'. So after deleting a user when i again create a new user, the 'userid' gets a value of the next integer. i want the table to consider the deletion and assign primary key value, numbers which have been deleted
..
example:
the 'userid' values in the table are - 1,2,3,4,5,6,7....
i deleted userid with value 3.
so now when i create a next record of user, the table should use the userid value '3' as it is no longer in use. how can i do that in phpmyadmin?
i want to do this to keep the no of values of userid minimum. the count may go upto a 5 digit value of the userid. hence if a 2 digit is available to use since its been deleted before, using this 2 digit value will save memory usage of the database
It is entirely possible to assign the ID that is no longer used by explicitely providing it in the next insert you make. AUTO_INCREMENT only assigns an id if you do not supply it yourself.
Be certain though that the ID is really not being used, otherwise the insertion will fail.
That being said, I would discourage doing this. I am not 100% certain, but I think that when you declare an integer in MySQL, it requires integer space, regardless of how many digits the integer has, but I am open to clarification on this point. In any case, I believe the minor benefit of potentially using a little less space is not worth risking failure by tinkering with your IDs.
In my experience, such little things have a tendency to haunt you later on, and I do not see the real benefit.
I suggest looking for other ways to improve memory usage if necessary.
I have to temporarily store lots of data in a table, an hour at a time. For example, each hour I write 20k (or some other large number) of records. I do some stuff with that data during the hour, and by the end of the hour the table is empty again. At the start of the new hour, I add 20k more records and continue the process.
I'm currently storing the data in a MySQL table with an INT auto-increment primary key.
Eventually, I will run out of numbers and I'm looking for a solution that solves the problem, rather than postpones it (e.g. using unsigned BIG INT). I can think of several solutions, but I'm not sure what the consequences are of each - specifically the effects on MySQL. For example, will doing this a particular way cause table corruption or performance problems, etc.
How should problems like this be solved?
Create and remove the table each hour
Reset the auto-increment column each hour
Do either #1 or #2, but at longer intervals
Don't use an auto-increment key (I can easily insert my own sequential key)
Other?
NOTE: For processing purposes, it doesn't matter that the key is re-used each hour. I only use it to distribute the processing of the data.
When you empty the table, reset its auto-increment counter:
ALTER TABLE foo AUTOINCREMENT = 1;
Clearing the rows from the table leaves the counter where it was, and eventually you run out of key space as you indicated. If you reset this counter, and it doesn't matter as each hour is a new set of data, this seems the most effective solution.
If you truly want to empty the table, you should use the TRUNCATE TABLE command, which efficiently wipes out the data and indexes, leaving the table structure intact.
Unless you forgot to mention something you won't run out of numbers any time soon... If the amount of records written stays at 20k records/hour you'll have 12+ years with your current implementation.
And if that is not enough:
Using a SERIAL (alias for BIGINT UNSIGNED NOT NULL AUTO_INCREMENT UNIQUE) and adding about 20k records/hour will still give you over 100 billion years to play with; without having to alter tables or reset the AUTOINCREMENT.
I have a 100 million rows, and it's getting too big.
I see a lot of gaps. (since I delete, add, delete, add.)
I want to fill these gaps with auto-increment.
If I do reset it..is there any harM?
If I do this, will it fill the gaps?:
mysql> ALTER TABLE tbl AUTO_INCREMENT = 1;
Potentially very dangerous, because you can get a number again that is already in use.
What you propose is resetting the sequence to 1 again. It will just produce 1,2,3,4,5,6,7,.. and so on, regardless of these numbers being in a gap or not.
Update: According to Martin's answer, because of the dangers involved, MySQL will not even let you do that. It will reset the counter to at least the current value + 1.
Think again what real problem the existence of gaps causes. Usually it is only an aesthetic issue.
If the number gets too big, switch to a larger data type (bigint should be plenty).
FWIW... According to the MySQL docs applying
ALTER TABLE tbl AUTO_INCREMENT = 1
where tbl contains existing data should have no effect:
To change the value of the
AUTO_INCREMENT counter to be used for
new rows, do this:
ALTER TABLE t2 AUTO_INCREMENT = value;
You cannot reset the counter to a
value less than or equal to any that
have already been used. For MyISAM, if
the value is less than or equal to the
maximum value currently in the
AUTO_INCREMENT column, the value is
reset to the current maximum plus one.
For InnoDB, if the value is less than
the current maximum value in the
column, no error occurs and the
current sequence value is not changed.
I ran a small test that confirmed this for a MyISAM table.
So the answers to you questions are: no harm, and no it won't fill the gaps. As other responders have said: a change of data type looks like the least painful choice.
Chances are you wouldn't gain anything from doing this, and you could easily screw up your application by overwriting rows, since you're going to reset the count for the IDs. (In other words, the next time you insert a row, it'll overwrite the row with ID 1, and then 2, etc.) What will you gain from filling the gaps? If the number gets too big, just change it to a larger number (such as BIGINT).
Edit: I stand corrected. It won't do anything at all, which supports my point that you should just change the type of the column to a larger integer type. The maximum possible value for a BIGINT is 2^64, which is over 18 quintillion. If you only have 100 million rows at the moment, that should be plenty for the foreseeable future.
I agree with musicfreak... The maximum for an integer (int(10)) is 4,294,967,295 (unsigned ofcoarse). If you need to go even higher, switching to BIGINT brings you up to 18,446,744,073,709,551,615.
Since you can't change the next auto-increment value, you have other options. The datatype switch could be done, but it seems a little unsettling to me since you don't actually have that many rows. You'd have to make sure your code can handle IDs that large, which may or may not be tough for you.
Are you able to do much downtime? If you are, there are two options I can think of:
Dump/reload the data. You can do this so it won't keep the ID numbers. For example you could use a SELECT ... INTO to copy the data, sans-IDs, to a new table with identical DDL. Then you drop the old table and rename the new table to the old name. Depending on how much data there is, this could take a noticeable about of time (and temporary disk space).
You could make a little program to issue UPDATE statements to change the IDs. If you let that run slowly, it would "defragment" your IDs over time. Then you could temporarily stop the inserts (just a minute or two), update the last IDs, then restart it. After updating the last IDs you can change the AUTO_INCREMENT value to be the next number and your hole will be gone. This shouldn't cause any real downtime (at least on InnoDB), but it could take quite a while depending on how aggressive your program is.
Of course, both of these ignore referential integrity. I'm assuming that's not a problem (log statements that aren't used as foreign keys, or some such).
Does it really matter if there are gaps?
If you really want to go back and fill them, you can always turn off auto increment, and manually scan for the next available id every time you want to insert a row -- remembering to lock the table to avoid race conditions, of course. But it's a lot of work to do for not much gain.
Do you really need a surrogate key anyway? Depending on the data (you haven't mentioned a schema) you can probably find a natural key.