I'm looking for an insight as I'm a bit stuck.
Right now I'm using auto increment ID's in all of my MySQL innoDB tables. Even for transaction tables (large volume tables).
In my application, someone enters a financial journal which contains of a few transaction lines which are saved in a transaction table.
Now I get requirement requests that the users want to be able to change those transactions rather then insert a total correction for that journal and insert new lines.
No problem for my procedures, but one question arises for me: what if the number of transaction lines for that journal particular changes? If they are the same I could just overwrite the lines with the current auto increment IDs that they already have.
But what if there are more or less journal lines than the first time they were inserted?
Sure, I can delete all previously inserted lines and insert new ones which the auto increment ID field handles fine. But it leaves a gap in the IDs.
I've read before about peoples opinions about the gap I'm talking about, where things go from 'don't worry about it, you'll never run out' to 'oh my, a problem'.
Is it a bad thing if i leave out the auto increment ID as I do not even need it in this trans-table? I always find the rows in the trans-table through the journal number, which is unique.
Are there any drawbacks with leaving out an auto increment ID field on this table?
All I can think if that it maybe may slow down queries on that table (and am not sure of that), but can't think of anything else.
For reference, my trans table looks somewhat like this:
ID --> auto increment bigint
Journal --> int(11)
JournalLineNumber --> int(3)
Description --> varchar (50)
Amount --> decimal
etc.
etc.
By the waym I still do not know how to insert things like tables for displaying mysql table info in my questions in stackoverflow, does anyone know? Googled it again just now but all kinds of confusing tools seems to be needed to simple enter a table of information....but hopefully I'm wrong.
The important thing is that you have a primary key, and that it is reasonably compact. UUID isn't too great because it's big; bigint is fine. int is great. If your have an int or bigint journal number, you can use that as the PK, and you don't need an auto_increment. As long as you have a PK and it isn't huge, performance will not be negatively impacted.
Note that signed (default) int(11) is only good for up to 2^31 (~2bn). Given that your auto_increment was bigint, will you need more than -2bn to +2bn range for this? Should it be unsigned for 0 to 4bn instead? Is 4bn going to be enough for the foreseeable future? ALTER TABLE on 4bn rows is going to take a while.
Paste the output of SHOW CREATE TABLE. To show it as a block, make sure each line is prefixed with at least 4 spaces. Or just use the formatting toolbar.
Related
I'm setting up a table that needs two auto-incrementing fields, 'id' and 'member#'.
I'll use AUTO_INCREMENT = 1001 on the latter for new data, as there is old data with member numbers less than 1000.
I'll use 'MAX(id)+1' on the 'id' field to auto-increment it.
But I'm not sure if this will do the job whenever there's an INSERT, or even where to put that bit of code. All I'm trying to do here is auto-increment the field, not SELECTing anything.
And out of curiosity, why is there only one AUTO_INCREMENTing field per table?
Surely, it can't be difficult to code AUTO_INCREMENT_2, AUTO_INCREMENT_3 etc.
All answers and assistance appreciated.
================================
ADDITIONAL INFORMATION AND LINKS
Sorry for the delay in my response, I've been doing additional research.
Ok so to explain further, we have people joining our group via the net. As such we need to assign a unique membership number to each person. Two John Does? Two different membership numbers. For this I've set the member# column as AUTO_INCREMENT, and then AUTO_INCREMENT = 1001 as a table option. Old membership numbers have three digits, new memberships have four. So each time someone registers as a new member on the web, there's an insert command that automatically assigns the next four digit membership number in the series to the new member.
member# INT(6) UNSIGNED NOT NULL UNIQUE KEY AUTO_INCREMENT
And as a table option AUTO_INCREMENT = 1001
I hope this is clear. Other situations where someone might want to use a similar strategy could be assigning consecutive invoice numbers, receipt numbers, account numbers, etc. So how does one guarantee a +1 result, ie consecutive numbers?
Now we also need a table id column. Lots of tables need a table id. It too needs to be assigned an AUTO_INCREMENT value, in our case, beginning with 1, and incrementing by 1 (the default), to identify and distinguish one row from another. But unfortunately there can be only one AUTO_INCREMENT column per table in MySQL. :-/
So this situation belongs to a class of problems known as MAX+1 problems. (It may also be related to ROW_COUNT and LAST_INSERT_ID solutions.) The limit of a single AUTO_INCREMENT field per table requires a MAX+1 workaround and I am looking for advice on the best way to implement this. For example, is there a way to set this up inside the CREATE TABLE command itself, when defining the id field? Or something else of an equally simple nature, such as writing a function. It is indeed preferable to optimize for efficiency and use only needed features rather than implement a series of commands. Typically a suggested work around might be:
Lock tables membership write;
$max = SELECT MAX(id) FROM membership;
INSERT INTO membership ( id, firstname, lastname )
VALUES ($max+1 , 'jane', 'smith')
unlock tables;
Is there something better?
As whether AUTO_INCREMENT_2 /_3... features should exist. Well, I'd have to point out that there are a lot of features in MySQL that I'll never use, but obviously someone needs them. Nevertheless, it would be convenient to have this for those (rare) occasions when you might need it. Perhaps there is a distinction to be drawn between having a feature available and using it on any given table. I doubt an unused feature requires much in the way of additional memory or clicks (which are pretty cheap these days anyways).
Some links that may prove useful in understanding this situation:
https://duckduckgo.com/?q=mysql+max%2B1+problems&t=ffab&atb=v1-1&ia=web
Insert and set value with max()+1 problems
Problem with MySql INSERT MAX()+1
https://bugs.mysql.com/bug.php?id=3575
All answers, advice and assistance appreciated.
Each InnoDB table has at most one counter for its auto-increment. This is part of the implementation. If you could define N auto-increment columns, in the same table, it would need more storage space to store N counters. It would require the auto-increment lock to last longer while you incremented N counters.
As for why is there only one per table, sure, it is possible that they could implement it to support more than one, but why?
It would make the implementation a lot more complex, and hinder performance, for cases that 99.99% of apps don't need.
They were trying to solve the needs for the majority of cases. In nearly every case of a table with an auto-increment, one per table is sufficient.
In nearly every case where someone like you thinks they need more than one per table, you'd be wise to step back and reconsider your design.
In MySQL the table structure cannot contain more than one auto_increment field. When you try to create a table with 2 autoincremented fields or alter the table in attempt to create second autoincrement, the query fails.
Autoincrement guarantees that each next value generated in the field will be greater than previous one in current connection. But it do NOT guarantee, that each next value generated in the field will be greater than previous value by 1. The "delta" may be 2 or even 1000... it cannot be negative or zero only.
I have a MySQL table that auto increments an id column every time an entry is added. About once per year, I delete all of the data in this table and start new. Right now, the id column's next auto increment value will be 27721.
Questions...
For this type of situation, would it be advisable to reset the auto incrementing to 1 when I delete the table data?
Does it matter?
How high is "too high" for ids like this?
Generally it would not be advisable. You will break things referencing specific IDs, but mostly because...
It doesn't matter.
The auto_increment limit is the sizes of the column. If it's an unsigned int, that's in the billions so if you only have 27721 after a couple years it's probably not a concern.
Moreover I would recommend that you not delete the data and just archive it instead. It would probably make it easier to restore if you had to.
I have to temporarily store lots of data in a table, an hour at a time. For example, each hour I write 20k (or some other large number) of records. I do some stuff with that data during the hour, and by the end of the hour the table is empty again. At the start of the new hour, I add 20k more records and continue the process.
I'm currently storing the data in a MySQL table with an INT auto-increment primary key.
Eventually, I will run out of numbers and I'm looking for a solution that solves the problem, rather than postpones it (e.g. using unsigned BIG INT). I can think of several solutions, but I'm not sure what the consequences are of each - specifically the effects on MySQL. For example, will doing this a particular way cause table corruption or performance problems, etc.
How should problems like this be solved?
Create and remove the table each hour
Reset the auto-increment column each hour
Do either #1 or #2, but at longer intervals
Don't use an auto-increment key (I can easily insert my own sequential key)
Other?
NOTE: For processing purposes, it doesn't matter that the key is re-used each hour. I only use it to distribute the processing of the data.
When you empty the table, reset its auto-increment counter:
ALTER TABLE foo AUTOINCREMENT = 1;
Clearing the rows from the table leaves the counter where it was, and eventually you run out of key space as you indicated. If you reset this counter, and it doesn't matter as each hour is a new set of data, this seems the most effective solution.
If you truly want to empty the table, you should use the TRUNCATE TABLE command, which efficiently wipes out the data and indexes, leaving the table structure intact.
Unless you forgot to mention something you won't run out of numbers any time soon... If the amount of records written stays at 20k records/hour you'll have 12+ years with your current implementation.
And if that is not enough:
Using a SERIAL (alias for BIGINT UNSIGNED NOT NULL AUTO_INCREMENT UNIQUE) and adding about 20k records/hour will still give you over 100 billion years to play with; without having to alter tables or reset the AUTOINCREMENT.
The database I am working on right now has records being added and removed by the 10s of thousands and because of this there are gaps in the auto-incremented key hundreds of thousands big and auto-increment numbers well into the billions.
These numbers are never stored to reference the individual record but are used to reference the record when doing on-the-fly calculations.
Is there any reason to remove these gaps and reset the auto-increment number or is this inconsequential?
The id field is an unsigned int, should I increase it to an unsigned big int? From what I understand, right now if it hits 4,294,967,295 it will break.
The only reason I'd worry about it is if you find yourself close to that 2^32 limit. If you're not using the column as a row id, then don't even worry about it.
EDIT If you are using this column for any kind of identifying information, then I'd switch the column over to a GUID or something, because you're gonna get overflow, and then you'll get duplicate values. And that's no bueno.
I don't know what the growth rate of your autoincrement field is, but it should be simple math for you to estimate when you will hit the 4294967295 limit.
If you still feel that you need to do something, you have the following options:
reset the current count to 1. Do this ideally by dropping the column and recreating it. Since you are not using this field for referential integrity, should be a quick and simple fix until the next time...
Change the datatype to an unsigned BIGINT. Now you can go up to 18446744073709551615. But you need more space in the heap to store this increased amount of data, and you have only postponed your problem.
Change from an autoincrement (INT / BIGINT) to a UUID. Then you can stop worrying about numbers and the nature of infinity, but you most likely will have to change all of your client code.
On a separate note, I sense a poor decision or two somewhere earlier up the line here.
Is there any good practice for this?
I wish I could solve the problem when primary key hit the limit and not to avoid it. Because this is what will happen in my specific problem.
If it's unavoidable... What can i do?
This is mysql question, is there Sybase sql anywhere same problem?
Why would you hit the limit on that field? If you defined it with a datatype large enough it should be able to hold all your records.
If you used an unsigned bigint you can have up to 18,446,744,073,709,551,615 records!
You should pick the correct type for the primary key, if you know you will have lots of rows you could use bigint instead of the commonly used int.
In mysql you can easily adjust the primary key collumn with the alter table statement to adjust the range.
you should also use the unsigned property on that collumn because an auto increment primary key is always positive.
when the limit is reached you could maybe create some algorithm to put inside the ON DUPLICATE KEY UPDATE statement of an INSERT
Well, depending on the autoincrement column's datatype.
Unsign int goes up to 4294967295.
If you want to prevent the error, you can check the value last autoincrement value: LAST_INSERT_ID()
If it's approaching the datatype's max, either do not allow insertion or handle it in other ways.
Other than that, I can only suggest you use bigint so you can almost not hit the max for most scenario.
Can't give you a foolproof answer though :)
I know this question might be too old, but I would like to answer as well.
It is actually impossible to make that scenario unavoidable. Just by
thinking there is a physical limit about how many storage drives
humankind is able to make. But this is certainly not likely to happen
to fill all available storage.
As others have told you, an UNSIGNED BIGINT is able to handle up to 18,446,744,073,709,551,615 records, probably the way to go in "most" cases.
Here is another idea: by triggering the number of records of your database (for example, 85% full) you could backup that table into another table/region/database, and scale your infrastructure accordingly. And then reset that initial table.
And my last approach: some companies opt to do a tiny change to their license and use agreement, that they will shutdown an account if the user do not log in for a certain amount of time (say, 6 months for free users, 6 years for pro users, 60 years for ultimate users... <-- hey! you can also use different tables for those too!).
Hope somebody find this useful.