There's an InnoDB table for storing comments for blog posts used by a custom built web application.
Recently I noticed that the auto incremented primary key values for the comments are incrementing by 2 instead of just 1.
I also noticed that in another MySQL table which is used for remembering the last few commenter's footprint signature (e.g. ip, session id, uagent string, etc) the name of the PHP session starts with "viruskinq" which is weird because I thought it should always be a hexadecimal md5-like string.
Google yields only a couple of results for "viruskinq", all in Turkish. It is interesting because approximately a year ago the website in question was defaced by Turkish villains. (I'm 100% sure that the attackers didn't succeed because of any security holes in my app, because other websites, hosted by the same company, were defaced too at that time.)
The site is on a shared host, using Linux.
Do you think it is possible that the server itself may still be under the influence of those hackers? Examining the comment's id values revealed that this doubling phenomena exists since this May, but the defacing happened almost a year ago.
What other causes could there be that explain the weird behavior of the auto increment value? The application hasn't been changed and at older comments the auto incremented primary key values are in order.
Edit: Summary of the solution
The hosting company informed me that the reason of the doubled auto increment value is because they use a Master-Slave MySQL architect and according to them this phenomena is normal.
They also admitted that various hackers are constantly attacking their servers, "especially the sessions" and they cannot do anything about it.
I think I better start packing my things and move to a better webhost.
I really, really doubt this is a virus. Double-check whether that really is the session ID that starts with that string (which would indeed be reason for some concern). My guess would be this is a kid who discovered how to alter the User Agent string in the browser, and you are seeing the results of that, which is entirely harmless.
In regards to the increment problem.
First, check the auto_increment_increment setting of your mySQL server. Maybe it was set to 2 for some reason?
Second, if it's not that, I would look at all DELETE operations that the comment system runs on the table. Do comments recognized as spam get deleted? Can you log deletions for a while, or switch to soft deletions?
Also, try to create some subsequent comments yourself. Does the same phenonmenon occur? What if you add records using mySQL manually?
Look through the PHP code inserting a submitted comment making really sure there is nothing that could lead to this behaviour.
Try moving the comment system to a different server - preferably a local one, maybe freshly set up - to see whether the behaviour persists there.
Could it just be that the table's auto-increment value is set to 2?
See: MySQL autoincrement column jumps by 10- why?
Related
I have a SQL server table RealEstate with columns - Id, Property, Property_Value. This table has about 5-10 million rows and can increase even more in the future. I want to insert a row only if a combination of Id, Property, Property_Value does not exist in this table.
Example Table -
1,Rooms,5
1,Bath,2
1,Address,New York
2,Rooms,2
2,Bath,1
2,Address,Miami
Inserting 2,Address,Miami should NOT be allowed. But, 2,Price,2billion is okay. I am curious to know which is the "best" way to do this and why. The why part is most important to me. The two ways of checking are -
At application level - The app should check if a row exists before it inserts a row.
At database level - Set unique constraints on all 3 columns and let the database
do the checking instead of person/app.
Is there any scenario where one would be better than the other ?
Thanks.
PS: I know there is a similar question already, but it does not answer my problem -
Unique constraint vs pre checking
Also, I think that UNIQUE is applicable to all databases, so I don't think I should remove the mysql and oracle tags.
I think it most cases the differences between that two are going to be small enough that the choice should mostly be driven by picking the implementation that ends up being most understandable to someone looking at the code for the first time.
However, I think exception handling has a few small advantages:
Exception handling avoids a potential race condition. The 'check, then insert' method might fail if another process inserts a record between your check and your insert. So, even if you're doing 'check then insert' you still want exception handling on the insert and if you're already doing exception handling anyways then you might as well do away with the initial check.
If your code is not a stored procedure and has to interact with the database via the network (i.e. the application and the db are not on the same box), then you want to avoid having two separate network calls (one for the check and the other for the insert) and doing it via exception handling provides a straightforward way of handling the whole thing with a single network call. Now, there are tons of ways to do the 'check then insert' method while still avoiding the second network call, but simply catching the exception is likely to be the simplest way to go about it.
On the other hand, exception handling requires a unique constraint (which is really a unique index), which comes with a performance tradeoff:
Creating a unique constraint will be slow on very large tables and it will cause a performance hit on every single insert to that table. On truly large databases you also have to budget for the extra disk space consumed by the unique index used to enforce the constraint.
On the other hand, it might make selecting from the table faster if your queries can take advantage of that index.
I'd also note that if you're in a situation where what you actually want to do is 'update else insert' (i.e. if a record with the unique value already exists then you want to update that record, else you insert a new record) then what you actually want to use is your particular database's UPSERT method, if it has one. For SQL Server and Oracle, this would be a MERGE statement.
Dependent on the cost of #1 (doing a lookup) being reasonable, I would do both. At least, in Oracle, which is the database I have the most experience with.
Rationale:
Unique/primary keys should be a core part of your data model design, I can't see any reason to not implement them - if you have so much data that performance suffers from maintaining the unique index:
that's a lot of data
partition it or archive it away from your OLTP work
The more constraints you have, the safer your data is against application logic errors.
If you check that a row exists first, you can easily extract other information from that row to use as part of an error message, or otherwise fork the application logic to cope with the duplication.
In Oracle, rolling back DML statements is relatively expensive because Oracle expects to succeed (i.e. COMMIT changes that have been written) by default.
This does not answer the question directly, but I thought it might be helpful to post it here since its better than wikipedia and the link might just become dead someday.
Link - http://www.celticwolf.com/blog/2010/04/27/what-is-a-race-condition/
Wikipedia has a good description of a race condition, but it’s hard to follow if you don’t understand the basics of programming. I’m going to try to explain it in less technical terms, using the example of generating an identifier as described above. I’ll also use analogies to human activities to try to convey the ideas.
A race condition is when two or more programs (or independent parts of a single program) all try to acquire some resource at the same time, resulting in an incorrect answer or conflict. This resource can be information, like the next available appointment time, or it can be exclusive access to something, like a spreadsheet. If you’ve ever used Microsoft Excel to edit a document on a shared drive, you’ve probably had the experience of being told by Excel that someone else was already editing the spreadsheet. This error message is Excel’s way of handling the potential race condition gracefully and preventing errors.
A common task for programs is to identify the next available value of some sort and then assign it. This technique is used for invoice numbers, student IDs, etc. It’s an old problem that has been solved before. One of the most common solutions is to allow the database that is storing the data to generate the number. There are other solutions, and they all have their strengths and weaknesses.
Unfortunately, programmers who are ignorant of this area or simply bad at programming frequently try to roll their own. The smart ones discover quickly that it’s a much more complex problem than it seems and look for existing solutions. The bad ones never see the problem or, once they do, insist on making their unworkable solution ever more complex without fixing the error. Let’s take the example of a student ID. The neophyte programmer says “to know what the next student number should be, we’ll just get the last student number and increment it.” Here’s what happens under the hood:
Betty, an admin. assistant in the admissions office fires up the student management program. Note that this is really just a copy of the program that runs on her PC. It talks to the database server over the school’s network, but has no way to talk to other copies of the program running on other PCs.
Betty creates a new student record for Bob Smith, entering all of the information.
While Betty is doing her data entry, George, another admin. assistant, fires up the student management program on his PC and begins creating a record for Gina Verde.
George is a faster typist, so he finishes at the same time as Betty. They both hit the “Save” button at the same time.
Betty’s program connects to the database server and gets the highest student number in use, 5012.
George’s program, at the same time, gets the same answer to the same question.
Both programs decide that the new student ID for the record that they’re saving should be 5013. They add that information to the record and then save it in the database.
Now Bob Smith (Betty’s student) and Gina Verde (George’s student) have the same student ID.
This student ID will be attached to all sorts of other records, from grades to meal cards for the dining hall. Eventually this problem will come to light and someone will have to spend a lot of time assigning one of them a new ID and sorting out the mixed-up records.
When I describe this problem to people, the usual reaction is “But how often will that happen in practice? Never, right?”. Wrong. First, when data entry is being done by your staff, it’s generally done during a relatively small period of time by everyone. This increases the chances of an overlap. If the application in question is a web application open to the general public, the chances of two people hitting the “Save” button at the same time are even higher. I saw this in a production system recently. It was a web application in public beta. The usage rate was quite low, with only a few people signing up every day. Nevertheless, six pairs of people managed to get identical IDs over the space of a few months. In case you’re wondering, no, neither I nor anyone from my team wrote that code. We were quite surprised, however, at how many times that problem occurred. In hindsight, we shouldn’t have been. It’s really a simple application of Murphy’s Law.
How can this problem be avoided? The easiest way is to use an existing solution to the problem that has been well tested. All of the major databases (MS SQL Server, Oracle, MySQL, PostgreSQL, etc.) have a way to increment numbers without creating duplicates. MS SQL server calls it an “identity” column, while MySQL calls it an “auto number” column, but the function is the same. Whenever you insert a new record, a new identifier is automatically created and is guaranteed to be unique. This would change the above scenario as follows:
Betty, an admin. assistant in the admissions office fires up the student management program. Note that this is really just a copy of the program that runs on her PC. It talks to the database server over the school’s network, but has no way to talk to other copies of the program running on other PCs.
Betty creates a new student record for Bob Smith, entering all of the information.
While Betty is doing her data entry, George, another admin. assistant, fires up the student management program on his PC and begins creating a record for Gina Verde.
George is a faster typist, so he finishes at the same time as Betty. They both hit the “Save” button at the same time.
Betty’s program connects to the database server and hands it the record to be saved.
George’s program, at the same time, hands over the other record to be saved.
The database server puts both records into a queue and saves them one at a time, assigning the next available number to them.
Now Bob Smith (Betty’s student) gets ID 5013 and Gina Verde (George’s student) gets id 5014.
With this solution, there is no problem with duplication. The code that does this for each database server has been tested repeatedly over the years, both by the manufacturer and by users. Millions of applications around the world rely on it and continue to stress test it every day. Can anyone say the same about their homegrown solution?
There is at least one well tested way to create identifiers in the software rather than in the database: uuids (Universally Unique Identifiers). However, a uuid takes the form of xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx where “x” stands for a hexadecimal digit (0-9 and a-f). Do you want to use that for an invoice number, student ID or some other identifier seen by the public? Probably not.
To summarize, a race condition occurs when two programs, or two independent parts of a program, attempt to access some information or access a resource at the same time, resulting in an error, be it an incorrect calculation, a duplicated identifier or conflicting access to a resource. There are many more types of race conditions than I’ve presented here and they affect many other areas of software and hardware.
The description of your problem is exactly why primary keys can be compound, e.g., they consist of multiple fields. That way, the database will handle the uniqueness for you, and you don't need to care about it.
In your case, the table definition could be something similar to the following like:
CREATE TABLE `real_estate` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`property` varchar(255) DEFAULT NULL,
`property_value` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `index_id_property_property_value` (`id`, `property`, `property_value`),
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
I have a question that I didn't find an answer. Yet ;-)
I have a Django/MySQL application that runs memcached in the background. One of my tables change every access. I mean, when the user access the page I have a "count" field that is incremented, this same table contains all data that is going to be displayed.
Is recommended to use memcached in this scenario? Or should I create a new relation table that will contain only "id" and "count" field?
Thanks!
Sure, that's a valid use for memcached. The basic rule is that anytime you update, or delete in the mysql sense of the words, you need to do something to keep the memcache record consistent. Usually that is done by either adjusting the value right there, or deleting it so the next access of it builds it and saves it.
In your case, I would just get the value, increment it, and then set it. Depending on how important accuracy is to you, and how much concurrent traffic you get, you should consider atomicity of the transactions outlined in this post.
I have a fairly large database that I use for tracking items installed in a home by our service reps. For programmatic simplicity I wrote the tracking page so that every time anyone updates, removes or adds a new installed item it totally clears that home's installed item list and rebuilds it from scratch.
This works very well and has been error free in actual use, but now I've come into a different problem that I'm a bit worried about. The primary key that is used to track each particular item in the home has grown exponentially, because for every update it clears out old numbers and starts again from the highest auto_increment. This means I have large gaps in my ids and my highest index is thousands of numbers higher than the actual count of installed measures.
For clarification: I don't care that there are gaps in the ids, I built my system to only use that number as a foreign key reference to the billing information for it and it's never displayed. My actual concern is that I'm going to run out of digits far, far sooner than should be possible.
I know that I could change my script around to be "more efficient" and not delete items that don't change and I may end up doing that in the future (this issue is a symptom of the purpose of my tracking radically changing in the middle of a project. Thanks, boss), but in the mean time I'd like to know if there is a way to "clean up" my ids. Everything that depends on those numbers is set to cascade so there shouldn't be an issue with updating the keys. Basically I'd like to start with 1, eliminate the gaps between the ids and avoid clashing with existing ids as the script runs.
I'm hoping that someone can provide a simple means of doing this, hopefully one that can be implemented as a stored procedure and run routinely.
There are two options to reset the auto_increment:
Truncate the table
Reset the auto_increment
This is done by:
ALTER TABLE tablename AUTO_INCREMENT=10000
So, you can just clear the auto_increment.
Otherwise, I would recommend you to increase the integer size. Use BIGINT, not INT.
In my hoptoad logs I will periodically see
Mysql::Error: Duplicate entry 'XXXX' for key 'YYY'
This happens for most of my models, about 6 in all, and I will see this error once every few hours on a site doing about 5k requests/minute according to newrelic.
I am doing an ActiveRecord.find_or_initialize_by in each of these cases. It's possible, but unlikely, that this is from a client in the field doing two simultaneous posts of the same data, because these are mobile clients and the codepath doesn't really lend itself to that (ie, this isn't a client clicking a submit button twice quickly).
Is there a known issue with find_or_initialize_by? Is it possible my mysql instance (Amazon RDS) is every so often just flaking out (though I would expect it to raise an exception in that case vs. returning no data)...
Moreover, is there a better way to be inserting records? If the record exists, I generally am only updating its updated_at field.
Thanks!
This is most likely occurring because "check validity then insert" isn't an atomic operation. There's no guarantee that someone else can't insert a row with the same value for a unique column in between the validity check and the insert.
The official docs mention this, but only in passing, and they really don't explain it very well. Rails Warts has a much better page on the problem.
And as it sounds like you're backing the uniqueness condition up with a unique index in your DB, you're already doing what you can to prevent it on the DB side. And I'm not sure if find_or_initialize_by / ON DUPLICATE KEY UPDATE is a good idea or not - it depends on what your users are editing, and the security consequences of them editing something they didn't necessarily intend to.
Hope this helps!
I have this site sinograms.com. I 'm screen scraping news sites to categorize Chinese Characters through frequency and I've found the same stuff your coming up with.
I've gone through many stages. The current one up is just the last test and it only has a few million indexes. They duplicates seem to come up quickly; i know this because I only find duplicates of Chinese Characters that are very common or indexed frequently in other words.
I am designing a system and I don't think it's a good idea to give the ability to the end user to delete entries in the database. I think that way because often then end user, once given admin rights, might end up making a mess in the database and then turn to me to fix it.
Of course, they will need to be able to do remove entries or at least think that they did if they are set as admin.
So, I was thinking that all the entries in the database should have an "active" field. If they try to remove an entry, it will just set the flag to "false" or something similar. Then there will be some kind of super admin that would be my company's team who could change this field.
I already saw that in another company I worked for, but I was wondering if it was a good idea. I could just make regular database backups and then roll back if they commit an error and adding this field would add some complexity to all the queries.
What do you think? Should I do it that way? Do you use this kind of trick in your applications?
In one of our databases, we distinguished between transactional and dictionary records.
In a couple of words, transactional records are things that you cannot roll back in real life, like a call from a customer. You can change the caller's name, status etc., but you cannot dismiss the call itself.
Dictionary records are things that you can change, like assigning a city to a customer.
Transactional records and things that lead to them were never deleted, while dictionary ones could be deleted all right.
By "things that lead to them" I mean that as soon as the record appears in the business rules which can lead to a transactional record, this record also becomes transactional.
Like, a city can be deleted from the database. But when a rule appeared that said "send an SMS to all customers in Moscow", the cities became transactional records as well, or we would not be able to answer the question "why did this SMS get sent".
A rule of thumb for distinguishing was this: is it only my company's business?
If one of my employees made a decision based on data from the database (like, he made a report based on which some management decision was made, and then the data report was based on disappeared), it was considered OK to delete these data.
But if the decision affected some immediate actions with customers (like calling, messing with the customer's balance etc.), everything that lead to these decisions was kept forever.
It may vary from one business model to another: sometimes, it may be required to record even internal data, sometimes it's OK to delete data that affects outside world.
But for our business model, the rule from above worked fine.
A couple reasons people do things like this is for auditing and automated rollback. If a row is completely deleted then there's no way to automatically rollback that deletion if it was in error. Also, keeping a row around and its previous state is important for auditing - a super user should be able to see who deleted what and when as well as who changed what, etc.
Of course, that's all dependent on your current application's business logic. Some applications have no need for auditing and it may be proper to fully delete a row.
The downside to just setting a flag such as IsActive or DeletedDate is that all of your queries must take that flag into account when pulling data. This makes it more likely that another programmer will accidentally forget this flag when writing reports...
A slightly better alternative is to archive that record into a different database. This way it's been physically moved to a location that is not normally searched. You might add a couple fields to capture who deleted it and when; but the point is it won't be polluting your main database.
Further, you could provide an undo feature to bring it back fairly quickly; and do a permanent delete after 30 days or something like that.
UPDATE concerning views:
With views, the data still participates in your indexing scheme. If the amount of potentially deleted data is small, views may be just fine as they are simpler from a coding perspective.
I prefer the method that you are describing. Its nice to be able to undo a mistake. More often than not, there is no easy way of going back on a DELETE query. I've never had a problem with this method and unless you are filling your database with 'deleted' entries, there shouldn't be an issue.
I use a combination of techniques to work around this issue. For some things adding the extra "active" field makes sense. Then the user has the impression that an item was deleted because it no longer shows up on the application screen. The scenarios where I would implement this would include items that are required to keep a history...lets say invoice and payment. I wouldn't want such things being deleted for any reason.
However, there are some items in the database that are not so sensitive, lets say a list of categories that I want to be dynamic...I may then have users with admin privileges be allowed to add and delete a category and the delete could be permanent. However, as part of the application logic I will check if the category is used anywhere before allowing the delete.
I suggest having a second database like DB_Archives whre you add every row deleted from DB. The is_active field negates the very purpose of foreign key constraints, and YOU have to make sure that this row is not marked as deleted when it's referenced elsewhere. This becomes overly complicated when your DB structure is massive.
There is an acceptable practice that exists in many applications (drupal's versioning system, et. al.). Since MySQL scales very quickly and easily, you should be okay.
I've been working on a project lately where all the data was kept in the DB as well. The status of each individual row was kept in an integer field (data could be active, deleted, in_need_for_manual_correction, historic).
You should consider using views to access only the active/historic/... data in each table. That way your queries won't get more complicated.
Another thing that made things easy was the use of UPDATE/INSERT/DELETE triggers that handled all the flag changing inside the DB and thus kept the complex stuff out of the application (for the most part).
I should mention that the DB was a MSSQL 2005 server, but i guess the same approach should work with mysql, too.
Yes and no.
It will complicate your application much more than you expect since every table that does not allow deletion will be behind extra check (IsDeleted=false) etc. It does not sound much but then when you build larger application and in query of 11 tables 9 require chech of non-deletion.. it's tedious and error prone. (Well yeah, then there are deleted/nondeleted views.. when you remember to do/use them)
Some schema upgrades will become PITA since you'll have to relax FK:s and invent "suitable" data for very, very old data.
I've not tried, but have thought a moderate amount about solution where you'd zip the row data to xml and store that in some "Historical" table. Then in case of "must have that restored now OMG the world is dying!1eleven" it's possible to dig out.
I agree with all respondents that if you can afford to keep old data around forever it's a good idea; for performance and simplicity, I agree with the suggestion of moving "logically deleted" records to "old stuff" tables rather than adding "is_deleted" flags (moving to a totally different database seems a bit like overkill, but you can easily change to that more drastic approach later if eventually the amount of accumulated data turns out to be a problem for a single db with normal and "old stuff" tables).