I have the following table:
CREATE TABLE `some_table` (
`ReferenceId` int(11) DEFAULT NULL,
`ten` int(10) DEFAULT NULL,
`so` bigint(18) DEFAULT NULL,
`mc` bigint(18) DEFAULT NULL,
`ev` bigint(18) DEFAULT NULL,
`sclso` bigint(18) DEFAULT NULL,
`sowbse` bigint(18) DEFAULT NULL,
`AsOfDate` date DEFAULT NULL,
`dud` date NOT NULL,
UNIQUE KEY `ReferenceId` (`ReferenceId`,`AsOfDate`),
KEY `fk_main_table` (`ReferenceId`),
CONSTRAINT `fk_main_table` FOREIGN KEY (`ReferenceId`) REFERENCES `some_other_table` (`Id`) ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
In this table I have added a multiple column UNIQUE index on columns ReferenceId and AsOfDate. But I just noticed that there is a duplicate entry in the table even when we have this constraint.
Check the 2 highlighted records. The constraint is applied on first and second-last columns, which are identical but should not have existed.
What could be the possible issues? The data in this table is not inserted/updated from any web/desktop application but only from 1 script that runs in background.
Edit: I have only identified this 1 index being duplicate and the script have been running for past 3 months.
Either one of two things is true:
You're mistaken
Your database is corrupt
To verify your assertion, write a query to show only the invalid condition:
select count(*) as N, ReferenceId, AsOfDate
from some_table
group by ReferenceId, AsOfDate
having count(*) > 1
(You can dispense with the unnecessary, nonstandard backtick-quotes, by the way. You'll find it makes SQL more pleasant to deal with.)
If that query produces any rows, your database is corrupt, by definition: the table cannot be declared unique on two columns and admit two rows of the same values. Find out what's wrong, and fix it.
If it doesn't produce any rows, it might still be corrupt, but that's evidence you're mistaken. You'll want to re-check your facts, and see if there's another explanation for what you're seeing. Get your hands on the verbatim SQL that produced that output (or is supposed to have done). Execute it, redirecting the output to a temporary table or file, and verify the duplication. If you don't find it, it's not there. If you do, see "corruption" in your friendly manual.
One last thing, just as an aside. This line:
KEY `fk_main_table` (`ReferenceId`),
is likely not doing you much good. You already have
UNIQUE KEY `ReferenceId` (`ReferenceId`,`AsOfDate`),
and your DBMS probably creates an index to enforce that, and probably uses that index to locate rows by ReferenceId.
Related
I have a table that needs a unique constraint on 3 columns, but, if the "date" column in for that insert transaction is a newer date than the current record's date, then I want to update that record (so the unique constraint is still true for the table).
Postgres has the concept of deferrable constraints, MySQL does not.
I do want to implement it with the SQL object tools available, though.
Here is my table DDL with column names obfuscated:
CREATE TABLE `apixio_results_test_sefath` (
`ID` int(11) NOT NULL AUTO_INCREMENT,
`number` varchar(20) DEFAULT NULL,
`insert_date` datetime DEFAULT NULL,
`item_id` int(5) DEFAULT NULL,
`rule` tinyint(4) DEFAULT NULL,
`another_column` varchar(20) DEFAULT NULL,
`another_column1` varchar(20) DEFAULT NULL,
PRIMARY KEY (`ID`),
KEY `insert_date_index` (`insert_date`),
KEY `number` (`number`),
) ENGINE=InnoDB AUTO_INCREMENT=627393 DEFAULT CHARSET=latin1
and here is the unique constraint statement
Alter Table dbname.table add unique constraint my_unique_constraint (number, item_id, rule);
but I can not add a condition here in this constraint (unless there is a way I'm not aware of?)
The logic I need to run before inserts are blocked by the constraint is to check if the three values: number, item_id, and rule are unique in the table, and if they aren't, then I want to compare the existing record's insert_date with the insert_date from the transaction, and only keep the record with the newest insert_date.
This could be achieved with a trigger I suppose, although I've heard triggers are only to be used if really needed. And on every insert, this trigger would be quite computationally taxing on the DB. Any advice? Any other sql tricks I can use? Or anything to help point me to how to make this trigger?
I tried the unique constraint statement
Alter Table dbname.table add unique constraint my_unique_constraint (number, item_id, rule);
But it will never update with the newer insert_date.
You can do this with an insert statement like:
insert into apixio_results_test_sefath (number, item_id, rule, insert_date, another_column, another_column1)
values (?,?,?,?,?,?)
on duplicate key update
another_column=if(insert_date>values(insert_date),another_column,values(another_column),
another_column1=if(insert_date>values(insert_date),another_column1,values(another_column1),
insert_date=greatest(insert_date,values(insert_date)
for each column besides the unique ones and insert_date, testing to see if the existing insert_date is greater than the value supplied with the insert and conditionally using the existing value or new value for the other column based on that, and ending with updating insert_date only if it is now greater.
mysql 8 has an alternate syntax it prefers to using the values function, but the values function still works.
If you want this to happen automatically for all inserts, you would need to use a trigger.
Example:
Here is the employee table:
CREATE TABLE `employees` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(50) NOT NULL,
`code` varchar(4) NOT NULL,
`deleted_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
);
The code is a simple login code of 4 characters. Soft delete is implemented using deleted_at field. Current employees are those with deleted_at=NULL.
We need to keep the code unique between the current employees.
Using a UNIQUE Constraints on the code field will prevent current employees from using codes that have been used by a soft-deleted employee.
How to enforce this constraint?
This is an example of the general problem of how to enforce consistency constraints in MySQL.
Edit:
The schema could be changed to make use of unique constraints as #bill-karwin suggests.
What about applying complex consistency constraints that may span multiple tables?
One way (if possible) is to change the schema in order to apply the constraints using foreign key constraint or unique constraint.
Is there another way to apply complex consistency constraints?
One relatively simple solution to your problem would be to change the deleted_at column to default to something other than NULL (e.g. '1900-01-01', or even the "zero" date '0000-00-00' if you have them enabled). You can then create a UNIQUE index on (code, deleted_at) which would prevent any employee from using a code which a current employee had (since you would get a match on (code,default)), but not exclude them using a code which a previous employee had used, since the default value would not match the deleted_at timestamp.
One solution is to create a nullable column is_active that is restricted to either NULL or a single non-NULL value. The columns code and is_active together must be unique.
CREATE TABLE `employees` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(50) NOT NULL,
`code` varchar(4) NOT NULL,
`is_active` enum('yes'),
`deleted_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY (`code`, `is_active`)
);
If is_active is NULL, then it allows any number of duplicates in the code column. If is_active is not NULL, then it allows only one value 'yes' and therefore each value in the code column must be unique.
deleted_at no longer indicates if the employee is currently active or not, only when then were inactivated.
Re your comment:
Constraints that span multiple tables are called ASSERTIONS in the SQL standard, but there is literally no RDBMS product that implements that feature from the standard.
Some implement constraints with triggers, but it's not always obvious how to design triggers to do what you want efficiently.
Honestly, most people resort to application logic for these sorts of constraints. This comes with some risk of race conditions. As soon as you do a SELECT statement to verify the data satisfies the constraints, some other concurrent session may commit data that spoils the constraint before your session can commit its changes.
The only solution is to use pessimistic locking to ensure no other session can jump ahead of you.
I have a composite unique key on two columns, "user_id" & "project_id".
When I try to run a DELETE query on single rows or multiple rows, I get an error.
ERROR 1062: 1062: Duplicate entry '87-1736' for key 'index_on_user_id_and_project_id'
SQL Statement:
DELETE FROM `members` WHERE `id`='39142'
The table has a single column primary key, 2 single column unique indexes (for user_id and project_id), and 1 composite unique index on user_id and project_id. No foreign keys in the database.
`id` int(11) NOT NULL AUTO_INCREMENT,
`user_id` int(11) NOT NULL DEFAULT '0',
`project_id` int(11) NOT NULL DEFAULT '0',
`created_on` datetime DEFAULT NULL,
`mail_notification` tinyint(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`id`),
UNIQUE KEY `index_on_user_id_and_project_id` (`user_id`,`project_id`),
KEY `index_members_on_user_id` (`user_id`),
KEY `index_members_on_project_id` (`project_id`)
This error only shows up for certain entries (a lot of entries) and it is consistently those entries that are problematic (e.g. 87 and 1736 pair shown above).
I have tried looking for duplicates and none were found. I was able to find some entries in there with "0"s in the fields and I removed those entries. No NULL fields were found.
I have tried:
looking for duplicates, found none.
looking for zero or NULL values in the index fields, deleted, but did not solve
removing the composite unique index, did not solve.
alter ignore table ... add unique index (user_id, project_id), it found no duplicates, threw a warning about IGNORE being deprecated, and did not solve
How do I delete these problematic entries?
It is impossible for a delete statement itself to generate a duplicate key error. At least, I cannot think of any way for that to happen in an unbroken database. After all, if you are removing a value, it can't conflict with another value.
That leaves the possibility that something else is going on. The only reasonable alternative is a trigger on the table. It is unfortunate that the error message doesn't specify the table name, but that is the only cause that I can readily think of.
I've bumped into this before when I've had the target table (t1) tied to a history table (t1_hist). t1_hist was populated by a trigger on changes to the t1 (any add/change/delete). Once I deleted the unwanted records from the t1_hist, I was able to delete from the t1. This required a second pass delete from the t1_hist table because it recorded my deletes from t1.
Simply, mine was: DELETE FROM t1 WHERE customer_number > 50000;
Same error. Did the same from t1_hist first (and last), then no problem.
Let's say we have a (InnoDB) table associations in a MySQL-Database which has the following structure:
CREATE TABLE `associations` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`fk_id_1` int(11) NOT NULL,
`fk_id_2` int(11) NOT NULL,
`fk_id_3` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `some_unique_constraint` (`fk_id_1`,`fk_id_2`),
KEY `fk_id_2_INDEX` (`fk_id_2`),
KEY `fk_id_3_INDEX` (`fk_id_3`),
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin$$
There are jumps in the column id (I know this is an issue of how the autoincremented value is generated while multiple threads try to get one). Since no other table is using the column id as a reference I plan to drop the column id and to create it again, hopefully the counting holes will be gone. I backed up my database and tested that. The result was a little confusing. The order of the rows seemed to have changed. If I am not mistaken the order is first by fk_id_1 then fk_id_2 then fk_id_3.
Is this the natural order in which MySQL sets the table, when assignung an new generated autoincrement key to the rows?
Is there more I should know, that happened during this process?
The reason, why I need to know about this is that I need to make the column id useful for another task I intend to accomplish where gaps are a no go.
There is no natural order to a table in any mainstream RDBS.
Only the outermost ORDER BY in a SELECT statement will guarantee the order of results.
If you want "order":
create a new table
INSERT..SELECT..ORDER BY fk_id_1, fk_id_2, fk_id_3
Drop old table
Rename new table
Or live with gaps... OCD isn't good for developers
Edit:
Question says "no dependency" on this value but turns out there is.
If gaps are not allowed then don't use autonumber and use fk_id_1, fk_id_2, fk_id_3 as your key, with a ROW_NUMBER emulation. Or code your downstream to deal with gaps.
Autonumbers will have gaps: immutable fact of life.
RESOLVED
From the developer: the problem was that a previous version of the code was still writing to the table which used manual ids instead of the auto increment. Note to self: always check for other possible locations where the table is written to.
We are getting duplicate keys in a table. They are not inserted at the same time (6 hours apart).
Table structure:
CREATE TABLE `table_1` (
`sales_id` int(10) unsigned NOT NULL auto_increment,
`sales_revisions_id` int(10) unsigned NOT NULL default '0',
`sales_name` varchar(50) default NULL,
`recycle_id` int(10) unsigned default NULL,
PRIMARY KEY (`sales_id`),
KEY `sales_revisions_id` (`sales_revisions_id`),
KEY `sales_id` (`sales_id`),
KEY `recycle_id` (`recycle_id`)
) ENGINE= MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=26759 ;
The insert:
insert into `table_1` ( `sales_name` ) VALUES ( "Blah Blah" )
We are running MySQL 5.0.20 with PHP5 and using mysql_insert_id() to retrieve the insert id immediately after the insert query.
I have had a few duplicate key error suddenly appear in MySql databases in the past even though the primary key is defined and auto_increment. Each and every time it has been because the table has become corrupted.
If it is corrupt performing a check tables should expose the problem. You can do this by running:
CHECK TABLE tbl_name
If it comes back as corrupt in anyway (Will usually say the size is bigger than it actually should be) then just run the following to repair it:
REPAIR TABLE tbl_name
Does the sales_id field have a primary (or unique) key? If not, then something else is probably making inserts or updates that is re-using existing numbers. And by "something else" I don't just mean code; it could be a human with access to the database doing it accidentally.
As the other said; with your example it's not possible.
It's unrelated to your question, but you don't have to make a separate KEY for the primary key column -- it's just adding an extra not-unique index to the table when you already have the unique (primary) key.
We are getting duplicate keys in a table.
Do you mean you are getting errors as you try to insert, or do you mean you have some values stored in the column more than once?
Auto-increment only kicks in when you omit the column from your INSERT, or try to insert NULL or zero. Otherwise, you can specify a value in an INSERT statement, over-riding the auto-increment mechanism. For example:
INSERT INTO table_1 (sales_id) VALUES (26759);
If the value you specify already exists in the table, you'll get an error.
Please post the results of this query:
SELECT `sales_id`, COUNT(*) AS `num`
FROM `table_1`
GROUP BY `sales_id`
HAVING `num` > 1
ORDER BY `num` DESC
If you have a unique key on other fields, that could be the problem.
If you have reached the highest value for your auto_increment column MySQL will keep trying to re-insert it. For example, if sales_id was a tinyint column, you would get duplicate key errors after you reached id 127.