I am trying to migrate data from Oracle table over to MariaDB table. I have csv files from the Oracle table unload and a load script to load these files into corresponding mariadb tables which I have created using the same table/schema definition as it is in Oracle side. I am getting the 1062 (230000) Duplicate entry error however I don't understand why MySQL/Mariadb treats the data as unique? The value in the 1st record which gets inserted is in lower case and in the 2nd insert which fails is in uppercase. It seems that both mysql and mariadb are treating the value the same and hence I am getting the Duplicate entry error for primary key. In order to show my error, I have created a test table and manually attempted to insert 2 rows. I have put all the relevant info below and would appreciate if there is some explanation. Thanks in advance for your help.
MariaDB [mytestdb]> create table test1 (id char(15) not null, recnum integer, primary key(id));
MariaDB [mytestdb]> show create table test1\G
*************************** 1. row ***************************
Table: test1
Create Table: CREATE TABLE `test1` (
`id` char(15) NOT NULL,
`recnum` int(11) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
MariaDB [mytestdb]> insert into test1 values ('abc123',1);
Query OK, 1 row affected (0.01 sec)
MariaDB [mytestdb]> insert into test1 values ('ABC123',2);
ERROR 1062 (23000): Duplicate entry 'ABC123' for key 'PRIMARY'
Thanks,
Sanjay
MySQL isn't case sensitive by default. You should define the column with one of the COLLATION options (*_cs) suitable for the datatype, or if none exists, the _bin collation. Or, if all else fails (eg, the _cs or _bin collations for your datatype don't exist), you may need to set the column to BINARY charset instead.
As a starting point, https://dev.mysql.com/doc/refman/5.7/en/charset-mysql.html might be useful.
Related
Using MySQL 5.6.40
This is the table definition:
CREATE TABLE IF NOT EXISTS `updated_tables` (
`table_name` VARCHAR(50) NOT NULL,
`updated_at` TIMESTAMP(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6),
PRIMARY KEY (`table_name`),
UNIQUE INDEX `table_name_UNIQUE` (`table_name` ASC))
ENGINE = InnoDB;
This is the query (Generated by Sequalize):
INSERT INTO `updated_tables` (`table_name`,`updated_at`) VALUES ('workdamnit',NULL) ON DUPLICATE KEY UPDATE `table_name`=VALUES(`table_name`), `updated_at`=VALUES(`updated_at`);
Simplified form of the same query:
INSERT INTO `updated_tables` (`table_name`,`updated_at`) VALUES ('workdamnit',NULL)
And it produces the following entry in table:
table_name: workdamnit
updated_at: 2018-07-05 14:27:17.142494
Now to the question.
Using MySQL 5.6.39-log on AWS RDS
Gives this error:
Error Code: 1048. Column 'updated_at' cannot be null
Is it since the MySQL versions are a bit different, or it has to do something with RDS?
During the creation of your table you have mentioned the following on your column: updated_at:
Do not allow NULL
If nothing is provided, use CURRENT_TIMESTAMP(6) as default.
Hence, It is giving the error when you are inserting NULL in your query.
So, If I understand what you are trying to do currectly, this should be your query:
INSERT INTO `updated_tables` (`table_name`) VALUES ('workdamnit')
instead of
INSERT INTO `updated_tables` (`table_name`,`updated_at`) VALUES ('workdamnit',NULL)
Hope it helps
You are trying to INSERT NULL on a field that does not allow it. Furthermore this field has a default value. so you can simply try:
INSERT INTO `updated_tables` (`table_name`) VALUES ('workdamnit') ...
After some research I found the answer. The RDS instance has a strict mode turned on by default, so it was responding properly. While my local instance, and also an instance of MySQL on EC2, were not using the strict mode.
I have the below table columns in MySQL.
id
user_primary_email
user_secondary_email
I want to make the combination of columns user_primary_email and user_secondary_email unique which I can achieve by using UNIQUE KEY unique_key_name (user_primary_email, user_secondary_email)
The above addition of unique key constraint will help me achieve the below scenario or rather just by adding a unique key to the individual column itself.
user_primary_email = 'xyz#gmail.com' AND user_secondary_email = 'pqr#gmail.com'
user_primary_email = 'xyz#gmail.com' AND user_secondary_email = 'pqr#gmail.com' //This will not be allowed to enter due to unique key constraint
Now the problem which I am facing is the same combination should not be allowed to add in a reverse way as mentioned below.
user_primary_email = 'pqr#gmail.com' AND user_secondary_email = 'xyz#gmail.com' //This should not be allowed to add since already same email id combination added once
id | user_primary_email | user_secondary_email
-------------------------------------------------------
1 | xyz#gmail.com | pqr#gmail.com
-------------------------------------------------------
2 | pqr#gmail.com | xyz#gmail.com
-------------------------------------------------------
In the above case during insert of row id 2 it should throw error as both the email id combination is already used in row id 1.
Any help would be great.
In any MariaDB:
CREATE TABLE `t` (
`id` int(11) NOT NULL,
`user_primary_email` varchar(64) DEFAULT NULL,
`user_secondary_email` varchar(64) DEFAULT NULL,
`mycheck` varchar(128) AS (IF(user_primary_email<user_secondary_email,CONCAT(user_primary_email,user_secondary_email),CONCAT(user_secondary_email,user_primary_email))) PERSISTENT,
PRIMARY KEY (`id`),
UNIQUE KEY `mycheck` (`mycheck`)
);
MariaDB [test]> insert into t values (1,'a','b',null);
Query OK, 1 row affected (0.03 sec)
MariaDB [test]> insert into t values (2,'b','a',null);
ERROR 1062 (23000): Duplicate entry 'ab' for key 'mycheck'
There is no direct support for that, but you can use a workaround to create your bidirectional key: You need a unique key on an ordered version of your two columns.
Fortunately, you can very easily do that. MySQL 5.7.6+ supports generated columns and unique indexes for them, which you can use to order your two values and to enforce uniqueness.
create table testBiDirKey (
a varchar(100),
b varchar(100),
a_ordered varchar(100) as (least(a, b)) STORED,
b_ordered varchar(100) as (greatest(a, b)) STORED,
unique key unqBi_test_ab (a_ordered, b_ordered)
);
insert into testBiDirKey(a,b) values('a', 'b');
insert into testBiDirKey(a,b) values('b', 'a');
Error Code: 1062. Duplicate entry 'a-b' for key 'unqBi_test_ab'
This will treat null exactly as your current normal unique key, so
insert into testBiDirKey(a,b) values('a', null);
insert into testBiDirKey(a,b) values('a', null);
insert into testBiDirKey(a,b) values(null, 'a');
are all allowed. You can add coalesce(x,'') to only allow one empty value (either null OR '') if you want. If you verify your values before you add them (e.g. if they don't contain a ,), you can combine the two columns to just one, concatenated with an , - although with little benefit apart from just having 1 additional column.
For 5.7.8+, you don't need the STORED keyword anymore to be able to use these columns in an index. That keyword effects if the values are stored (using disk space) or calculated when required (default).
Before MySQL 5.7.6, you can use a trigger (on update and insert) to update the two columns with the these values, the same logic applies, it's just a little more code.
I have Table called url_info and the structure of the table is:
url_info:
url_id ( auto_increment, primary key )
url ( unique,varchar(500) )
When I insert into table like this:
INSERT INTO url_info(url) VALUES('Tom');
INSERT INTO url_info(url) VALUES('Jerry');
The output is:
1 Tom
2 Jerry
When I insert like this
INSERT INTO url_info(url) VALUES('Tom');
INSERT INTO url_info(url) VALUES('Tom');
INSERT INTO url_info(url) VALUES('Jerry');
The output is
1 Tom
3 Jerry
The auto-increment id is incremented when I try to insert to duplicate entry. I have also tried Insert Ignore
How to prevent it from incrementing when I try to insert a duplicate entry?
It's probably worth creating a stored procedure to insert what you want into the table. But, in the stored procedure check what items you have already in the table. If these match what you're trying to insert, then the query should not even attempt the insert.
Ie. The procedure needs to contain something like this:
IF NOT EXISTS(SELECT TOP 1 url_id FROM url_info WHERE url = 'Tom')
INSERT INTO url_info(url) VALUES('Tom')
So, in your stored procedure, it would look like this (assuming the arguments/variables have been declared)
IF NOT EXISTS(SELECT TOP 1 url_id FROM url_info WHERE url = #newUrl)
INSERT INTO url_info(url) VALUES(#newUrl)
This is expected behaviour in InnoDB. The reason is that they want to let go of the auto_increment lock as fast as possible to improve concurrency. Unfortunately this means they increment the AUTO_INCREMENT value before resolving any constraints, such as UNIQUE.
You can read more about the idea in the manual on AUTO_INCREMENT Handling in InnoDB, but the manual is also unfortunately buggy and doesn't tell why your simple insert will give non-consecutive values.
If this is a real problem for you and you really need consecutive numbers, consider setting the innodb_autoinc_lock_mode option to 0 in your server, but this is not recommended as it will have severe effects on your database (you cannot do any inserts concurrently).
Auto_increment is performed updated by the engine. This is done before hand of checking a value is unique or not. And we can't roll back the operation to get back to former value of auto_increment.
Hence NO to start from where you last read on auto_increment.
And it is not an issue in loosing some intermediate values on auto_increment field.
The MAX value you can store into a SIGNED INT field is 2^31-1 equal to 2,147,483,647. If you read it loud, it sounds 2 billion+.
And I don't think it is small and won't suite your requirement.
CREATE TABLE `url_info` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`url` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=4 ;
When I execute:
INSERT INTO url_info(url) VALUES('Tom');
INSERT INTO url_info(url) VALUES('Tom');
INSERT INTO url_info(url) VALUES('Jerry');
I get:
Make sure you ID column is UNIQUE too.
As the manual says:
A UNIQUE index creates a constraint such that all values in the index
must be distinct. An error occurs if you try to add a new row with a
key value that matches an existing row. This constraint does not apply
to NULL values except for the BDB storage engine. For other engines, a
UNIQUE index permits multiple NULL values for columns that can contain
NULL. If you specify a prefix value for a column in a UNIQUE index,
the column values must be unique within the prefix.
I know this has been discussed before but when I read the other threads, they don't seem to address my problem.
When I try to run the SQL query in PhpMyAdmin, I get the error :
#1062 - Duplicate entry 'button_buynow' for key 'PRIMARY'
I am sure the table was empty prior to me running the query so I don't know what's going on. Can somebody shed a light?
CREATE TABLE IF NOT EXISTS `buttons` (
`name` varchar(255) NOT NULL default '',
`value` text NOT NULL,
PRIMARY KEY (`name`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
--
-- Dumping data for table `buttons`
--
INSERT INTO `buttons` (`name`, `value`) VALUES
('button_buynow', 'buynowCC_LG.gif'),
('button_addtocart', 'x-click-but41.gif'),
('button_viewcart', 'viewcart_LG.gif'),
('button_freedownload', 'downloadnow.jpg');
I am sure the table was empty prior to me running the query so I don't know what's going on.
If you're sure that the table was empty you might have a trigger defined on this table that is the cause of this error.
You can check it this way
SELECT *
FROM information_schema.triggers
WHERE trigger_schema = schema()
AND event_object_table = 'buttons'
If you do in fact have a trigger then you either fix it or just drop it.
In my MySQL table I've created an ID column which I'm hoping to auto-increment in order for it to be the primary key.
I've created my table:
CREATE TABLE `test` (
`id` INT( 11 ) NOT NULL AUTO_INCREMENT PRIMARY KEY ,
`name` VARCHAR( 50 ) NOT NULL ,
`date_modified` DATETIME NOT NULL ,
UNIQUE (
`name`
)
) TYPE = INNODB;
then Inserted my records:
INSERT INTO `test` ( `id` , `name` , `date_modified` )
VALUES (
NULL , 'TIM', '2011-11-16 12:36:30'
), (
NULL , 'FRED', '2011-11-16 12:36:30'
);
I'm expecting that my ID's for the above are 1 and 2 (respectively). And so far this is true.
However when I do something like this:
insert into test (name) values ('FRED')
on duplicate key update date_modified=now();
then insert a new record, I'm expecting it to be 3, however now I'm shown an ID of 4; skipping the place spot for 3.
Normally this wouldn't be an issue but I'm using millions of records which have thousands of updates every day.. and I don't really want to even have to think about running out of ID's simply because I'm skipping a ton of numbers..
Anyclue to why this is happening?
MySQL version: 5.1.44
Thank you
My guess is that the INSERT itself kicks off the code that generates the next ID number. When the duplicate key is detected, and ON DUPLICATE KEY UPDATE is executed, the ID number is abandoned. (No SQL dbms guarantees that automatic sequences will be without gaps, AFAIK.)
MySQL docs say
In general, you should try to avoid using an ON DUPLICATE KEY UPDATE
clause on tables with multiple unique indexes.
That page also says
If a table contains an AUTO_INCREMENT column and INSERT ... ON
DUPLICATE KEY UPDATE inserts or updates a row, the LAST_INSERT_ID()
function returns the AUTO_INCREMENT value.
which stops far short of describing the internal behavior I guessed at above.
Can't test here; will try later.
Is it possible to change your key to unsigned bigint - 18,446,744,073,709,551,615 is a lot of records - thus delaying the running out of ID's
Found this in mysql manual http://dev.mysql.com/doc/refman/5.1/en/example-auto-increment.html
Use a large enough integer data type for the AUTO_INCREMENT column to hold the
maximum sequence value you will need. When the column reaches the upper limit of
the data type, the next attempt to generate a sequence number fails. For example,
if you use TINYINT, the maximum permissible sequence number is 127.
For TINYINT UNSIGNED, the maximum is 255.
More reading here http://dev.mysql.com/doc/refman/5.6/en/information-functions.html#function_last-insert-id it could be inferred that the insert to a transactional table is a rollback so the manual says "LAST_INSERT_ID() is not restored to that before the transaction"
What about for a possible solution to use a table to generate the ID's and then insert into your main table as the PK using LAST_INSERT_ID();
From the manual:
Create a table to hold the sequence counter and initialize it:
mysql> CREATE TABLE sequence (id INT NOT NULL);
mysql> INSERT INTO sequence VALUES (0);
Use the table to generate sequence numbers like this:
mysql> UPDATE sequence SET id=LAST_INSERT_ID(id+1);
mysql> SELECT LAST_INSERT_ID();
The UPDATE statement increments the sequence counter and causes the next call to
LAST_INSERT_ID() to return the updated value. The SELECT statement retrieves that
value. The mysql_insert_id() C API function can also be used to get the value.
See Section 20.9.3.37, “mysql_insert_id()”.
It's really a bug how you can see here: http://bugs.mysql.com/bug.php?id=26316
But, apparently, they fixed it on 5.1.47 and it was declared as INNODB plugin problem.
A duplicate, but same problem, you can see here too: http://bugs.mysql.com/bug.php?id=53791 referenced to the first page mentioned here in this answer.