mysql update a table with LOAD DATA - mysql

So I have someone working on a csv (generated from my table) where they will be updating some fields by hand. This may happen multiple times. I'd like to take the modified csv and update my existing table.
From my understanding, I will need to create a tmp table and then use that to update the existing table. So I can create the temporary table, but how can I iterate through that table and use it to update the existing table?
My sql querying skills are pretty basic. I think its possible, but I'm not sure where to start.

You don't need a temporary table. Just make sure the CSV file includes the primary key of the table. Then you can use the REPLACE modifier in LOAD DATA INFILE. From the documentation:
If you specify REPLACE, input rows replace existing rows. In other words, rows that have the same value for a primary key or unique index as an existing row.

In the CSV you are generating that gets edited, you must include a unique value that will allow you to match the edited record to the original record. Make sure the user doesn't change that column! Also, make sure you have a unique key on that column.
You can then import the edited data into a table with the same (or at least very similar) structure as the original table.
Once the data is imported, you can use an INSERT ... ON DUPLICATE KEY UPDATE ... statement to update the original table. Here's an example:
Main data table:
DROP TABLE IF EXISTS `my_table`;
CREATE TABLE IF NOT EXISTS `my_table` (
`id` INT(10) UNSIGNED NOT NULL AUTO_INCREMENT COMMENT 'Primary Key',
`fld1` VARCHAR(100) NULL,
`fld2` VARCHAR(100) NULL,
`fld3` VARCHAR(100) NULL,
PRIMARY KEY (`id`)
)
ENGINE=MyISAM
AUTO_INCREMENT=1
DEFAULT CHARSET=utf8
COLLATE=utf8_unicode_ci;
Temporary table for edited CSV import:
DROP TABLE IF EXISTS `import_table`;
CREATE TABLE IF NOT EXISTS `import_table` (
`n_id` INT(10) NOT NULL COMMENT 'Original Primary Key',
`n_fld1` VARCHAR(100) NULL,
`n_fld2` VARCHAR(100) NULL,
`n_fld3` VARCHAR(100) NULL,
PRIMARY KEY (`id`)
)
ENGINE=MyISAM
AUTO_INCREMENT=1
DEFAULT CHARSET=utf8
COLLATE=utf8_unicode_ci;
Simulated data before export and editing:
INSERT INTO `my_table`
(`fld1`,`fld2`,`fld3`)
VALUES
('John','Doe','Atlanta'),
('Jane','Smith','New York'),
('Bill','Howe','San Antonio'),
('Harry','Fields','Paris');
Simulate the imported, edited records:
INSERT INTO `import_table`
(`n_id`,`n_fld1`,`n_fld2`,`n_fld3`)
VALUES
(1,'John','Doe','Decatur, IL'),
(2,'Jane','Smithsonian','New York, NY'),
(3,'Bill','Bellweather','San Antonio, TX'),
(4,'Harry','Belefonte','Houston, TX');
Merge the imported, edited records bak into the main table:
INSERT INTO `my_table`
(`id`,`fld1`,`fld2`,`fld3`)
SELECT `n_id`,`n_fld1`,`n_fld2`,`n_fld3`
FROM `import_table`
ON DULPICATE KEY UPDATE
`fld1` = `n_fld1`,
`fld2` = `n_fld2`,
`fld3` = `n_fld3`;

Related

How to suppress unique key checking while sql insert

I got a MySQL database with some tables.
In one of these tables i want to insert by a SQL script some new rows.
Unfortunately i have to insert in two columns an empty string and the two columns are part of an unique key for that table.
So i tried to set UNIQUE_CHECKS before and after the insert, but i'm getting errors because of duplicate entries.
Here is the definition of the table:
CREATE TABLE `Table_A` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(100) NOT NULL,
`number` varchar(25) DEFAULT NULL,
`changedBy` varchar(150) DEFAULT NULL,
`changeDate` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
UNIQUE KEY `name` (`name`,`number`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
And the INSERT statement which causes error:
SET UNIQUE_CHECKS = 0;
INSERT INTO `Table_A`
(`name`, `number`, `changedBy`, `changeDate`)
SELECT DISTINCT '', 'myUser', CURRENT_TIMESTAMP
FROM Table_A
AND id NOT IN
(
SELECT DISTINCT id
FROM Table_A
);
SET UNIQUE_CHECKS = 1;
As You can see, i'm using UNIQUE_CHECKS.
But as i said this doesn't work properly.
Any help or suggestion would be appreciated.
Patrick
Switching off Unique Keys for the insert operation doesn't indicate that it will check uniqueness only for the operations that happen after you switch it on again. It just means that database will not waste time to check the constraint during the time it is switch off but it will check the constraint when you switch it on again.
What it measn is that you nead to ensure that column has unique value in a columns with Unique Keys before you can turn it on. Which you don't do.
If you want to maintain Uniqueness somehow for new records you insert after some point in time you would need to create trigger and manually check the new records against already existing data. The same possibly goes for updates. But I don't recommend it - you should probably redesign data so either the Unique Key is not there or the data is truly unique for all the records there are and will be.

MySQL: Enforce an unique column without using an unique key

I have a column with data that exceeds MySQL's index length limit. Therefore, I can't use an unique key.
There's a solution here to the problem without using an unique key: MySQL: Insert record if not exists in table
However, in the comments, people are having issues with inserting the same value into multiple columns. In my case, a lot of my values are 0, so I'll get duplicate values very often.
I'm using Node and node-mysql to access the database. I'm thinking I can have a variable that keeps track of all values that are currently being inserted. Before inserting, I check if the value is currently being inserting. If so, I'll wait until it finishes inserting, then continue execution as if the value was originally inserted. However, I feel like this will be very error prone.
Here's part of my table schema:
CREATE TABLE `links` (
`id` int(10) UNSIGNED NOT NULL,
`url` varchar(2083) CHARACTER SET latin1 COLLATE latin1_general_cs NOT NULL,
`likes` int(10) UNSIGNED NOT NULL,
`tweets` int(10) UNSIGNED NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
ALTER TABLE `links`
ADD PRIMARY KEY (`id`),
ADD KEY `url` (`url`(50));
I cannot put an unique key on url because it can be 2083 bytes, which is over MySQL's key size limit. likes and tweets will often be 0, so the linked solution will not work.
Is there another possible solution?
If you phrase your INSERT in a certain way, you can make use of WHERE NOT EXISTS to check first if the URL does not exist before completing the insert:
INSERT INTO links (`url`, `likes`, `tweets`)
SELECT 'http://www.google.com', 10, 15 FROM DUAL
WHERE NOT EXISTS
(SELECT 1 FROM links WHERE url='http://www.google.com');
This assumes that the id column is a primary key/auto increment, and MySQL will automatically assign a value to it.

Best way to add column to a big table with longblob

I have the following table:
CREATE TABLE IF NOT EXISTS PDF_STORAGE (
ID_PDF_STORAGE bigint(20) NOT NULL AUTO_INCREMENT,
DESC_FILE varchar(255) DEFAULT NULL,
PDF_FILE longblob,
LINK_FILE varchar(255) DEFAULT NULL,
VERSION int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (ID_PDF_STORAGE)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=1 ;
Where PDF_FILE is a file of 10 MB on average. Today it has about 50.000 rows. I need to add a new column to this table but it is taking a long time, more than 10 min, some times giving a 401 error in PhpMyAdmin, so I'd like to know what is the proper way to achieve this...
I already tried:
ALTER TABLE PDF_STORAGE ADD VERSION INT NOT NULL DEFAULT '0' AFTER LINK_FILE ;
and
SET FOREIGN_KEY_CHECKS=0;
CREATE TABLE PDF_STORAGE_new LIKE PDF_STORAGE;
ALTER TABLE PDF_STORAGE_new ADD VERSION INT NOT NULL DEFAULT '0' AFTER LINK_FILE ;
INSERT INTO PDF_STORAGE_new (PDF_STORAGE, DESC_FILE, ID_PDF_STORAGE, LINK_FILE) SELECT * FROM PDF_STORAGE;
RENAME TABLE PDF_STORAGE TO PDF_STORAGE_old, PDF_STORAGE_new TO PDF_STORAGE;
DROP TABLE PDF_STORAGE_old;
SET FOREIGN_KEY_CHECKS = 1;
but they are also slow.. is there a better way?
Thanks
What you are doing now ALTER TABLE is the best approach to my knowledge. You can try making this change when there is not much transaction (or) DB operation going on. I mean say, do the changes in idle time.
ALTER TABLE PDF_STORAGE ADD VERSION INT NOT NULL DEFAULT '0' AFTER LINK_FILE ;
You can as well create a new table same as this table schema along with the new column.
Insert all the records from this table to the newly created table.
Rename the new table to the old table name.
delete the old table.

MySQL stop inserting when duplicate key condition is met

I have a table constructed by the followinng:
CREATE TABLE IF NOT EXISTS test_table (
ID int(11) NOT NULL AUTO_INCREMENT,
ProfileID int(11) NOT NULL,
ForeignID int(11) NOT NULL,
PRIMARY KEY (ProfileID,ForeignID) )
ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=1 ;
I want to do something a little peculiar though, say there are 4 records in the database:
RecA, RecB, RecC, RecD
I would like to run the following query and have the insert behavior stop when a duplicate key was encountered:
INSERT IGNORE INTO test_table (ProfileID, ForeignID) VALUES(RecE, RecF, RecA, RecB, RecG);
So the query would only insert RecE and RecF, is there a way to do this in MySQL, perhaps using ON DUPLICATE KEY? Ideally the execution would just be terminated once a duplicate has been found, I am not too familiar with SQL syntax though.
Where RecG was explicitly not inserted.

Resolving a duplicate primary key on mysql import

I'm looking to append a comments table from one WordPress site to another. The users are different. When I import the comments from site B to A, I run into a duplicate key issue; comment_id is already taken.
So how can I resolve this and append the table with a simple .sql file? Would I have to take the user information, generate a new user, check for comments made on site B, pull the content and postID, then go back to site A and recreate the comment for the newly created user!?
What a headache! THanks.
if your only problem is a duplicate key issue, go to the end of your sql file after
ENGINE=MyISAM
and make it
ENGINE=MyISAM AutoIncrement=a nubmer above the last id in the new database
or
Query database A for the last id then add one and use it on a new insert query.
Example 1:
CREATE TABLE IF NOT EXISTS `movies` (
`id` int(255) NOT NULL AUTO_INCREMENT,
`title` varchar(255) NOT NULL,
`year` int(4) NOT NULL,
`size` varchar(255) NOT NULL,
`added` date NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `title` (`title`,`year`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=4 ;
The Inserts From My Dump:
INSERT INTO `movies` (`title`, `year`, `size`, `added`) VALUES
('[REC] 2', 0, '716688', '2011-09-23'),
('5 Days of War', 0, '1435406', '2012-01-09'),
('[REC]', 0, '1353420800', '2011-11-06');
See how i didnt include the PRIMARY KEY (id) in my includes, but it will still check against my UNIQUE KEY and see if the title exists. Just a little demo that hopefully helps out. If your table already exists on the new database then just skip to the inserts and dont include the primary key and it will be auto set on a new insert to the next available value.