SQL insert query so slow - mysql

Each 10,000 rows take about 124 second to insert ,how this can be faster
This is the table i insert to
CREATE TABLE `orders`
(`oid` int(11) NOT NULL AUTO_INCREMENT,
`countryCode` varchar(10) NOT NULL,
`date` datetime NOT NULL,
`id` bigint(20) NOT NULL,
`productId` bigint(20) NOT NULL,
PRIMARY KEY (`oid`),
UNIQUE KEY `id` (`id`),
KEY `date` (`date`),
KEY `productId` (`productId`)
) ENGINE=InnoDB AUTO_INCREMENT=4833010 DEFAULT CHARSET=latin1
this is the query i used
ALTER TABLE `orders` DISABLE KEYS;
SET FOREIGN_KEY_CHECKS=0;
INSERT IGNORE INTO `orders` (`countryCode`, `date`, `id`,`productId`)
VALUES
('ru','2019-04-09 06:59',100453324298986,32829863707) ,
('fr','2019-04-09 05:59',100645420835625,32829863707) ,
('ru','2019-04-08 12:04',704482263524094,32829863707)
.......etc 10,000 rows here at once

Try to write insert statement as
START TRANSACTION;
INSERT INTO `insert_table` (`datetime`, `uid`, `content`, `type`) VALUES ('0', 'userid_0', 'content_0', 0);
INSERT INTO `insert_table` (`datetime`, `uid`, `content`, `type`) VALUES ('1', 'userid_1', 'content_1', 1);
...
COMMIT;

I think it's slow because you trying to INSERT bulk records at once.
You have several methods to speedup the process.
1) Insert records batch wise. ( 500-1000 records at a time )
2) Increase Memory parameters in MYSQL. so the memory will be increased.
(http://www.geeksengine.com/database/data-manipulation/bulk-insert.php)
Try This for batch Update
SET autocommit=0;
SET unique_checks=0;
SET foreign_key_checks=0;
SET GLOBAL bulk_insert_buffer_size =1024*1024*512;
START TRANSACTION;
INSERT IGNORE INTO `orders` (`countryCode`, `date`, `id`,`productId`)
VALUES
('ru','2019-04-09 06:59',100453324298986,32829863707) ,
('fr','2019-04-09 05:59',100645420835625,32829863707) ,
('ru','2019-04-08 12:04',704482263524094,32829863707)
.......etc 600 rows here at once
COMMIT;
START TRANSACTION;
INSERT IGNORE INTO `orders` (`countryCode`, `date`, `id`,`productId`)
VALUES
('ru','2019-04-09 06:59',100453324298986,32829863707) ,
('fr','2019-04-09 05:59',100645420835625,32829863707) ,
('ru','2019-04-08 12:04',704482263524094,32829863707)
.......etc 600 rows here at once
COMMIT;
Note:
1) If this is slow Try changing bulk_insert_buffer_size and no of rows per insert.
2) Check your PC Free Memory/CPU before executing the query . try to free it as much as possible

Related

Duplicate entry '111-222' for key 'PRIMARY' when inserting a new value to MySQL database

I have a MySQL table running on AWS RDS with structure like the following:
CREATE TABLE `my_table` (
`col1` int(11) NOT NULL,
`col2` int(11) NOT NULL DEFAULT '0',
`f_name` varchar(45) DEFAULT NULL,
`l_name` varchar(45) DEFAULT NULL,
PRIMARY KEY (`col1`,`col2`),
KEY `idx_col1` (`col1`),
KEY `idx_col2` (`col2`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
The query
SELECT * FROM my_table WHERE col1=111 AND col2=222;
returns 0 row.
But when I run an insert query
INSERT INTO my_table
(col1, col2, f_name, l_name)
VALUES (111, 222, 'John', 'Doe')
I got an error saying
Duplicate entry '111-222' for key 'PRIMARY'.
Why does this happen? The table doesn't contain a row with col1=111 and col2=222.
There's already a row with values col1=111, col2=111, f_name='John', and l_name='Doe'. But I don't think this would cause a duplicate entry error.
=========================== EDIT ======================================
There's a trigger that generates the duplicate error. Here's the script to reproduce the error.
# Initialize the tables
CREATE TABLE `my_table` (
`col1` int(11) NOT NULL,
`col2` int(11) NOT NULL DEFAULT '0',
`f_name` varchar(45) DEFAULT NULL,
`l_name` varchar(45) DEFAULT NULL,
PRIMARY KEY (`col1`,`col2`),
KEY `idx_col1` (`col1`),
KEY `idx_col2` (`col2`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE `triggered_table` (
`col1` int(11) NOT NULL,
`col2` int(11) NOT NULL DEFAULT '0',
`update_date` bigint(20) DEFAULT NULL,
PRIMARY KEY (`col1`,`col2`),
KEY `idx_col1` (`col1`),
KEY `idx_col2` (`col2`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
# Insert the data that cause duplicate error
INSERT INTO triggered_table (col1, col2) VALUES (111, 222);
# Create the trigger
DELIMITER $$
CREATE TRIGGER weird_trigger AFTER INSERT
ON my_table
FOR EACH ROW
BEGIN
INSERT INTO triggered_table
(col1, col2)
VALUES (NEW.col1, NEW.col2);
END$$
DELIMITER ;
# Create the duplicate error
INSERT INTO my_table
(col1, col2, f_name, l_name)
VALUES (111, 222, 'John', 'Doe');
I really don't understand why the developers created the triggered_table table. Why didn't they put update_date column to my_table?
This is so weird.
All you have to do is:
Truncate your table then run (Assuming that you have only a test data but if not, you have to do some backup first)
INSERT INTO my_table
(col1, col2, f_name, l_name)
VALUES (111, 222, 'John', 'Doe')
Now if the error still exists, this is a pretty much problem.
Your error seems like you concatinated col1 and col2 as your primary key ('111-222')
You can try
select * from yourTable where FieldPrimary = '111-222' if it is already exists
The duplicate key error does not come from the my_table table but from the triggered_table table instead. When you add a row in triggered_table for the key (111, 222) and then add a new row in the my_table table (with the same key), your trigger will also try to add a new row with the key (111, 222) in your triggered_table. However there is already such a key in use and you will get the duplicate key error.
Depending on what you want to do with the my_table and triggered_table tables, you might want to change the trigger to use REPLACE INTO instead of INSERT INTO. Or you run a check with SELECT first to see if you need to add a new row or not. After that you can run an UPDATE query to change the value of update_date. But to answer your question, the duplicate key error comes from the duplicate key in the triggered_table table.

MySQL-8.0.11 "...LIMIT 1 FOR UPDATE SKIP LOCKED" does not works with multiple sorts?

I have a table with a task queue.
Each task can have a status of "new" or "ready."
I use several threads, each of which should take one "new" task and after some time update it to the "finished" one.
Each thread uses this request:
....WHERE `status` = 'new' ORDER BY `service` LIMIT 1 FOR UPDATE SKIP LOCKED
And it works.
But if I add 1 more sorting criteria, the query from the first thread returns one record, as expected, But the other threads get an empty result.
It looks like the second sorting criteria locks all the records in the table.
How to get around this?
thank
steps to reproduce:
CREATE TABLE `test` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`service` varchar(32) NOT NULL,
`address` varchar(32) NOT NULL,
`status` enum('new','done') NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB;
INSERT INTO `test` (`id`, `service`, `adders`, `status`) VALUES
(1, 'service_1', 'adders_1', 'new'),
(2, 'service_2', 'adders_2', 'new');
Request for each thread:
BEGIN;
SELECT * FROM `test` WHERE `status` = 'new'
ORDER BY `service`, `address` LIMIT 1 FOR UPDATE SKIP LOCKED
Then pause like sleep().
Then
UPDATE `test` SET.....
then
COMMIT;

how to get average time 'xx:xx:xx' on mysql?

please see the the test data bellow. I want to get the avgtime (=timeonsite/visits) and display as "xx:xx:xx" result in mysql. how can I get it?
SET FOREIGN_KEY_CHECKS=0;
-- ----------------------------
-- Table structure for `t`
-- ----------------------------
DROP TABLE IF EXISTS `t`;
CREATE TABLE `t` (
`id` int(11) NOT NULL auto_increment,
`timeOnsite` time default NULL,
`visits` int(11) default NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=4 DEFAULT CHARSET=utf8;
-- ----------------------------
-- Records of t
-- ----------------------------
INSERT INTO `t` VALUES ('1', '04:05:30', '20');
INSERT INTO `t` VALUES ('2', '03:00:00', '10');
INSERT INTO `t` VALUES ('3', '00:01:30', '17');
You can use TIME_TO_SEC function to change xx:xx:xx format to seconds.
SELECT TIME_TO_SEC('00:01:30') / 17; # return 5.2941
And then through SEC_TO_TIME you can convert seconds to time back as below :
SELECT SEC_TO_TIME(TIME_TO_SEC('00:01:30') / 17); # return 00:00:05
Are sure that you calculate avgtime in such way?
If yes, mysql select below:
select id, timeOnsite,visits, SEC_TO_TIME(TIME_TO_SEC(timeOnsite)/visits) as avgtime
from t

Can't create table 'e-learningsystem.chapter' (errno: 121) [duplicate]

I have troubles with forward engineering my MySQL database into WAMP server..
I was going to post an image of the schema but as this is my first post I can't.
Below is the executed script..
use aquaticstar;
SET #OLD_UNIQUE_CHECKS=##UNIQUE_CHECKS, UNIQUE_CHECKS=0;
SET #OLD_FOREIGN_KEY_CHECKS=##FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0;
SET #OLD_SQL_MODE=##SQL_MODE, SQL_MODE='TRADITIONAL,ALLOW_INVALID_DATES';
-- -----------------------------------------------------
-- Table `Students`
-- -----------------------------------------------------
DROP TABLE IF EXISTS `Students` ;
CREATE TABLE IF NOT EXISTS `Students` (
`id` VARCHAR(10) NOT NULL ,
`studentName` VARCHAR(45) NOT NULL ,
`gender` CHAR NOT NULL ,
`birthDate` DATETIME NOT NULL ,
`mNo` VARCHAR(10) NOT NULL ,
`contactName` VARCHAR(45) NOT NULL ,
`contactEmail` VARCHAR(45) NOT NULL ,
`contactPhone` INT(10) NOT NULL ,
`startDate` DATETIME NOT NULL ,
`remarks` VARCHAR(200) NULL ,
PRIMARY KEY (`id`) )
ENGINE = InnoDB;
-- -----------------------------------------------------
-- Table `Waiting List`
-- -----------------------------------------------------
DROP TABLE IF EXISTS `Waiting List` ;
CREATE TABLE IF NOT EXISTS `Waiting List` (
`wait_id` VARCHAR(5) NOT NULL ,
`name` VARCHAR(45) NULL ,
`contactName` VARCHAR(45) NULL ,
`contactPhone` INT(10) NULL ,
`contactEmail` VARCHAR(45) NULL ,
`status` CHAR NULL ,
`remarks` VARCHAR(200) NULL ,
PRIMARY KEY (`wait_id`) )
ENGINE = InnoDB;
-- -----------------------------------------------------
-- Table `Schedule`
-- -----------------------------------------------------
DROP TABLE IF EXISTS `Schedule` ;
CREATE TABLE IF NOT EXISTS `Schedule` (
`lesson_id` VARCHAR(10) NOT NULL ,
`day` VARCHAR(3) NOT NULL ,
`branch` VARCHAR(30) NOT NULL ,
`level` VARCHAR(30) NOT NULL ,
`time` TIME NOT NULL ,
`ae` VARCHAR(45) NOT NULL ,
PRIMARY KEY (`lesson_id`) )
ENGINE = InnoDB;
-- -----------------------------------------------------
-- Table `Link`
-- -----------------------------------------------------
DROP TABLE IF EXISTS `Link` ;
CREATE TABLE IF NOT EXISTS `Link` (
`link_id` VARCHAR(10) NOT NULL ,
`id` VARCHAR(10) NOT NULL ,
`lesson_id` VARCHAR(10) NOT NULL ,
PRIMARY KEY (`link_id`) ,
INDEX `id_idx` (`id` ASC) ,
INDEX `lesson_id_idx` (`lesson_id` ASC) ,
CONSTRAINT `id`
FOREIGN KEY (`id` )
REFERENCES `Students` (`id` )
ON DELETE NO ACTION
ON UPDATE NO ACTION,
CONSTRAINT `lesson_id`
FOREIGN KEY (`lesson_id` )
REFERENCES `Schedule` (`lesson_id` )
ON DELETE NO ACTION
ON UPDATE NO ACTION)
ENGINE = InnoDB;
-- -----------------------------------------------------
-- Table `Attendance`
-- -----------------------------------------------------
DROP TABLE IF EXISTS `Attendance` ;
CREATE TABLE IF NOT EXISTS `Attendance` (
`date` DATETIME NOT NULL ,
`attendance` VARCHAR(5) NOT NULL ,
`link_id` VARCHAR(10) NOT NULL ,
INDEX `link_id_idx` (`link_id` ASC) ,
CONSTRAINT `link_id`
FOREIGN KEY (`link_id` )
REFERENCES `Link` (`link_id` )
ON DELETE NO ACTION
ON UPDATE NO ACTION)
ENGINE = InnoDB;
SET SQL_MODE=#OLD_SQL_MODE;
SET FOREIGN_KEY_CHECKS=#OLD_FOREIGN_KEY_CHECKS;
SET UNIQUE_CHECKS=#OLD_UNIQUE_CHECKS;
-- -----------------------------------------------------
-- Data for table `Students`
-- -----------------------------------------------------
START TRANSACTION;
INSERT INTO `Students` (`id`, `studentName`, `gender`, `birthDate`, `mNo`, `contactName`, `contactEmail`, `contactPhone`, `startDate`, `remarks`) VALUES ('s001', 'Sam Khew', 'm', '12/12/1991', 'nm', 'May Khew', 'may#gmail.com', 0198829387, '12/07/2011', NULL);
INSERT INTO `Students` (`id`, `studentName`, `gender`, `birthDate`, `mNo`, `contactName`, `contactEmail`, `contactPhone`, `startDate`, `remarks`) VALUES ('s002', 'Joe Biden', 'm', '13/03/2003', 'nm', 'Layla Biden', 'layla#gmail.com', 0199283763, '14/05/2011', NULL);
INSERT INTO `Students` (`id`, `studentName`, `gender`, `birthDate`, `mNo`, `contactName`, `contactEmail`, `contactPhone`, `startDate`, `remarks`) VALUES ('s003', 'Bob Builder', 'm', '14/02/2002', 'LK920K', 'Mama Builder', 'mama#yahoo.com', 0167728376, '29/02/2012', NULL);
INSERT INTO `Students` (`id`, `studentName`, `gender`, `birthDate`, `mNo`, `contactName`, `contactEmail`, `contactPhone`, `startDate`, `remarks`) VALUES ('s004', 'Kenny Koh', 'm', '18/02/1999', 'MM992', 'Lisa Koh', 'lk#hotmail.com', 0123160231, '19/01/2012', NULL);
INSERT INTO `Students` (`id`, `studentName`, `gender`, `birthDate`, `mNo`, `contactName`, `contactEmail`, `contactPhone`, `startDate`, `remarks`) VALUES ('s005', 'Jane Doe', 'f', '29/09/1999', 'nm', 'Jackie Doe', 'jackied#gmail.com', 0127736254, '02/03/2012', NULL);
INSERT INTO `Students` (`id`, `studentName`, `gender`, `birthDate`, `mNo`, `contactName`, `contactEmail`, `contactPhone`, `startDate`, `remarks`) VALUES ('s006', 'Lola Lai', 'f', '02/05/2004', 'nm', 'Mark Lai', 'mark#gmail.com', 0198827365, '11/09/2011', NULL);
COMMIT;
-- -----------------------------------------------------
-- Data for table `Schedule`
-- -----------------------------------------------------
START TRANSACTION;
INSERT INTO `Schedule` (`lesson_id`, `day`, `branch`, `level`, `time`, `ae`) VALUES ('sat1_s4', 'Sat', 'Sunway', 'basic', '4pm', 'Aini');
INSERT INTO `Schedule` (`lesson_id`, `day`, `branch`, `level`, `time`, `ae`) VALUES ('sat1_s5', 'Sat', 'Sunway', 'basic', '5pm', 'Aini');
INSERT INTO `Schedule` (`lesson_id`, `day`, `branch`, `level`, `time`, `ae`) VALUES ('sat1_s6', 'Sat', 'Sunway', 'basic', '6pm', 'Aini');
INSERT INTO `Schedule` (`lesson_id`, `day`, `branch`, `level`, `time`, `ae`) VALUES ('sat2_s4', 'Sat', 'Sunway', 'advance', '4pm', 'Nina');
INSERT INTO `Schedule` (`lesson_id`, `day`, `branch`, `level`, `time`, `ae`) VALUES ('sat2_s5', 'Sat', 'Sunway', 'advance', '5pm', 'Nina');
INSERT INTO `Schedule` (`lesson_id`, `day`, `branch`, `level`, `time`, `ae`) VALUES ('sat3_s6', 'Sat', 'Sunway', 'pre-comp', '6pm', 'Marcus');
COMMIT;
-- -----------------------------------------------------
-- Data for table `Link`
-- -----------------------------------------------------
START TRANSACTION;
INSERT INTO `Link` (`link_id`, `id`, `lesson_id`) VALUES ('L001', 's001', 'sat1_s4');
INSERT INTO `Link` (`link_id`, `id`, `lesson_id`) VALUES ('L002', 's002', 'sat1_s5');
INSERT INTO `Link` (`link_id`, `id`, `lesson_id`) VALUES ('L003', 's003', 'sat1_s6');
INSERT INTO `Link` (`link_id`, `id`, `lesson_id`) VALUES ('L004', 's004', 'sat2_s4');
INSERT INTO `Link` (`link_id`, `id`, `lesson_id`) VALUES ('L005', 's005', 'sat1_s5');
COMMIT;
-- -----------------------------------------------------
-- Data for table `Attendance`
-- -----------------------------------------------------
START TRANSACTION;
INSERT INTO `Attendance` (`date`, `attendance`, `link_id`) VALUES ('26/9/2012', '1', NULL);
COMMIT;
But then I get this error:
Executing SQL script in server
ERROR: Error 1005: Can't create table 'aquaticstar.link' (errno: 121)
I can't figure out why. Can anyone help me?
I searched quickly for you, and it brought me here. I quote:
You will get this message if you're trying to add a constraint with a
name that's already used somewhere else
To check constraints use the following SQL query:
SELECT
constraint_name,
table_name
FROM
information_schema.table_constraints
WHERE
constraint_type = 'FOREIGN KEY'
AND table_schema = DATABASE()
ORDER BY
constraint_name;
Look for more information there, or try to see where the error occurs. Looks like a problem with a foreign key to me.
Foreign Key Constraint Names Have to be Unique Within a Database
Both #Dorvalla’s answer and this blog post mentioned above pointed me into the right direction to fix the problem for myself; quoting from the latter:
If the table you're trying to create includes a foreign key constraint, and you've provided your own name for that constraint, remember that it must be unique within the database.
I wasn’t aware of that. I have changed my foreign key constraint names according to the following schema which appears to be used by Ruby on Rails applications, too:
<TABLE_NAME>_<FOREIGN_KEY_COLUMN_NAME>_fk
For the OP’s table this would be Link_lession_id_fk, for example.
You can login to mysql and type
mysql> SHOW INNODB STATUS\G
You will have all the output and you should have a better idea of what the error is.
I faced this error (errno 121) but it was caused by mysql-created intermediate tables that had been orphaned, preventing me from altering a table even though no such constraint name existed across any of my tables. At some point, my MySQL had crashed or failed to cleanup an intermediate table (table name starting with a #sql-) which ended up presenting me with an error such as: Can't create table '#sql-' (errno 121) when trying to run an ALTER TABLE with certain constraint names.
According to the docs at http://dev.mysql.com/doc/refman/5.7/en/innodb-troubleshooting-datadict.html , you can search for these orphan tables with:
SELECT * FROM INFORMATION_SCHEMA.INNODB_SYS_TABLES WHERE NAME LIKE '%#sql%';
The version I was working with was 5.1, but the above command only works on versions >= 5.6 (manual is incorrect about it working for 5.5 or earlier, because INNODB_SYS_TABLES does not exist in such versions). I was able to find the orphaned temporary table (which did not match the one named in the message) by searching my mysql data directory in command line:
find . -iname '#*'
After discovering the filename, such as #sql-9ad_15.frm, I was able to drop that orphaned table in MySQL:
USE myschema;
DROP TABLE `#mysql50##sql-9ad_15`;
After doing so, I was then able to successfully run my ALTER TABLE.
For completeness, as per the MySQL documentation linked, "the #mysql50# prefix tells MySQL to ignore file name safe encoding introduced in MySQL 5.1."
If you have a foreign key definition in some table and the name of the foreign key is used elsewhere as another foreign key you will have this error.
If you want to fix quickly, Forward Engineer again and check "Generate DROP SCHEMA" option and proceed.
I assume the database doesn't contain data, so dropping it won't affect.
Something I noticed was that I had "other_database" and "Other_Database" in my databases.
That caused this problem as I actually had same reference in other database which caused this mysterious error!
mysql> SHOW ENGINE INNODB STATUS;
But in my case only this way could help:
1. Make backup of current DB
2. Drop DB (not all tables, but DB)
3. Create DB (check that you still have previleges)
4. Restore DB from backup

How to refact UPDATE via SELECT

I have this structure of my db:
CREATE TABLE IF NOT EXISTS `peoples` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(100) COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=1 ;
For customers.
CREATE TABLE IF NOT EXISTS `peoplesaddresses` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`people_id` int(10) unsigned NOT NULL,
`phone` varchar(20) COLLATE utf8_unicode_ci NOT NULL,
`address` varchar(100) COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=1 ;
For their addresses.
CREATE TABLE IF NOT EXISTS `peoplesphones` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`people_id` int(10) unsigned NOT NULL,
`phone` varchar(20) COLLATE utf8_unicode_ci NOT NULL,
`address` varchar(100) COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=1 ;
For their phones.
UPD4
ALTER TABLE peoplesaddresses DISABLE KEYS;
ALTER TABLE peoplesphones DISABLE KEYS;
ALTER TABLE peoplesaddresses ADD INDEX i_phone (phone);
ALTER TABLE peoplesphones ADD INDEX i_phone (phone);
ALTER TABLE peoplesaddresses ADD INDEX i_address (address);
ALTER TABLE peoplesphones ADD INDEX i_address (address);
ALTER TABLE peoplesaddresses ENABLE KEYS;
ALTER TABLE peoplesphones ENABLE KEYS;
END UPD4
CREATE TABLE IF NOT EXISTS `order` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`people_id` int(10) unsigned NOT NULL,
`name` varchar(255) CHARACTER SET utf8 NOT NULL,
`phone` varchar(255) CHARACTER SET utf8 NOT NULL,
`adress` varchar(255) CHARACTER SET utf8 NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=12 ;
INSERT INTO `order` (`id`, `people_id`, `name`, `phone`, `adress`) VALUES
(1, 0, 'name1', 'phone1', 'address1'),
(2, 0, 'name1_1', 'phone1', 'address1_1'),
(3, 0, 'name1_1', 'phone1', 'address1_2'),
(4, 0, 'name2', 'phone2', 'address2'),
(5, 0, 'name2_1', 'phone2', 'address2_1'),
(6, 0, 'name3', 'phone3', 'address3'),
(7, 0, 'name4', 'phone4', 'address4'),
(8, 0, 'name1_1', 'phone5', 'address1_1'),
(9, 0, 'name1_1', 'phone5', 'address1_2'),
(11, 0, 'name1', 'phone1', 'address1'),
(10, 0, 'name1', 'phone1', 'address1');
Production base have over 9000 records. Is there way to execute this 3 update query's little more faster, than now (~50 min on dev machine).
INSERT INTO peoplesphones( phone, address )
SELECT DISTINCT `order`.phone, `order`.adress
FROM `order`
GROUP BY `order`.phone;
Fill peoplesphones table with unique phones
INSERT INTO peoplesaddresses( phone, address )
SELECT DISTINCT `order`.phone, `order`.adress
FROM `order`
GROUP BY `order`.adress;
Fill peoplesaddresses table with unique adress.
The next three querys are very slow:
UPDATE peoplesaddresses, peoplesphones SET peoplesaddresses.people_id = peoplesphones.id WHERE peoplesaddresses.phone = peoplesphones.phone;
UPDATE peoplesaddresses, peoplesphones SET peoplesphones.people_id = peoplesaddresses.people_id WHERE peoplesaddresses.address = peoplesphones.address;
UPDATE `order`, `peoplesphones` SET `order`.people_id = `peoplesphones`.people_id where `order`.phone = `peoplesphones`.phone;
Finally fill people table, and clear uneccessary fields.
INSERT INTO peoples( id, name )
SELECT DISTINCT `order`.people_id, `order`.name
FROM `order`
GROUP BY `order`.people_id;
ALTER TABLE `peoplesphones`
DROP `address`;
ALTER TABLE `peoplesaddresses`
DROP `phone`;
So, again: How can I make those UPDATE query's a little more faster? THX.
UPD: I forgott to say: I need to do it at once, just for migrate phones and adresses into other tables since one people can have more than one phone, and can order pizza not only at home.
UPD2:
UPD3:
Replace slow update querys on this (without with) get nothing.
UPDATE peoplesaddresses
LEFT JOIN
peoplesphones
ON peoplesaddresses.phone = peoplesphones.phone
SET peoplesaddresses.people_id = peoplesphones.id;
UPDATE peoplesphones
LEFT JOIN
`peoplesaddresses`
ON `peoplesaddresses`.address = `peoplesphones`.address
SET `peoplesphones`.people_id = `peoplesaddresses`.people_id;
UPDATE `order`
LEFT JOIN
`peoplesphones`
ON `order`.phone = `peoplesphones`.phone
SET `order`.people_id = `peoplesphones`.people_id;
UPD4 After adding code at the top (upd4), script takes a few seconds for execute. But on ~6.5k query it terminate with text: "The system cannot find the Drive specified".
Thanks to All. Especially to xQbert and Brent Baisley.
50 minutes for 9000 records is a bit ridiculous, event without indexes. You might as well put the 9000 records in Excel and do what you need to do. I think there is something else going on with your dev machine. Perhaps you have mysql configured to use very little memory? Maybe you can post the results of this "query":
show variables like "%size%";
Just this morning I did an insert(ignore)/select on 2 tables (one into another), both with over 400,000 records. 126,000 records were inserted into the second table, it took a total of 2 minutes 13 seconds.
I would say put indexes on any of the fields you are joining or grouping on, but this seems like a one time job. I don't think the lack of indexes is your problem.
All write operations are slow in relational databases. Especially indexes make them slow, since they have to be recalculated.
If you're using a WHERE in your statements, you should place an index on the fields referenced.
GROUP BY is always very slow, and so is DISTINCT, since they have to do a lot of checks that don't scale linearly. Always avoid them.
You may like to choose a different database engine for what you're doing. 9000 records in 50 minutes is very slow. Experiment with a few different engines, such as MyISAM and InnoDB. If you're using temporary tables a lot, MEMORY is really fast for those.
Update: Also, updating multiple tables in one statement probably shouldn't be done.