I am building a MySQL accounts database. I have a routine which imports the entire database from a json file. The routine runs inside a single transaction,
drops all the records from every table, calls:
ALTER TABLE <tablename> AUTO_INCREMENT = 1
on each table, then loads the tables from data using INSERT statements like
INSERT INTO Journal (idJournal, DocumentId, AccountId, Memo, JournalNum,
Amount, Outstanding, Cleared, NameAddressId)
VALUES (11423, 4454, 14, 'Deposit', 1, 53.33, 53.33, 'X', 292)
(Note that the tables are loaded in the right order, so the related records already exist).
I am finding the insert statements are taking a long time (some up to 70 msec), especially compared with loading the same data from a mysqldump.
Is there anything I can do to speed this up (other than using mysqldump instead, which I don't want to do because I want the backup data to be in json format)?
Table definition:
CREATE TABLE IF NOT EXISTS `accounts`.`Journal` (
`idJournal` INT NOT NULL AUTO_INCREMENT,
`DocumentId` INT NOT NULL,
`AccountId` INT NOT NULL,
`Memo` VARCHAR(45) NULL,
`JournalNum` INT NOT NULL,
`Amount` DECIMAL(10,2) NOT NULL,
`Outstanding` DECIMAL(10,2) NOT NULL DEFAULT 0,
`Cleared` CHAR(1) NOT NULL,
`NameAddressId` INT NULL,
PRIMARY KEY (`idJournal`),
INDEX `fk_Journal_Document1_idx` (`DocumentId` ASC),
INDEX `fk_Journal_Account1_idx` (`AccountId` ASC),
UNIQUE INDEX `Document_Num` (`DocumentId` ASC, `JournalNum` ASC),
INDEX `fk_Journal_NameAddress1_idx` (`NameAddressId` ASC),
CONSTRAINT `fk_Journal_Document1`
FOREIGN KEY (`DocumentId`)
REFERENCES `accounts`.`Document` (`idDocument`)
ON DELETE NO ACTION
ON UPDATE NO ACTION,
CONSTRAINT `fk_Journal_Account1`
FOREIGN KEY (`AccountId`)
REFERENCES `accounts`.`Account` (`idAccount`)
ON DELETE NO ACTION
ON UPDATE NO ACTION,
CONSTRAINT `fk_Journal_NameAddress1`
FOREIGN KEY (`NameAddressId`)
REFERENCES `accounts`.`NameAddress` (`idNameAddress`)
ON DELETE NO ACTION
ON UPDATE NO ACTION)
ENGINE = InnoDB
When performing multiple inserts, it's more performant to insert multiple lines in a single statement, especially if you have a lot of secondary indexes:
INSERT INTO Journal (idJournal, DocumentId, AccountId, Memo, JournalNum,
Amount, Outstanding, Cleared, NameAddressId)
VALUES (11423, 4454, 14, 'Deposit', 1, 53.33, 53.33, 'X', 292),
(11424, 4455, 15, 'Deposit', 1, 23.33, 23.33, 'X', 293),
...
That way, the indexes are only updated once (at the end of the statement), rather than after each insert.
If your database grows beyond 1GB, you're going to have problems because you'll reach the limit of max_allowed_packet. In that case, you may have to break it up into multiple queries.
When you say "drops all the records", I hope you mean drop table or truncate table rather than actually deleting all the records.
This works with "no" interruption of usage of the table:
CREATE TABLE new LIKE real;
Load new by batched INSERTs (or whatever).
RENAME TABLE real TO old, new TO real; -- Instantaneous and atomic
DROP TABLE old;
No TRUNCATE, no AUTO_INCREMENT=1
Related
I have a table that needs a unique constraint on 3 columns, but, if the "date" column in for that insert transaction is a newer date than the current record's date, then I want to update that record (so the unique constraint is still true for the table).
Postgres has the concept of deferrable constraints, MySQL does not.
I do want to implement it with the SQL object tools available, though.
Here is my table DDL with column names obfuscated:
CREATE TABLE `apixio_results_test_sefath` (
`ID` int(11) NOT NULL AUTO_INCREMENT,
`number` varchar(20) DEFAULT NULL,
`insert_date` datetime DEFAULT NULL,
`item_id` int(5) DEFAULT NULL,
`rule` tinyint(4) DEFAULT NULL,
`another_column` varchar(20) DEFAULT NULL,
`another_column1` varchar(20) DEFAULT NULL,
PRIMARY KEY (`ID`),
KEY `insert_date_index` (`insert_date`),
KEY `number` (`number`),
) ENGINE=InnoDB AUTO_INCREMENT=627393 DEFAULT CHARSET=latin1
and here is the unique constraint statement
Alter Table dbname.table add unique constraint my_unique_constraint (number, item_id, rule);
but I can not add a condition here in this constraint (unless there is a way I'm not aware of?)
The logic I need to run before inserts are blocked by the constraint is to check if the three values: number, item_id, and rule are unique in the table, and if they aren't, then I want to compare the existing record's insert_date with the insert_date from the transaction, and only keep the record with the newest insert_date.
This could be achieved with a trigger I suppose, although I've heard triggers are only to be used if really needed. And on every insert, this trigger would be quite computationally taxing on the DB. Any advice? Any other sql tricks I can use? Or anything to help point me to how to make this trigger?
I tried the unique constraint statement
Alter Table dbname.table add unique constraint my_unique_constraint (number, item_id, rule);
But it will never update with the newer insert_date.
You can do this with an insert statement like:
insert into apixio_results_test_sefath (number, item_id, rule, insert_date, another_column, another_column1)
values (?,?,?,?,?,?)
on duplicate key update
another_column=if(insert_date>values(insert_date),another_column,values(another_column),
another_column1=if(insert_date>values(insert_date),another_column1,values(another_column1),
insert_date=greatest(insert_date,values(insert_date)
for each column besides the unique ones and insert_date, testing to see if the existing insert_date is greater than the value supplied with the insert and conditionally using the existing value or new value for the other column based on that, and ending with updating insert_date only if it is now greater.
mysql 8 has an alternate syntax it prefers to using the values function, but the values function still works.
If you want this to happen automatically for all inserts, you would need to use a trigger.
I have the following table with a unique index by field "position_in_list":
CREATE TABLE `planned_operation` (
`id` bigint(20) NOT NULL,
`position_in_list` bigint(20) NOT NULL,
`name` varchar(255) not null
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
ALTER TABLE `planned_operation`
ADD PRIMARY KEY (`id`),
ADD UNIQUE KEY `position_in_list` (`position_in_list`);
ALTER TABLE `planned_operation`
MODIFY `id` bigint(20) NOT NULL AUTO_INCREMENT, AUTO_INCREMENT=3;
INSERT INTO `planned_operation` (`id`, `position_in_list`, `name`) VALUES
(1, 1, 'first'),
(2, 2, 'second');
Then I have a trivial task, this is a change in position when updating the list. Accordingly, you need to update the list of items before which the record was inserted. In order not to perform thousands of updates, I execute one query:
update planned_operation
set position_in_list = case position_in_list
when 2 then 3
when 1 then 2
end
where position_in_list in (1, 2)
But when executing an error is issued:
#1062 - Duplicate entry '1' for key 'position_in_list'
Is there any way to avoid an error? Without disabling the unique index
You want deferrable constraints.
Unfortunately, MySQL does not implement deferrable constraint checks -- an integral part of SQL that few database engines implement.
As far as I know only PostgreSQL and Oracle (partial) do implement them.
In simple words, this means that MySQL checks the unique constraint on every single row change inside an UPDATE statement. With deferrable constraints you could defer this check to the end of the statement, or even to the end of the database transaction.
Now, you would need to switch to PostgrSQL or Oracle to defer contraints checks to the end of the statement (as you seem to want). I guess that's way out of the scope for you, but it's a theoretical option.
For a more in depth discussion you could look into Deferrable Constraints answer.
EDIT **********
I couldn't get the below code to work so I have tried to run it in the diagram tool on Workbench. This is my code
CREATE DEFINER = CURRENT_USER TRIGGER `MeasureUp_Data`.`bmd_results_BEFORE_INSERT` BEFORE INSERT ON `bmd_l_arm` FOR EACH ROW
BEGIN
INSERT INTO bmd_results VALUES ( new.bmd_l_arm_typ
CASE new.l_arm_tscore
WHEN < -2.5 THEN 'osteoporosis'
WHEN > -2.4 < -1.0 THEN 'osteopenia'
WHEN > -1.0 THEN 'normal'
ELSE NULL
END);
END
Im getting red crosses all though the code. Basically the values will be automatically entered into data.bmd_l_arm and I want the field in data.bmd_results labelled bmd_l_arm_typ to have the text entry based on the numerical result pushed into l_arm_tscore
Thanks for you ongoing help!
I am very new to the coding game and have been creating a Db on MySQL to add some value to my company. I have created the DB schema and want to have a field in one of my table insert a value based on a entry which comes into another table. Basically this is for a medical reporting database and I would like the diagnosis to appear in text on in as a field entry.
This is the table which I would like the values to appear, more specially i would like the column (for example) bmd_sp_typ to inset a VARCHAR (I will specify the result) based on a numerical result in a different table.
CREATE TABLE IF NOT EXISTS `MeasureUp_Data`.`bmd_results` (
`access_no` INT NOT NULL,
`bmd_sp_typ` VARCHAR(45) NULL,
PRIMARY KEY (`access_no`),
INDEX `pat_id_FK_idx` (`pat_id` ASC),
INDEX `fk_site_bmd_site_idx` (`site_id` ASC),
UNIQUE INDEX `pat_id_UNIQUE` (`pat_id` ASC),
UNIQUE INDEX `site_id_UNIQUE` (`site_id` ASC),
UNIQUE INDEX `access_no_UNIQUE` (`access_no` ASC),
CONSTRAINT `fk_demo_results_id`
FOREIGN KEY (`pat_id`)
REFERENCES `MeasureUp_Data`.`patient_demo` (`pat_id`)
ON DELETE NO ACTION
ON UPDATE NO ACTION,
CONSTRAINT `fk_site_bmd_site`
FOREIGN KEY (`site_id`)
REFERENCES `MeasureUp_Data`.`site_cont` (`site_id`)
ON DELETE NO ACTION
ON UPDATE NO ACTION)
ENGINE = InnoDB
The table where the numerical result will be found is below. Basically is the value for spine_total_bmd is X then Y will be inserted into bmd_sp_type, alternatively, if spine_total_bmd is A then B will be inserted.
CREATE TABLE IF NOT EXISTS `MeasureUp_Data`.`bmd_spine` (
`access_no` INT NOT NULL,
`spine_total_bmd` INT NULL,
`spine_total_tscore` INT NULL,
`spine_total_zscore` INT NULL,
`spine_total_peakref` INT NULL,
`spine_total_agemat` INT NULL,
PRIMARY KEY (`access_no`),
CONSTRAINT `fk_bmd_spine_access`
FOREIGN KEY (`access_no`)
REFERENCES `MeasureUp_Data`.`bmd_results` (`access_no`)
ON DELETE NO ACTION
ON UPDATE NO ACTION)
ENGINE = InnoDB
Any help for this coding noob will be greatly appreciated.
You can write an INSERT trigger -
CREATE TRIGGER trigger
BEFORE INSERT
ON bmd_spine
FOR EACH ROW
BEGIN
INSERT INTO bmd_results VALUES (
new.access_no,
CASE new.spine_total_bmd
WHEN 1 THEN 'y'
WHEN 2 THEN 'b'
ELSE NULL
END);
END
Also, you can just select data from the table and change values on the fly, and do not store duplicated values. Here it is an example -
SELECT
access_no,
CASE spine_total_bmd
WHEN 1 THEN 'y'
WHEN 2 THEN 'b'
ELSE NULL
END
FROM
bmd_spine;
I have a table in MySQL (50 million rows) new data keep inserting periodically.
This table has following structure
CREATE TABLE values (
id double NOT NULL AUTO_INCREMENT,
channel_id int(11) NOT NULL,
val text NOT NULL,
date_time datetime NOT NULL,
PRIMARY KEY (id),
KEY channel_date_index (channel_id,date_time)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Two rows must never have duplicate channel_id and date_time, but if such insert occurs it is important to keep the newest value.
Is there a procedure to check for duplicates realtime before the insert or should I keep inserting all data while doing periodic checks for duplicity in a different cycle.
Realtime speed is important here, because 100 inserts occur per second.
To prevent future duplicates:
Change KEY channel_date_index (channel_id,date_time) to UNIQUE (channel_id,date_time)
Change the INSERT to INSERT ... ON DUPLICATE KEY UPDATE ... to change the timestamp when that pair exists.
To fix the existing table, you could do ALTER IGNORE TABLE ... ADD UNIQUE(...). However that would not give you the latest timestamps.
For minimum downtime (not maximum speed), use pt-online-schema-change.
A standard problem in applications is to insert a record if one doesn't exist - or update if it does. In cases where the PRIMARY KEY is unknown this is usally solved by issuing a SELECT and then running either an INSERT or UPDATE if the record was found.
However, there seems to be at least three ways I know of that you can insert a record into a database even when a record already exists. Personally, I would rather drop the new insert request if one already exists, but then there might be cases where you would rather drop the record in the database and use the new one.
CREATE TABLE IF NOT EXISTS `table` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`foo` int(10) unsigned NOT NULL,
`bar` int(10) unsigned NOT NULL,
PRIMARY KEY (`id`),
KEY `row` (`foo`,`bar`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Here are the three methods:
INSERT IGNORE INTO table (foo, bar) VALUES (2,3);
INSERT INTO table (foo, bar) VALUES (2,3) ON DUPLICATE KEY UPDATE;
REPLACE INTO table (foo, bar) VALUES (2,3);
At what times should each of these methods be used?
Can someone give some examples of correct usage scenarios?
INSERT should be used when you just want to insert a new row
Lets say you are storing log entries, you'll want to log every event, use INSERT.
INSERT IGNORE should be used when you just want there to be a specific key exists in the table, it doesn't matter if it's the current insert that creates it, or if it's already present.
Let's say you have a table of phone-numbers and the number of uses, you find a new phone number that you are not sure exists in the table, but you want it to be there.
You use INSERT IGNORE to make sure that it's there.
REPLACE INTO should be used when you want to make sure that a specific key exists in the table, if it exists you'd like the new values to be used, instead of that present.
You have another table with phone-numbers, this time you find a new number and a name to associate it with.
You use REPLACE INTO to find and update a complete record, or just insert the new information.
INSERT INTO ... ON DUPLICATE KEY UPDATE ...
Please not that this is not an alternative method of writing REPLACE INTO, the above should be used whenever you'd like to make sure that a specific key exists, but if it does update some of the columns, not all of them.
For example if you are storing the numbers of visits from a certain IP, and the first page the user ever visited.
INSERT INTO visitors (ip,visits,first_page) VALUES (<ip>,1,<current_page>) ON DUPLICATE KEY visits = visits +1;
In your question, you have
INSERT INTO `table` (foo, bar) VALUES (2,3) ON DUPLICATE KEY UPDATE;
That can only work if the row index was UNIQUE:
CREATE TABLE IF NOT EXISTS `table` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`foo` int(10) unsigned NOT NULL,
`bar` int(10) unsigned NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `row` (`foo`,`bar`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Otherwise, why have just an index?
Also the ON DUPLICATE KEY clauses allows you to update non-indexed columns.
As for REPLACE, keep in mind that REPLACE is actually DELETE and INSERT under the hood.