I've created a stored procedure. I fetch cursor with huge amount of data(around 1L rows).
After that I call another procedure in it to do all calculations related to needed data.
I create temporary table and try to insert this calculated data in it. But it takes too long
about 9.5 mins.
I want to know how to insert bulk data by using least "INSERT" queries as 1L insert queries cause poorest performance. Can anyone help me??
You can use the following SQL statement for bulk insert:
INSERT INTO TABLE_A (A, B, C, D) VALUES
(1,1,1,1),
(2,2,2,2),
(3,3,3,3),
(4,4,4,4);
Your question is a bit vague but you can BULK load data into mysql using
load data infile...
Check the following links:
http://dev.mysql.com/doc/refman/5.1/en/load-data.html
MySQL load data infile - acceleration?
If you need to process the data during the load it might be better to first bulk load the data into a temporary table then run some stored procedures against it which process and populate your main tables.
Hope this helps.
also you can insert record to in-memory table.after complete inserting to in-memory table following code for insert many rows from in-memory table.
INSERT INTO TABLE_A (A, B, C, D)
SELECT A,B,C,D FROM INMEMORY_TABLE
DELETE FROM INMEMORY_TABLE
Related
I am developer and working on performance improvment on MySQL queries. The flow is full load in 2 stages. First query will read FILES table and load into TABLE_STAGE table with full delete and in 2nd level also it read from TABLE_STAGE to TABLE_MAIN table. First delete it and then select all the records.
delete from TABLE_STAGE
select from FILES
delete from TABLE_MAIN ;
insert into TABLE_MAIN from select * from TABLE_STAGE
As a first part of step i have replaced delete with truncate . It improved performance immediately , but when i using delete again , performance is same ,the time is not increasing. I am not getting the reason behind this why it is showing the same result..
Don't use DELETE, it is slow.
CREATE TABLE new LIKE real;
LOAD DATA INFILE INTO new (or use batched insert)
RENAME TABLE real TO old,
new TO real;
DROP TABLE old;
(I don't understand where your FILES table is used.)
I want to add some 1000 records into my table for creating a database. Inserting each record manually is not at all practical. Is there a proper way to do this?
In MySQL you can insert multiple rows with a single insert statement.
insert into table values (data-row-1), (data-row-2), (data-row-3)
If you run a mysqldump on your database, you will see that this is what the output does.
The insert is then run as a single "transaction", so it's much, much faster than running 1000 individual inserts
I have a situation in which I have to insert over 10 million separate records into one table. Normally a batch insert split into chunks does the work for me. The problem however is that this over 3gig file contains over 10 million separate insert statements. Since every query takes 0.01 till 0.1 seconds, it will take over 2 days to insert everything.
I'm sure there must be a way to optimize this by either lowering the insert time drasticly or somehow import in a different way.
I'm now just using the cli
source /home/blabla/file.sql
Note: It's a 3th party that is providing me this file. I'm
Small update
I removed any indexes
Drop the indexes, then re-index when you are done!
Maybe you can parse the file data and combine several INSERT queries to one query like this:
INSERT INTO tablename (field1, field2...) VALUES (val1, val2, ..), (val3, val4, ..), ...
There are some ways to improve the speed of your INSERT statements:
Try to insert many rows at once if this is an option.
An alternative can be to insert the data into a copy of your desired table without indexes, insert the data there, then add the indexes and rename your table.
Maybe use LOAD DATA INFILE, if this is an option.
The MySQL manual has something to say about that, too.
Hi I have a huge unnormalized mysql database with (~100 million) urls (~20% dupes) divided into identical split tables of 13 million rows each.
I want to move the urls into a normalized database on the same mySql server.
The old database table is unnormalized, and the url's have no index
It look like this:
entry{id,data,data2, data3, data4, possition,rang,url}
And i'm goin to slit it up into multiple tables.
url{id,url}
data{id,data}
data1{id,data}
etc
The first thing I did was
INSERT IGNORE INTO newDatabase.url (url)
SELECT DISTINCT unNormalised.url FROM oldDatabase.unNormalised
But the " SELECT DISTINCT unNormalised.url" (13 million rows) took ages, and I figured that that since "INSERT IGNORE INTO" also do a comparison, it would be fast to just do a
INSERT IGNORE INTO newDatabase.url (url)
SELECT unNormalised.url FROM oldDatabase.unNormalised
Without the DISTINCT, is this assumption Wrong?
Any way it still takes forever and i need some help, is there a better way of dealing withe this huge quantity of unnormalized data?
Whould it be best if i did a SELECT DISTINCT unNormalised.url" on the entire 100 milion row database, and exported all the id's, and then moved only those id's to the new database with lets say a php script?
All ideas are welcomed, i have no clue how to port all this date without it taking a year!
ps it is hosted on a rds amazon server.
Thank you!
As the MySQL Manual states that LOAD DATA INFILE is quicker than INSERT, the fastest way to load your data would be:
LOCK TABLES url WRITE;
ALTER TABLE url DISABLE KEYS;
LOAD DATA INFILE 'urls.txt'
IGNORE
INTO TABLE url
...;
ALTER TABLE url ENABLE KEYS;
UNLOCK TABLES;
But since you already have the data loaded into MySQL, but just need to normalize it, you might try:
LOCK TABLES url WRITE;
ALTER TABLE url DISABLE KEYS;
INSERT IGNORE INTO url (url)
SELECT url FROM oldDatabase.unNormalised;
ALTER TABLE url ENABLE KEYS;
UNLOCK TABLES;
My guess is that INSERT IGNORE ... SELECT will be faster than INSERT IGNORE ... SELECT DISTINCT but that's just a guess.
I noticed that the concept of temporary tables in these two systems is different, and I have a musing.. I have the following scenario in MySQL:
Drop temporary table 'a' if exists
Create temporary table 'a'
Populate it with data through a stored procedure
Use the data in another stored procedure
How can I implement the same scenario in Oracle? Can I (in one procedure preferable) create a temporary table, populate it, and insert data in another (non-temporary) table?
I think that I can use a (global) temporary table which truncates on commit, and avoid steps 1&2, but I need someone else's opinion too.
In Oracle, you very rarely need a temporary table in the first place. You commonly need temporary tables in other databases because those databases do not implement multi-version read consistency and there is the potential that someone reading data from the table would be blocked while your procedure runs or that your procedure would do a dirty read if it didn't save off the data to a separate structure. You don't need global temporary tables in Oracle for either of these reasons because readers don't block writers and dirty reads are not possible.
If you just need a temporary place to store data while you perform PL/SQL computations, PL/SQL collections are more commonly used than temporary tables in Oracle. This way, you're not pushing data back and forth from the PL/SQL engine to the SQL engine and back to the PL/SQL engine.
CREATE PROCEDURE do_some_processing
AS
TYPE emp_collection_typ IS TABLE OF emp%rowtype;
l_emps emp_collection_type;
CURSOR emp_cur
IS SELECT *
FROM emp;
BEGIN
OPEN emp_cur;
LOOP
FETCH emp_cur
BULK COLLECT INTO l_emps
LIMIT 100;
EXIT WHEN l_emps.count = 0;
FOR i IN 1 .. l_emps.count
LOOP
<<do some complicated processing>>
END LOOP;
END LOOP;
END;
You can create a global temporary table (outside of the procedure) and use the global temporary table inside your procedure just as you would use any other table. So you can continue to use temporary tables if you so desire. But I can count on one hand the number of times I really needed a temporary table in Oracle.
You are right, temporary tables will work work you.
If you decide stick with regular tables you may want to use the advice #Johan gave, along with
ALTER TABLE <table name> NOLOGGING;
to make this perform a bit faster.
I see no problem in the scheme your are using.
Note that it doesn't have to be a temp-table, you can use a sort of kind of memory table as well.
Do this by creating a table as usual, then do
ALTER TABLE <table_name> CACHE;
This will prioritize the table for storage in memory.
As long as you fill and empty the table in short order you don't need to do step 1 & 2.
Remember the cache modifier is just a hint. The table still ages in the cache and will be pushed out of memory eventually.
Just do:
Populate cache-table with data through a stored procedure
Use the data in another stored procedure, but don't wait to long.
2a. Clear the data in the cache table.
In your MySQL version, I didn't see a step 5 to drop the table a. So, if you want or don't mind having the data in the table persist you could also use a materialized view and simply refresh on demand. With a materialized view you do not need to manage any INSERT statements, just include the SQL:
CREATE MATERIALIZED VIEW my_mv
NOCACHE -- NOCACHE/CACHE: Optional, cache places the table in the most recently used part of the LRU blocks
BUILD IMMEDIATE -- BUILD DEFERRED or BUILD IMMEDIATE
REFRESH ON DEMAND
WITH PRIMARY KEY -- Optional: creates PK column
AS
SELECT *
FROM ....;
Then in your other stored procedure, call:
BEGIN
dbms_mview.refresh ('my_mv', 'c'); -- 'c' = Complete
END;
That said, a global temporary table will work as well, but you manage the insert and exceptions.