I creating a data mart system where i should pull some data from another system and push it here.To insert into data mart tables I am writing a procedure, which will be run by EOD. Initially all the data will be pushed, later for subsequent pulls only modified or newly added will be pushed into my tables.
My Procedure should be fast in terms of performance. This is what I have written. I wanted to know any other best way.
SET #rowCount = (SELECT COUNT(1) FROM ST_Student)
IF(#rowCount ==0)
BEGIN
-- Insert all 10000 records
END
ELSE
BEGIN
Delete from table where ID is exist.
Insert all newly added records,( max 10 record)
END
What is the best to do this. So that I can ignore two transactions.
Related
This question already has an answer here:
How to rename two tables in one atomic operation in MySQL
(1 answer)
Closed 2 years ago.
Getting some "myTable does not exist" errors while renaming a table to an updated version of itself. Not sure if I'm doing it right.
I have a web site where users run queries against a table that is replaced once every 5 minutes with an updated copy of itself. The table has 600,000 rows and needs to be built from scratch once every few minutes so that it is internally consistent.
This is how I do the table update:
// not shown: bunch of code to build newTable from scratch; takes 90 seconds
// while this is happening users are querying myTable
// Then, on a 5 minute mark, this happens:
START TRANSACTION
RENAME TABLE myTable TO oldTable // fast, like 0.005 seconds
RENAME TABLE newTable TO myTable // fast, like 0.005 seconds
COMMIT
DROP oldTable // a bit slow... like 0.5 to 1.0 seconds
I put the DROP outside the transaction because I'm trying to minimize the time when myTable doesn't exist.
During this transition period (which happens every 5 min) I'm getting a 1 to 3 mysql errors "myTable does not exist".
I'm not sure if some users are just starting a query exactly during the time when myTable has been renamed (and therefore does not exist) before newTable has been renamed to myTable? It's a pretty tiny window; I think the transaction takes 0.01 seconds and there are maybe 20-30 users on the site at one time (according to Google Analytics) running queries.
Or maybe there are some longer queries in progress just before I rename myTable to oldTable? Does a query from another thread fail if you take its table away in another thread?
Should I even be using START TRANSACTION / COMMIT for this use case?
All tables are InnoDB. Mysql version is "Ver 8.0.22-0ubuntu0.20.04.3 for Linux on x86_64 ((Ubuntu))"
Any suggestions on how I can get rid of the "myTable does not exist" errors while I'm in the middle of renaming the tables once every 5 minutes?
Im not sure if this will work for your use case, but one way you could achieve the same result without rebuilding the table every few minutes would be like this:
Only use one table, but add a column to it that defines the version number or something similar.
In your 'rebuild' step, add rows to the table with a new version id.
While this is happening query the table for the previous version id
At the 5 minute mark, start querying for the new version id.
Remove all rows from the table that have the old version id
Updated:
I think using two tables with same structure between which you can switch would solve you problem. Let's say we have TableA and TableB, and currently your application is reading data from tableA. Once you receive a new snapshot and process the data, try inserting it in the non-used table(TableB).
Once data is inserted, you may switch your reference from tableA to tableB.
You may perform a select with limit 1, to get the timestamp from both tables, to check which onc contains stale data and should be overridden. And same logic can be used to reference the correct table.
Similar to having two partitions and switching b/w the two OS
Same logic can be abstracted in a Stored Procedure, which you can call to insert or fetch data. And the Stored Procedure would decide which table to refer. This would reduce multiple DB calls from the application.
To start with,
START TRANSACTION won't help as RENAME is a DDL command
And the syntax is incorrect (oldTableName then newTableName):
RENAME TABLE oldTable TO newTable;
You can use a store procedure, which first checks if table exists before renaming, to avoid errors and silently fail the queries.
DELIMITER $$
CREATE PROCEDURE SP_RENAME_TABLE(IN newTable VARCHAR(64), IN oldTable VARCHAR(64))
BEGIN
IF EXISTS(
SELECT 1 FROM information_schema.tables
WHERE
table_schema = DATABASE() AND
table_name = oldTable LIMIT 1)
THEN
SET #query = CONCAT('RENAME TABLE ', oldTable, ' TO ', newTable, ';');
PREPARE stmt FROM #query;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
END IF;
END$$
DELIMITER ;
CALL SP_RENAME_TABLE('new_table_name', 'old_table_name');
I have 2 tables (1. Staging Table and 2. Master Table).
Staging Table (temp_Customer).
Master Table (Customer_Master)
I want to insert records from Staging table to Master table one by one and update those records as sync_status = 1 in the staging table.So that next time I would be able to know that which records are not in my master table and I will pick only those records where sync_status is 0.
My boss has suggested me to do it with loop but not to use it with the curser.
and I don't know how to use loops in sql..
I recommend using a transaction which will first attempt to insert all records from staging into master, and then will mark the staging records as having been transferred.
BEGIN TRANSACTION [yourTransaction]
BEGIN TRY
INSERT INTO Master ([STK_Code], [Cust_Code], [Cust_Add], ...) -- other columns
SELECT [STK_Code], [Cust_Code], [Cust_Add], ...
FROM Staging
WHERE sync_status = 0
UPDATE Staging
SET sync_status = 1
WHERE sync_status = 0
COMMIT TRANSACTION [yourTransaction]
END TRY
BEGIN CATCH
ROLLBACK TRANSACTION [Tran1]
END CATCH
I am wondering if it is possible to perform a SQL query then update another table with the generated ID and continue through all of the rows?
I have this SQL query that works but what I need to do is after each row is added to cards to then update merged.cars_id with the last generated ID so they are linked. normally I would do this with PHP but ideally I would like to just do it with MySQL if possible.
MAIN QUERY
INSERT INTO cards (first_contact_date, card_type, property_id, user_id)
SELECT first_contact_date, 'P', property_id, user_id FROM merged
THEN I NEED WITH MATCHING ROWS (Roughly)
UPDATE merged SET merged.card_id = LAST_INSERT_ID (FROM ABOVE) into the matching record..
Is something like this possible and how do I do it?
I would recommend using MySQL triggers to do this
http://dev.mysql.com/doc/refman/5.0/en/create-trigger.html
A trigger is a function that will be executed AFTER or BEFORE the INSERT or DELETE or UPDATE is done over any record of your table.
In your case you need to do a AFTER INSERT on cards that just updates the merged table. Make sure its AFTER insert as you wont be able to access the new row's ID otherwise.
The code would look something like this, assuming the id field from the cards table its named "id"
delimiter |
CREATE TRIGGER updating_merged AFTER INSERT ON cards
FOR EACH ROW BEGIN
UPDATE merged SET card_id = NEW.id;
END;
|
delimiter ;
May I suggest Stored Procedures?
http://dev.mysql.com/doc/refman/5.0/en/create-procedure.html
--EDIT--
Ah yes, triggers. For this particular situation, Jimmy has the answer. I will leave this post for the sake of the link.
I would set up a trigger to do this. For mysql, read http://dev.mysql.com/doc/refman/5.0/en/triggers.html. This is what triggers are designed to handle.
I'm new to SSIS and need help on this one. I found an article which describes how to detect rows which exist and which have changed. The part that I'm missing is how to update rows that changed. I found some articles which say that it's also good solution to delete records which have changed and insert new recordset. The thing is I don't know how to do that step of deleting (red box).
Any suggestions?
If you have to delete the rows within Data Flow Task, then you need to use the OLE DB Command transformation and write a DELETE statement like DELETE FROM dbo.Table WHERE ColumnName = ?. Then in the column mappings of the OLE DB Command transformation, you will map the parameter represented by the question mark with the data that comes from the previous transformation. In your case, the data that comes from Union All 2.
However, I wouldn't recommend that option because OLE DB Command executes for every row and it might slow down your package if there are too many rows.
I would recommend something like this:
Redirect the output from the Union All 2 to a temporary staging table (say dbo.Staging) using OLE DB Destination.
Let's us assume that your final destination table is dbo.Destination. Now, your Staging table has all the records that should be deleted from the table Destination.
On the Control Flow tab, place an Execute SQL Task after the Data Flow Task. In the Execute SQL Task, write an SQL statement or use a stored procedure that would call an SQL statement to join the records between Staging and Destination to delete all the matching rows from Destination table.
Also, place another Execute SQL Task before the Data Flow Task. In this Execute SQL Task, delete/truncate rows from the Staging table.
Something like this might work to delete the rows:.
DELETE D
FROM dbo.Destination D
INNER JOIN dbo.Staging S
ON D.DestinationId = S.StagingId
Hope that helps.
In addition to user756519 answer. If you have millions of records to delete the last step (4) for ExecuteSQL Delete statement can be done in batches with something like this:
WHILE (1=1)
BEGIN
DELETE D
from dbo.Destination D
inner join
(
-- select ids that should be removed from table
SELECT TOP(10000) DestinationId
FROM
(
SELECT
D1.DestinationId,
S.StagingId
from
dbo.Destination as D1
LEFT JOIN
dbo.Staging as S
ON
D1.DestinationId = S.StagingId
) AS G
WHERE
StagingId IS NULL
) as R
on D.DestinationId = R.DestinationId;
IF ##ROWCOUNT < 1 BREAK
-- info message
DECLARE #timestamp VARCHAR(50)
SELECT #timestamp = CAST(getdate() AS VARCHAR)
RAISERROR ('Chunk deleted %s', 10, 1,#timestamp) WITH NOWAIT
END
Hey guys, here is one I am not able to figure out. We have a table in database, where PHP inserts records. I created a trigger to compute a value to be inserted as well. The computed value should be unique. However it happens from time to time that I have exact same number for few rows in the table. The number is combination of year, month and day and a number of the order for that day. I thought that single operation of insert is atomic and table is locked while transaction is in progress. I need the computed value to be unique...The server is version 5.0.88. Server is Linux CentOS 5 with dual core processor.
Here is the trigger:
CREATE TRIGGER bi_order_data BEFORE INSERT ON order_data
FOR EACH ROW BEGIN
SET NEW.auth_code = get_auth_code();
END;
Corresponding routine looks like this:
CREATE FUNCTION `get_auth_code`() RETURNS bigint(20)
BEGIN
DECLARE my_auth_code, acode BIGINT;
SELECT MAX(d.auth_code) INTO my_auth_code
FROM orders_data d
JOIN orders o ON (o.order_id = d.order_id)
WHERE DATE(NOW()) = DATE(o.date);
IF my_auth_code IS NULL THEN
SET acode = ((DATE_FORMAT(NOW(), "%y%m%d")) + 100000) * 10000 + 1;
ELSE
SET acode = my_auth_code + 1;
END IF;
RETURN acode;
END
I thought that single operation of
insert is atomic and table is locked
while transaction is in progress
Either table is locked (MyISAM is used) or records may be locked (InnoDB is used), not both.
Since you mentioned "transaction", I assume that InnoDB is in use.
One of InnoDB advantages is absence of table locks, so nothing will prevent many triggers' bodies to be executed simultaneously and produce the same result.