I want to update a MySQL table from matlab in bulk. The current logic that I use iterates over the array and inserts it one-by-one which takes way too long.
Here is my current implementation-
function update_table(customer_id_list, cluster_id_list, write_conn)
num_customers = size(customer_id_list, 1);
for idx=1:num_customers+1
customer_id = customer_id_list(idx);
cluster_id = cluster_id_list(idx);
sql = strcat(sql, 'UPDATE table SET cluster_id = ', num2str(cluster_id), ' WHERE customer_id = ', num2str(customer_id));
exec(write_conn, sql);
end
end
Tried to look for documentation to do bulk update/insert, but haven't found anything yet.
Do an "upjoin" using a temporary table.
Build your update specification as a Matlab table array with all the cluster_id and customer_id pairs that specify the new values.
Create a SQL temporary table that contains columns for the key columns you'll be matching on and the columns to update.
CREATE TEMPORARY TABLE my_temp_table SELECT customer_id, cluster_id FROM table WHERE 1 = 0
Batch-insert your update specification data from Matlab into the temporary table using Matlab Database Toolbox's datainsert or sqlwrite.
Update the target table en masse by joining it to the temp table: UPDATE table SET targ.cluster_id = upd.cluster_id FROM table targ INNER JOIN my_temp_table upd ON targ.customer_id = upd.customer_id.
Drop the temp table.
Boom. If you're going to do this a lot, wrap it up in a generic upjoin() function.
See the Matlab documentation for datainsert and sqlwrite. Do not use fastinsert; despite its name, it is much slower than datainsert and sqlwrite.
Related
I have a table with many columns in SQL Server and I have move part of data into MySQL. I made a view or function on the table in SQL Server and these two databases must be synced once a day through job. Because the data of this view may change every day.
View return a table with 3 columns: (char, varchar, varchar) that none of them are unique or primary key.
My solution is:
create a job
execute view on SQL Server
return result of view
create temp table with 3 column in MySQL
move result view from SQL Server to temp table
move records from temp table to new table one by one if not exist before
delete temp table
To transfer without using the temp table, I wanted to use below type of query but could not find the correct query. That's why I used the temp table:
insert into new_table
values (array of records) where record if not exist in new table.
And for the solution I mentioned above, I used the following query:
insert into new_table
select *
from temp_table
where not exist new_table.column = temp_table.column
Do you have a better suggestion that new records can be fetch and added to previous records?
It should look more like this:
insert into new_table
select *
from temp_table
where not exists (
select 1
from new_table
where new_table.column = temp_table.column
)
or maybe this:
insert into new_table
select *
from temp_table
where not exists (
select 1
from new_table
where new_table.column = temp_table.column
and new_table.column2 = temp_table.column2
and new_table.column3 = temp_table.column3
)
I have a table Provision with this structure:
ONT_ID varchar(12) PK
neID set('7360-1','7360-2','7360-3','5000-1','5000-2') not null
and some other crap
I have loaded a temporary table called tempTable that has the same structure and the same data. Prior to trying what I'm trying the neID in the Provision table was a varchar field. The values that were not the same as the set were deleted. (I've done this before without a problem.)
Neither this:
UPDATE Provision P
INNER JOIN tempTable TT ON TT.ONT_ID = P.ONT_ID
SET P.neID = TT.NE_ID
Nor this (broken up for readability):
update Provision P
set P.neID = (
select TT.NE_ID from tempTable TT where TT.ONT_ID = P.ONT_ID
)
...accomplishes what they are supposed to. What is going on?
The Provision table has a record of the ONT_ID and the neID is an empty string. The temp table has the same ONT_ID and a pertinent NE_ID. I'm trying to update the neID in the Provision table with the value that is in the temporary table.
Data in tempTable was wrong. Changed data and updated Provision table.
I am trying to insert records into MySQL database from a MS SQL Server using the "OPENQUERY" but what I am trying to do is ignore the duplicate keys messages. so when the query run into a duplicate then ignore it and keep going.
What ideas can I do to ignore the duplicates?
Here is what I am doing:
pulling records from MySQL using "OpenQuery" to define MySQL "A.record_id"
Joining those records to records in MS SQL Server "with a specific criteria and not direct id" from here I find a new related "B.new_id" record identifier in SQL Server.
I want to insert the found results into a new table in MySQL like so A.record_id, B.new_id Here in the new table I have A.record_id set as a primary key for that table.
The problem is that when joining table A to Table B some times I find 2+ records into table B matching the criteria that I am looking for which causes the value A.record_id to 2+ times in my data set before inserting that into table A which causes the problem. Note I can use aggregate function to eliminate the records.
I don't think there is a specific option. But it is easy enough to do:
insert into oldtable(. . .)
select . . .
from newtable
where not exists (select 1 from oldtable where oldtable.id = newtable.id)
If there is more than one set of unique keys, you can add additional not exists statements.
EDIT:
For the revised problem:
insert into oldtable(. . .)
select . . .
from (select nt.*, row_number() over (partition by id order by (select null)) as seqnum
from newtable nt
) nt
where seqnum = 1 and
not exists (select 1 from oldtable where oldtable.id = nt.id);
The row_number() function assigns a sequential number to each row within a group of rows. The group is defined by the partition by statement. The numbers start at 1 and increment from there. The order by clause says that you don't care about the order. Exactly one row with each id will have a value of 1. Duplicate rows will have a value larger than one. The seqnum = 1 chooses exactly one row per id.
If you are on SQL Server 2008+, you can use MERGE to do an INSERT if row does not exist, or an UPDATE.
Example:
MERGE
INTO dataValue dv
USING tmp_holding_DataValue t
ON t.dateStamp = dv.dateStamp
AND t.itemId = dv.itemId
WHEN NOT MATCHED THEN
INSERT (dateStamp, itemId, value)
VALUES (dateStamp, itemId, value)
I am working with a database table that stores, among other things, an AssociateID field and a DocumentName field. The table is used to keep records of which associates are missing which documents.
I am trying to write a stored procedure that will remove a row for 'Restrictive Covenant Letters' if an entry for 'Termination Statement' cannot be found for the same associate. Below is the body of what I have so far:
DELETE from tbl_HR_Auditing_Reports
WHERE DocumentName = 'Restrictive Covenant Letters'
AND NOT EXISTS
(
SELECT TS.AssociateID
FROM
(SELECT AssociateID FROM tbl_HR_Auditing_Reports WHERE DocumentName = 'Termination Statement (*)') TS,
(SELECT AssociateID FROM tbl_HR_Auditing_Reports WHERE DocumentName = 'Restrictive Covenant Letters') RCL
WHERE TS.AssociateID = RCL.AssociateID
)
I think I am close, but sql isn't really my forte. If anyone could possibly help, I would really appreciate it!
According to the MySQL manual:
Currently, you cannot delete from a table and select from the same table in a subquery.
To get around this you can use a temporary table, inserting the relevant rows into the temporary table, then referencing it in the delete statement.
CREATE TEMPORARY TABLE tmp (AssociateID INT);
INSERT INTO tmp (AssociateID)
SELECT AssociateID
FROM tbl_HR_Auditing_Reports
WHERE DocumentName = 'Termination Statement (*)';
DELETE
FROM tbl_HR_Auditing_Reports
WHERE DocumentName = 'Restrictive Covenant Letters'
AND NOT EXISTS
( SELECT 1
FROM tmp
WHERE tmp.AssociateID = tbl_HR_Auditing_Reports.AssociateID
)
Example on SQL Fiddle
I have a query written very poorly in SQL Server 2008
UPDATE PatientChartImages
SET PatientChartImages.IsLockDown = #IsLockdown
WHERE PatientChartImages.IsLockDown = #IsNotLockdown
AND PatientChartId IN (
SELECT PatientCharts.PatientChartId
FROM PatientCharts
WHERE ( PatientCharts.ChartStatusID = #ChartCompletedStatusID
OR PatientCharts.ChartStatusID = #ChartOnBaseStatusID
)
AND PatientCharts.IsLockDown = #IsNotLockdown
AND PatientCharts.CompletedOn IS NOT NULL
AND DATEDIFF(MINUTE, PatientCharts.CompletedOn, GETUTCDATE()) >= ( SELECT
tf.LockUpInterval
FROM
#tblFacCOnf tf
WHERE
tf.facilityId = PatientCharts.FacilityId
) )
This query locks the main table and results in TimeOut. IF i create a CTE first of all the updateable records and then update the main table by joining to the CTE. Will it help ??
First thing i advice you to do is to substitute IN condition with EXISTS. Second is to move all this conditional logic into CTE. Third is to substitute sub-select with #tblFacCOnf with join.
Last advice depends on your business logic and is not so important in my opinion
So at the end you will get something as
WITH search_cte as (
SELECT PatientCharts.PatientChartId
FROM PatientCharts
JOIN #tblFacCOnf tf on tf.facilityId = PatientCharts.FacilityId
WHERE ( PatientCharts.ChartStatusID = #ChartCompletedStatusID
OR PatientCharts.ChartStatusID = #ChartOnBaseStatusID
)
AND PatientCharts.IsLockDown = #IsNotLockdown
AND PatientCharts.CompletedOn IS NOT NULL
AND DATEDIFF(MINUTE, PatientCharts.CompletedOn, GETUTCDATE()) >= tf.LockUpInterval
) --cte end
UPDATE PatientChartImages
SET PatientChartImages.IsLockDown = #IsLockdown
WHERE PatientChartImages.IsLockDown = #IsNotLockdown
AND EXISTS (select 1 from PatientChartImages where PatientChartImages.PatientChartId = search_cte.PatientChartId)
One additional thing I might suggest if the other suggestions don't get you enough speed is not to use a table variable. Temp Tables are often faster for large data sets and can be indexed if need be.
The update lock is being held the time it takes to compute the CTE and the time for the update. The CTE time is probably causing the time out.
To reduce the lock time to the minimum required to update the target table. I suggest you create a temp table with two columns. Col1 is the primary key or cluster key of the target table and Col2 is the value you want in the target table. Wrap the temp table creation and fill the table with values according to your business logic within a transaction. Update the target table using a join to the temp table and the value from the temp table in a seperate transaction. After update drop the temp table.
I think you should create an SQL script (or stored procedure, if you will use it from a higher level) where you store the results of your selection into a cursor (you'll only have to find the PatientCartId's of the rows to be updated) and then you should use it in your update, so, the answer is yes.
It's easy to test this, you should put these commands into a transaction, rollback the transaction and before the rollback you should perform a selection to test your results. Good luck.