I'm using MySQL 5.6 and I have this issue.
I'm trying to improve my bulk update strategy for this case.
I have a table, called reserved_ids, provided by an external company, to assign unique IDs to its invoices. There is no other way to make this; I can't use auto_increment fields or simulated sequences.
I have this PL pseudocode to make this assignment:
START TRANSACTION;
OPEN invoice_cursor;
read_loop: LOOP
FETCH invoice_cursor INTO internalID;
IF done THEN
LEAVE read_loop;
END IF;
SELECT MIN(SECUENCIAL)
INTO v_secuencial
FROM RESERVED_IDS
WHERE COUNTRY_CODE = p_country_id AND INVOICE_TYPE = p_invoice_type;
DELETE FROM RESERVED_IDS WHERE SECUENCIAL = v_secuencial;
UPDATE MY_INVOICE SET RESERVED_ID = v_secuencial WHERE INVOICE_ID = internalID;
END LOOP read_loop;
CLOSE invoice_cursor;
COMMIT;
So, it's take one - remove - assign, then take next - remove - assign... and so on.
This works, but it's very very slow.
I don't know if there is any approach to make this assignment in a faster way.
I'm looking for something like INSERT INTO SELECT..., but with UPDATE statement, to assign 1000 or 2000 IDs directly, and no one by one.
Please, any suggestion is very helpful for me.
Thanks a lot.
EDIT 1: I have added WHERE clause details, because it was requested by user #vmachan . In the UPDATE...INVOICE clause, I don't filter by other criteria, because I have the direct and indexed invoice ID, which I want to update. Thanks
Finally, I have this solution. It's much faster than my initial approach.
The UPDATE query is
set #a=0;
set #b=0;
UPDATE MY_INVOICE
INNER JOIN
(
select
F.invoice_id,
I.secuencial as RESERVED_ID,
CONCAT_WS(/* format your final invoice ID */) AS FINAL_MY_INVOICE_NUMBER
FROM
(
select if(#a, #a:=#a+1, #a:=1) as current_row, internal_id
from MY_INVOICE
where reserved_id is null
order by internal_id asc
limit 2000
) F
INNER JOIN
(
SELECT if(#b, #b:=#b+1, #b:=1) as current_row, secuencial
from reserved_ids
order by secuencial asc
limit 2000
) I USING (CURRENT_ROW)
) TEMP MY_INVOICE.internal_id=TEMP.INTERNAL_ID
SET MY_INVOICE.RESERVED_ID = TEMP.RESERVED_ID, MY_INVOICE.FINAL_MY_INVOICE_NUMBER=TEMP.FINAL_MY_INVOICE_NUMBER
So, with autogenerated and correlated secuencial numbers #a and #b, we can join two different and no related tables like MY_INVOICE and RESERVED_IDs.
If you want to check this solution, please execute this tricky update following these steps:
Execute #a and then the first inner select in an isolated way: select if(#a, #a:=#a+1, ...
Execute #b and then the second inner select in an isolated way: select if(#b, #b:=#b+1, ...
Execute #a, #b and the big select that builds the TEMP auxiliar table: select F.invoice_id, ...
Execute the UPDATE
Finally, remove the assigned IDs from RESERVED_ID table.
Assignation time reduced drastically. My initial solution was one by one; with this, you assign 2000 (or more) in one single (ok, and a little tricky) update.
Hope this helps.
Related
I am trying to reduce the number of queries my application uses to build the dashboard and so am trying to gather all the info I will need in advance into one table. Most of the dashboard can be built in javascript using the JSON which will reduce server load doing tons of PHP foreach, which was resulting in excess queries.
With that in mind, I have a query that pulls together user information from 3 other tables, concatenates the results in JSON group by family. I need to update the JSON object any time anything changes in any of the 3 tables, but not sure what the "right " way to do this is.
I could set up a regular job to do an UPDATE statement where date is newer than the last update, but that would miss new records, and if I do inserts it misses updates. I could drop and rebuild the table, but it takes about 16 seconds to run the query as a whole, so that doesn't seem like the right answer.
Here is my initial query:
SET group_concat_max_len = 100000;
SELECT family_id, REPLACE(REPLACE(REPLACE(CONCAT("[", GROUP_CONCAT(family), "]"), "\\", ""), '"[', '['), ']"', ']') as family_members
FROM (
SELECT family_id,
JSON_OBJECT(
"customer_id", c.id,
"family_id", c.family_id,
"first_name", first_name,
"last_name", last_name,
"balance_0_30", pa.balance_0_30,
"balance_31_60", pa.balance_31_60,
"balance_61_90", pa.balance_61_90,
"balance_over_90", pa.balance_over_90,
"account_balance", pa.account_balance,
"lifetime_value", pa.lifetime_value,
"orders", CONCAT("[", past_orders, "]")
) AS family
FROM
customers AS c
LEFT JOIN accounting AS pa ON c.id = pa.customer_id
LEFT JOIN (
SELECT patient_id,
GROUP_CONCAT(
JSON_OBJECT(
"id", id,
"item", item,
"price", price,
"date_ordered", date_ordered
)
) as past_orders
FROM orders
WHERE date_ordered < NOW()
GROUP BY customer_id
) AS r ON r.customer_id = c.id
where c.user_id = 1
) AS results
GROUP BY family_id
I briefly looked into triggers, but what I was hoping for was something like:
create TRIGGER UPDATE_FROM_ORDERS
AFTER INSERT OR UPDATE
ON orders
(EXECUTE QUERY FROM ABOVE WHERE family_id = orders.family_id)
I was hoping to create something like that for each table, but at first glance it doesn't look like you can run complex queries such as that where we are creating nested JSON.
Am I wrong? Are triggers the right way to do this, or is there a better way?
As a demonstration:
DELIMITER $$
CREATE TRIGGER orders_au
ON orders
AFTER UPDATE
FOR EACH ROW
BEGIN
SET group_concat_max_len = 100000
;
UPDATE target_table t
SET t.somecol = ( SELECT expr
FROM ...
WHERE somecol = NEW.family_id
ORDER BY ...
LIMIT 1
)
WHERE t.family_id = NEW.family_id
;
END$$
DELIMITER ;
Notes:
MySQL triggers are row level triggers; a trigger is fired for "for each row" that is affected by the triggering statement. MySQL does not support statement level triggers.
The reference to NEW.family_id is a reference to the value of the family_id column of the row that was just updated, the row that the trigger was fired for.
MySQL trigger prohibits the SQL statements in the trigger from modifying any rows in the orders table. But it can modify other tables.
SQL statements in a trigger body can be arbitrarily complex, as long as its not a bare SELECT returning a resultset, or DML INSERT/UPDATE/DELETE statements. DDL statements (most if not all) are disallowed in a MySQL trigger.
I need to limit records based on percentage but MYSQL does not allow that. I need 10 percent User Id of (count(User Id)/max(Total_Users_bynow)
My code is as follows:
select * from flavia.TableforThe_top_10percent_of_the_user where `User Id` in (select distinct(`User Id`) from flavia.TableforThe_top_10percent_of_the_user group by `User Id` having count(distinct(`User Id`)) <= round((count(`User Id`)/max(Total_Users_bynow))*0.1)*count(`User Id`));
Kindly help.
Consider splitting your problem in pieces. You can use user variables to get what you need. Quoting from this question's answers:
You don't have to solve every problem in a single query.
So... let's get this done. I'll not put your full query, but some examples:
-- Step 1. Get the total of the rows of your dataset
set #nrows = (select count(*) from (select ...) as a);
-- --------------------------------------^^^^^^^^^^
-- The full original query (or, if possible a simple version of it) goes here
-- Step 2. Calculate how many rows you want to retreive
-- You may use "round()", "ceiling()" or "floor()", whichever fits your needs
set #limrows = round(#nrows * 0.1);
-- Step 3. Run your query:
select ...
limit #limrows;
After checking, I found this post which says that my above approach won't work. There's, however, an alternative:
-- Step 1. Get the total of the rows of your dataset
set #nrows = (select count(*) from (select ...) as a);
-- --------------------------------------^^^^^^^^^^
-- The full original query (or, if possible a simple version of it) goes here
-- Step 2. Calculate how many rows you want to retreive
-- You may use "round()", "ceiling()" or "floor()", whichever fits your needs
set #limrows = round(#nrows * 0.1);
-- Step 3. (UPDATED) Run your query.
-- You'll need to add a "rownumber" column to make this work.
select *
from (select #rownum := #rownum+1 as rownumber
, ... -- The rest of your columns
from (select #rownum := 0) as init
, ... -- The rest of your FROM definition
order by ... -- Be sure to order your data
) as a
where rownumber <= #limrows
Hope this helps (I think it will work without a quirk this time)
When I have about 30 actors this query is taking 20+ seconds. Is there a way to speed this up dramatically? I am sure there is, I just don't know in what way or where to start.
REPEAT
FETCH actors INTO a;
IF a != '' THEN
IF !(SELECT COUNT(*) FROM movieactor WHERE actor = a) THEN
INSERT INTO movieactor (actor)
VALUES (a);
END IF;
END IF;
UNTIL done END REPEAT;
movieactor table has just an actor and an id.
You really need to show the table definitions, indexes, query plan, and triggers to understand performance. However, this query has some obvious suggestions.
Why not use set operations, such as:
insert into movieactor(actor)
select a
from actors
where a <> '' and
a not in (select * from (select actor from movieactor))
(The double select is something that I think is needed for MySQL to parse this.)
Second, you should have an index on movieactor.actor. That will probably speed up the query a lot.
I want to update multiple rows based on a SELECT sql query.
I want to do it ALL IN AN SQL SHELL!
Here is my select:
SELECT #myid := id, #mytitle := title
FROM event
WHERE pid>0 GROUP BY title
ORDER BY start;
Then, I want to do an update with this pseudocode:
foreach($mytitle as $t)
BEGIN
UPDATE event
SET pid=$myid
WHERE title=$t;
END
But I don't know how to ake a loop in SQL.
Maybe there's a way to make it in a single sql query?
I DON'T WANT ANY PHP!!! ONLY SQL SHELL CODE!!!
I want to update every rows with a pid with the id of the first occurence of an event. Start is a timestamp
I think this should do what you want, but if it doesn't (I'm not sure about joining a subquery in an UPDATE query) then you can use a temporary table instead.
UPDATE
event
JOIN (
SELECT
MIN(pid) AS minPID,
title
FROM
event
WHERE
pid > 0
GROUP BY
title
) AS findPIDsQuery ON event.title = findPIDsQuery.title
SET
event.pid = findPIDsQuery.minPID
Pure SQL doesn't really have "loops", per se: it's a set-based descriptive language. I believe the following update will do what you want (though your problem statements leaves much to be desired—we know nothing about the underlying schema).
update event t
set pid = ( select min(id)
from event x
where x.title = t.title
and x.pid > 0
group by x.title
having count(*) > 1
)
Cheers!
I need to select data when a page is viewed and update the 'views' column is there a way to do this in one query, or do I have to use to distinct queries?
If you do not want/need to use a transaction, you could create a stored procedure that first updates the view count and then selects the values and return them to the user.
You would have to do this in two statements in one transaction
Begin Tran
Update Pages Set Views = Views + 1 Where ID = #ID
Select Columns From Pages Where ID = #ID
Commit Tran
It would help if you listed the RDBMS you are using
SQL Server has the OUTPUT statement
Example
USE AdventureWorks;
GO
DECLARE #MyTestVar table (
OldScrapReasonID int NOT NULL,
NewScrapReasonID int NOT NULL,
WorkOrderID int NOT NULL,
ProductID int NOT NULL,
ProductName nvarchar(50)NOT NULL);
UPDATE Production.WorkOrder
SET ScrapReasonID = 4
OUTPUT DELETED.ScrapReasonID,
INSERTED.ScrapReasonID,
INSERTED.WorkOrderID,
INSERTED.ProductID,
p.Name
INTO #MyTestVar
FROM Production.WorkOrder AS wo
INNER JOIN Production.Product AS p
ON wo.ProductID = p.ProductID
AND wo.ScrapReasonID= 16
AND p.ProductID = 733;
SELECT OldScrapReasonID, NewScrapReasonID, WorkOrderID,
ProductID, ProductName
FROM #MyTestVar;
GO
PostgreSQL's UPDATE statement has the RETURNING clause that will return a result set like a SELECT statement:
UPDATE mytable
SET views = 5
WHERE id = 16
RETURNING id, views, othercolumn;
I'm pretty sure this is not standard though. I don't know if any other databases implement it.
Edit: I just noticed that your question has the "MySQL" tag. Maybe you should mention it in the question itself. It's a good generic database question though - I would like to see how to do it in other databases.
I used this trick with Java and SQL Server will also let you send two commands in a single PreparedStatement.
update tablex set y=z where a=b \r\n select a,b,y,z from tablex
This will need to be in a read committed transaction to work like you think it should though.