I've had quite a few problems and know that I can get some good answers here!
Ok kinda 2 part question.
Part 1 I'm doing some really big updating of data, kind rejiging the tables mostly.
so the question is should I be using a mysql stored procedure or mysql/php like normal.
I'm currently on the stored producure frame of mind.
Reasons are
a) Quicker
b) No timeouts.
If anyone has any other opinions let me know.
P.S we are talking about a big heap of data. LIKE over 1.5 million rows
2nd part.
In stored procedures how do I make a query that will only return one row just give me that row. Also the query is a little dynamic so like
SET tag_query = concat('SELECT tag_id FROM tags WHERE tag = "',split_string_temp,'"');
Any clues?
I can't seem to find anything just easy about this language!
Thanks in advance for your help.
Richard
Your question is a little vague, so I'll just respond to the one piece of code you included.
If you want to get a tag_id from a tag name, I would recommend a stored function instead of a stored procedure.
Something like this:
DELIMITER $$
DROP FUNCTION IF EXISTS GET_TAG_ID $$
CREATE FUNCTION GET_TAG_ID(P_TAG_NAME varchar(255)) RETURNS int
BEGIN
DECLARE v_return_val INT;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET v_return_val = -1;
IF (P_TAG_NAME IS NULL)
THEN
RETURN NULL;
END IF;
select tag_id
into v_return_val
from TAGS
where tag = P_TAG_NAME;
RETURN v_return_val;
END $$
DELIMITER ;
To update data once (not as a regular task) I would prefer using a gui admin like phpmyadmin or sqlyog issuing SQL commands directly (with a good backup of course!) as you can see the results quickly and don't need to worry with other things than your main task.
Related
I'm using mysql and I want to use the if statement in a way you are doing in a stored procedure. Something like this:
delimiter $$
if #myVariable is null then
drop temporary table tmp_buffer;
select 'cannot proceed without variable #myVariable';
else
update my_table as t
set t.name = #myVariable;
end;
end$$
delimiter ;
When I execute this code it does nothing.
I tried to google this but I only find select if(....) explanations what doesn’t fit to my requirements.
I know that it is possible to create a stored procedure and put the code there and then call the procedure, but I'm looking for a way to do it without stored procedures.
Is this possible? If yes, what is wrong in my code?
Thanks for reading this
Felix
but I'm looking for a way to do it without stored procedures. Is this
possible?
NO; as already commented above, you cann't use if .. else construct block like the way you are intend to use in a normal SQL query. You will have to wrap it inside a procedural block which could be a stored procedure or a function.
I have inherited a MySQL InnoDB table with around 500 million rows. The table has IP numbers and the name of the ISP to which that number belongs, both as strings.
Sometimes, I need to update the name of an ISP to a new value, after company changes such as mergers or rebranding. But, because the table is so big, a simple UPDATE...WHERE statement doesn't work - The query usually times out, or the box runs out of memory.
So, I have written a stored procedure which uses a cursor to try and make the change one record at a time. When I run the procedure on a small sample table, it works perfectly. But, when I try to run it against the whole 500 million row table in production, I can see a temporary table gets created (because a /tmp/xxx.MYI and /tmp/xxx.MYD file appear). The temporary table file keeps growing in size until it uses all available disk space on the box (around 40 GB).
I'm not sure why this temporary table is necessary. Is the server trying to maintain some kind of rollback state? My real question is, can I change the stored procedure such that the temporary table is not created? I don't really care if some, but not all of the records get updated - I can easily add some reporting and just keep running the proc until no records are altered.
At this time, architecture changes are not really an option – I can't change the structure of the table, for example.
Thanks in advance for any help.
David
This is my stored proc;
DELIMITER $$
DROP PROCEDURE IF EXISTS update_isp;
CREATE PROCEDURE update_isp()
BEGIN
DECLARE v_finished INT DEFAULT 0;
DECLARE v_num VARCHAR(255) DEFAULT "";
DECLARE v_isp VARCHAR(255) DEFAULT "";
DECLARE ip_cursor CURSOR FOR
SELECT ip_number, isp FROM ips;
DECLARE CONTINUE HANDLER
FOR NOT FOUND SET v_finished = 1;
OPEN ip_cursor;
get_ip: LOOP
IF v_finished = 1 THEN
LEAVE get_ip;
END IF;
FETCH ip_cursor INTO v_num, v_isp;
IF v_isp = 'old name' THEN
UPDATE ips SET isp = 'new name' WHERE ip_number = v_num;
END IF;
END LOOP get_ip;
CLOSE ip_cursor;
END$$
DELIMITER ;
CALL update_isp();
I have also tried wrapping the update statement in a transaction. It didn't make any difference.
[EDIT] My assumption below, that a simple counting procedure does not create a temporary table, was wrong. The temporary table is still created, but it grows more slowly and the box does not run out of disk space before the procedure completes.
So the problem seems to be that any use of a cursor in a stored procedure results in a temporary table being created. I have no idea why, or if there is any way to prevent this.
If your update is essentially:
UPDATE ips
SET isp = 'new name'
WHERE isp = OLDNAME;
I am guessing that this update -- without the cursor -- will work better if you have an index on isp(isp):
create index idx_isp_isp on isp(isp);
Your original query should be fine once this index is created. There should be no performance issue updating a single row even in a very large table. The issue is in all likelihood finding the row, not updating it.
I don't think there is a solution to this problem.
From this page; http://spec-zone.ru/mysql/5.7/restrictions_cursor-restrictions.html
In MySQL, a server-side cursor is materialized into an internal
temporary table. Initially, this is a MEMORY table, but is converted
to a MyISAM table when its size exceeds the minimum value of the
max_heap_table_size and tmp_table_size system variables.
I misunderstood how cursors work. I assumed that my cursor functioned as a pointer to the underlying table. But, it seems MySQL must build the full result set first, and then give you a pointer to that. So, I don't really understand the benefits of cursors in MySQL. Thanks to everyone who tried to help.
David
If the table has some numerical index also you can specify a
WHERE myindex > 123 AND myindex < 456
in your update query and do that for a couple of intevals (with a loop for example) until the whole table is covered.
(sorry, my rep is too low to ask in the comment section, so I'll just post my guess-answer here to be able to comment on)
You could try to fake a numerical index with
SELECT ROW_NUMBER() as n, thetable.* FROM thetable ORDER BY oneofyourcolumns;
and then try what I suggested above.
I have Postgresql Function which has to INSERT about 1.5 million data into a table. What I want is I want to see the table getting populated with every one records insertion. Currently what is happening when I am trying with say about 1000 records, the get gets populated only after the complete function gets executed. If I stop the function half way through, no data gets populated. How can I make the record committed even if I stop after certain number of records have been inserted?
This can be done using dblink. I showed an example with one insert being committed you will need to add your while loop logic and commit every loop. You can http://www.postgresql.org/docs/9.3/static/contrib-dblink-connect.html
CREATE OR REPLACE FUNCTION log_the_dancing(ip_dance_entry text)
RETURNS INT AS
$BODY$
DECLARE
BEGIN
PERFORM dblink_connect('dblink_trans','dbname=sandbox port=5433 user=postgres');
PERFORM dblink('dblink_trans','INSERT INTO dance_log(dance_entry) SELECT ' || '''' || ip_dance_entry || '''');
PERFORM dblink('dblink_trans','COMMIT;');
PERFORM dblink_disconnect('dblink_trans');
RETURN 0;
END;
$BODY$
LANGUAGE plpgsql VOLATILE
COST 100;
ALTER FUNCTION log_the_dancing(ip_dance_entry text)
OWNER TO postgres;
BEGIN TRANSACTION;
select log_the_dancing('The Flamingo');
select log_the_dancing('Break Dance');
select log_the_dancing('Cha Cha');
ROLLBACK TRANSACTION;
--Show records committed even though we rolled back outer transaction
select *
from dance_log;
What you're asking for is generally called an autonomous transaction.
PostgreSQL does not support autonomous transactions at this time (9.4).
To properly support them it really needs stored procedures, not just the user-defined functions it currently supports. It's also very complicated to implement autonomous tx's in PostgreSQL for a variety of internal reasons related to its session and process model.
For now, use dblink as suggested by Bob.
If you have the flexibility to change from function to procedure, from PostgreSQL 12 onwards you can do internal commits if you use procedures instead of functions, invoked by CALL command. Therefore your function will be changed to a procedure and invoked with CALL command: e.g:
CREATE PROCEDURE transaction_test2()
LANGUAGE plpgsql
AS $$
DECLARE
r RECORD;
BEGIN
FOR r IN SELECT * FROM test2 ORDER BY x LOOP
INSERT INTO test1 (a) VALUES (r.x);
COMMIT;
END LOOP;
END;
$$;
CALL transaction_test2();
More details about transaction management regarding Postgres are available here: https://www.postgresql.org/docs/12/plpgsql-transactions.html
For Postgresql 9.5 or newer you can use dynamic background workers provided by pg_background extension. It creates autonomous transaction. Please, refer the github page of the extension. The sollution is better then db_link. There is a complete guide on Autonomous transaction support in PostgreSQL. There is a third way to start autonomous transaction in Postgres, but some patching neede. Please see Peter's Eisentraut patch proposal for OracleDB-style transactions.
I am great with Microsoft's SQL Server and SQL Server Management Studio (SSMS).
I'm trying to get things I used to do there to work in MySQL Workbench, but it is giving me very unhelpful errors.
Currently, I am trying to write an INSERT statement. I want to declare my variables and test it with a few values, then turn the end result into a stored procedure.
Right now, I have a syntax error that is not allowing me to continue, and the error message is not helpful either:
syntax error, unexpected DECLARE_SYM
There is no error list at the bottom and no way to copy the text of that error to the clipboard, so it all has to be studied on one screen, then flip to this screen so I can write it down.
Irritating!
The MySQL documentation surely has what I'm looking for, but I can learn much faster by doing than spending weeks reading their online manual.
DELIMITER $$
declare cGroupID char(6) DEFAULT = 'ABC123';
declare subGroupRecords int;
declare nDocTypeID int(11);
declare bDocActive tinyint(1) DEFAULT '1';
declare cDocID varchar(256) DEFAULT NULL;
insert into dbo_connection.documents
(group_id, subgroup_id, type_id, active, title, doc_id, priority, ahref, description, last_modified)
values
(cGroupID,cSubGroupID,nDocTypeID,bDocActive,cTitle,cDocID,0,ahref1, docDesc,NOW());
select * from dbo_connection.documents where group_id='ABC123';
END
So, for right now, I'm looking for why MySQL does not like my declare statement.
For the long term, I'm interested in finding a short article that shows a cookbook approach to doing some of the basic tasks that SQL developers would need (i.e. skips the Hello World program and discussion on data types).
DECLARE is only valid within stored programs. In other words, unlike T-SQL, you can't build up your query and then wrap CREATE PROCEDURE around it to turn the end result into a stored procedure, you have to build it up as a stored procedure from the get-go.
When inserting a new row in a table T, I would like to check if the table is larger than a certain threshold, and if it is the case delete the oldest record (creating some kind of FIFO in the end).
I thought I could simply make a trigger, but apparently MySQL doesn't allow the modification of the table on which we are actually inserting :
Code: 1442 Msg: Can't update table 'amoreAgentTST01' in stored function/trigger because it is already used by statement which invoked this stored function/trigger.
Here is the trigger I tried :
Delimiter $$
CREATE TRIGGER test
AFTER INSERT ON amoreAgentTST01
FOR EACH ROW
BEGIN
DECLARE table_size INTEGER;
DECLARE new_row_size INTEGER;
DECLARE threshold INTEGER;
DECLARE max_update_time TIMESTAMP;
SELECT SUM(OCTET_LENGTH(data)) INTO table_size FROM amoreAgentTST01;
SELECT OCTET_LENGTH(NEW.data) INTO new_row_size;
SELECT 500000 INTO threshold;
select max(updatetime) INTO max_update_time from amoreAgentTST01;
IF (table_size+new_row_size) > threshold THEN
DELETE FROM amoreAgentTST01 WHERE max_update_time = updatetime; -- and check if not current
END IF;
END$$
delimiter ;
Do you have any idea on how to do this within the database ?
Or it is clearly something to be done in my program ?
Ideally you should have a dedicated archive strategy in a separate process that runs at off-peak times.
You could implement this either as a scheduled stored procedure (yuck) or an additional background worker thread within your application server, or a totally separate application service. This would be a good place to put other regular housekeeping jobs.
This has a few benefits. Apart from avoiding the trigger issue you're seeing, you should consider the performance implications of anything happening in a trigger. If you do many inserts, that trigger will do that work and effectively half the performance, not to mention the lock contention that will arise as other processes try to access the same table.
A separate process that does housekeeping work minimises lock contention, and allows the work to be carried out as a high-performance bulk operation, in a transaction.
One last thing - you should possibly consider archiving records to another table or database, rather than deleting them.