I'm creating a function in postgres and getting strange error. What am I doing wrong? I also would like to see your variants how to do it
CREATE OR REPLACE FUNCTION export_csv(request TEXT, filename VARCHAR(255))
RETURNS VOID AS
$$
BEGIN
EXECUTE 'COPY (' || request || ') TO "/home/r90t/work/study/etl/postgres_etl/export/' || filename || '" WITH CSV;';
END
$$
LANGUAGE plpgsql;
REQUEST:
SELECT export_csv('SELECT * FROM orders', 'orders.csv')
ERROR:
psql:/tmp/vUp267V/dbext.sql:2: ERROR: syntax error at or near ""/home/r90t/work/study/etl/postgres_etl/export/orders.csv""
LINE 1: COPY (SELECT * FROM orders) TO "/home/r90t/work/study/etl/po...
^$
QUERY: COPY (SELECT * FROM orders) TO "/home/r90t/work/study/etl/postgres_etl/export/orders.csv" WITH CSV;
CONTEXT: PL/pgSQL function export_csv(text,character varying) line 3 at EXECUTE statement
Oh boy.....
First, because you COPY TO FILE, your function must run as superuser, and you are inlining an SQL query provided by the user into that file. At least you must run the query as superuser and you haven't set SECURITY DEFINER on it. But the whole point of your function is SQL injection and for very little gain. I get that this is a bit for personal study but there is nothing to be gained by doing it in a way that would put a business's data at risk in the future.
In particular, I wonder what would happen if I do something like:
SELECT export_csv('SELECT * FROM ORDERS TO STDOUT; DROP DATABASE critical_db; --', 'foo');
or
SELECT export_csv('SELECT * FROM ORDERS', '../../../../../../../var/lib/pgsql/data/log/postgresql-Tue.log');
Really, really, really bad stuff are possible with your function. Don't do it. these are contained now but as soon as someone does the following, you are totally:
ALTER FUNCTION export_csv SET SECURITY DEFINER;
A better approach would be to take a single argument that can be quoted and processed in place. Something like:
CREATE OR REPLACE FUNCTION export_csv(relation name, columns name[])
RETURNS VOID AS
$$
DECLARE column_list text;
BEGIN
SELECT array_to_string(cols, ', ') INTO column_list
FROM (SELECT array_agg(quote_literal(col)) as cols
FROM unnest(columns) col
) a;
EXECUTE 'COPY (SELECT ' || column_list || ' FROM ' || quote_ident(relation) || ')
TO ' || quote_literal('/home/r90t/work/study/etl/postgres_etl/export/' || relation) || ' WITH CSV;';
END
$$
LANGUAGE plpgsql;
This would give you protection against sql injection, and if you need the date added on to the end, you can do that with something inside the quote_literal call.
Related
Hello guys ive been trying to build an software using postgresql and python.
Basically i want increment and/or dynamically expand the json
example: at first the field will be empty then:
#insert (toyota,honda,nissan)
{"toyota":1,
"honda":1,
"nissan":1}
#insert (toyota)
{"toyota":2,
"honda":1,
"nissan":1}
#insert (honda,mitsubitshi)
{"toyota":2,
"honda":2,
"nissan":1,
"mitsubitshi":1}
Yes i know it can be done by first retrieving json doing it via python but i dont it that way:
I dont have much experience with postgresql procedure or trigger feature.
Any Help will be apreciated: :-)
Normalized tables would be more performant, however json solution may be quite comfortable using this function:
create or replace function add_cars(cars jsonb, variadic car text[])
returns jsonb language plpgsql as $$
declare
new_car text;
begin
foreach new_car in array car loop
cars = cars || jsonb_build_object(new_car, coalesce(cars->>new_car, '0')::int+ 1);
end loop;
return cars;
end $$;
Find the full example in DbFiddle.
Please check function below. Hopefully it meets your requirement!
CREATE FUNCTION sp_test(json)
RETURNS VOID AS
$BODY$
DECLARE
var_sql varchar;
BEGIN
IF (EXISTS (
SELECT json_object_keys($1)
EXCEPT
SELECT column_name FROM information_schema.columns WHERE table_schema = 'your schema' AND table_name = 'test_table'
)) THEn
RAISE EXCEPTION 'There is column(s) does not exists on table'; -- Checking structure.
END IF;
var_sql := 'Update test_table t SET ' || (SELECT string_agg(CONCAT(t.key, ' = (t.', t.key, ' + ', t.value,')'),', ') FROM json_each($1) t);
EXECUTE (var_sql);
END;
$BODY$
LANGUAGE plpgsql;
I have a Users table and I'm getting user details (usual way) with id.
This is how I'm getting user details;
select id,kullaniciadi,FotoGet(K.Foto,K.Gender) from kullanicilar K where id=1;
FotoGet function always returning string value like 'Photos/ssss.jpg'
So far I have not trouble to use this way but I'm wondering how can I do this with a single function?
Like
set #uid=1;
Select UserGet(#uid);
I will put the id inside parentheses and the function will run my select query and return all user detail columns. id, kullaniciadi, FotoGet(K.id,K.Foto,K.Gender)
Can I use this way to get details?
A MySQL stored function cannot return multiple columns, it can only return a single scalar value.
But you could design a stored procedure that returns a result set:
CREATE PROCEDURE UserGet(uid INT)
SELECT id, kullaniciadi, FotoGet(K.Foto,K.Gender) FROM kullanicilar K WHERE id=uid;
Then call it like so:
CALL UserGet(1234);
It returns a result set just like a SELECT query would.
DELIMITER //
CREATE PROCEDURE UserGet(IN uid INT, IN column_name VARCHAR(64))
BEGIN
SET #sql = CONCAT('SELECT id, ', column_name, ' FROM kullanicilar K WHERE id=?');
PREPARE stmt FROM #sql;
SET #uid = uid;
EXECUTE stmt USING #uid;
END//
DELIMITER ;
Call it like so:
CALL UserGet(1234, 'kullaniciadi');
Remember that it's your responsibility to write code to pass a valid column name as the procedure argument. If you allow untrustworthy content to be passed, then it might be an SQL injection vulnerability.
Re your additional comment:
This should work fine:
CALL UserGet(1234, 'FotoGet(Foto,Gender)');
Can we select inside a PostgreSQL function without using EXECUTE?
I'm trying to use quote_ident() to create dynamic SQL but it doesn't work.
CREATE OR REPLACE FUNCTION select_server(p_id text)
RETURNS integer AS $$
DECLARE
serialnum_value INTEGER;
STATEMENT TEXT;
BEGIN
STATEMENT := 'tbl' || substr($1, 1, 4);
SELECT serialnum INTO serialnum_value FROM quote_ident(STATEMENT ) WHERE id = $1;
RETURN serialnum;
END;
Does anybody have idea how to select from dynamic table in a PostgreSQL function without using EXECUTE?
You can't avoid execute here, per igor's comment. The best alternative is to create a function that creates a function, which you can then call as needed. Example:
create function gen_select_server(p_id text)
returns boolean
as $gen$
begin
execute $exec$
create function $exec$ || quote_ident('select_server_' || p_id) || $exec$
returns integer
as $body$
begin
return serialnum
from $exec$ || quote_ident('tbl_' || p_id) || $exec$
where id = $exec$ || quote_literal('id_' || p_id) || $exec$;
end;
$body$ language plpgsql;
$exec$;
return true;
end;
$gen$ language plpgsql;
-- usage:
select gen_select_server('1234'); -- call this only once; uses execute
select * from select_server_1234(); -- no execute here
As the above highlights, it can get messy with string delimiters. Also note that the above isn't your original function -- it's primarily to illustrate how to quote things properly within the big $exec$ "string" block.
I'd recommend to stick to using execute, that being said. Or using an ORM, for that matter. Maintaining these micro-optimizations is potentially a pain worse than the superficial gain in performance.
You cannot execute dynamic SQL without EXECUTE in PL/pgSQL. That's what makes it dynamic in the first place.
But you can streamline your function quite a bit:
CREATE OR REPLACE FUNCTION select_server(p_id text, OUT serial_value integer) AS
$func$
BEGIN
EXECUTE 'SELECT serialnum FROM tbl' || left($1::text, 4)) || ' WHERE id = $1'
USING $1
INTO serial_value;
END
$func$ LANGUAGE plpgsql STRICT;
Normally you would have to sanitize any identifier.
But you do not need quote_ident() in this special case, since the only dynamic component are 4 digits from an integer. Neither SQL injection nor illegal identifiers are possible this way.
Pass the value of p_id as value using the USING clause.
Reduce the number of assignments. Those are comparatively expensive in plpgsql. You only need a single SQL statement to do everything.
The OUT parameter helps to shorten the syntax.
I also made the function STRICT (RETURNS NULL ON NULL INPUT), since it would not make sense with NULL as input.
left() is slightly faster than substr().
We have a monitor on our databases to check for ids approaching max-int or max-bigint. We just moved from MySQL, and I'm struggling to get a similar check working on PostgreSQL. I'm hoping someone can help.
Here's the query in MySQL
SELECT table_name, auto_increment FROM information_schema.tables WHERE table_schema = DATABASE();
I'm trying to get the same results from PostgreSQL. We found a way to do this with a bunch of calls to the database, checking each table individually.
I'd like to make just 1 call to the database. Here's what I have so far:
CREATE OR REPLACE FUNCTION getAllSeqId() RETURNS SETOF record AS
$body$
DECLARE
sequence_name varchar(255);
BEGIN
FOR sequence_name in SELECT relname FROM pg_class WHERE (relkind = 'S')
LOOP
RETURN QUERY EXECUTE 'SELECT last_value FROM ' || sequence_name;
END LOOP;
RETURN;
END
$body$
LANGUAGE 'plpgsql';
SELECT last_value from getAllSeqId() as(last_value bigint);
However, I need to somehow add the sequence_name to each record so that I get output in records of [table_name, last_value] or [sequence_name, last_value].
So I'd like to call my function something like this:
SELECT sequence_name, last_value from getAllSeqId() as(sequence_name varchar(255), last_value bigint);
How can I do this?
EDIT: In ruby, this creates the output we're looking for. As you can see, we're doing 1 call to get all the indexes, then 1 call per index to get the last value. Gotta be a better way.
def perform
find_auto_inc_tables.each do |auto_inc_table|
check_limit(auto_inc_table, find_curr_auto_inc_id(auto_inc_table))
end
end
def find_curr_auto_inc_id(table_name)
ActiveRecord::Base.connection.execute("SELECT last_value FROM #{table_name}").first["last_value"].to_i
end
def find_auto_inc_tables
ActiveRecord::Base.connection.execute(
"SELECT c.relname " +
"FROM pg_class c " +
"WHERE c.relkind = 'S'").map { |i| i["relname"] }
end
Your function seems quite close already. You'd want to modify it a bit to:
include the sequences names as literals
returns a TABLE(...) with typed columns instead of SET OF RECORD because it's easier for the caller
Here's a revised version:
CREATE OR REPLACE FUNCTION getAllSeqId() RETURNS TABLE(seqname text,val bigint) AS
$body$
DECLARE
sequence_name varchar(255);
BEGIN
FOR sequence_name in SELECT relname FROM pg_class WHERE (relkind = 'S')
LOOP
RETURN QUERY EXECUTE 'SELECT ' || quote_literal(sequence_name) || '::text,last_value FROM ' || quote_ident(sequence_name);
END LOOP;
RETURN;
END
$body$
LANGUAGE 'plpgsql';
Note that currval() is not an option since it errors out when the sequence has not been set in the same session (by calling nextval(), not sure if there's any other way).
Would something as simple as this work?
SELECT currval(sequence_name) from information_schema.sequences;
If you have sequences that aren't keys, I guess you could use PG's sequence name generation pattern to try to restrict it.
SELECT currval(sequence_name) from information_schema.sequences
WHERE sequence_name LIKE '%_seq';
If that is still too many false positives, you can get table names from the information_schema (or the pg_* schemata that I don't know very well) and refine the LIKE parameter.
I am trying to have a conditional change in a parameter for update statement.
I am getting the following error when I try the following function
/home/y/bin/mysql -u root < testpri.sql > out
ERROR 1415 (0A000) at line 4: Not allowed to return a result set from a function
Contents of testpri.sql are as follows:
use `zestdb`;
DROP FUNCTION IF EXISTS UPDATEPASSWD;
DELIMITER //
CREATE FUNCTION UPDATEPASSWD(n INT) RETURNS varchar(255) DETERMINISTIC
BEGIN
DECLARE mypasswd varchar(255);
IF (n = 1) THEN
SET mypasswd = '12ccc1e5c3c9203af7752f937fca4ea6263f07a5';
SELECT 'n is 1' AS ' ';
ELSE
SET mypasswd = '1a7bc371cc108075cf8115918547c3019bf97e5d';
SELECT 'n is 0' AS ' ';
END IF;>
SELECT CONCAT('mypasswd is ', mypasswd) AS ' ';
RETURN mypasswd;
END //
DELIMITER ;
CALL UPDATEPASSWD(0);
What am I missing?
I think it's actually your debugging SELECT calls.
From the docs:
Statements that return a result set can be used within a stored procedure but not within a stored function. This prohibition includes SELECT statements that do not have an INTO var_list clause...
I arrived in search of answers to the same question, and found another way to work around the issue, so that I can use the SELECT statement that is the heart and soul of the MySQL function that elicited the warning.
Consider the following snippet.
SET intNMatches = ( SELECT COUNT(*) ...
SET coerces the SELECT statement to return its one and only column, a row count, into intNMatches, a local variable cast to BIGINT. Since it contains trade secrets, I can't show the rest of the query. Suffice it to say that the query installs without causing the MySQL engine to issue a warning.