After reading Postgres manual and many posts here, I wrote this function tacking in mind all I found regarding security. It works great and does everything I looked for.
takes a json, each key has an array [visible, filter, arg1, optional arg2]
SELECT public.goods__list_json2('{"name": [true, 7, "Ad%"], "category": [true], "stock": [false, 4, 0]}', 20, 0);
returns a json array with requested data.
[{"name": "Adventures of TRON", "category": "Atari 2600"}, {"name": "Adventure", "category": "Atari 2600"}]
My question is, how could I really be sure that when I create the query using user input arguments, passing them as %L with format is injection safe?
By my db design, all is done through functions, running most of them as security definer only allowing certain roles to execute them.
Being secure, my intention is to convert old functions to this dynamic logic and save myself to write a lot of lines of code creating new or specific queries.
I would really appreciate a experienced Postgres developer could give me an advice on this.
I'm using Postgres 13.
CREATE FUNCTION public.goods__list_json (IN option__j jsonb, IN limit__i integer, IN offset__i integer)
RETURNS jsonb
LANGUAGE plpgsql
VOLATILE
STRICT
SECURITY DEFINER
COST 1
AS $$
DECLARE
table__v varchar := 'public.goods_full';
column__v varchar[] := ARRAY['id', 'id__category', 'category', 'name', 'barcode', 'price', 'stock', 'sale', 'purchase'];
filter__v varchar[] := ARRAY['<', '>', '<=', '>=', '=', '<>', 'LIKE', 'NOT LIKE', 'ILIKE', 'NOT ILIKE', 'BETWEEN', 'NOT BETWEEN'];
select__v varchar[];
where__v varchar[];
sql__v varchar;
key__v varchar;
format__v varchar;
temp__v varchar;
temp__i integer;
betw__v varchar;
result__j jsonb;
BEGIN
FOR key__v IN SELECT jsonb_object_keys(option__j) LOOP
IF key__v = ANY(column__v) THEN
IF (option__j->key__v->0)::bool THEN
select__v := array_append(select__v, key__v);
END IF;
temp__i := (option__j->key__v->1)::int;
IF temp__i > 0 AND temp__i <= array_length(filter__v, 1) THEN
temp__v := (option__j->key__v->>2)::varchar;
IF temp__i >= 11 THEN
betw__v := (option__j->key__v->>3)::varchar;
format__v := format('%I %s %L AND %L', key__v, filter__v[temp__i], temp__v, betw__v);
ELSE
format__v := format('%I %s %L', key__v, filter__v[temp__i], temp__v);
END IF;
where__v := array_append(where__v, format__v);
END IF;
END IF;
END LOOP;
sql__v := 'SELECT jsonb_agg(t) FROM (SELECT '
|| array_to_string(select__v, ', ')
|| format(' FROM %s WHERE ', table__v)
|| array_to_string(where__v, ' AND ')
|| format(' OFFSET %L LIMIT %L', offset__i, limit__i)
|| ') t';
RAISE NOTICE 'SQL: %', sql__v;
EXECUTE sql__v INTO result__j;
RETURN result__j;
END;
$$;
A word of warning: this style with dynamic SQL in SECURITY DEFINER functions can be elegant and convenient. But don't overuse it. Do not nest multiple levels of functions this way:
The style is much more error prone than plain SQL.
The context switch with SECURITY DEFINER has a price tag.
Dynamic SQL with EXECUTE cannot save and reuse query plans.
No "function inlining".
And I'd rather not use it for big queries on big tables at all. The added sophistication can be a performance barrier. Like: parallelism is disabled for query plans this way.
That said, your function looks good, I see no way for SQL injection. format() is proven good to concatenate and quote values and identifiers for dynamic SQL. On the contrary, you might remove some redundancy to make it cheaper.
Function parameters offset__i and limit__i are integer. SQL injection is impossible through integer numbers, there is really no need to quote them (even though SQL allows quoted string constants for LIMIT and OFFSET). So just:
format(' OFFSET %s LIMIT %s', offset__i, limit__i)
Also, after verifying that each key__v is among your legal column names - and while those are all legal, unquoted column names - there is no need to run it through %I. Can just be %s
I'd rather use text instead of varchar. Not a big deal, but text is the "preferred" string type.
Related:
Format specifier for integer variables in format() for EXECUTE?
Function to return dynamic set of columns for given table
COST 1 seems too low. The manual:
COST execution_cost
A positive number giving the estimated execution cost for the
function, in units of cpu_operator_cost. If the function
returns a set, this is the cost per returned row. If the cost is not
specified, 1 unit is assumed for C-language and internal functions,
and 100 units for functions in all other languages. Larger values
cause the planner to try to avoid evaluating the function more often
than necessary.
Unless you know better, leave COST at its default 100.
Single set-based operation instead of all the looping
The whole looping can be replaced with a single SELECT statement. Should be noticeably faster. Assignments are comparatively expensive in PL/pgSQL. Like this:
CREATE OR REPLACE FUNCTION goods__list_json (_options json, _limit int = NULL, _offset int = NULL, OUT _result jsonb)
RETURNS jsonb
LANGUAGE plpgsql SECURITY DEFINER AS
$func$
DECLARE
_tbl CONSTANT text := 'public.goods_full';
_cols CONSTANT text[] := '{id, id__category, category, name, barcode, price, stock, sale, purchase}';
_oper CONSTANT text[] := '{<, >, <=, >=, =, <>, LIKE, "NOT LIKE", ILIKE, "NOT ILIKE", BETWEEN, "NOT BETWEEN"}';
_sql text;
BEGIN
SELECT concat('SELECT jsonb_agg(t) FROM ('
, 'SELECT ' || string_agg(t.col, ', ' ORDER BY ord) FILTER (WHERE t.arr->>0 = 'true')
-- ORDER BY to preserve order of objects in input
, ' FROM ' || _tbl
, ' WHERE ' || string_agg (
CASE WHEN (t.arr->>1)::int BETWEEN 1 AND 10 THEN
format('%s %s %L' , t.col, _oper[(arr->>1)::int], t.arr->>2)
WHEN (t.arr->>1)::int BETWEEN 11 AND 12 THEN
format('%s %s %L AND %L', t.col, _oper[(arr->>1)::int], t.arr->>2, t.arr->>3)
-- ELSE NULL -- = default - or raise exception for illegal operator index?
END
, ' AND ' ORDER BY ord) -- ORDER BY only cosmetic
, ' OFFSET ' || _offset -- SQLi-safe, no quotes required
, ' LIMIT ' || _limit -- SQLi-safe, no quotes required
, ') t'
)
FROM json_each(_options) WITH ORDINALITY t(col, arr, ord)
WHERE t.col = ANY(_cols) -- only allowed column names - or raise exception for illegal column?
INTO _sql;
IF _sql IS NULL THEN
RAISE EXCEPTION 'Invalid input resulted in empty SQL string! Input: %', _options;
END IF;
RAISE NOTICE 'SQL: %', _sql;
EXECUTE _sql INTO _result;
END
$func$;
db<>fiddle here
Shorter, faster and still safe against SQLi.
Quotes are only added where necessary for syntax or to defend against SQL injection. Burns down to filter values only. Column names and operators are verified against the hard-wired list of allowed options.
Input is json instead of jsonb. Order of objects is preserved in json, so you can determine the sequence of columns in the SELECT list (which is meaningful) and WHERE conditions (which is purely cosmetic). The function observes both now.
Output _result is still jsonb. Using an OUT parameter instead of the variable. That's totally optional, just for convenience. (No explicit RETURN statement required.)
Note the strategic use of concat() to silently ignore NULL and the concatenation operator || so that NULL makes the concatenated string NULL. This way, FROM, WHERE, LIMIT, and OFFSET are only inserted where needed. A SELECT statement works without either of those. An empty SELECT list (also legal, but I suppose unwanted) results in a syntax error. All intended.
Using format() only for WHERE filters, for convenience and to quote values. See:
String concatenation using operator "||" or format() function
The function isn't STRICT anymore. _limit and _offset have default value NULL, so only the first parameter _options is required. _limit and _offset can be NULL or omitted, then each is stripped from the statement.
Using text instead of varchar.
Made constant variables actually CONSTANT (mostly for documentation).
Other than that the function does what your original does.
I tried to put all that I learned here and I came up with this below and new questions =D.
Is there any advantage declaring _oper this way '{LIKE, "NOT LIKE"}' instead of ARRAY['LIKE', 'NOT LIKE']?
Casting as int _limit and _offset, I'm assuming no SQLi, right?
Is it an elegant way for 'IN' and 'NOT IN' CASE? I wonder why string_agg() is allowed nested in concat() but not there where I needed to use a sub query.
This is a naive private function.
Edit: Removed "SECURITY DEFINER" as identified dangerous.
CREATE FUNCTION public.crud__select (IN _tbl text, IN _cols text[], IN _opts json, OUT _data jsonb)
LANGUAGE plpgsql STRICT AS
$$
DECLARE
_oper CONSTANT text[] := '{<, >, <=, >=, =, <>, LIKE, "NOT LIKE", ILIKE, "NOT ILIKE", BETWEEN, "NOT BETWEEN", IN, "NOT IN"}';
BEGIN
EXECUTE (
SELECT concat('SELECT jsonb_agg(t) FROM ('
, 'SELECT ' || string_agg(e.col, ', ' ORDER BY ord) FILTER (WHERE e.arr->>0 = 'true')
, ' FROM ', _tbl
, ' WHERE ' || string_agg(
CASE
WHEN (e.arr->>1)::int BETWEEN 1 AND 10 THEN
format('%s %s %L', e.col, _oper[(e.arr->>1)::int], e.arr->>2)
WHEN (e.arr->>1)::int BETWEEN 11 AND 12 THEN
format('%s %s %L AND %L', e.col, _oper[(e.arr->>1)::int], e.arr->>2, e.arr->>3)
WHEN (e.arr->>1)::int BETWEEN 13 AND 14 THEN
format('%s %s (%s)', e.col, _oper[(e.arr->>1)::int], (
SELECT string_agg(format('%L', ee), ',') FROM json_array_elements_text(e.arr->2) ee)
)
END, ' AND ')
, ' OFFSET ' || (_opts->>'_offset')::int
, ' LIMIT ' || (_opts->>'_limit')::int
, ') t'
)
FROM json_each(_opts) WITH ORDINALITY e(col, arr, ord)
WHERE e.col = ANY(_cols)
) INTO _data;
END;
$$;
Then for table or view, I create wrapper function executable for some roles.
CREATE FUNCTION public.goods__select (IN _opts json, OUT _data jsonb)
LANGUAGE sql STRICT SECURITY DEFINER AS
$$
SELECT public.crud__select(
'public.goods_full',
ARRAY['id', 'id__category', 'category', 'name', 'barcode', 'price', 'stock', 'sale', 'purchase'],
_opts
);
$$;
SELECT public.goods__select('{"_limit": 10, "name": [true, 9, "a%"], "id__category": [true, 13, [1, 2]], "category": [true]}'::json);
[{"name": "Atlantis II", "category": "Atari 2600", "id__category": 1}, .. , {"name": "Amidar", "category": "Atari 2600", "id__category": 1}]
I have a very specific question.
I have to build a SQL statement that builds a table where some columns are merged together. These columns shall be formatted with delimiters like '\n' or ' ' or ' - '. These delimiters shall be added only if the column before is not empty or null. This should prevent empty lines or unneeded delimiters.
Here is how I started:
SELECT
any_table.table_id,
CONCAT(any_table.text1, '\n', any_table.text2) AS text1_2,
FROM
any_table
WHERE any_table.use = 'true'
This code concats text1 and text2 as a new column text1_2 and uses a line feed as delimiter. The missing part is that line feed shall just be added if any_table.text1 is not null or empty.
Is there an elegant way in doing this with SQL?
thx
Some databases support a very handy function called concat_ws() which does exactly what you want:
CONCAT_WS('\n', NULLIF(any_table.text1, ''), NULLIF(any_table.text2, '')) AS text1_2,
In standard SQL, you can do:
TRIM(LEADING '\n' FROM CONCAT( '\n', || NULLIF(any_table.text1, ''),
'\n' || NULLIF(any_table.text2, '')
)
)
It is possible that your database supports neither of these constructs.
if you'r under SQL SERVER you can use,
SELECT id, CONCAT(colonne1 + ' - ', colonne2) FROM "table"
if you'r under Oracle : you shoul use || for concaténation like
SELECT id, CONCAT(colonne1 || ' - ', colonne2) FROM "table"
I have a table 'car_purchases' with a 'description' column. The column is a string that includes first name initial followed by full stop, space and last name.
An example of the Description column is
'Car purchased by J. Blow'
I am using 'substring_index' function to extract the letter preceding the '.' in the column string. Like so:
SELECT
Description,
SUBSTRING_INDEX(Description, '.', 1) as TrimInitial,
SUBSTRING_INDEX(
SUBSTRING_INDEX(Description, '.', 1),' ', -1) as trimmed,
length(SUBSTRING_INDEX(
SUBSTRING_INDEX(Description, '.', 1),' ', -1)) as length
from car_purchases;
I will call this query 1.
picture of the result set (Result 1) is as follows
As you can see the problem is that the 'trimmed' column in the select statement starts counting the 2nd delimiter ' ' instead of the first from the right and produces the result 'by J' instead of just 'J'. Further the length column indicates that the string length is 5 instead of 4 so WTF?
However when I perform the following select statement;
select SUBSTRING_INDEX(
SUBSTRING_INDEX('Car purchased by J. Blow', '.', 1),' ', -1); -- query 2
Result = 'J' as 'Result 2'.
As you can see from result 1 the string in column 'Description' is exactly (as far as I can tell) the same as the string from 'Result 2'. But when the substring_index is performed on the column (instead of just the string itself) the result ignores the first delimiter and selects a string from the 2nd delimiter from the right of the string.
I've racked my brains over this and have tried 'by ' and ' by' as delimiters but both options do not produce the desired result of a single character. I do not want to add further complexity to query 1 by using a trim function. I've also tried the cast function on result column 'trimmed' but still no success. I do not want to concat it either.
There is an anomaly in the 'length' column of query 1 where if I change the length function to char_length function like so:
select length(SUBSTRING_INDEX(
SUBSTRING_INDEX(Description, '.', 1),' ', -1)) as length -- result = 5
select char_length(SUBSTRING_INDEX(
SUBSTRING_INDEX(Description, '.', 1),' ', -1)) as length -- result = 4
Can anyone please explain to me why the above select statement would produce 2 different results? I think this is the reason why I am not getting my desired result.
But just to be clear my desired outcome is to get 'J' not 'by J'.
I guess I could try reverse but I dont think this is an acceptable compromise. Also I am not familiar with collation and charset principles except that I just use the defaults.
Cheers Players!!!!
CHAR_LENGTH returns length in characters, so a string with 4 2-byte characters would return 4. LENGTH however returns length in bytes, so a string with 4 2-byte characters would return 8. The discrepancy in your results (including SUBSTRING_INDEX) says that the "space" between by and J is not actually a single-byte space (ASCII 0x20) but a 2-byte character that looks like a space. To workaround this, you could try replacing all unicode characters with spaces using CONVERT and REPLACE. In this example, I have an en-space unicode character in the string between by and J. The CONVERT changes that to a ?, and the REPLACE then converts that to a space:
SELECT SUBSTRING_INDEX( SUBSTRING_INDEX("Car purchased by J. Blow", '.', 1),' ', -1)
Output:
by J
With CONVERT and REPLACE:
SELECT SUBSTRING_INDEX( SUBSTRING_INDEX(REPLACE(CONVERT("Car purchased by J. Blow" USING ASCII), '?', ' '), '.', 1),' ', -1)
Output
J
For your query, you would replace the string with your column name i.e.
SELECT SUBSTRING_INDEX( SUBSTRING_INDEX(REPLACE(CONVERT(description USING ASCII), '?', ' '), '.', 1),' ', -1)
Demo on DBFiddle
There are a lot of rows with multiple spaces in column title and I want to replace them with a single space.
update abc set title = REPLACE(title, " ", " ");
Nothing is replaced.
I'm using phpMyAdmin.
I noticed (clicking on the button Simulate query that my query is transformed into:
update abc set title = REPLACE(title, " ", " ");
so replace single space with single space.
Any help?
While checking this page it came to my attention that to replacing all the double spaces from the database you might have triple space or more on the single record.
The thing that the some solution didn't take in consideration.So you need to make sure that your statement replace them all. Doing one time or two time replacement of double space with single space might not cover all the corrupted data.
For example having a record value as 'A B C'; what you can do is:
first replace all the single space with open/closed characters
like <> , or [] or {}...
Then replace the back to back reversed order characters (closed/open)
with empty value, so all >< or ][ or }{ will be removed.
Final step is to restore the single spaces by replacing the remaining open/close
characters with single space, for example <> will be changed back to ' '
I always use something like following to fix my data:
UPDATE Table1 SET Column1 = REPLACE(REPLACE(REPLACE(Column1, ' ', '<>'), '><', ''),'<>',' ');
Number of consecutive space characters can either be odd or even. You can replace two space characters with one space character, and do a similar replace again on the modified string to cover all the odd/even cases.
UPDATE abc SET title = REPLACE(REPLACE(title, ' ', ' '), ' ', ' ');
Explanation:
2 spaces: First replace will convert to 1 space. Second replace will not modify further.
3 spaces: First replace will convert (2+1) spaces to (1+1). Second will convert (1+1 = 2) spaces to 1 space.
4 spaces: First replace will convert (2+2) spaces to (1+1). Second will convert (1+1 = 2) spaces to 1 space.
and so on...
DEMO:
mysql> select
-> dt.test_str,
-> REPLACE(REPLACE(dt.test_str, ' ', ' '), ' ', ' ') AS modified
-> FROM
-> (SELECT 'thi s is a weird string' AS test_str) AS dt ;
+--------------------------------+--------------------------+
| test_str | modified |
+--------------------------------+--------------------------+
| thi s is a weird string | thi s is a weird string |
+--------------------------------+--------------------------+
1 row in set (0.00 sec)
You can try the following SELECT example, with REGEXP_REPLACE is used. For example:
SELECT 'ab asd asd a qeqw q qwe qweqw qw' AS `text 1`, REGEXP_REPLACE('ab asd asd a qeqw q qwe qweqw qw', ' \+', ' ') AS `text 2`;
REGEXP_REPLACE documentation
You can use REGEXP_REPLACE in an UPDATE statement:
UPDATE abc SET title = REGEXP_REPLACE(title, ' \+', ' ');
Here is my approach
SELECT ARRAY_TO_STRING(ARRAY_AGG(VALUE::varchar),' ') FROM TABLE(flatten(split(REGEXP_REPLACE('','\n'),' '))) WHERE VALUE <> '' ORDER BY INDEX;
might be a long route but does the job.
Following is a Function created in PostgreSQL 9.2 ,
CREATE OR REPLACE FUNCTION getRows()
RETURNS TABLE (
"Name" character varying
,"Address" text
,"Phone" text
,"Email" character varying
,"Tin" character varying
,"DL.No1, DL.No2" text
,"Area" character varying
) AS
$func$
BEGIN
select acname as "Name", coalesce(nullif(concat_ws(', ', nullif(add1, ''),
nullif(add2, '')), ''), 'NIL') "Address", coalesce(nullif(concat_ws(', ',
nullif(phoneoff, ''), nullif(mobile, '')), ''), 'NIL') "Phone",case when email
<>'' then email else 'NIL' end as "Email",case when email <>'' then tin else
'NIL' end as "Tin",case when dlno1 ||', '||dlno2=', ' then 'NIL' else dlno1 ||',
'||dlno2 end as "DL.No1, DL.No2",areaname as "Area" from gtab12 left join
gtab47 using(areaid) where acgrcode = '204' order by areaname,"Name";
END
$func$ LANGUAGE plpgsql;
When select getRows() an error is occurring
ERROR: query has no destination for result data
HINT: If you want to discard the results of a SELECT, use PERFORM instead.
CONTEXT: PL/pgSQL function getRows() line 4 at SQL statement
given below is actually what the select query inside the function getRow returns.
You cannot use a SELECT statement in a plpgsql function without the INTO clause (you must use PERFORM instead).
You also forget to RETURN with something.
But, if your function is that simple, you could use a plain sql function (or even a view) instead.
To return a set of something, you need to use RETURN NEXT (to return a row at a time) or RETURN QUERY (to return a number of rows resulting from a query).
See the manual section relating to returning values.
Your example function would need to have:
RETURN QUERY SELECT....;
RETURN;
Note the need for the second RETURN - this is needed because there may be multiple RETURN QUERY statements before the function exits.