PostgreSQL: preventing sql injection on multiinsertion - json

I'm looking for the fastest way to parse, validate and insert data in table(Postgresql 9.3).
The data is an json-array which contains 1..N items.
[{"name":"a","value":"1"},{"name":"b","value":"2"}]
The table looks like:
CREATE TABLE logs
(
id serial NOT NULL,
name text ,
value text,
CONSTRAINT "log_Pkey" PRIMARY KEY (id)
);
For that i have stored procedure:
CREATE OR REPLACE FUNCTION insert_logs(v json)
RETURNS integer AS
$BODY$
DECLARE
sql text;
i json;
logs_part_id int;
BEGIN
SELECT INTO logs_part_id id from another_table_with_that_id where some_condition.
sql = '';
FOR i IN SELECT * FROM json_array_elements(v)
LOOP
sql = sql||'insert into logs_'||logs_part_id ||'
(name, value)
values( ' ||quote_literal(i->>'name')||' , ' ||quote_literal(i->>'value')||' );';
END LOOP;
raise notice '%',sql;
EXECUTE sql;
return 1;
END
$BODY$
LANGUAGE plpgsql VOLATILE
COST 100;
(function returns integer as a response status)
Function call:
select * from insert_logs('[{"name":"a","value":"1"},{"name":"b","value":"2"}]'::json);
Actually the "insert.." statement is quite bigger - 15 columns to insert and aparently some of them should be checked in order to prevent sql injection.
Question:
Is there any way to rewrite this stored procedure in order to improve performance?
Should I use prepared statements?
EDIT.
The reason i build sql string because the table name is unknown because of the tables partitioning. The table name format is: logs_id where id - int which is obtained just before insert.

If you need to speed up your query, json_populate_recordset() does exactly what you need:
insert into logs
select * from json_populate_recordset(null::logs, '[...]')
As, for SQL-injection: you should always use prepared statements, or at least execute your sql with parameters sent separately (f.ex. with PQexecParams() if you use libpq directly).

Why are you building an SQL multi-statement string then EXECUTEing it at all?
Just:
insert into logs (name, value)
values( i->>name , i->>value );
There's no need for explicit quoting because i->>name is a text value that's inserted as a bound parameter into the insert by PL/PgSQL. It's never parsed as SQL.
If you must build the statement dynamically (e.g. varying table name, per comment) use EXECUTE ... USING with format:
EXECUTE format('insert into %I (name, value) values( $1, $2 );', 'logs_'||log_partition_id)
USING i->>name , i->>value;
in your case

Related

Should it be possible to execute an SQL function in a check constraint within DB2 z/OS

Simple version of the DDL:
create function rm00dv1.no_concurrent_schedules()
returns integer
LANGUAGE SQL
READS SQL DATA
NO EXTERNAL ACTION
NOT DETERMINISTIC
BEGIN
declare num_overlaps integer;
select count(*)
into num_overlaps
from
rm00dv1.schedules a
where
a.id != 0
and
exists (
select 1
from rm00dv1.schedules b
where
b.id = 0 -- matches the key of a given record
and rm00dv1.isConcurrent(b.schdl_eff_dt, b.schdl_trm_dt, a.schdl_eff_dt, a.schdl_trm_dt) != 0
);
return num_overlaps;
end;
Table:
create table rm00dv1.schedules (
id int not null,
schdl_eff_dt date not null,
schdl_trm_dt date not null,
info_chg_ts timestamp(6) not null with default
)
in RM00DV1.TSRMDV01 ;
alter table rm00dv1.schedules add constraint no_schedule_overlap
check ((schdl_trm_dt < '01/01/2015')
or
rm00dv1.no_concurrent_schedules() <= 0);
I am getting an SQL00551N - no execution privilege and that is odd because I can execute the function in a select statement.
Any idea to solve this problem?
Thanks.
Looks like you can't. I'm looking at the DB2 10 for z/OS reference for ALTER TABLE reference and it says the following under CHECK (check-condition): "A check-condition is a search condition, with the following restrictions: ... must not contain... Built-in or user-defined functions...".
Since your function looks like it won't convert to a check condition, defining triggers on the table might be the next best option.
I learned that AFTER triggers do not get a -746 like BEFORE triggers do. I had really wanted to use a CONSTRAINT because that best captures the intent for people who come after me, with a BEFORE trigger to terminate the active schedules. But, it looks like a sequence of triggers is going to be the way to go. It is a bit clunky because the triggers all have to be created separately and you have to look at them together to get the intent, and because correct behavior is dependent on their creation order. Yes, it is documented that they will be executed in the order of their creation.
Happy path termination of rows without a specified termination date:
CREATE TRIGGER terminate_no_trm
after
INSERT ON schedules
referencing new as new
FOR EACH ROW
MODE DB2SQL
BEGIN ATOMIC
update schedules
set
schdl_trm_dt = max(schdl_eff_dt, new.schdl_eff_dt - 1 days) -- prob not necessary, but don't set the trm before the eff
, info_chg_ts = new.info_chg_ts
where
new.keyCombo = keyCombo
and
schdl_trm_dt = '9999-12-31'
and schdl_eff_dt < new.schdl_eff_dt;
end
Prevent insert of rows if that insert causes an overlap:
CREATE TRIGGER no_overlapping_schedules_i
after
insert ON schedules
referencing new as n
FOR EACH ROW
MODE DB2SQL
when (num_concurrent_schedules(n.keyCombo) > 0)
begin atomic
SIGNAL SQLSTATE '75001' (
'Concurrent schedules detected: '
concat ' ' concat cast(n.keyCombo as varchar(32))
concat ': ' concat cast(n.schdl_eff_dt as varchar(32))
concat ' to ' concat cast(n.schdl_trm_dt as varchar(32))
);
end
and prevent UPDATE if that would result in an overlap
CREATE TRIGGER no_overlapping_schedules_u
after
update ON schedules
referencing new as n
FOR EACH ROW
MODE DB2SQL
when (num_concurrent_schedules(n.keyCombo) > 0)
begin atomic
SIGNAL SQLSTATE '75001' (
'Concurrent schedules detected: '
concat ' ' concat cast(n.keyCombo as varchar(32))
concat ': ' concat cast(n.schdl_eff_dt as varchar(32))
concat ' to ' concat cast(n.schdl_trm_dt as varchar(32))
);
end
Thanks for the ideas.

Postgres JSON Array data processing

Have a table "json_test" and inserted the following record:
create table json_test ( v json);
insert into json_test values ('{"facilityId": ["20","30","40","50","51"]}')
SELECT trim(json_array_elements_text(v->'facilityId') ) from json_test
The above select lists the facility ID as individual rows.
I need the same rows in a Postgres function to insert the record into another table. I wrote the following code to return i. The output of the v_status when checked is (20,,,,,,,,,,,,). I need to get just 20, but I am unable to get that.
for i in SELECT json_array_elements_text(v->'facilityId') from json_test
loop
v_status:= i;
end loop;
You have not specified entire function definition in your question.
Assuming you have DDL:
CREATE TABLE json_test(
id SERIAL PRIMARY KEY,
v JSON
);
INSERT INTO json_test(v) VALUES
('{"facilityId": ["20","30","40","50","51"]}'::JSON);
You can check full PL/pgSQL guide as a reference, but your function may be defined as the following:
CREATE OR REPLACE FUNCTION get_facility_ids(rid INTEGER)
RETURNS SETOF INTEGER AS $$
DECLARE
t TEXT;
BEGIN
FOR t IN SELECT json_array_elements_text(v->'facilityId')
FROM json_test WHERE id = rid
LOOP
RETURN NEXT t;
END LOOP;
END;
$$ LANGUAGE plpgsql;
SELECT get_facility_ids(1) AS facultyId;
Just for your information, you can INSERT records from SELECT statements. Check the documentation.

MySql: Can I Create array using sql

I am Using Mysql DB. my question is : can i create array using sql?
if Yes then how and how to populate this array with output of following query -- "Select column_name1 From tableName".
Help me, Thanks in Advance
As I mentioned in my comment, MySQL does not support arrays by itself. That kind of structures are supported by other programming languages (like PHP, Java, Python, etcetera) and you can write a program capable of connecting to a MySQL database, read data from it and populate arrays (I think PostgreSQL supports an array data type, but I'm not sure).
What you can do is use cursors in a stored procedure to retreive data from a query and store it into variables.
Example:
delimiter $$
create procedure my_procedure()
begin
declare value varchar(100);
declare done int default false;
declare cur cursor for
select column_name1 from your_table;
declare continue handler for not found set done = true;
open cur; -- This line will open the row set and place the cursor
-- on the first row.
loop_data: loop
fetch cur into value; -- This line will fetch the current row
-- into the variable and move the cursor
-- to the next row.
if done then -- If there are no more rows in the
leave loop_data; -- row set, the loop is terminated here
end if; -- and the execution moves to the next
-- instruction after "end loop;"
-- do whatever you need to do with the retrieved value
end loop;
close cur;
end $$
delimiter ;
If you want to use an array in a high level programming language, you can do it using the appropriate methods. Here's an example using Java (read The Java tutorials: JDBC Database access for more info):
public class SomeClass {
/*
Retrieve data from a database and return an array with it.
Parameters:
- conn: Connection to the database.
*/
public String[] getValues(Connection conn) {
String[] ans = new String[10];
int i;
try(
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(
"select column_name1 from your_table limit 10"
);
) {
rs.beforeFirst();
i = 0;
while(rs.next()) {
ans[i] = rs.getString("column_name1");
i++;
}
} catch(SQLException e) {
// Code to handle the SQL exception
}
return ans;
}
}
References:
MySQL reference manual: Cursors
You can use an variable to simply select your row values into a string. Not precisely an array, but it allows you to store all your values into a single variable:
-- load test data
create table tableName (column_name1 varchar(5));
insert into tableName values
('abcde');
insert into tableName values
('fghij');
insert into tableName values
('klmno');
insert into tableName values
('pqrst');
insert into tableName values
('uvwzy');
insert into tableName values
('z');
-- build "array"
set #array := '';
select #array := concat(#array,column_name1) as array
from tableName;
select #array;

SSRS multi-value parameter using a stored procedure

I am working on a SSRS report that uses a stored procedure containing a few parameters. I am having problems with two of the parameters because I want to have the option of selecting more than one item.
Here's a condensed version of what I have:
CREATE PROCEDURE [dbo].[uspMyStoredProcedure]
(#ReportProductSalesGroupID AS VARCHAR(MAX)
,#ReportProductFamilyID AS VARCHAR(MAX)
,#ReportStartDate AS DATETIME
,#ReportEndDate AS DATETIME)
--THE REST OF MY QUERY HERE WHICH PULLS ALL OF THE NEEDED COLUMNS
WHERE DateInvoicedID BETWEEN #ReportStartDate AND #ReportEndDate
AND ProductSalesGroupID IN (#ReportProductSalesGroupID)
AND ProductFamilyID IN (#ReportProductFamilyID)
When I try to just run the stored procedure I only return values if I enter only 1 value for #ReportProductSalesGroupID and 1 value #ReportProductFamilyID. If I try to enter two SalesGroupID and/or 2 ProductFamilyID it doesn't error, but I return nothing.
-- Returns data
EXEC uspMyStoredProcedure 'G23', 'NOF', '7/1/2009', '7/31/2009'
-- Doesn't return data
EXEC uspMyStoredProcedure 'G23,G22', 'NOF,ALT', '7/1/2009', '7/31/2009'
In SSRS I get an error that says:
Incorrect syntax near ','
It appears that the , separator is being included in the string instead of a delimiter
You need three things:
In the SSRS dataset properties, pass the multi-value param to the stored procedure as a comma-delimited string
=Join(Parameters!TerritoryMulti.Value, ",")
In Sql Server, you need a table-value function that can split a comma-delimited string back out into a mini table (eg see here). edit: Since SQL Server 2016 you can use the built-in function STRING_SPLIT for this
In the stored procedure, have a where clause something like this:
WHERE sometable.TerritoryID in (select Item from dbo.ufnSplit(#TerritoryMulti,','))
... where ufnSplit is your splitting function from step 2.
(Full steps and code in my blog post 'SSRS multi-value parameters with less fail'):
Let us assume that you have a multi value list #param1
Create another Internal Parameter on your SSRS report called #param2 and set the default value to:
=Join(Parameters!param1.value, 'XXX')
XXX can be any delimiter that you want, EXCEPT a comma (see below)
Then, you can pass #param2 to your query or stored procedure.
If you try to do it any other way, it will cause any string function that uses commas to separate arguments, to fail. (e.g. CHARINDEX, REPLACE).
For example Replace(#param2, ',', 'replacement') will not work. You will end up with errors like "Replace function requires 3 arguments".
Finally I was able to get a simple solution for this problem. Below I have provided all (3) steps that I followed.
I hope you guys will like it :)
Step 1 - I have created a Global Temp Table with one column.
CREATE GLOBAL TEMPORARY TABLE TEMP_PARAM_TABLE(
COL_NAME VARCHAR2(255 BYTE)
) ON COMMIT PRESERVE ROWS NOCACHE;
Step 2 - In the split Procedure, I didn't use any array or datatable, I have directly loaded the split values into my global temp table.
CREATE OR REPLACE PROCEDURE split_param(p_string IN VARCHAR2 ,p_separator IN VARCHAR2
)
IS
v_string VARCHAR2(4000);
v_initial_pos NUMBER(9) := 1;
v_position NUMBER(9) := 1;
BEGIN
v_string := p_string || p_separator;
delete from temp_param_policy;
LOOP
v_position :=
INSTR(v_string, p_separator, v_initial_pos, 1);
EXIT WHEN(NVL(v_position, 0) = 0);
INSERT INTO temp_param_table
VALUES (SUBSTR(v_string, v_initial_pos
, v_position - v_initial_pos));
v_initial_pos := v_position + 1;
END LOOP;
commit;
END split_param;
/
Step 3 - In the SSRS dataset parameters, I have used
=Join(Parameters!A_COUNTRY.Value, ",")
Step 4: In the start of your stored procedure executes the Procedure
Exec split_param(A_Country, ‘,’);
Step 5: In your stored procedure sql use the condition like below.
Where country_name in (select * from TEMP_PARAM_TABLE)
When SSRS passes the parameter it is in the form: Param1,Param2,Param3.
In the procedure, you just need to put identifiers around each parameter. And also identifiers around the value that is returned by the dataset. In my case, I used semicolons.
CREATE OR REPLACE PROCEDURE user.parameter_name (
i_multivalue_parameter
)
AS
l_multivalue_parameter varchar2(25555) := ';' || replace(i_multivalue_parameter,',',';') || ';';
BEGIN
select something
from dual
where (
instr(l_multivalue_parameter, ';' || database_value_that_is_singular || ';') > 0
)
END;
i_multivalue_parameter is passed in via SSRS.
l_multivalue_parameter reads the parameter passed in via SSRS and puts identifiers around each value.
database_value_that_is_singular is the value returned for each record.
So if 'Type1,Type2,Type3'is passed in via SSRS:
i_multivalue_parameter is: Type1,Type2,Type3
l_multivalue_parameter is: ;Type1;Type2;Type3;
database_value_that_is_singular is: ;Type1; or ;Type2; or ;Type3;
Instr will return a value over 0 if the parameter matches.
This works even if each parameters are similar. EG: "Type A" and "Type AA". That is "Type A" will not match "Type AA".
I found a simple way for my solution. Define the parameter value in the report as an expression like this
="'" + Join(Parameters!parm.Value,"','") + "'"
(in case you can't read it the first and last literals are double quote, single quote, double quote. The join literal is double quote, single quote, comma, single quote, double quote)
Then in the stored procedure you can use dynamic sql to create your statement. I did this to create a temp table of values to join to in a later query, like this:
CREATE #nametable (name nvarchar(64))
SET #sql = N'SELECT Name from realtable where name in (' + #namelist + ')'
INSERT INTO #nametable exec sp_executesql #sql
#namelist would be the name of the stored procedure parameter.

Perl DBI execute not maintaining MySQL stored procedure results

I'm having a problem with executing a stored procedure from Perl (using the DBI Module). If I execute a simple SELECT * FROM table there are no problems.
The SQL code is:
DROP FUNCTION IF EXISTS update_current_stock_price;
DELIMITER |
CREATE FUNCTION update_current_stock_price (symbolIN VARCHAR(20), nameIN VARCHAR(150), currentPriceIN DECIMAL(10,2), currentPriceTimeIN DATETIME)
RETURNS INT
DETERMINISTIC
BEGIN
DECLARE outID INT;
SELECT `id` INTO outID FROM `mydb449`.`app_stocks` WHERE `symbol` = symbolIN;
IF outID > 0 THEN
UPDATE `mydb449`.`app_stocks`
SET `currentPrice` = currentPriceIN, `currentPriceTime` = currentPriceTimeIN
WHERE `id` = outID;
ELSE
INSERT INTO `mydb449`.`app_stocks`
(`symbol`, `name`, `currentPrice`, `currentPriceTime`)
VALUES (symbolIN, nameIN, currentPriceIN, currentPriceTimeIN);
SELECT LAST_INSERT_ID() INTO outID;
END IF;
RETURN outID;
END|
DELIMITER ;
The Perl code:
$sql = "select update_current_stock_price('$csv_result[0]', '$csv_result[1]', '$csv_result[2]', '$currentDateTime') as `id`;";
My::Extra::StandardLog("SQL being used: ".$sql);
my $query_handle = $dbh->prepare($sql);
$query_handle->execute();
$query_handle->bind_columns(\$returnID);
$query_handle->fetch();
If I execute select update_current_stock_price('aapl', 'Apple Corp', '264.4', '2010-03-17 00:00:00') asid; using the mysql CLI client it executes the stored function correctly and returns an existing ID, or the new ID.
However, the Perl will only return a new ID, (incrementing by 1 on each run). It also doesn't store the result in the database. It looks like it's executing a DELETE on the new id just after the update_current_stock_price function is run.
Any help? Does Perl do anything funky to procedures I should know about?
Before you ask, I don't have access to binary logging, sorry.
Perhaps you're doing it in a transaction and it's getting rolled back? The row is inserted but never becomes committed and cannot be seen.
I'd try it on your dev server and enable general query log, if in doubt.
Also you may want to know about the INSERT ... ON DUPLICATE KEY UPDATE syntax, which can probably do what you're trying to do anyway.
try
$query_handle->dump_results(15, "\n", '|');
before the bind_columns call to see if it is actually getting the results back, you could also try replace SELECT storedprocedure with SELECT * FROM storedprocedure
You should check that you are running the latest version of DBD::mysql (which is the MySQL-driver used by DBI). There used to be several issues with stored procedures, at least some are fixed in recent versions. Maybe these ressources are also helpful:
http://www.perlmonks.org/?node_id=609098
http://www.perlmonks.org/?node_id=830585