Is there a way to change the default date input format - mysql

I have a source of data from where I extract some fields, among the fields there are some date fields and the source sends their values like this
#DD/MM/YYYY#
almost all the fields can be sent into the query with no modificaction, except this of course.
I have written a program the gets the data from an internet connection and sends it to the MySQL server and it's sending everything as it should, I am sure because I enabled general logging in the MySQL server and I can see all the queries are correct, except the ones with date fields.
I would like to avoid parsing the fields this way because it's a lot of work since it's all written in c, but if there is no way to do it, I understand and would accept that as an answer of course.
As an example suppose we had the following
INSERT INTO sometable VALUES ('#12/10/2015#', ... OTHER_VALUES ..., '#11/10/2015#');
in this case I send the whole thing as a query using mysql_query() from libmysqlclient.
In other cases I can split the parts of the message in something that is like an instruction and the parameters, something like this
iab A,B,C,#12/10/2015#,X,Y,#11/10/2015#
which could mean INSERT INTO table_a_something_b_whatever VALUES, and in this situation of course, I capture all the parameters and send a single query with a list of VALUES in it. Also in this situation, it's rather simple because I can handle the date like this
char date[] = "#11/10/2015#";
int day;
int month;
int year;
if (sscanf(date, "#%d/%d/%d#", &day, &month, &year) == 3)
{
/* it's fine, build a sane YYYY-MM-DD */
}
So the question is:
How can I tell the MySQL server in what format the date fields are?
Clarification to: Comment 1
Not necessarily INSERT, it's more complex than that. They are sometimes queries with all their parameters in it, sometimes they are just the parameters and I have to build the query. It's a huge mess but I can't do anything about it because it's a paid database and I must use it for the time being.
The real problem is when the query comes from the source and has to be sent as it is, because then there can be many occurrences. When I split the parameters one by one there is no real problem because parsing the above date format and generating the appropriate value of MySQL is quite simple.

You can use STR_TO_DATE() in MySQL:
SELECT STR_TO_DATE('#08/10/2015#','#%d/%m%Y#');
Use this as part of your INSERT process:
INSERT INTO yourtable (yourdatecolumn) VALUES (STR_TO_DATE('#08/10/2015#','#%d/%m%Y#'));

The only Thing I could imagine at the Moment would be to Change your Column-Type from DateTime to varchar and use a BEFORE INSERT Trigger to fix "wrong" Dates.
Something like this:
DELIMITER //
CREATE TRIGGER t1 BEFORE INSERT on myTable FOR EACH ROW
BEGIN
IF (NEW.myDate regexp '#[[:digit:]]+\/[[:digit:]]+\/[[:digit:]]+#') THEN
SET NEW.myDate = STR_TO_DATE(NEW.myDate,'#%d/%m/%Y#');
END IF;
END; //
DELIMITER ;
If you are just Need to run the Import in question once, use the Trigger to generate a "proper" dateTimeColumn out of the inserts - and drop the varchar-column afterwards:
('myDate' := varchar column to be dropped afterwards;`'myRealDate' := DateTime Column to Keep afterwards)
DELIMITER //
CREATE TRIGGER t1 BEFORE INSERT on myTable FOR EACH ROW
BEGIN
IF (NEW.myDate regexp '#[[:digit:]]+\/[[:digit:]]+\/[[:digit:]]+#') THEN
SET NEW.myRealDate = STR_TO_DATE(NEW.myDate,'#%d/%m/%Y#');
else
#assume a valid date representation
SET NEW.myRealDate = NEW.myDate;
END IF;
END; //
DELIMITER ;
Unfortunately you cannot use a Trigger to work on the datetime-column itself, because mysql will already mess up the NEW.myDate-Column.

Related

Running Multiple Sql Commands in a Loop

I successfully executed the sql code below from an msql editor (phpmyadmin), testing it with one customer (where Customer No=1). I need to now run the sql script for all the customers.
START TRANSACTION;
INSERT INTO `addresses` (`AddressLine1`,`CityID`,`ProvStateCode`,`AddressPostCode`,`CountryIso`) SELECT `Bill To Address`,`Bill To City`,`Bill To Province`,`Bill Code`,`Country` FROM `pdx_customers` where `Customer No`=1;
SELECT #last_id := LAST_INSERT_ID();
SELECT `Customer No` FROM `pdx_customers`
INSERT INTO `customer_addresses` (`CustID`,`AddressID`,`AddressTypeID`) Values(1,#last_id,1);
COMMIT;
It seems I would need to create a stored procedure ? In a loop, I need to get the Customer No dynamically for each row in the pdx_customers table, and enter into the Values clause in the insert command, i.e Values(#CustID,#last_id,1). Not sure how I would do this ?
Any help would be appreciated.
Thanks
This is a really common problem, and I would say that doing a loop in sql is almost never a good idea. Here is another option, which you may or may not consider good as it does introduce a new column. I've used it in some apps I've done, and its made things very simple. Does depend on your use case though so it wont be for everyone.
1) Firstly, add a new column to the address table, call it something that wont be confused by anyone looking at the table like TempInsertId.
2) When writing the new address, include the CustomerId in the TempInsertId column
3) Now you can easily read the AddressId and CustomerId back and write it into the CustomerAddress table
4) If you wish, do a final update to set the TempInsertId back to null.
As I said, not advocating in all cases, but it can be a very simple solution to the problem.
You can use the below statement to create a loop:
start transaction;
while counter < max do
insert into . . . ;
set counter=counter+1;
end while;

Filter Dataset using SQL Query

I am using Zeos and mysql in my delphi project.
what I would like to do is filter dataset using a textbox.
to do that, I am using following query in textbox 'OnChange' Event:
ZGrips.Active := false;
ZGrips.SQL.Clear;
ZGrips.SQL.Add('SELECT Part_Name, Description, OrderGerman, OrderEnglish FROM Part');
ZGrips.SQL.Add('WHERE Part_Name LIKE ' + '"%' + trim(txt_search.Text) + '%"');
ZGrips.Active := true;
after I run and type first character in textbox, I get empty dataset in my DBGrid,
so DBGrid is showing nothing, then If I type second character I get some result in DBGrid. and even more strange behavior: if I will use AS Clause in my SQL Query like:
Part_Name AS blablabla,
Description AS blablabla,
OrderGerman AS OG,
OrderEnglish AS OE
in that case DBGrid is showing only 2 columns: Part_Name and Description, I dont understand why it is ignoring 3rd and 4th columns.
thanks for any help in advance.
Always use parameters
Firstly you need to use parameters, otherwise your query will break or worse when the user enters the wrong characters in the search box.
See: How does the SQL injection from the "Bobby Tables" XKCD comic work?
Parameters also makes you query faster, because the database engine only have to decode the query once.
If you change a parameter the engine will know that the query itself has not changed and will not re-decode it.
Don't use clear and add
Just supply the SQL as text in one go, it's faster.
This is esp. true in a loop, outside the loop you will not notice the difference.
Your code should read something like:
procedure TForm1.SetupSearch; //run this only once.
var
SQL: string;
begin
ZGrips.Active:= false;
SQL:= 'SELECT Part_Name, Description, OrderGerman, OrderEnglish FROM Part' +
'WHERE Part_Name LIKE :searchtext'); //note no % here.
ZGrips.SQL.Text:= SQL; //don't use clear and don't use SQL.Add.
end;
//See: http://docwiki.embarcadero.com/Libraries/XE2/en/Vcl.StdCtrls.TEdit.OnChange
procedure TForm1.Edit1Change(Sender: TObject);
begin
if Edit1.Modified then begin
Timer1.Active:= true;
end;
end;
procedure TForm1.Timer1Timer(Sender: TObject);
begin
Timer1.Active:= false;
if Edit1.Text <> ZGrips.Params[0].AsString then begin
ZGrips.Params[0].AsString:= Edit1.Text + '%'
ZGrips.Active:= true;
end;
end;
Use a timer
As per #MartinA's suggestion, use a timer and start the query only ever so often.
The wierd behaviour you're getting maybe because you're stopping and reactivating a new query before the old one has had time to finish.
The Params[index: integer] property is a bit faster than the ParamsByName property.
Although this does not really matter outside a loop.
Allow the database to use an index!
Using only a trailing wildcard % is faster than using a leading wildcard because the database can only use an index is there is a trailing wildcard.
If you want to use a leading wildcard, then consider storing the data in reverse order and use a trailing wildcard instead.
Full-text indexes are much better than like
Of course if you use both a leading and a trailing wild card then you have to use a full-text index.
In MySQL you'll than use the MATCH AGAINST syntax,
see: Differences between INDEX, PRIMARY, UNIQUE, FULLTEXT in MySQL?
and: Which SQL query is better, MATCH AGAINST or LIKE?
The lastest versions of MySQL support full-text indexes in InnoDB.
Remember to never use MyISAM, it's unreliable.

Custom format example for serial number creation in mysql

I have recently switched over from OpenOffice Base to MySQL community. In ooBase you can customize integers upon entering them into the database. For example I could type 2013_00000 and then every number in this column would be formatted in this way (2013_00001, 2013_00002, ...). After playing around with MySQL community for a while I noticed that there is no obvious way to format a custom integer like this. I might be overlooking something basic, but if anyone knows how to do this please let me know. Thank you
In order to manipulate data before inserting them you need a trigger.
Something like that:
CREATE TRIGGER ins_sn BEFORE INSERT ON tbl
FOR EACH ROW
SET NEW.sn = CONCAT(LEFT(NEW.sn, 4), '_', RIGHT(NEW.sn,5));
If you need to update your rows, you need a second trigger:
CREATE TRIGGER upd_sn BEFORE UPDATE ON tbl
FOR EACH ROW
SET NEW.sn = CONCAT(LEFT(NEW.sn, 4), '_', RIGHT(NEW.sn,5));
See http://sqlfiddle.com/#!2/01428/1 for the complete example.
Here, I use the CONCAT(), LEFT() and RIGHT() functions for simplicity. Maybe your data require more complex manipulations (i.e.: padding). MySQL has a rich set of string functions. Anyway this is the principle.
Not the most flexible (here locked down to an underscore separator), but you can do it with a trigger, something like this;
CREATE TRIGGER trig_MyTable BEFORE INSERT ON MyTable
FOR EACH ROW
BEGIN
SELECT seq into #seq FROM MyTable_Seq FOR UPDATE;
SET NEW.id = #seq, #cutoff = LOCATE('_', #Seq);
UPDATE MyTable_Seq SET Seq =
CONCAT(LEFT(#seq, #cutoff),
LPAD(SUBSTR(#seq, #cutoff+1)+1, LENGTH(#seq)-#cutoff, '0'));
END;
//
An SQLfiddle to test with.

Show me an Injection Attack for this Stored Procedure

I notice that many people have said it's possible to create an injection attack, but my understanding is that is if someone is creating a query from a string, not parameters. In order to test the statement that Stored Procedures do not protect you against Injection Attacks, I am putting this example up in the hopes someone can show me a vulnerability if there is one.
Please note that I have built the code this way to easily insert a function that calls a procedure and embed it in a SELECT query. That means I cannot create a Prepared Statement. Ideally I'd like to keep my setup this way, as it is dynamic and quick, but if someone can create an injection attack that works, obviously that is not going to happen.
DELIMITER $$
#This procedure searches for an object by a unique name in the table.
#If it is not found, it inserts. Either way, the ID of the object
#is returned.
CREATE PROCEDURE `id_insert_or_find` (in _value char(200), out _id bigint(20))
BEGIN
SET #_value = _value;
SET #id = NULL;
SELECT id INTO _id FROM `table` WHERE name=_value;
IF _id IS NULL THEN
BEGIN
INSERT INTO `table` (`name`) VALUE (_value);
SELECT LAST_INSERT_ID() INTO _id;
END;
END IF;
END$$
CREATE FUNCTION `get_id` (_object_name char(200)) RETURNS INT DETERMINISTIC
BEGIN
SET #id = NULL;
call `id_insert_or_find`(_object_name,#id);
return #id;
END$$
The PHP Code
The PHP code I use here is:
(note, Boann has pointed out the folly of this code, below. I am not editing it for the sake of honoring the answer, but it will certainly not be a straight query in the code. It will be updated using ->prepare, etc. I still welcome any additional comments if new vulnerabilities are spotted.)
function add_relationship($table_name,$table_name_child) {
#This table updates a separate table which has
#parent/child relationships listed.
$db->query("INSERT INTO table_relationships (`table_id`,`tableChild_id`) VALUES (get_id('{$table_name}'),get_id('{$table_name_child}')");
}
The end result is
table `table`
id name
1 oak
2 mahogany
Now if I wanted to make oak the child of mahogany, I could use
add_relationship("mahogany","oak");
And if I wanted to make plastic the child of oak, I could use
add_relationship("oak","plastic");
Hopefully that helps give some framework and context.
It is not necessarily the stored procedure that is unsafe but the way you call it.
For example if you do the following:
mysqli_multi_query("CALL id_insert_or_find(" + $value + ", " + $id + ")");
then the attacker would set $value="'attack'" and id="1); DROP SCHEMA YOUR_DB; --"
then the result would be
mysqli_multi_query("CALL id_insert_or_find('attack', 1); DROP SCHEMA YOUR_DB; --)");
BOOM DEAD
Strictly speaking, that query should be written to escape the table names:
$db->query("INSERT INTO table_relationships (`table_id`,`tableChild_id`) " .
"VALUES (get_id(" . $db->quote($table_name) + ")," .
"get_id(" . $db->quote($table_name_child) . "))");
Otherwise, it would break out of the quotes if one of the parameters contained a single quote. If you only ever call that function using literal strings in code (e.g., add_relationship("mahogany", "oak");) then it is safe to not escape it. If you might ever call add_relationship using data from $_GET/$_POST/$_COOKIE or other database fields or files, etc, it's asking for trouble. I would certainly not let it pass a code review.
If a user could control the table name provided to that function then they could do, for example:
add_relationship("oak", "'+(SELECT CONCAT_WS(',', password_hash, password_salt) FROM users WHERE username='admin')+'");
Now you might say that there's no practical way to then extract that information if the resulting table name doesn't exist, but even then you could still extract information one binary bit at a time using a binary search and separate queries, just by breaking the query. Something like this (exact syntax not tested):
add_relationship("oak", "plastic'+(IF(ORD(SUBSTR(SELECT password_hash FROM users WHERE username='admin'),1,1)>=128, 'foo', ''))+'");
Really, it's easier to just escape the parameters and then you don't have to worry.

Sanitisation of input for a stored MySQL database procedure

I am using a database someone else produced (and I am not really authorised to change it). However, as I was looking into the stored procedures within the database I noticed the following procedure:
DELIMITER $$
CREATE PROCEDURE `logIn`(userName varChar(50), userPass varChar(50))
BEGIN
declare userID int;
SELECT
u.userID INTO userID
FROM
users u
WHERE
u.userName=userName
AND u.userPassword=MD5(userPass);
IF (IFNULL(uID,-1) > 0) THEN
select 1 as outMsg;
ELSE
select 0 as outMsg;
END IF;
END$$
with the corresponding table users having three columns: userID INT, userName VARCHAR(50) and userPassword VARCHAR(50).
As I am not very good at this, could someone let me know whether the input for such a function needs to be sanitised as to not allow any SQL injections and if not - why? A general rule of thumb would be very much appreciated.
P.S. This function will be called from a JS script on a form submit.
There are a few rules of thumb here that depend on the underlying datatype and how it's inserted into the database.
First, Parameterized queries are always best for SQL Injection protection.. but.. if you can't change that..
String type:
Remove any single quotes
OR
Replace any single quotes with the single quote twice.
Replace any of the following characters with their encoded alternative;
>
<
"
;
(chr 34)
)
(
For example.. ) is replaced with & #x29;
-(the space in the above example is so you'll see the code, remove it to get ")")
For a datatype other then string, check that the datatype is sane and remove any character that shouldn't be in the datatype. If it's an integer, make sure the string that you're passing in is an integer. This can commonly be done by casting to the type in code. The cast will either work.. or cause an error. It's also good to check that the datatype min and maxes have not been exceeded. For example.. If I was checking for an integer, I might use code similar to this:
var myInt = parseInt(param);
Then I might check it's bounds to be sure it's less then the maximum integer value and greater then the minimum integer value.
That should be good enough to prevent a SQL Injection attack...
And.. since you have not posted the code that actually interfaces with the database... As an added precaution.. you may also want to remove --,`,%,",", "".
You only want 'sane' values getting to the database call.. so an integer like, $309 wouldn't make sense, you'd want to remove the $.. . probably by using a regex replace for any non numeric characters a comma and a period.
[^[0-9,.]]
Be extra cautious.
Yes, the input must be sanitized before trying to run the procedure.
You might want to share the actual calling point for the procedure to get more help here, since there is no way that the procedure is called directly from JS on form submit. You probably have a Servlet, PHP page or some HTTP friendly intermediary to make the database call somehow.