I have a set of queries with randoms data that i want to insert in database. Randoms data may have any special characters.
for example:
INSERT INTO tablename VALUES('!^'"twco\dq');
Are there any scope that escape all special characters?
please help.
No, there is no "scope" in MySQL to automatically escape all special characters.
If you have a text file containing statements that were created with potentially unsafe "random values" like this:
INSERT INTO tablename VALUES('!^'"twco\dq');
^^^^^^^^^^^
You're basically screwed. MySQL can't unscramble a scrambled egg. There's no "mode" that makes MySQL work with a statement like that.
Fortunately, that particular statement will throw an error. More tragic would be some nefariously random data,
x'); DROP TABLE students; --
if that random string got incorporated into your SQL text without being escaped, the result would be:
INSERT INTO tablename VALUES('x'); DROP TABLE students; --');
The escaping of special characters has to be done before the values are incorporated into SQL text.
You'd need to take your random string value:
!^'"twco\dq
And run it through a function that performs the necessary escaping to make that value safe for including that as part of the the SQL statement.
MySQL provides the real_escape_string_function as part of their C library. Reference https://dev.mysql.com/doc/refman/5.5/en/mysql-real-escape-string.html. This same functionality is exposed through the MySQL Connectors for several languages.
An even better pattern that "escaping" is to use prepared statements with bind placeholders, so your statement would be a static literal, like this:
INSERT INTO tablename VALUES ( ? )
You can use \ character to escape special characters like below. See this DEMO if in doubt.
INSERT INTO tablename VALUES('\!\^\'\"twco\\dq');
Per MySQL documentation, below are the defined escape sequences
Table 9.1 Special Character Escape Sequences
\0 An ASCII NUL (0x00) character.
\' A single quote (“'”) character.
\" A double quote (“"”) character.
\b A backspace character.
\n A newline (linefeed) character.
\r A carriage return character.
\t A tab character.
\Z ASCII 26 (Control+Z). See note following the table.
\\ A backslash (“\”) character.
\% A “%” character. See note following the table.
\_ A “_” character. See note following the table.
Related
Im trying to escape both parentheses using MySQL. It's giving me an error starting at A.T.
INSERT INTO `schools`(`schoolid`, `name`) VALUES (1,"A.T. Still University of Health Sciences(AZ / MO)")
You don't need to "escape" anything in that statement. You just need to use single-quotes for strings in SQL:
'A.T. Still University of Health Sciences(AZ / MO)'
It looks like double-quotes can be supported in MySQL, depending on the ANSI_QUOTES setting. But unless there's a compelling reason to use them, I'd recommend sticking with single-quotes as it's more standard and more commonly used.
You need to use single quotes for strings instead of double quotes. Try this:
INSERT INTO `schools`(`schoolid`, `name`)
VALUES (1,'A.T. Still University of Health Sciences(AZ / MO)')
SQL FIDDLE DEMO
Use Special Character Escape Sequences in the Insert statement
Escape Sequence Character Represented by Sequence
\0 An ASCII NUL (0x00) character
\' A single quote (“'”) character
\" A double quote (“"”) character
\b A backspace character
\n A newline (linefeed) character
\r A carriage return character
\t A tab character
\Z ASCII 26 (Control+Z); see note following the table
\ A backslash (“\”) character
\% A “%” character; see note following the table
_ A “_” character; see note following the table
Postgres escapes quotes incorrectly when creating a JSON export. Note the double quotes in the below update...
UPDATE models SET column='"hello"' WHERE id=1;
COPY (SELECT row_to_json(models)
FROM (SELECT column FROM shaders WHERE id=1) shaders)
TO '/output.json';
The contents of output.json:
{"column":"\\"hello\\""}
You can see that the quotes are escaped improperly and it creates invalid JSON.
It should be:
{"column":"\"hello\""}
How can I fix this Postgres bug or work around it?
This is not JSON related. It's about the way text format (default) in COPY command handles backslashes. From the PostgreSQL documentation - COPY:
Backslash characters (\) can be used in the COPY data to quote data characters that might otherwise be taken as row or column delimiters. In particular, the following characters must be preceded by a backslash if they appear as part of a column value: backslash itself, newline, carriage return, and the current delimiter character.
(Emphasis mine.)
You can solve it by using CSV-format and changing the quote character from doublequote to something else.
To demonstrate:
SELECT row_to_json(row('"hello"'))
| "{"f1":"\"hello\""}" |
COPY (SELECT row_to_json(row('"hello"'))) TO '/output.json';
| {"f1":"\\"hello\\""} |
COPY (SELECT row_to_json(row('"hello"'))) TO '/output.json' CSV QUOTE '$';
| {"f1":"\"hello\""} |
The answer by Simo Kivistö works if you are certain that the character $, or whatever the special quote character you chose does not appear in your strings. In my case, I had to export a very large table and there was no particular character which didn't appear in the strings.
To work around this issue, I piped the output of the COPY command to sed to revert the double escaping of quotes:
psql -c "COPY (SELECT row_to_json(t) from my_table as t) to STDOUT;" |
sed 's/\\"/\"/g' > my_table.json
The sed expression I am piping to simply replaces occurrences of \\" with \".
please look here:
UPDATE cars_tbl
SET description = '{\rtf1'
WHERE (ID=1)
Description field is "blob", where my RTF document is to be stored.
When I check updated data I always find
{
tf1
\r simply disapears. I tried to find solution on the web, but no success. My rtf files are corrupted on many places, because the escape characters used in the string are substituted. How to suppress this substitution and update field with string as is?
Thanx for advice
Lyborko
Backslash is an escape character, so to keep it you need a double backslash:
UPDATE cars_tbl
SET description = '{\\rtf1'
WHERE (ID=1)
As an aside \r is a carriage return.. and it hasn't disappeared in your data; it is responsible for tf1 appearing on the line below the {.
You can achieve this with a more generic approach
use of QUOTE() in mysql
MySQL QUOTE() produces a string which is a properly escaped data value in an SQL statement, out of an user supplied string as argument.
The function achieve this by enclosing the string with single quotes, and by preceding each single quote, backslash, ASCII NUL and control-Z with a backslash.
example
UPDATE cars_tbl
SET description = QUOTE('{\rtf1')
WHERE (ID=1)
UPDATE
to escape your RTF you can also just use REPLACE this way all your \ will become \\
Example
UPDATE cars_tbl
SET description = REPLACE('{\rtf1', '\', '\\')
WHERE (ID=1)
I have a table where a column allows special characters like '/' (forward slash) and '' (back slash).
Now when I try to search such records from table, I am unable to get those.
For example: abc\def or abc/def
I am generating a search query like:
select * from table1_1 where column10 like '%abc\def%'
It is returning 0 rows, but actually there is 1 record existing that should be returned. How do I write the query in this case?
The trick is to double escape ONLY the backslash; for string escapes only a single escape is needed.
For example
The single quote ' only needs escaping once LIKE '%\'%'
But to query backslash \ you need to double escape to LIKE '%\\\\%'
If you wanted to query backslash+singlequote \' then LIKE '%\\\\\'%' (with 5 backslashes)
Explanation Source
excerpt:
Because MySQL uses C escape syntax in strings (for example, “\n” to
represent a newline character), you must double any “\” that you use
in LIKE strings. For example, to search for “\n”, specify it as “\n”.
To search for “\”, specify it as “\\”; this is because the
backslashes are stripped once by the parser and again when the pattern
match is made, leaving a single backslash to be matched against.
In MySQL, this works:
select * from Table1
where column10 like '%abc\\\\def%'
FIDDLE
Backslash is an escape prefix for both strings and LIKE patterns. So you need to double it once for LIKE, and again for string literal syntax.
You have to escape the \ with another \
select * from table1_1 where column10 like '%abc\\def%'
Escaping the LIKE value didn't work when I tried it on MySQL v5.5. After a few tries, what did work was a regex (and I had to escape there too, plus hide it in a character class). I finally got it to work like this:
select * from table1_1 where column10 rlike 'abc[\\]def'
These didn't work:
... column10 like '%abc\\def%'
... column10 like concat('%abc', char(92), 'def%')
... column10 rlike 'abc\\def'
Note you also have this tagged as PL/SQL, which is Oracle. The query you posted works as-is on Oracle. Is the PL/SQL tag a mistake?
Use escaping.
https://dev.mysql.com/doc/refman/5.0/en/string-literals.html
Table 9.1. Special Character Escape Sequences
Escape Sequence Character Represented by Sequence
\0 An ASCII NUL (0x00) character.
\' A single quote (“'”) character.
\" A double quote (“"”) character.
\b A backspace character.
\n A newline (linefeed) character.
\r A carriage return character.
\t A tab character.
\Z ASCII 26 (Control+Z). See note following the table.
\ A backslash (“\”) character.
\% A “%” character. See note following the table.
_ A “_” character. See note following the table.
If you use PHP sprintf() to put together your query where you want:
AND column10 LIKE '%/%'
This worked for me inside sprintf():
AND column10 LIKE '%%/%%'
(MySSQL 8 - WAMP)
We all know that we should use prepared statements or the appropriate replacement/formatting rules in order to prevent sql injection in our applications.
However, when taking a look at MySQL's list of character literals, I noticed that it includes the following characters:
\0 An ASCII NUL (0x00) character.
\' A single quote (') character.
\" A double quote (") character.
\b A backspace character.
\n A newline (linefeed) character.
\r A carriage return character.
\t A tab character.
\Z ASCII 26 (Ctrl+Z). See note following the table.
\\ A backslash (\) character.
\% A % character.
\_ A _ character.
Now, while the % and _ characters need to be escaped in order to prevent injection of unwanted wildcards into LIKE statements, and while the ' (single quote), \ (backslash), and " (double quote) all need to be escaped in order to prevent injection of arbitrary SQL - could having any of these other characters unescaped lead directly to a SQL injection vulnerability that would not otherwise be present? Does anyone have any real world examples of such an exploit?
Let's assume we are building our query like:
SELECT * FROM users WHERE username='$user'
Is there any value for $user where the only unescaped character literals are \b (backspace), \0 (NUL), \n (newline), \r (linefeed), \t (tab) or \Z (Ctrl+Z) that allows the injection of arbitrary SQL into this query?
Considering the below lines from mysql_real_escape_string() manual :
MySQL requires only that backslash and the quote character used to quote the string in the query be escaped. mysql_real_escape_string() quotes the other characters to make them easier to read in log files.
SQL injection in MySQL should not be possible with these special characters alone by themselves : \b \0 \n \r \t \Z .
However String Literals manual states the following but the reasons specified ( or not ) does not relate to SQL injection :
If you want to insert binary data into a string column (such as a BLOB column), you should represent certain characters by escape sequences. Backslash (“\”) and the quote character used to quote the string must be escaped. In certain client environments, it may also be necessary to escape NUL or Control+Z. The mysql client truncates quoted strings containing NUL characters if they are not escaped, and Control+Z may be taken for END-OF-FILE on Windows if not escaped.
Furthermore , in a simple test , irrespective of weather the above listed special characters are escaped or not , MySQL yielded same results . In other words MySQL did not even mind :
$query_sql = "SELECT * FROM `user` WHERE user = '$user'";
The above query worked similarly for non-escaped and escaped versions of those above listed characters as put below :
$user = chr(8); // Back Space
$user = chr(0); // Null char
$user = chr(13); // Carriage Return
$user = chr(9); // Horizontal Tab
$user = chr(26); // Substitute
$user = chr(92) .chr(8); // Escaped Back Space
$user = chr(92) .chr(0); // Escaped Null char
$user = chr(92) .chr(13); // Escaped Carriage Return
$user = chr(92) .chr(9); // Escaped Horizontal Tab
$user = chr(92) .chr(26); // Escaped Substitute
Test table and data used in the simple test :
-- Table Structure
CREATE TABLE IF NOT EXISTS `user` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`user` varchar(10) CHARACTER SET utf8 NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
-- Table Data
INSERT INTO `user` ( `user` ) VALUES
( char( '8' ) ),
( char( '0' ) ),
( char( '10' ) ),
( char( '13' ) ),
( char( '9' ) ),
( char( '26' ) );
An obligatory addendum from 2020:
Dealing with characters was proven to be inefficient and obsoleted
You must use prepared statements and forget about escaping, "dangerous characters" or any of that business.
Using parameterized queries is considered the only proper way to protect from SQL injections, for the reasons provided in the original answer below:
Which characters are actually capable of causing SQL injection in mysql
There are no such characters.
It is not "characters" that cause the SQL injections. But improper formatting. Any character, depends on the circumstances, could be either "dangerous" or absolutely harmless. Limiting your protection to some subset is a dangerous delusion that will actually lead to SQL injection sooner or later.
There are two wrong statements in your question that led you to a confusion:
We all know that we should use ... the appropriate replacement rules in order to prevent sql injection in our applications.
This statement is wrong. Not replacement but formatting. The difference is essential. Replacement alone does not protect from injections, while formatting does. Note that every distinct part of the query require different formatting which being useless for any other part. Say, there is another character, essential for injection protection - a backtick (`). But you didn't list it because it has nothing to do with string literals.
the ' (single quote), \ (backslash), and " (double quote) all need to be escaped in order to prevent injection
That's a gravely wrong statement. Escaping do not prevent injections. These characters need to be escaped in order to format strings and has absolutely nothing to do with injections. While it is true that properly formatted query part is invulnerable. But the truth is - you have to format dynamical query parts just for sake of it, to follow the syntax rules and not because of whatever injections. And you will have your query impenetrable just as a side effect.
Now you can see why your last statement,
why all of these other characters are vulnerable enough to be escaped via mysql_real_escape_string, as it is not immediately obvious to me.
is wrongly put:
It is string formatting rules require these characters, not whatever "vulnerability". Some of them are escaped just for convenience, some for readability, some for the obvious reason of escaping a delimiter. That's all.
To answer recent questions from comments:
I really want an answer to this, as PHP's mysql_real_escape_string does not quote these literals either.
Again: although in the mind of average PHP user mysql_real_escape_string() is strongly connected to whatever scaring injection, in reality it doesn't. There are no "dangerous" characters. Not a single one. There are some service characters with special meaning. They have to be escaped in some circumstances, depends on the context.
Thus, there is no connection between characters escaped by this function, and whatever "danger". The moment you start thinking that mysql_real_escape_string()'s purpose is to escape "dangerous" characters, you are indeed putting yourself in a danger. While as long as you are using this function only to escape strings (and doing it unconditionally) - you may consider yourself safe (of course if you don't forget to format all other literals too, using their respective formatting rules)
I want to know if the "%" character can lead to anything more than extra results in a LIKE clause.
No.