SQL - Incorrect string value: '\xEF\xBF\xBD' [duplicate] - mysql

I have a table I need to handle various characters. The characters include Ø, ® etc.
I have set my table to utf-8 as the default collation, all columns use table default, however when I try to insert these characters I get error: Incorrect string value: '\xEF\xBF\xBD' for column 'buyerName' at row 1
My connection string is defined as
string mySqlConn = "server="+server+";user="+username+";database="+database+";port="+port+";password="+password+";charset=utf8;";
I am at a loss as to why I am still seeing errors. Have I missed anything with either the .net connector, or with my MySQL setup?
--Edit--
My (new) C# insert statement looks like:
MySqlCommand insert = new MySqlCommand( "INSERT INTO fulfilled_Shipments_Data " +
"(amazonOrderId,merchantOrderId,shipmentId,shipmentItemId,"+
"amazonOrderItemId,merchantOrderItemId,purchaseDate,"+ ...
VALUES (#amazonOrderId,#merchantOrderId,#shipmentId,#shipmentItemId,"+
"#amazonOrderItemId,#merchantOrderItemId,#purchaseDate,"+
"paymentsDate,shipmentDate,reportingDate,buyerEmail,buyerName,"+ ...
insert.Parameters.AddWithValue("#amazonorderId",lines[0]);
insert.Parameters.AddWithValue("#merchantOrderId",lines[1]);
insert.Parameters.AddWithValue("#shipmentId",lines[2]);
insert.Parameters.AddWithValue("#shipmentItemId",lines[3]);
insert.Parameters.AddWithValue("#amazonOrderItemId",lines[4]);
insert.Parameters.AddWithValue("#merchantOrderItemId",lines[5]);
insert.Parameters.AddWithValue("#purchaseDate",lines[6]);
insert.Parameters.AddWithValue("#paymentsDate",lines[7]);
insert.ExecuteNonQuery();
Assuming that this is the correct way to use parametrized statements, it is still giving an error
"Incorrect string value: '\xEF\xBF\xBD' for column 'buyerName' at row 1"
Any other ideas?

\xEF\xBF\xBD is the UTF-8 encoding for the unicode character U+FFFD. This is a special character, also known as the "Replacement character". A quote from the wikipedia page about the special unicode characters:
The replacement character � (often a black diamond with a white question mark) is a symbol found in the Unicode standard at codepoint U+FFFD in the Specials table. It is used to indicate problems when a system is not able to decode a stream of data to a correct symbol. It is most commonly seen when a font does not contain a character, but is also seen when the data is invalid and does not match any character:
So it looks like your data source contains corrupted data. It is also possible that you try to read the data using the wrong encoding. Where do the lines come from?
If you can't fix the data, and your input indeed contains invalid characters, you could just remove the replacement characters:
lines[n] = lines[n].Replace("\xFFFD", "");

Mattmanser is right, never write a sql query by concatenating the parameters directly in the query. An example of parametrized query is:
string lastname = "Doe";
double height = 6.1;
DateTime date = new DateTime(1978,4,18);
var connection = new MySqlConnection(connStr);
try
{
connection.Open();
var command = new MySqlCommand(
"SELECT * FROM tblPerson WHERE LastName = #Name AND Height > #Height AND BirthDate < #BirthDate", connection);
command.Parameters.AddWithValue("#Name", lastname);
command.Parameters.AddWithValue("#Height", height);
command.Parameters.AddWithValue("#Name", birthDate);
MySqlDataReader reader = command.ExecuteReader();
...
}
finally
{
connection.Close();
}

To those who have a similar problem using PHP, try the function utf8_encode($string). It just works!

I have this some problem, when my website encoding is utf-u and I tried to send in form CP-1250 string (example taken by listdir dictionaries).
I think you must send string encoded like website.

Related

weird escape behaviour when writing string from node to mysql db

I'm on node and want to write this in my mysql db:
var x = JSON.stringify(['aa"a']);
console.log(x);
mysqlConnection.query("UPDATE `table` SET field = '" + x + "' WHERE id = 1");
The console.log() produces: ["aa\"a"]
When I read the string from the db later, I get: ["aa"a"]
The backslash is missing, making the string useless, as calling JSON.parse() would produce an error.
You're mashing your SQL together as a string. \ is an escape character (in SQL as well as JSON), so it escapes the " when passed to the SQL engine.
Use placeholders (whichever MySQL API library you are using should have a way of using them) instead of manually shoving variables into the string of SQL.

Regex doesn't find a match with a pattern

In a C# project, Regex is behaving weirdly for me.
I have this method:
string RegTest()
{
string HTML = "<input type=\"hidden\" name=\"authenticity_token\" value=\"d27956cca6b75db4d8dd502d0569dd246455131c\">";
Regex AuthRegex = new Regex(#"name=""authenticity_token"" value=""([A-Ba-b0-9/-]+)""");
string Auth = AuthRegex.Match(HTML).Value;
return Auth;
}
For a reason I don't understand at all, the Regex doesn't find any match with this pattern. It just returns "".
How can I fix this?
The problem is:
[A-Ba-b0-9/-]+
What character ranges (x-y) basically do is get a set of all characters in between. In other words, a-b = all letters between a and b, aka only a and b. However,
d27956cca6b75db4d8dd502d0569dd246455131c
looks like a hex. Therefore, you should use
[A-Fa-f0-9-]+
instead.

VB.NET: Populating ComboBox Using MySQL Query

I am trying to populate a Combobox from a MySQL database; but I don't get anything.
Below is the code.
Table: Class
Columns: Code, State
sqlstr = "SELECT * FROM Class WHERE State= Not Started"
DBCmd = New MySql.Data.MySqlClient.MySqlCommand(sqlstr, DBConn)
DBDR = DBCmd.ExecuteReader
While (DBDR.Read())
CB_Class.Items.Add(DBDR("Code"))
End While
DBCmd.Dispose()
DBDR.Close()
I believe the result is wrong, because there are at least 2 records with their state values set as "Not Started". What is wrong? Is there anything wrong with the way I've written "State= Not Started" ?
The command text doesn't seems correct. It lacks the single quote around the string to search
sqlstr = "SELECT * FROM Class WHERE State='Not Started'"
^ ^
If State is a string field then every search over it should be enclosed in single quotes.
Beware of the potential problems when the string to search contains a single quote.
In this simple case you could use directly the string constant, but if you render your search dynamic with user input then you should use parametrized queries.

Incrementing numerical value and changing string in SQL

I have a database that has stored values in a complicated, serialized array where one component is a string and another is the length of the characters of the string, in this format:
s:8:"test.com"
Where "s" holds the character length of the string in the quotations.
I would like to change the string from "test.com" to "testt.com", and I'm using the following statement in SQL:
UPDATE table SET row=(REPLACE (row, 'test.com','testt.com'))
However, this breaks the script in question, because it doesn't update the character length in the "s" preceding the string where "test.com" is stored.
I was wondering if there is a query I can use that would replace the string, and then also increment the value of this "s" preceding to where the replacement occurs, something like this:
UPDATE table SET row=(REPLACE (row, 's:' number 'test.com','s:' number+1 'testt.com'))
Does anyone know if this kind of query is even possible?
UPDATE table set row = concat('s:',length('testt.com'),':"testt.com"');
If you need to change exact string, then use exact query -
UPDATE table SET row = 's:9:"testt.com"' WHERE row = 's:8:"test.com"';
The string is a "serialized string".
If there are multiple strings to be replaced, it might be easier to create a script to handle this.
In PHP, it goes something like this:
$searchfor = serialize('test.com');
$replaceby = serialize('testt.com');
// strip last semicolon from serialized string
$searchfor = trim($searchfor,';');
$replaceby = trim($replaceby,';');
$query = "UPDATE table SET field = '$replaceby' WHERE field = '$searchfor';";
This way, you can create an exact query string with what you need.
Do fill in the proper code for db connection if necessary.

How to escape single quotes in MySQL

How do I insert a value in MySQL that consist of single or double quotes. i.e
This is Ashok's Pen.
The single quote will create problems. There might be other escape characters.
How do you insert the data properly?
Put quite simply:
SELECT 'This is Ashok''s Pen.';
So inside the string, replace each single quote with two of them.
Or:
SELECT 'This is Ashok\'s Pen.'
Escape it =)
' is the escape character. So your string should be:
This is Ashok''s Pen
If you are using some front-end code, you need to do a string replace before sending the data to the stored procedure.
For example, in C# you can do
value = value.Replace("'", "''");
and then pass value to the stored procedure.
See my answer to "How to escape characters in MySQL"
Whatever library you are using to talk to MySQL will have an escaping function built in, e.g. in PHP you could use mysqli_real_escape_string or PDO::quote
Use this code:
<?php
$var = "This is Ashok's Pen.";
mysql_real_escape_string($var);
?>
This will solve your problem, because the database can't detect the special characters of a string.
If you use prepared statements, the driver will handle any escaping. For example (Java):
Connection conn = DriverManager.getConnection(driverUrl);
conn.setAutoCommit(false);
PreparedStatement prepped = conn.prepareStatement("INSERT INTO tbl(fileinfo) VALUES(?)");
String line = null;
while ((line = br.readLine()) != null) {
prepped.setString(1, line);
prepped.executeQuery();
}
conn.commit();
conn.close();
There is another way to do this which may or may not be safer, depending upon your perspective. It requires MySQL 5.6 or later because of the use of a specific string function: FROM_BASE64.
Let's say you have this message you'd like to insert:
"Ah," Nearly Headless Nick waved an elegant hand, "a matter of no importance. . . . It's not as though I really wanted to join. . . . Thought I'd apply, but apparently I 'don't fulfill requirements' -"
That quote has a bunch of single- and double-quotes and would be a real pain to insert into MySQL. If you are inserting that from a program, it's easy to escape the quotes, etc. But, if you have to put that into a SQL script, you'll have to edit the text (to escape the quotes) which could be error prone or sensitive to word-wrapping, etc.
Instead, you can Base64-encode the text, so you have a "clean" string:
SWtGb0xDSWdUbVZoY214NUlFaGxZV1JzWlhOeklFNXBZMnNnZD
JGMlpXUWdZVzRnWld4bFoyRnVkQ0JvWVc1a0xDQWlZU0J0WVhS
MFpYCklnYjJZZ2JtOGdhVzF3YjNKMFlXNWpaUzRnTGlBdUlDNG
dTWFFuY3lCdWIzUWdZWE1nZEdodmRXZG9JRWtnY21WaGJHeDVJ
SGRoYm5SbApaQ0IwYnlCcWIybHVMaUF1SUM0Z0xpQlVhRzkxWj
JoMElFa25aQ0JoY0hCc2VTd2dZblYwSUdGd2NHRnlaVzUwYkhr
Z1NTQW5aRzl1SjMKUWdablZzWm1sc2JDQnlaWEYxYVhKbGJXVn
VkSE1uSUMwaUlBPT0K
Some notes about Base64-encoding:
Base64-encoding is a binary encoding, so you'd better make sure that you get your character set correct when you do the encoding, because MySQL is going to decode the Base64-encoded string into bytes and then interpret those. Be sure base64 and MySQL agree on what the character encoding is (I recommend UTF-8).
I've wrapped the string to 50 columns for readability on Stack Overflow. You can wrap it to any number of columns you want (or not wrap at all) and it will still work.
Now, to load this into MySQL:
INSERT INTO my_table (text) VALUES (FROM_BASE64('
SWtGb0xDSWdUbVZoY214NUlFaGxZV1JzWlhOeklFNXBZMnNnZD
JGMlpXUWdZVzRnWld4bFoyRnVkQ0JvWVc1a0xDQWlZU0J0WVhS
MFpYCklnYjJZZ2JtOGdhVzF3YjNKMFlXNWpaUzRnTGlBdUlDNG
dTWFFuY3lCdWIzUWdZWE1nZEdodmRXZG9JRWtnY21WaGJHeDVJ
SGRoYm5SbApaQ0IwYnlCcWIybHVMaUF1SUM0Z0xpQlVhRzkxWj
JoMElFa25aQ0JoY0hCc2VTd2dZblYwSUdGd2NHRnlaVzUwYkhr
Z1NTQW5aRzl1SjMKUWdablZzWm1sc2JDQnlaWEYxYVhKbGJXVn
VkSE1uSUMwaUlBPT0K
'));
This will insert without any complaints, and you didn't have to manually-escape any text inside the string.
You should escape the special characters using the \ character.
This is Ashok's Pen.
Becomes:
This is Ashok\'s Pen.
If you want to keep (') apostrophe in the database use this below code:
$new_value = str_replace("'","\'", $value);
$new_value can store in database.
You can use this code,
<?php
$var = "This is Ashok's Pen.";
addslashes($var);
?>
if mysqli_real_escape_string() does not work.
In PHP, use mysqli_real_escape_string.
Example from the PHP Manual:
<?php
$link = mysqli_connect("localhost", "my_user", "my_password", "world");
/* check connection */
if (mysqli_connect_errno()) {
printf("Connect failed: %s\n", mysqli_connect_error());
exit();
}
mysqli_query($link, "CREATE TEMPORARY TABLE myCity LIKE City");
$city = "'s Hertogenbosch";
/* this query will fail, cause we didn't escape $city */
if (!mysqli_query($link, "INSERT into myCity (Name) VALUES ('$city')")) {
printf("Error: %s\n", mysqli_sqlstate($link));
}
$city = mysqli_real_escape_string($link, $city);
/* this query with escaped $city will work */
if (mysqli_query($link, "INSERT into myCity (Name) VALUES ('$city')")) {
printf("%d Row inserted.\n", mysqli_affected_rows($link));
}
mysqli_close($link);
?>
$var = mysqli_real_escape_string($conn, $_POST['varfield']);
If you are using PHP, just use the addslashes() function.
PHP Manual addslashes
For programmatic access, you can use placeholders to automatically escape unsafe characters for you.
In Perl DBI, for example, you can use:
my $string = "This is Ashok's pen";
$dbh->do("insert into my_table(my_string) values(?)",undef,($string));
Use either addslahes() or mysql_real_escape_string().
The way I do, by using Delphi:
TheString to "escape":
TheString=" bla bla bla 'em some more apo:S 'em and so on ";
Solution:
StringReplace(TheString, #39,'\'+#39, [rfReplaceAll, rfIgnoreCase]);
Result:
TheString=" bla bla bla \'em some more apo:S \'em and so on ";
This function will replace all Char(39) with "\'" allowing you to insert or update text fields in MySQL without any problem.
Similar functions are found in all programming languages!
Maybe you could take a look at function QUOTE in the MySQL manual.
This is how my data as API response looks like, which I want to store in the MYSQL database. It contains Quotes, HTML Code , etc.
Example:-
{
rewardName: "Cabela's eGiftCard $25.00",
shortDescription: '<p>adidas gift cards can be redeemed in over 150 adidas Sport Performance, adidas Originals, or adidas Outlet stores in the US, as well as online at adidas.com.</p>
terms: '<p>adidas Gift Cards may be redeemed for merchandise on adidas.com and in adidas Sport Performance, adidas Originals, and adidas Outlet stores in the United States.'
}
SOLUTION
CREATE TABLE `brand` (
`reward_name` varchar(2048),
`short_description` varchar(2048),
`terms` varchar(2048),
) ENGINE=InnoDB AUTO_INCREMENT=6 DEFAULT CHARSET=latin1;
While inserting , In followed JSON.stringify()
let brandDetails= {
rewardName: JSON.stringify(obj.rewardName),
shortDescription: JSON.stringify(obj.shortDescription),
term: JSON.stringify(obj.term),
}
Above is the JSON object and below is the SQL Query that insert data into MySQL.
let query = `INSERT INTO brand (reward_name, short_description, terms)
VALUES (${brandDetails.rewardName},
(${brandDetails.shortDescription}, ${brandDetails.terms})`;
Its worked....
If nothing works try this :
var res = str.replace(/'/g, "\\'");
var res = res.replace(/"/g, "\\\"");
It adds the \ escape character to all(every) occurrences of ' and "
Not sure if its the correct/professional way to fix the issue
I'm guessing it will work but in actual content, every single and double quotes will be replaced with \ character
As a Python user I replace the quote with a raw "'":
don_not_work_str = "This is Ashok's Pen."
work_str = don_not_work_str.replace("'", r"\'")
For Python,I tried many ways to escape '"', not work, cuz I have various input.
Finally, I just use these code, the data in database was same like the input.
tmp = val.replace('\\', r'\\')
ret = tmp.replace('"', r'\"')