I am trying to return an MD5 encoded string of a value from my database, but it just returns a blank result (not null, just blank).
I have tried just running this query and get the same result:
SELECT MD5('test');
I have tried restarting the MySQL server, MySQL Workbench, etc. But get the same result.
If I try running the same command on a different database/server, it returns the hash string just fine.
What am I doing wrong? Is there a setting I disabled on accident?
Prior to MySQL v5.5.3, MD5() returned a binary string.
By default, MySQL Workbench does not display binary strings (to avoid accidental misinterpretation); however it is possible to display binary string values in output grids: View > Edit > Preferences > SQL Editor > Treat BINARY/VARBINARY as nonbinary character string.
Alternatively, either upgrade your MySQL server or transcode the result to a non-binary character set:
SELECT CONVERT(MD5('test') USING utf8)
Related
I keep finding that MySQL Workbench shows query results as BLOB. e.g:
SELECT INET_NTOA(167773449) --> BLOB
If I select to 'view value' I can determine the text value is '10.0.5.9' but it's pretty irritating when I SELECT multiple rows and want to glance at the contents.
Is there a way around this or is it a limitation of the tool?
Background:
This problem occurs when the binary string values (BINARY/VARBINARY type) are returned in the results. The binary strings contain the zero bytes and for some reason, apparently security, have not been shown by default. More details about binary strings here.
Even in the reported example SELECT INET_NTOA(167773449), the function returns binary string. Check this for reference.
Solution:
Since MySQL Workbench v5.2.22, it can be set through preferences whether to SHOW or HIDE such values.
In MySQL Workbench, go to: "Edit -> Preferences... -> SQL Queries" OR "Edit -> Preferences... -> SQL Editor -> SQL Execution" (depending upon what version of Workbench you have).
Check the option 'Treat BINARY/VARBINARY as nonbinary character string' to show the actual value.
Reference:
The original issue has been reported and answered with fix here.
What you can do is Cast your BLOB type to a string. This will simply allow you to glance at whats in your BLOB type when browsing your select statement.
SELECT CAST(`blob_column_name` AS CHAR(10000) CHARACTER SET utf8) FROM `table_name`;
Another (slightly shorter) solution
SELECT CONVERT(FROM_BASE64(myMEDIUMTEXTcol) using utf8) notAblobject FROM myTable;
Unrelated background
My table contains a MEDIUMTEXT column which stores a Base64 string. In my frontend JS code I kept getting an Object returned instead of a string. The Object appeared to contain an array of ASCII values.
Testing the associated query in phpMyAdmin indicated the result was a BLOB. I updated my SQL query, as above, in order to get a simple string returned instead of an Object.
I have a MySQL table with a VARCHAR(100) column, using the utf8_general_ci collation.
I can see rows where this column contains arbitrary byte sequences (i.e. data that contains invalid UTF8 character sequences), but I can't figure out how to write an UPDATE or INSERT statement that allows this type of data to be entered.
For example, I've tried the following:
UPDATE DataTable SET Data = CAST(BINARY(X'16d7a4fca7442dda3ad93c9a726597e4') AS CHAR(100)) WHERE Id = 1;
But I get the error:
Incorrect string value: '\xFC\xA7D-\xDA:...' for column 'Data' at row 1
How can I write an INSERT or UPDATE statement that bypasses the destination column's collation, allowing me to insert arbitrary byte sequences?
Have you considered using one of the Blob data types instead of varchar? I believe that this'd take a lot of the pain away from your use-case.
EDIT: Alternatively, there is the HEX and UNHEX functions, which MySQL supports. Hex takes either a str or a numeric argument and returns the hexadecimal representation of your argument as a string. Unhex does the inverse; taking a hexadecimal string and returning a binary string.
The short answer is that it shouldn't be possible to insert values with invalid UTF8 characters into VARCHAR column declared to use UTF8 characterset.
That's the design goal of MySQL, to disallow invalid values. When there's an attempt to do that, MySQL will return either an error or a warning, or (more leniently?) silently truncate the supplied value at the first invalid character encountered.
The more usual variety of characterset issues are with MySQL performing a characterset conversion when a characterset conversion isn't required.
But the issue you are reporting is that invalid characters were inserted into a UTF8 column. It's as if a latin1 (ISO-8859) encoding was supplied, and a characterset conversion was required, but was not performed.
As far as working around that... I believe it was possible in earlier versions of MySQL. I believe it was possible to cast a value to BINARY, and then warp that in CONVERT( ... USING UTF8), and MySQL wouldn't perform a validation of the characterset. I don't know if that's still possible with the current MySQL Connectors.
If it is possible, then that's (IMO) a bug in the Connector.
The only way I can think of getting around that characterset check/validation would be to get the MySQL sever to trust the client, and determine that no check of the characterset is required. (That would also mean the MySQL server wouldn't be doing a characterset conversion, the client lying to the server, the client telling the server that it's supplying valid UTF8 characters.
Basically, the client would be telling the server "Hey server, I'm going to be sending UTF8 character encodings".
And the server says "Okay. I'll not do any characterset conversion then, since we match. And I'll just trust that what you send is valid UTF8".
And then the client mischievously chuckles to itself, "Heh, heh, I lied. I'm actually sending character encodings that aren't valid UTF8".
And I think it's much more likely to be able to achieve such mischief using prepared statements with the old school MySQL C API (mysql_stmt_prepare, mysql_stmt_execute), supplying nvalid UTF8 encodings as values for string bind parameters. (The onus is really on the client to supply valid values for bind parameters.)
You should base64 encode your value beforehand so you can generate a valid SQL with it:
UPDATE DataTable SET Data = from_base64('mybase64-encoded-representation-of-my-value') WHERE Id = 1;
I am working on encrypting text strings that contain sensitive data. I need to save these encrypted strings to a MySQL db. The strings are cipher text, and all character (printable ASCII to control characters to null are equally likely).
These strings are not long (< 40 characters). I am using Ruby 2.1 (no Rails) along with the encryptor gem with a custom salt and iv to do the encryption. The encryptor gem is a wrapper for Ruby's openssl.
For many strings this works fine. However, I have run into a small number of these strings that, once encrypted, contain illegal or improperly quoted characters. As a result, when the string is saved I get an error.
What is the best way to handle the encrypted value so it can be reliably saved to MySQL?
Here is my encryption command:
require 'encryptor'
encrypted_value = Encryptor.encrypt(#sensitive_string,
:key => #config["encryption"]["key"],
:iv = #config["encryption"]["iv"],
:salt => #config["encryption"]["salt"])
Here is the encrypted value:
encrypted_value: /:Z`߉Nc??"v'??\??؟??????Oa?jR
and a screenshot since some of the characters did not copy correctly:
MySQL update statement:
query = "UPDATE db.table
SET `key` = mysql_real_escape_string(#{encrypted_value})"
With the value in the query it looks like:
query UPDATE db.table
SET `key` = mysql_real_escape_string(/:Z`߉Nc??"v'??\??؟??????Oa?jR)
I have tried both the MySQL Quote and mysql_real_escape_string functions. I get the same error with both and also wrapping the encrypted_value in double and single quotes.
Then I get this error:
wrong number of arguments (0 for 2) (ArgumentError)
What is the best way to tackle this? Any advice is appreciated.
You can perform base64 encoding of encrypted string, and store the encoded value in DB.
When retrieving the value, you can decode it back to binary and decrypt it.
require "base64"
enc = Base64.encode64('Send reinforcements')
# -> "U2VuZCByZWluZm9yY2VtZW50cw==\n"
plain = Base64.decode64(enc)
# -> "Send reinforcements"
http://ruby-doc.org/stdlib-2.2.0/libdoc/base64/rdoc/Base64.html
I encrypted my password field and inserted it into a MySQL table.
After that I decrypted that column by
select fld_user,fld_pwd,AES_DECRYPT(fld_encryptedpwd,'key')
from users
where fld_id='1903';
But, the result is showing "BLOB". I used Varbinary() as the datatype for encrypted column. What I should do ?
Solution for in MySQL Workbench is to toggle the following option for the SQL Editor: "Treat BINARY/VARBINARY as nonbinary character string." At least on MacOS X, you'll need to restart Workbench for the option to take effect. (You can also right click the value and do 'Open Value in Viewer').
If you don't want to change the options in Workbench, you can use the CAST() function to return the result of AES_DECRYPT() as a string:
SELECT fld_user, fld_pwd, CAST(AES_DECRYPT(fld_encryptedpwd, 'key') AS CHAR)
FROM users WHERE fld_id='1903';
A trick to check/make sure what data type a function would return is to do the following using the MySQL CLI:
mysql> CREATE TALBE tmp1 AS SELECT AES_DECRYPT(fld_encryptedpwd, 'key')
FROM users WHERE fld_id='1903';
mysql> DESC tmp1;
That will show which data type will be returned.
(Again, storing encrypted passwords: not so good.)
Here's my scenario. I save a bunch of Strings containing asian characters in MySQL using Hibernate. These strings are written in varbinary columns. Everything works fine during the saving operation. The DB contains the correct values (sequence of bytes). If I query (again using Hibernate) for the Strings that I saved I get the correct results. But when Hibernate fills the entity to which the Strings belong with the values from the DB I get different values then the ones I used in the query that retrieved them. Instead of receiving the correct values I receive a bunch of FFFD replacement characters.
For example: if I store "하늘" in the DB and then I query for it, the resulting String will be \uFFFD\uFFFD\uFFFD\uFFFD\uFFFD\uFFFD.
the DB connection has the following parameters set useUnicode=true&characterEncoding=UTF-8,
I've tried using the following configurations for Hibernate but that didn't solve the problem:
- connection.useUnicode = true
- connection.characterEncoding = UTF-8
By the way, this all works fine if the MySQL columns are of type varchar.
What am I missing? Any suggestions?
Thanks
Set the connection character set to be binary too:
SET NAMES 'binary';