I am moving my data from SQL Server to MySQL. The data is moving without any issues, but I am trying to come up with a way to validate the integrity of the data between SQL Server and MySQL. I am having issues with regards datatypes VARCHAR and NVARCHAR. Since these are ranging from 255 to MAX. I am trying to do hash comparison between these datatypes but they are not matching.
For testing purpose I tried this:
mysql> SET #test := repeat('t',50000);
Query OK, 0 rows affected (0.00 sec)
mysql> select md5(#test);
+----------------------------------+
| md5(#test) |
+----------------------------------+
| a9cf60d59fe2fb94a84bc106eca028be |
+----------------------------------+
1 row in set (0.01 sec)
Trying the same in SQL Server gives something different:
DECLARE #Test NVARCHAR(MAX) = REPLICATE(CONVERT(NVARCHAR(MAX), N't'), 50000);
SELECT HashBytes('MD5', CONVERT(varchar, #Test));
0x3C48C287BC783516AC89297848A104FE
select LOWER(CONVERT(VARCHAR(32), HashBytes('MD5', CONVERT(varchar, #Test)), 2));
3c48c287bc783516ac89297848a104fe
SELECT CONVERT(VARCHAR(32),HashBytes('MD5', #Test),2);
40752EB301B41EEAEB309348CE9711D6
As you can see none of the output in SQL Server is not matching with MySQL. Hence, looking for some guidance here as to how can I do this data comparison.
One page as a unit of data storage for sql server database file is only 8KB and the limit for varchar LEN is 8000, so following your test example, you won't be able to get the hex result of 50000 characters per page. Instead, it is the result of your #Test example that is calculated on a string length of 8000 characters.
Related
I have table with:
1) Encrypted_ID varchar (256)
2) Initialization Vector(iv)varchar(256).
I would like to decrypt the column value using the key
I am using:
select Cast(AES_DECRYPT(Encrypted_ID,'Key',InitializationVector_iv)as CHAR ) as DecryptedValue from MyTable;
The result is Null.
I Also tried:
select Cast(AES_DECRYPT(AES_ENCRYPT(Encrypted_ID,'Key',InitializationVector_iv),'Key') as CHAR ) as DecryptedValue from MyTable;
The result is blob for few rows.
I'm not sure what I'm doing wrong here. Can any one help with the syntax to decrypt the column when I have:
Key
Initialization Vector value
Encrypted Column
There's actually nothing wrong with your first query, syntactically it's spot on as this worked example demonstrates.
mysql> SET ##SESSION.block_encryption_mode = 'aes-256-cbc';
mysql> create table MyTable(
-> Encrypted_ID varbinary(256),
-> InitializationVector_iv varbinary(16)
-> );
Query OK, 0 rows affected (0.93 sec)
mysql> SET #iv = RANDOM_BYTES(16);
Query OK, 0 rows affected (0.00 sec)
mysql> INSERT INTO MyTable SET Encrypted_ID = AES_ENCRYPT('hello','key', #iv), InitializationVector_iv = #iv;
Query OK, 1 row affected (0.17 sec)
mysql> SELECT CAST(AES_DECRYPT(Encrypted_ID,'key', InitializationVector_iv) AS CHAR) from MyTable;
+------------------------------------------------------------------------+
| CAST(AES_DECRYPT(Encrypted_ID,'key', InitializationVector_iv) AS CHAR) |
+------------------------------------------------------------------------+
| hello |
+------------------------------------------------------------------------+
1 row in set (0.00 sec)
As for why it's not working, I managed to get the query to return NULL in 2 scenarios. One, you get NULL returned if you use a different iv for encryption and decryption, so you might want to look at how you are storing as the iv. Two, you get NULL where you have the block_encryption_mode variable set differently when storing and trying to retrieve the value, check that you're not accidentally reverting to the default 'aes-128-ebc between sessions. There may be others...
The second query will fail because you need to supply the iv to both of he encryption and decryption functions, you only use it to encrypt. Also, since you are taking the values from the MyTable, Encrypted_ID will already be encrypted and the effect of this query would be to encrypt it again, before reversing that to get you back to the stored (encrypted) value.
Finally, AES is only going to use 16 bytes of the iv so you might as well make that VARBINARY(16).
AES doesn't work with MySQL Workbench in my case.
I have to use the mysql console.
MySql Workbench worked for me.
In my Case, encrypted value was encoded in base 64.
So I had to decode base 64 value and IV Using "From_base64" function.
SET block_encryption_mode = 'aes-256-cbc';
set #k = 'Key';
set #iv = From_base64('InitializationVector');
set #v = from_base64('EncryptedValue');
select CAST(AES_DECRYPT(#v, #k, #iv) AS CHAR);
Please make sure the encryption type, base 64 encoding, Hex/Unhex of the values/Iv are correct before you start working on the decryption.
Review MYSql functions
https://dev.mysql.com/doc/refman/8.0/en/string-functions.html
Hope this helps for someone.
In MySQL, an overflow creates only a warning and MySQL destroys the data you feed it:
mysql> create table tmp(data tinyint primary key);
Query OK, 0 rows affected (7.83 sec)
mysql> insert into tmp set data = 200;
Query OK, 1 row affected, 1 warning (2.27 sec)
mysql> select * from tmp;
+------+
| data |
+------+
| 127 |
+------+
1 row in set (0.00 sec)
This is a huge problem, especially when using MySQL via a programming language, where one (usually) does not check any MySQL-warnings. This may cause not only data-corruption, but data-corruption that is hidden and may remain undetected - one of the worst things that may happen in a database.
Is there a way to configure MySQL so that an overflow causes an ERROR (similar to a syntax error) and not only a warning?
See Mysql documentation
you have to activate the strict SQL mode:
If strict SQL mode is enabled, MySQL rejects the out-of-range value with an error, and the insert fails, in accordance with the SQL standard.
If no restrictive modes are enabled, MySQL clips the value to the appropriate endpoint of the range and stores the resulting value instead.
I've got a table which contains a varchar(1024) field, which in that contains strings which has hex encoded strings. This table is filled automatically and I have to provide an SP to allow users to download this, therefore, I need to change the hex back into human readable form.
If I manually run this statement (taking the Hex data from the field), it works just fine:
SELECT X'5468697320697320612074657374206D6573736167652031323334353637383930';
But I cannot find a working example of getting this to work when calling the field/column name. I've found a few examples, but these just return a null or 0.
I've tried X and UnHex() and neither give me a result.
Where am I going wrong?
Thanks
EDIT:
Okay, after doing a bit more testing, it appears it must be the way it's being written to the database in the first place.
It's a Classic ASP page that calls an SP, which creates the database entry. In this method, the write to the DB works, and I can see the HEX content in the field. Copying the content of the field, and putting this into a Select X'123123' gives me the ASCII values, as I want.
If I try this as a Select, this fails, giving me a zero or Null return.
SELECT Message_Body_Hex, UNHEX(Message_Body_Hex) FROM messages_inbound
returns:
Message_Body_Hex......unhex(Message_Body_Hex)
417265612032........(NULL)
Still confused! :)
I realize this is an old question but I ran into this same problem today and solved it using a combination of HEX and CAST. Using your example, try this:
SELECT HEX(CAST(X'5468697320697320612074657374206D6573736167652031323334353637383930' AS CHAR(33)))
When pulling from a table you'd substitute the field name:
SELECT HEX(CAST(binary_field AS CHAR(33)))
I've seen other answers recommending to use MAX in place of the 33 but this appears to work fine. Here are some sources I used:
SQL Server converting varbinary to string
and
How to convert from varbinary to char/varchar in mysql
Using the UNHEX() function seems to work fine on MySQL 5.5.29-1:
mysql> create table t1 ( f1 varchar(1024) );
Query OK, 0 rows affected (0.03 sec)
mysql> insert into t1 values('5468697320697320612074657374206D6573736167652031323334353637383930');
Query OK, 1 row affected (0.02 sec)
mysql> select f1 from t1;
+--------------------------------------------------------------------+
| f1 |
+--------------------------------------------------------------------+
| 5468697320697320612074657374206D6573736167652031323334353637383930 |
+--------------------------------------------------------------------+
1 row in set (0.00 sec)
mysql> select unhex(f1) from t1;
+-----------------------------------+
| unhex(f1) |
+-----------------------------------+
| This is a test message 1234567890 |
+-----------------------------------+
1 row in set (0.00 sec)
If I try to search for a value in mysql database and the string value contains dot in it, query returns 0 rows. Example:
SELECT * FROM table WHERE `username`='marco.polo' --> 0 rows
SELECT * FROM table WHERE `username` LIKE '%.polo%' --> 0 rows
SELECT * FROM table WHERE `username` LIKE 'polo' --> Success
This appeared after moving server and database to another place. I know that dot is a set of extended regular expressions, but it should not apply to equal nor LIKE operator, simply because I don't use REGEXP in query.
I've tested the same query on my local database and it works fine.
Could there be a special setting in mysql that treats dot differently than it usually does?
user1084605, I tried to replicate the problem (using MySQL version 5.1.37), but got exactly the opposite results as you. See below:
mysql> create table test (username varchar(100));
Query OK, 0 rows affected (0.01 sec)
mysql> insert into test values ('marco.polo');
Query OK, 1 row affected (0.00 sec)
mysql> SELECT * FROM test WHERE `username`='marco.polo';
+------------+
| username |
+------------+
| marco.polo |
+------------+
1 row in set (0.00 sec)
mysql> SELECT * FROM test WHERE `username` LIKE '%.polo%';
+------------+
| username |
+------------+
| marco.polo |
+------------+
1 row in set (0.00 sec)
mysql> SELECT * FROM test WHERE `username` LIKE 'polo';
Empty set (0.00 sec)
According to the MySQL docs, the only special characters when using the LIKE operator are "%" (percent: matches 0, 1, or many characters) and "_" (underscore: matches one and only one character).
http://dev.mysql.com/doc/refman/5.0/en/string-comparison-functions.html
A "." (period) does have special meaning for MySQL's REGEXP operator, but it should still match a literal period in your column.
http://dev.mysql.com/doc/refman/5.0/en/regexp.html
Can you replicate the SQL statements I ran above and paste your results in reply?
As #cen already mentioned, character set can causes that problem.
I have had this sample:
`email` VARCHAR(45) CHARACTER SET 'armscii8' NOT NULL,
this is was in the .sql dump, which I receive.
So, when I was trying to fetch object with this email
I couldn't get it.
The below query takes care of the scenario when we have only DOT operator in the columns.
SELECT * FROM test WHERE `username` LIKE '%.%';
I'm using MySQL in strict mode (SET sql_mode = 'STRICT_TRANS_TABLES') to convert all warnings to errors. However, I have a query that is expected to create warnings because it tries to convert a VARCHAR field that might be empty or contain letters to an integer.
Example:
mysql> select CAST("123b" AS SIGNED);
+------------------------+
| CAST("123b" AS SIGNED) |
+------------------------+
| 123 |
+------------------------+
1 row in set, 1 warning (0.00 sec)
mysql> show warnings;
+---------+------+-------------------------------------------+
| Level | Code | Message |
+---------+------+-------------------------------------------+
| Warning | 1292 | Truncated incorrect INTEGER value: '123b' |
+---------+------+-------------------------------------------+
1 row in set (0.00 sec)
Is there a way to suppress the warning caused by the CAST() without disabling strict mode? Or alternatively, can the strict mode be disabled for a single query or function (something like the # operator in PHP) without calling SET twice to temporarily switch off the strict mode?
Background: I have a table with street numbers. Most of them are numeric but some contain letters at the end. To implement a simplistic "natural sort" I'd like to use ORDER BY CAST (StreetNr AS SIGNED), StreetNr and the value returned by CAST() would be just fine for 1st level sorting.
I'm assuming the problem is that you are trying to insert data from one table into another using a query like this:
INSERT INTO ...
SELECT ...
FROM ...
WHERE ...
ORDER BY ...
And the inserts are failing because of the CAST() problem you described in your question.
Is that accurate?
If so, the easiest way around this is to use INSERT IGNORE. That syntax is useful for ignoring duplicate key errors, but it can also be used to ignore the CAST() errors that are affecting you.
Your updated query would look something like this:
INSERT IGNORE INTO target_table
SELECT ...
FROM source_table
WHERE ...
ORDER BY CAST (StreetNr AS SIGNED), StreetNr
I had the same issue with ordering and used regexp first:
ORDER BY CAST(CONCAT('0', REGEXP_REPLACE(StreetNr, '[^0-9]', '')) AS INTEGER)
The concat-0 is to handle an empty string.
(My server is MariaDB, but it should be the same as MySQL.)