I just upgraded the MySQL that I am using to MySQL version 5.6.14. When I issued this query,
SELECT FROM_BASE64(TO_BASE64('MySQL'));
I received 4d7953514c (hex value) as the answer instead of 'MySQL'. What is actually the problem? Is there anything that I have to do to unhex it?
NOTE: The UNHEX function in my MySQL also returns the same thing. If a hex value is given to UNHEX function, I will receive the same hex value again.
TQ in advance
MySQL client displays binary data using hexadecimal notation by default. The result of TO_BASE64 is a VAR_STRING with binary collation and, given that the original collation/character set has been already lost, FROM_BASE64 has no way of guessing what character set was used when producing those Bytes. As such, this particular client application (others may do it differently) chooses the safer route and just display their hexadecimal notation.
You can change this behavior by providing the option --skip-binary-as-hex when starting the mysql client. Just mind that if you don't know the source of the data this yield unexpected results. See the documentation for --binary-as-hex for additional ways of how to workaround this.
I'd also recommend turning on display result set metadata as it allows you to better understand what is causing this behavior.
Related
I'm searching for cases in MySQL/MariaDB where the value transmitted when storing will differ from the value that can be retrieved later on. I'm only interested in fields with non-binary string data types like VARCHAR and *TEXT.
I'd like to get a more comprehensive understanding on how much a stored value can be trusted. This would especially be interesting for cases where the output just lacks certain characters (like with the escape character example below) as this is specifically dangerous when validating.
So, this boils down to: Can you create an input string (and/or define an environment) where this doesn't output <value> in the second statement?
INSERT INTO t SET v = <value>, id = 1; // success
SELECT v FROM t WHERE id = 1;
Things I can think of:
strings containing escaping (\a → a)
truncated if too long
character encoding of the table not supporting the input
If something fails silently probably also depends on how strict the SQL mode is set (like with the last two examples).
Thanks a lot in advance for your input!
you can trust that all databases do, what the standards purpose, with strings and integer it is simple, because it saves the binary representation of that number or character in your choosen character set.
Decimal Double and single values are different, because the can't be saved directly and so it comes to fractals see decimal representation
That also follows standards, but you have to account with it.
If I execute a query against the MySQL Connector/C library the data I'm getting back all appears to be in straight char * format, including numerical data types.
For example, if I execute a query that returns 4 columns, all of which are INTEGER in MySQL, rather than getting back 4 bytes worth of data (each byte representing a single column row value), I'm actually getting back 4 ASCII encoded character bytes, where 1 is actually a byte with the numeric value 49 in it (ASCII for 1).
Is this accurate or am I just missing something complete?
Do I really need to then atoi that returned byte into an int in my code or is there a mechanism to get the native C data types out of the MySQL client directly?
I guess my real question is: is the mysql_store_result structure converting that data to ASCII encoded representations in a way that can be bypassed by my application code?
I believe the data is sent on the wire as text in the MySQL protocol (I just confirmed this with Wireshark). So that means mysql_store_result() is not converting the data, it's just simply passing the data on as it was received. MySQL actually sends integers as text. I agree this always seemed like an odd design to me as well.
MySQL originally only offered the Text Protocol that you are currently using, in which (as you note) results are encoded as strings. MySQL v4.1 (released in April 2003) introduced the Prepared Statement protocol, which (amongst other things) transmits results in a binary format.
See C API Prepared Statements for more information on how to use the latter protocol with Connector/C.
In mysql, if I do something like
round((amount * '0.75'),2)
it seem to work just fine like without single quotes for 0.75. Is there a difference in how mysql process this?
In the hope to close out this question, here's a link that explains type conversion in expression evaluation: https://dev.mysql.com/doc/refman/5.5/en/type-conversion.html
When an operator is used with operands of different types, type
conversion occurs to make the operands compatible. Some conversions
occur implicitly. For example, MySQL automatically converts numbers to
strings as necessary, and vice versa.
mysql> SELECT 1+'1';
-> 2
In your case, MySQL sees arithmetic and performs implicit conversion on any string contained in the expression. There is going to be an overheard in converting a string to number, but it's negligible. My preference is to explicitly type out a number instead of quoting it. That method has helped me in code clarity and maintainability.
I have a weird encoding problem from my PyQt app to my mysql database.
I mean weird in the sense that it works in one case and not the other ones, even though I seem to be doing the exact same thing for all.
My process is the following:
I have some QFocusOutTextEdit elements in which I write text possibly containing accents and stuff (é,à,è,...)
I get the text written with :
text = self.ui.text_area.toPlainText()
text = text.toUtf8()
Then to insert it in my database I do :
text= str(text).decode('unicode_escape').encode('iso8859-1').decode('utf8')
I also set the character set of my database, the specific tables and the specific columns of the table to utf8.
It is working for one my text areas, and for the other ones it puts weird characters instead in my db.
Any hint appreciated on this !
RESOLVED :
sorry for the disturbance, apparently I had some fields in my database that weren't up to date and this was blocking the process of encoding somehow.
You are doing a lot of encoding, decoding, and reencoding which is hard to follow even if you know what all of it means. You should try to simplify this down to just working natively with Unicode strings. In Python 3 that means str (normal strings) and in Python 2 that means unicode (u"this kind of string").
Arrange for your connection to the MySQL database to use Unicode on input and output. If you use something high-level like Sqlalchemy, you probably don't need to do anything. If you use MySQLdb directly make sure you pass charset="UTF8" (which implies use_unicode) to the connect() method.
Then make sure the value you are getting from PyQT is a unicode value. I don't know PyQT. Check the type of self.ui.text_area or self.ui.text_area.toPlainText(). Hopefully it is already a Unicode string. If yes: you're all set. If no: it's a byte string which is probably encoded in UTF-8 so you can decode it with theresult.decode('utf8') which will give you a Unicode object.
Once your code is dealing with all Unicode objects and no more encoded byte strings, you don't need to do any kind of encoding or decoding anymore. Just pass the strings directly from PyQT to MySQL.
Using SQL Server Integration Services (SSIS) to perform incremental data load, comparing a hash of to-be-imported and existing row data. I am using this:
http://ssismhash.codeplex.com/
to create the SHA512 hash for comparison. When trying to compare data import hash and existing hash from database using a Conditional Split task (expression is NEW_HASH == OLD_HASH) I get the following error upon entering the expression:
The data type "DT_BYTES" cannot be used with binary operator "==". The type of one or both of the operands is not supported for the operation. To perform this operation, one or both operands need to be explicitly cast with a cast operator.
Attempts at casting each column to a string (DT_WSTR, 64) before comparison have resulted in a truncation error.
Is there a better way to do this, or am I missing some small detail?
Thanks
Have you tried expanding the length beyond 64? I believe DT_BYTES is valid up to 8000 characters. I verified the following are legal cast destinations for DT_BYTES based on the books online article:
DT_I4
DT_UI4
DT_I8
DT_UI8
DT_STR
DT_WSTR
DT_GUID
DT_IMAGE
I also ran a test in BIDS and verified it had no problem comparing the values once I cast them to a sufficiently long data type.
SHA512 is a bit much as your chances of actually colliding are 1 in 2^256. SHA512 always outputs 512 bits which is 64 bytes. I have a similar situation where I check the hash of an incoming binary file. I use a Lookup Transformation instead of a Conditional Split.
This post is older but in order to help other users...
The answer is that in SSIS you cannot compare binary data using the == operator.
What I've seen is that people will most often convert (and store) the hashed value as varchar or nvarchar which can be compared in SSIS.
I believe the other users have answered your issue with "truncation" correctly.