I've a db table with a varchar(64) field to store PHashing data, as 64 chars (1's and 0's ascii characters). I need to calculate hamming distance with a test hasta, and it seems that the most efficient way to do is using mysql bit_count function. The problem is that I haven't found any way to convert/cast/whatever the hash field in order to be interpreted as a byte(8) instead of varchar(64). Like:
> select hash from data;
"0000000101100111111100011110000011100000111100011011111110011011"
> select convert_to_binary(hash) from data;
0b0000000101100111111100011110000011100000111100011011111110011011
I cannot alter the data and convert all previous data into a binary field. Is there any way to force mysql to re-interpret a field as a literal, or any other alternative solution?
I think you should be able to use it like this:
SELECT BIT_COUNT(CAST(hash AS BINARY)) FROM data;
Use CAST to convert the field to BINARY and pass the result to BIT_COUNT. Casting the field hash to binary will turn it into a string with the binary data type, but BIT_COUNT is able to handle that.
Working code:
SELECT BIT_COUNT( CONV( hash, 2, 10 ) ^
0b0000000101100111111100011110000011100000111100011011111110011011 )
I am using mysql's GROUP_CONCAT() for one of my query. But I am getting data in bytes.
To convert the bytes into VARCHAR I used the CONVERT() function.
But I dont see any difference even after that. I am still getting data in bytes.
Below is what I have tried:
CONVERT(GROUP_CONCAT((T1.CASE NODE_ID WHEN 103 THEN CONCAT(T1.CUSTOMER_ATTRIBUTE,'~',T2.DISPLAY_TEXT) ELSE NULL END)) USING LATIN1)AS HEADER,
but I am getting the output as:
HEADER
----------
Name [4B]
Help needed.
Thanks.
Use like this CONCAT(GROUP_CONCAT(your_column)) this will give as the string as output
cheers..
This is because different types are passed into CONCAT function. MySQL converts them into appropriate type, in your case it is BLOB or something else.
The solution is to convert all non-string types to string, or just convert result of CONCAT function to string once. For example -
SELECT CAST(CONCAT(id, '~', column1, column2) AS CHAR)
I am using a 3rd party .NET library (Rhino Security) that stores it's identifiers as guids in binary(16) fields in my mysql db. Everything works perfectly from the application but when I attempt to manually run a query via a query editor (TOAD for mysql) no rows are returned for identifiers I know to exist. For instance, if i run the following query, i get no results:
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
where EntitySecurityKey = '02a36462-49b7-406a-a3b6-d5accd6695e5'
Running the same query with no filter returns many results, including one with the above GUID in the EntitySecurityKey field. Is there another way to write a query to search on a guid/binary field?
Thanks!
EDIT
I found it interesting that TOAD returned a string and not an ugly blob. Using a different editor to return the results (for the unfiltered query) I get the raw binary data. I would have assume that my query would work using the binary keyword but neither of the following worked:
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
where EntitySecurityKey = BINARY '02a36462-49b7-406a-a3b6-d5accd6695e5'
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
where BINARY EntitySecurityKey = '02a36462-49b7-406a-a3b6-d5accd6695e5'
I am not familiar with Rhino Security, but it uses NHibernate to store to the database as far as I know.
The .net Guid uses 1 Int32, 2 Int16 and 8 Bytes internally to store the 128-bit Guid value.
When NHibernate (3.2) stores/retrieves a Guid value to/from a BINARY(16) column, it uses the .Net ToByteArray() method and Guid(Byte[] ...) constructor respectively. These methods basically swap the order of the bytes for the Int32 and Int16 values. You can reflect on the methods to see the exact code, but here is a simple example of the first 4 bytes:
guidBytes[0] = (byte)this._int32member;
guidBytes[1] = (byte)(this._int32member >> 8),
guidBytes[2] = (byte)(this._int32member >> 16),
guidBytes[3] = (byte)(this._int32member >> 24);
This may be the cause of your issue.
For example, your Guid is stored in MySql differently.
02a36462-49b7-406a-a3b6-d5accd6695e5 - Your Guid from Guid.ToString()
6264a302-b749-6a40-a3b6-d5accd6695e5 - Your Guid in MySql using MySql hex(Guid)
Note: hex(x) does not add the dashes; I added dashes manually for comparison purposes.
Notice that the order of the first 8 bytes is different.
To test if this is your issue, run the following query:
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
where EntitySecurityKey = UNHEX(REPLACE(6264a302-b749-6a40-a3b6-d5accd6695e5,'-',''));
If so, some other items I have noticed in my own travels:
Toad for MySql complicates this too, since it will display the Guid as the .net equivalent in a select statements output. It does not apply the same conversion if you include that Guid in a select's where clause (at least from my experience). You would need to use the Guid as stored by MySql.
You will either need to manually convert the Guids when performing DB queries or you may need to introduce a custom NHibernate type to convert the Guid differently.
You can refer to:
How to read a .NET Guid into a Java UUID
http://pastebin.com/M07wnK5K (I haven't used this code, but found it useful for reference)
Update:
I just tried the custom NH type on the pastebin link above. The byte order is incorrect, at least for my environment. I would need:
private static readonly int[] ByteOrder = new[] { 3, 2, 1, 0, 5, 4, 7, 6, 8, 9, 10, 11, 12, 13, 14, 15 };
to make by .Net output match the select hex(Guid column) from table output.
What strikes me as interesting is that you're storing your GUIDs in fields that are binary(16), emphasis on the 16. Per the manual, a binary field's length is in bytes and will truncate anything that goes over it (only in strict mode though). Is it possible that your GUID's are being truncated? With your sample GUID, 02a36462-49b7-406a-a3b6-d5accd6695e5, try querying the database with the first 16 characters:
WHERE EntitySecurityKey = '02a36462-49b7-40'
Per the accepted answer to this question, a field should be char(16) binary to store a GUID, not just binary(16). However, I couldn't get this to work in my sample table.
What did work for me were using char(36) and also binary(36). Try updating your field-lengths to 36 instead of 16 (I'd do this on a test-table first, just to be safe).
Here was my test table:
CREATE TABLE test_security_keys (
test_key1 char(16) not null,
test_key2 char(16) binary not null,
test_key3 char(36) not null,
test_key4 binary(16) not null,
test_key5 binary(36) not null
);
Then, I ran a script to insert numerous GUIDs (the same one for each column in a row). You can test it with your sample GUID:
INSERT INTO test_security_keys
VALUES ('02a36462-49b7-406a-a3b6-d5accd6695e5', '02a36462-49b7-406a-a3b6-d5accd6695e5', '02a36462-49b7-406a-a3b6-d5accd6695e5', '02a36462-49b7-406a-a3b6-d5accd6695e5', '02a36462-49b7-406a-a3b6-d5accd6695e5');
Using a simple SELECT * FROM test_security_keys will show all of the columns except the ones with size 36 to be truncated. Also, binary or not, I was able to successfully query the columns with a regular string-comparison:
SELECT * FROM test_security_keys WHERE test_key3 = '02a36462-49b7-406a-a3b6-d5accd6695e5';
SELECT * FROM test_security_keys WHERE test_key5 = '02a36462-49b7-406a-a3b6-d5accd6695e5';
If you've confirmed that your current columns with binary(16) aren't truncating, I would then-suggest to use a CAST() in your WHERE clause. The following should work (with your sample query):
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
WHERE
EntitySecurityKey = CAST('02a36462-49b7-406a-a3b6-d5accd6695e5' AS binary(16));
If you CAST(.. AS binary(16)) and the input-data is longer than 16 bytes, MySQL should issue a warning (should be unseen and not affect anything) stating that it had to truncated the data (try SHOW WARNINGS; if you get them). This is to be expected, but also means that you can't use binary(16) to store the GUIDs and you'll need to use binary(36) or char(36).
* I have not tried any of this using TOAD, but I've used both command-line MySQL and Navicat and have the same results for both.
I came across this question when I was evaluating how others "searched" for records where the primary key is a binary type (mainly the binary(16) indicative of GUID's) to compare it to my own. I must admit I was a little taken a back that the answers quickly diverged from the users desire to do manual searches based on the binary field but instead focused on changing the column types themselves to characters which, in this case, is murder on InnoDB storage.
An attempted solution for the lack of results would be to search against the Id column instead of the EntitySecurityKey column. Given JP's original query, the modified, working version would be:
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
where Id = UNHEX(REPLACE('02a36462-49b7-406a-a3b6-d5accd6695e5', '-', ''));
Again, this assumes the Id field is the binary(16) type. (I wasn't able to determine which column: Id, EntitySecurityKey was the binary(16)). If the EntitySecurityKey is the binary(16) column, then:
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
where EntitySecurityKey = UNHEX(REPLACE('02a36462-49b7-406a-a3b6-d5accd6695e5', '-', ''));
If that failed to yield the desired results, per this question (How to read a .NET Guid into a Java UUID), it would be of interest to try the same query but with the endianness of the first three parts of the GUID reversed:
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
where Id = UNHEX(REPLACE('6264a302-b749-6a40-a3b6-d5accd6695e5', '-', ''));
The only other thing I can think of is that EntitySecurityKey is a virtual column based on Id, but I don't have enough information on the table set-up to validate this statement.
Cheers.
From your EDIT, which I've a bit modifed:
Please, try
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
WHERE EntitySecurityKey = X'02a3646249b7406aa3b6d5accd6695e5'
Since the strings have 16 hexadecimal byes each, it may works...
Have you checked what the GUID data actually looks like in the table? Since it's the primary key, it quite possibly isn't stored as a GUID string. If that's the case querying the string value clearly won't work.
Possibly it's in binary format? Or possibly it's a string but without the hyphens? Or maybe it's been converted some other way? I don't know without seeing your data. But whatever the case, if a GUID that you know exists isn't being found when you query it, then whatever format it is in, it clearly isn't stored in the format you're querying.
The way I would solve this would be by searching for a different value in a record that you know exists. Or even just query the top few records without a where clause at all.
This will show you what the data actually looks like. With any luck, that will be sufficient for you to work out how it's been formatted.
If that doesn't help, one possible clue might come from this question: Why are binary Guids different from usual representation
Hope that helps.
Not sure how your database is set up, but I would try these two variants:
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
where EntitySecurityKey = x'02a3646249b7406aa3b6d5accd6695e5'
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
where EntitySecurityKey = x'6264a302b7496a40b6a3d5accd6695e5'
What really is the difference between MySQL UNHEX and X when dealing with hexadecimal values in a database?
Eg.
SELECT * FROM test WHERE guidCol IN (UNHEX('hexadecimalstring'));
SELECT * FROM test WHERE guidCol IN (X'hexadecimalstring');
Both gives me exact result set. So is there any difference? Performance implications?
Edit: the underlying type of guidCol is binary of course
UNHEX() is a function, therefore you can do something like
SET #var = '41';
SELECT UNHEX(#var);
SELECT UNHEX(hex_column) FROM my_table;
X, on the other hand, is the syntax for a hexadecimal litteral. You cannot do this:
SET #var = '41';
SELECT X#var; -- error (string litteral expected)
SELECT X'#var'; -- error (`#` is not a hexadecimal digit)
SELECT X(#var); -- returns NULL, not too sure about the reason... [edit: but this is probably why you are inserting NULL values]
SELECT X(hex_column) FROM my_table; -- returns NULL as well
This explains why you always get better performance with X: you are using a language construct instead of a function call. X does not need to evaluate a variable, since it expects a litteral string.
Note that even in MySQL 5.6, the X'' notation has a length limit in the reference mysql client and UNHEX() does not (appear to). I do not know what the limit is for X'', as it is not officially documented but I have encountered it when trying to INSERT a BLOB. With X'' literal, mysql client threw a syntax error with a sufficiently long hex sequence while UNHEX() of the same sequence did not. Obviously, length is not an issue when it comes to an actual GUID, but I figured this is useful for anyone else using this question to answer mysql insertion of binary data in the general case.
Suppose I have a query
SELECT * FROM table WHERE x > y;
where "y" is a user-input value. I'd like to enclose y in quotes. So far as I've tested, the query works properly when the quotes are added. Is the behaviour defined? Is it known to result in an efficiency drop?
Note that the query is an example fabricated for simplicity. Also, this is not an attempt to deal with SQL injection.
If the field that you are trying to compare with is a numeric field ie int, then all the possible combinations must work in mysql
select * from users where id = 20;
select * from users where id = "20";
select * from users where id = '20';
Refer the MySQL DOC for more depth
The conversion behavior of MySQL when comparing different types is well-defined. When a string and a number are compared, both are converted to floats. It's outlined in the manual, ยง 11.2.
IMO, you can try :-
cast( "-10" as signed )
This is casting in mysql (the function name is cast, obvious?).