Can anyone help me understand the following problem with a BIT(64) column in MySQL (5.7.19).
This simple example works fine and returns the record from the temporary table:
CREATE TEMPORARY TABLE test (v bit(64));
INSERT INTO test values (b'111');
SELECT * FROM test WHERE v = b'111';
-- Returns the record as expected
When using all the 64 bits of the column it no longer works:
CREATE TEMPORARY TABLE test (v bit(64));
INSERT INTO test values (b'1111111111111111111111111111111111111111111111111111111111111111');
SELECT * FROM test WHERE v = b'1111111111111111111111111111111111111111111111111111111111111111';
-- Does NOT return the record
This only happens when using a value with 64 bits. But I would expect that to be possible.
Can anyone explain this to me?
Please do not respond by advising me not to use BIT columns. I am working on a database tool that should be able to handle all the data types of MySQL.
The problem seems to be, that the value b'11..11' in the WHERE clause is considered to be a SIGNED BIGINT which is -1 and is compared to the value in your table which is considered to be an UNSIGNED BIGINT which is 18446744073709551615. This is always an issue when the first of 64 bits is 1. IMHO this is a bug or a design flaw, because I expect an expression in the WHERE clause to match a row if the same expression has been used in the INSERT satement (at least in this case).
One workaround would be to cast the value to UNSIGNED:
SELECT *
FROM test
WHERE v = CAST(b'1111111111111111111111111111111111111111111111111111111111111111' as UNSIGNED);
Or (if your application language supports it) convert it to something like long uint or decimal:
SELECT * FROM test WHERE v = 18446744073709551615;
Bits are returned as binary, so to display them, either add 0, or use a function such as HEX, OCT or BIN to convert them https://mariadb.com/kb/en/library/bit/ or Bit values in result sets are returned as binary values, which may not display well. To convert a bit value to printable form, use it in numeric context or use a conversion function such as BIN() or HEX(). High-order 0 digits are not displayed in the converted value. https://dev.mysql.com/doc/refman/8.0/en/bit-value-literals.html
Related
In my table, I've got a column call mobile and I need this mobile field value to be normalized and save to another column call formatted_phone. For this purpose, I am using the below MySQL query and unfortunately, it is not working. I am putting my query here, please someone correct it. Thank you.
UPDATE hiring_detail
SET formatted_phone = replace(replace(
replace(replace(replace(replace(mobile,'-',''),'+',''),')',''),'(',''),' ',''),'.','')
WHERE mobile IS NOT NULL;
Error what it throws:
SQL Error (1265):Data truncated for column 'formatted_phone' at row 3
mobile column: varchar 50
formatted_phone: bigint 15
Usually, beofre blindly executing UPDATE commands, we do a simply What If analysis first, just run the query as a SELECT so you can inspect the output and importantly, you can compare it to the existing values:
http://sqlfiddle.com/#!9/47723a/2
SELECT mobile, formatted_phone
, REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(mobile,'-',''),'+',''),')',''),'(',''),' ',''),'.','') as test
FROM hiring_detail
WHERE mobile IS NOT NULL;
If that logic works for you (and it does in my tests) then there should be no issue with your UPDATE logic.
UPDATE:
If your error is
SQL Error (1265):Data truncated for column 'formatted_phone' at row 3
Then that means that your formatted phone numbers are longer than the column width for formatted_phone. If you know what the length is, you can truncate your formatted numbers, but with phone numbers, if we remove the actual number, this usually results in phone numbers that cannot be called.
I would recommend instead that you increase the width of the formatted_phone field.
This is an example of a forced truncation:
UPDATE hiring_detail
SET formatted_phone = RIGHT(replace(replace(
replace(replace(replace(replace(mobile,'-',''),'+',''),')',''),'(',''),' ',''),'.',''),10)
WHERE mobile IS NOT NULL;
Update #2
given that the column is an int, we need to convert the value into an integer.
UPDATE hiring_detail
SET formatted_phone = CAST(replace(replace(
replace(replace(replace(replace(mobile,'-',''),'+',''),')',''),'(',''),' ',''),'.','') as UNSIGNED INT)
WHERE mobile IS NOT NULL;
Warning: It is NOT advisable to store phone numbers as integers, the leading zeros can be significant in area codes in many localities, by storing as a numeric value this can have significant effects and can result in los of data. It also makes it hard to search for partial matches on the numbers. Almost all operations that you can think of (including sorting) on phone numbers will involve string manipulations, not mathematical or numerical.
The query executed should match the story_id with the provided string but when I execute the query it's giving me a wrong result. Please refer to the screenshot.
story_id column in your case is of INT (or numeric) datatype.
MySQL does automatic typecasting in this case. So, 5bff82... gets typecasted to 5 and thus you get the row corresponding to story_id = 5
Type Conversion in Expression Evaluation
When an operator is used with operands of different types, type
conversion occurs to make the operands compatible. Some conversions
occur implicitly. For example, MySQL automatically converts strings to
numbers as necessary, and vice versa.
Now, ideally your application code should be robust enough to handle this input. If you expect the input to be numeric only, then your application code can use validation operations on the data (to ensure that it is only a number, without typecasting) before sending it to MySQL server.
Another way would be to explicitly typecast story_id as string datatype and then perform the comparison. However this is not recommended approach as this would not be able to utilize Indexing.
SELECT * FROM story
WHERE (CAST story_id AS CHAR(12)) = '5bff82...'
If you run the above query, you would get no results.
you can also use smth like this:
SELECT * FROM story
WHERE regexp_like(story_id,'^[1-5]{1}(.*)$');
for any story_ids starting with any number and matching any no of charatcers after that it wont match with story_id=5;
AND if you explicitly want to match it with a string;
My mysql version is 5.7.14
I have 1 table with two column
1). price_val_float with float data type
2). price_val_double with double data type
Table structure
CREATE TABLE test (
price_val_float FLOAT(6,2),
price_val_double DOUBLE(6,2)
);
Same value in both column
INSERT INTO test VALUES
(78.59, 78.59),
(78.60, 78.60),
(78.61, 78.61);
Now I set one variable as follow
SET #priceValue=78.6;
Now I want to get all record from test table where price_val_float >= #priceValue;
SELECT price_val_float FROM test WHERE price_val_float>= #priceValue;
above query return only 78.61
But if I run same query of price_val_double column
SELECT price_val_double FROM test WHERE price_val_double>= #priceValue;
This return
78.60
78.61
I am not getting why mysql return different result as only data type is different.
Does anyone knows about this ?
Here is Fiddle for testing
Thanks in advance.
This might sound strange to say but this is because decimal numbers are approximates values. This is an issue across all programming due to the nature of storing large numbers. Even the mysql documentation calls these "approximate" values:
https://dev.mysql.com/doc/refman/5.5/en/floating-point-types.html
For example: MySQL performs rounding when storing values, so if you insert 999.00009 into a FLOAT(7,4) column, the approximate result is 999.0001.
This is explained in the mysql documentation here:
https://dev.mysql.com/doc/refman/5.5/en/problems-with-float.html
Or as an additional case explained in Python here:
https://docs.python.org/2/tutorial/floatingpoint.html
The way to get around this is identify the precision you want and store the value as an integer.
Float is a single precision and Double is for double precision that why your getting the difference.
This is happening because the difference between the numbers shows up around the tenth decimal or so, depending on factors such as computer architecture or the compiler version or optimization level. For example, different CPUs may evaluate floating-point numbers differently.
You need to use DECIMAL data type for more accurate results. Also check this for more details
That is because Float point values are not stored as exact values. If you need exact value you can use Decimal data type. You can read about it here
I am using a 3rd party .NET library (Rhino Security) that stores it's identifiers as guids in binary(16) fields in my mysql db. Everything works perfectly from the application but when I attempt to manually run a query via a query editor (TOAD for mysql) no rows are returned for identifiers I know to exist. For instance, if i run the following query, i get no results:
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
where EntitySecurityKey = '02a36462-49b7-406a-a3b6-d5accd6695e5'
Running the same query with no filter returns many results, including one with the above GUID in the EntitySecurityKey field. Is there another way to write a query to search on a guid/binary field?
Thanks!
EDIT
I found it interesting that TOAD returned a string and not an ugly blob. Using a different editor to return the results (for the unfiltered query) I get the raw binary data. I would have assume that my query would work using the binary keyword but neither of the following worked:
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
where EntitySecurityKey = BINARY '02a36462-49b7-406a-a3b6-d5accd6695e5'
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
where BINARY EntitySecurityKey = '02a36462-49b7-406a-a3b6-d5accd6695e5'
I am not familiar with Rhino Security, but it uses NHibernate to store to the database as far as I know.
The .net Guid uses 1 Int32, 2 Int16 and 8 Bytes internally to store the 128-bit Guid value.
When NHibernate (3.2) stores/retrieves a Guid value to/from a BINARY(16) column, it uses the .Net ToByteArray() method and Guid(Byte[] ...) constructor respectively. These methods basically swap the order of the bytes for the Int32 and Int16 values. You can reflect on the methods to see the exact code, but here is a simple example of the first 4 bytes:
guidBytes[0] = (byte)this._int32member;
guidBytes[1] = (byte)(this._int32member >> 8),
guidBytes[2] = (byte)(this._int32member >> 16),
guidBytes[3] = (byte)(this._int32member >> 24);
This may be the cause of your issue.
For example, your Guid is stored in MySql differently.
02a36462-49b7-406a-a3b6-d5accd6695e5 - Your Guid from Guid.ToString()
6264a302-b749-6a40-a3b6-d5accd6695e5 - Your Guid in MySql using MySql hex(Guid)
Note: hex(x) does not add the dashes; I added dashes manually for comparison purposes.
Notice that the order of the first 8 bytes is different.
To test if this is your issue, run the following query:
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
where EntitySecurityKey = UNHEX(REPLACE(6264a302-b749-6a40-a3b6-d5accd6695e5,'-',''));
If so, some other items I have noticed in my own travels:
Toad for MySql complicates this too, since it will display the Guid as the .net equivalent in a select statements output. It does not apply the same conversion if you include that Guid in a select's where clause (at least from my experience). You would need to use the Guid as stored by MySql.
You will either need to manually convert the Guids when performing DB queries or you may need to introduce a custom NHibernate type to convert the Guid differently.
You can refer to:
How to read a .NET Guid into a Java UUID
http://pastebin.com/M07wnK5K (I haven't used this code, but found it useful for reference)
Update:
I just tried the custom NH type on the pastebin link above. The byte order is incorrect, at least for my environment. I would need:
private static readonly int[] ByteOrder = new[] { 3, 2, 1, 0, 5, 4, 7, 6, 8, 9, 10, 11, 12, 13, 14, 15 };
to make by .Net output match the select hex(Guid column) from table output.
What strikes me as interesting is that you're storing your GUIDs in fields that are binary(16), emphasis on the 16. Per the manual, a binary field's length is in bytes and will truncate anything that goes over it (only in strict mode though). Is it possible that your GUID's are being truncated? With your sample GUID, 02a36462-49b7-406a-a3b6-d5accd6695e5, try querying the database with the first 16 characters:
WHERE EntitySecurityKey = '02a36462-49b7-40'
Per the accepted answer to this question, a field should be char(16) binary to store a GUID, not just binary(16). However, I couldn't get this to work in my sample table.
What did work for me were using char(36) and also binary(36). Try updating your field-lengths to 36 instead of 16 (I'd do this on a test-table first, just to be safe).
Here was my test table:
CREATE TABLE test_security_keys (
test_key1 char(16) not null,
test_key2 char(16) binary not null,
test_key3 char(36) not null,
test_key4 binary(16) not null,
test_key5 binary(36) not null
);
Then, I ran a script to insert numerous GUIDs (the same one for each column in a row). You can test it with your sample GUID:
INSERT INTO test_security_keys
VALUES ('02a36462-49b7-406a-a3b6-d5accd6695e5', '02a36462-49b7-406a-a3b6-d5accd6695e5', '02a36462-49b7-406a-a3b6-d5accd6695e5', '02a36462-49b7-406a-a3b6-d5accd6695e5', '02a36462-49b7-406a-a3b6-d5accd6695e5');
Using a simple SELECT * FROM test_security_keys will show all of the columns except the ones with size 36 to be truncated. Also, binary or not, I was able to successfully query the columns with a regular string-comparison:
SELECT * FROM test_security_keys WHERE test_key3 = '02a36462-49b7-406a-a3b6-d5accd6695e5';
SELECT * FROM test_security_keys WHERE test_key5 = '02a36462-49b7-406a-a3b6-d5accd6695e5';
If you've confirmed that your current columns with binary(16) aren't truncating, I would then-suggest to use a CAST() in your WHERE clause. The following should work (with your sample query):
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
WHERE
EntitySecurityKey = CAST('02a36462-49b7-406a-a3b6-d5accd6695e5' AS binary(16));
If you CAST(.. AS binary(16)) and the input-data is longer than 16 bytes, MySQL should issue a warning (should be unseen and not affect anything) stating that it had to truncated the data (try SHOW WARNINGS; if you get them). This is to be expected, but also means that you can't use binary(16) to store the GUIDs and you'll need to use binary(36) or char(36).
* I have not tried any of this using TOAD, but I've used both command-line MySQL and Navicat and have the same results for both.
I came across this question when I was evaluating how others "searched" for records where the primary key is a binary type (mainly the binary(16) indicative of GUID's) to compare it to my own. I must admit I was a little taken a back that the answers quickly diverged from the users desire to do manual searches based on the binary field but instead focused on changing the column types themselves to characters which, in this case, is murder on InnoDB storage.
An attempted solution for the lack of results would be to search against the Id column instead of the EntitySecurityKey column. Given JP's original query, the modified, working version would be:
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
where Id = UNHEX(REPLACE('02a36462-49b7-406a-a3b6-d5accd6695e5', '-', ''));
Again, this assumes the Id field is the binary(16) type. (I wasn't able to determine which column: Id, EntitySecurityKey was the binary(16)). If the EntitySecurityKey is the binary(16) column, then:
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
where EntitySecurityKey = UNHEX(REPLACE('02a36462-49b7-406a-a3b6-d5accd6695e5', '-', ''));
If that failed to yield the desired results, per this question (How to read a .NET Guid into a Java UUID), it would be of interest to try the same query but with the endianness of the first three parts of the GUID reversed:
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
where Id = UNHEX(REPLACE('6264a302-b749-6a40-a3b6-d5accd6695e5', '-', ''));
The only other thing I can think of is that EntitySecurityKey is a virtual column based on Id, but I don't have enough information on the table set-up to validate this statement.
Cheers.
From your EDIT, which I've a bit modifed:
Please, try
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
WHERE EntitySecurityKey = X'02a3646249b7406aa3b6d5accd6695e5'
Since the strings have 16 hexadecimal byes each, it may works...
Have you checked what the GUID data actually looks like in the table? Since it's the primary key, it quite possibly isn't stored as a GUID string. If that's the case querying the string value clearly won't work.
Possibly it's in binary format? Or possibly it's a string but without the hyphens? Or maybe it's been converted some other way? I don't know without seeing your data. But whatever the case, if a GUID that you know exists isn't being found when you query it, then whatever format it is in, it clearly isn't stored in the format you're querying.
The way I would solve this would be by searching for a different value in a record that you know exists. Or even just query the top few records without a where clause at all.
This will show you what the data actually looks like. With any luck, that will be sufficient for you to work out how it's been formatted.
If that doesn't help, one possible clue might come from this question: Why are binary Guids different from usual representation
Hope that helps.
Not sure how your database is set up, but I would try these two variants:
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
where EntitySecurityKey = x'02a3646249b7406aa3b6d5accd6695e5'
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
where EntitySecurityKey = x'6264a302b7496a40b6a3d5accd6695e5'
What really is the difference between MySQL UNHEX and X when dealing with hexadecimal values in a database?
Eg.
SELECT * FROM test WHERE guidCol IN (UNHEX('hexadecimalstring'));
SELECT * FROM test WHERE guidCol IN (X'hexadecimalstring');
Both gives me exact result set. So is there any difference? Performance implications?
Edit: the underlying type of guidCol is binary of course
UNHEX() is a function, therefore you can do something like
SET #var = '41';
SELECT UNHEX(#var);
SELECT UNHEX(hex_column) FROM my_table;
X, on the other hand, is the syntax for a hexadecimal litteral. You cannot do this:
SET #var = '41';
SELECT X#var; -- error (string litteral expected)
SELECT X'#var'; -- error (`#` is not a hexadecimal digit)
SELECT X(#var); -- returns NULL, not too sure about the reason... [edit: but this is probably why you are inserting NULL values]
SELECT X(hex_column) FROM my_table; -- returns NULL as well
This explains why you always get better performance with X: you are using a language construct instead of a function call. X does not need to evaluate a variable, since it expects a litteral string.
Note that even in MySQL 5.6, the X'' notation has a length limit in the reference mysql client and UNHEX() does not (appear to). I do not know what the limit is for X'', as it is not officially documented but I have encountered it when trying to INSERT a BLOB. With X'' literal, mysql client threw a syntax error with a sufficiently long hex sequence while UNHEX() of the same sequence did not. Obviously, length is not an issue when it comes to an actual GUID, but I figured this is useful for anyone else using this question to answer mysql insertion of binary data in the general case.