I want to create a table column that stores a 500 byte bitmap (500 bytes * 8 bits per byte = 4000 bits) and do operations to mutate bits (set to 1 or 0) at certain indexes in the bitmap.
However, the documentation page on bitmaps is mostly empty leaving me with the raw bit functions as the only guide. How do you create, count, read, and mutate a bitmap as a column type in MySQL?
Using bin and lpad you can print a 64 bit number out as a binary string.
LPAD(BIN(34), 64, '0')
0000000000000000000000000000000000000000000000000000000000100010
However, how do you print out a binary/blob/varbinary string that might be 4000 bits long?
(Note: not talking about bitmap indexes)
First, upgrade to MySQL 8.0. This won't work on earlier versions of MySQL.
You should use BINARY, VARBINARY, or BLOB, depending on the length of the bitfields you want to store.
mysql> create table mytable ( bits binary(500) );
ERROR 1074 (42000): Column length too big for column 'bits' (max = 255); use BLOB or TEXT instead
mysql> create table mytable ( bits blob(500) );
Query OK, 0 rows affected (0.02 sec)
Use UNHEX() to form bitfields from hexadecimal strings. It's too difficult to work with binary bytes directly.
mysql> insert into mytable set bits = unhex(repeat('00', 500));
Query OK, 1 row affected (0.00 sec)
You can use bitwise operators like |, &, ^, and ~ with bitfields. But the strings must be the same length!
mysql> update mytable set bits = bits | b'01000';
ERROR 3513 (HY000): Binary operands of bitwise operators must be of equal length
It's inconvenient, but you must form strings that are the right length using CONCAT():
mysql> update mytable set bits = bits | unhex(concat(repeat('00', 498), 'ffff'));
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0
Reading a bit string (I have shortened it for this display):
mysql> select hex(bits) from mytable\G
*************************** 1. row ***************************
hex(bits): 000...000FFFF
The BIT_COUNT() function works:
mysql> select bit_count(bits) from mytable;
+-----------------+
| bit_count(bits) |
+-----------------+
| 16 |
+-----------------+
mysql> select bit_count(~bits) from mytable;
+------------------+
| bit_count(~bits) |
+------------------+
| 3984 |
+------------------+
By this time, you should sit back and think about whether SQL is the best tool for this task, even with the features of MySQL 8.0. You will find it more flexible and powerful to do bitwise work in most other programming languages.
Use BLOB as data type and write a function wich will:
extract the byte which needs to be updated
change the bit in the byte
insert the changed byte into the blob at the original position
Here is one implementation:
delimiter //
create function set_bit(b blob, pos int, val int) returns blob reads sql data
comment 'changes the bit at position <pos> (0: right most bit) to <val> in the blob <b>'
begin
declare len int; -- byte length of the blob
declare byte_pos int; -- position of the affected byte (1: left most byte)
declare bit_pos int; -- position within the affected byte (0: right most bit)
declare byte_val int; -- value of the affected byte
set len = length(b);
set byte_pos = len - (pos div 8);
set bit_pos = pos mod 8;
set byte_val = ord(substring(b, byte_pos, 1)); -- read the byte
set byte_val = byte_val & (~(1 << bit_pos)); -- set the bit to 0
set byte_val = byte_val | (val << bit_pos); -- set the bit to <val>
return insert(b, byte_pos, 1, char(byte_val)); -- replace the byte and return
end //
delimiter ;
A simple test:
create table test(id int, b blob);
insert into test(id, b) select 1, 0x000000;
insert into test(id, b) select 2, 0xffffff;
We have two blob bitmasks (3 bytes each) - One full of zeros and one full of ones. In both we set the bit at position 10 (11th bit from right) to 1 and the bit at position 11 (12th bit from right) to 0.
update test set b = set_bit(b, 10, 1);
update test set b = set_bit(b, 11, 0);
select id, hex(b), to_base2(b) from test;
Result:
| id | hex(b) | to_base2(b) |
| --- | ------ | -------------------------- |
| 1 | 000400 | 00000000 00000100 00000000 |
| 2 | FFF7FF | 11111111 11110111 11111111 |
View on DB Fiddle
Note: to_base2() is a custom function that returns a string with a bit representation of a BLOB and is only used for presentation purpose.
This works for MySQL 5.x as well as for 8.0.
It is possible to implement it inline in a single expression (without the need of a function) - But that is rather unreadable:
update test t
cross join (select 10 as pos, 1 as val) i -- input
set t.b = insert(
t.b,
length(t.b) - (i.pos div 8),
1,
char(ord(
substring(t.b, length(t.b) - (i.pos div 8), 1))
& ~(1 << (i.pos mod 8))
| (i.val << (i.pos mod 8)
))
);
View on DB Fiddle
In MySQL 8.0 it's a bit simpler, since we don't need to extract the byte and can use bit operations on blobs. But we need to make sure, that the oprerands are of the same length:
update test t
cross join (select 10 as pos, 1 as val) i -- input
set t.b = t.b
& (~(concat(repeat(0x00,length(t.b)-1),char(1)) << i.pos))
| (concat(repeat(0x00,length(t.b)-1),char(i.val)) << i.pos)
View on DB Fiddle
Another way:
update test t
cross join (select 10 as pos, 1 as val) i -- input
set t.b =
case when i.val = 1
then t.b | concat(repeat(0x00,length(t.b)-1),char(1)) << i.pos
else t.b & ~(concat(repeat(0x00,length(t.b)-1),char(1)) << i.pos)
end
View on DB Fiddle
I need to mask integer field in mysql such that 9999911111 becomes 9900001111. I want to keep first 2 digits and last 4 digits and need to mark rest of the digits as 0 for the integers stored in the field.
I have created a query and it's working but I am not sure whether this is right way to do for integers or not.
update table_name
set field_name=CONCAT(SUBSTR(field_name, 1, 2),
REPEAT('0', CHAR_LENGTH(field_name) - 6),
SUBSTR(field_name, CHAR_LENGTH(field_name)-3, CHAR_LENGTH(field_name)));
Just trying a different approach .
SET #myVar = 344553543534;
SELECT #myVar - (SUBSTRING(#myVar, 4, LENGTH(#myVar) - 7) * 10000) ;
Above mentioned formula will give 344000003534 as the result. Tried with different combination and found it working.
So your query need to change as given below
UPDATE table_name
SET field_name=
(field_name - (SUBSTRING(field_name, 4, LENGTH(field_name) - 7) * 10000));
Explanation :
Consider Number, a = 344553543534;
Expected Result, b = 344000003534;
c = (a - b) = 344553543534 - 344000003534 = 553540000;
Now if you consider the result, c, 55354 is the numbers where masking required, and 0000 indicates the last 4 number to be left open.
So to get masked value, we can use the formula, b = a - c;
So now to get c, used SUBSTRING(a, 4, LENGTH(a) - 7) * 10000
EDIT : To keep only first two numbers, use 3 instead of 4 and 6 instead of 7. I assumed that you needed to keep first 3.
SET #myVar = 344553543534;
SELECT #myVar - (SUBSTRING(#myVar, 3, LENGTH(#myVar) - 6) * 10000) ;
I need to create table in other database based on select result types. Query results can map to actual columns in table or not f.e. Select 1, c from char_length_test.
How to get actual column size after select statement using mysql C api?
I have created such table:
CREATE TABLE char_length_test (c char(22))
And using mysql_real_query to execute this query
SELECT c from char_length_test
Right after that I execute mysql_fetch_fields to get length of c field and expect it to be 22 as in create table statement. Unfortunately length contains value of 66 (3 times more then I expect). Tried different sizes but result is the same, length is always 3 times bigger.
Also used gdb to see if there is any other data field containing expected value:
{
name = "c",
org_name = "c",
table = "char_length_test",
org_table = "char_length_test",
db = "database",
catalog = "def",
def = 0x0,
length = 66,
max_length = 0,
name_length = 1,
org_name_length = 1,
table_length = 16,
org_table_length = 16,
db_length = 6,
catalog_length = 3,
def_length = 0,
flags = 0,
decimals = 0,
charsetnr = 33,
type = MYSQL_TYPE_STRING,
extension = 0x0
}
mysql_fetch_fields() and mysql_field_len() will return a number of bytes required to store a VARCHAR value, not the number of characters. For UTF-8 columns, this will return 3 times the actual column size, even though the documentation says otherwise. This is so that your C code will know how much memory to allocate. If you set your MySQL connection (not the table structure!) to a different character set, you will get different results.
Edit:
You can change the character set of a MySQL connection by calling mysql_set_character_set(). If you use an 8bit character set, you should get a number of bytes that matches the width of the database column, e.g.:
mysql_set_character_set(&mysql,'latin1');
I want to sort the following data items in the order they are presented below (numbers 1-12):
1
2
3
4
5
6
7
8
9
10
11
12
However, my query - using order by xxxxx asc sorts by the first digit above all else:
1
10
11
12
2
3
4
5
6
7
8
9
Any tricks to make it sort more properly?
Further, in the interest of full disclosure, this could be a mix of letters and numbers (although right now it is not), e.g.:
A1
534G
G46A
100B
100A
100JE
etc....
Thanks!
update: people asking for query
select * from table order by name asc
People use different tricks to do this. I Googled and find out some results each follow different tricks. Have a look at them:
Alpha Numeric Sorting in MySQL
Natural Sorting in MySQL
Sorting of numeric values mixed with alphanumeric values
mySQL natural sort
Natural Sort in MySQL
Edit:
I have just added the code of each link for future visitors.
Alpha Numeric Sorting in MySQL
Given input
1A 1a 10A 9B 21C 1C 1D
Expected output
1A 1C 1D 1a 9B 10A 21C
Query
Bin Way
===================================
SELECT
tbl_column,
BIN(tbl_column) AS binray_not_needed_column
FROM db_table
ORDER BY binray_not_needed_column ASC , tbl_column ASC
-----------------------
Cast Way
===================================
SELECT
tbl_column,
CAST(tbl_column as SIGNED) AS casted_column
FROM db_table
ORDER BY casted_column ASC , tbl_column ASC
Natural Sorting in MySQL
Given input
Table: sorting_test
-------------------------- -------------
| alphanumeric VARCHAR(75) | integer INT |
-------------------------- -------------
| test1 | 1 |
| test12 | 2 |
| test13 | 3 |
| test2 | 4 |
| test3 | 5 |
-------------------------- -------------
Expected Output
-------------------------- -------------
| alphanumeric VARCHAR(75) | integer INT |
-------------------------- -------------
| test1 | 1 |
| test2 | 4 |
| test3 | 5 |
| test12 | 2 |
| test13 | 3 |
-------------------------- -------------
Query
SELECT alphanumeric, integer
FROM sorting_test
ORDER BY LENGTH(alphanumeric), alphanumeric
Sorting of numeric values mixed with alphanumeric values
Given input
2a, 12, 5b, 5a, 10, 11, 1, 4b
Expected Output
1, 2a, 4b, 5a, 5b, 10, 11, 12
Query
SELECT version
FROM version_sorting
ORDER BY CAST(version AS UNSIGNED), version;
Just do this:
SELECT * FROM table ORDER BY column `name`+0 ASC
Appending the +0 will mean that:
0,
10,
11,
2,
3,
4
becomes :
0,
2,
3,
4,
10,
11
I hate this, but this will work
order by lpad(name, 10, 0) <-- assuming maximum string length is 10
<-- you can adjust to a bigger length if you want to
I know this post is closed but I think my way could help some people. So there it is :
My dataset is very similar but is a bit more complex. It has numbers, alphanumeric data :
1
2
Chair
3
0
4
5
-
Table
10
13
19
Windows
99
102
Dog
I would like to have the '-' symbol at first, then the numbers, then the text.
So I go like this :
SELECT name, (name = '-') boolDash, (name = '0') boolZero, (name+0 > 0) boolNum
FROM table
ORDER BY boolDash DESC, boolZero DESC, boolNum DESC, (name+0), name
The result should be something :
-
0
1
2
3
4
5
10
13
99
102
Chair
Dog
Table
Windows
The whole idea is doing some simple check into the SELECT and sorting with the result.
This works for type of data:
Data1,
Data2, Data3 ......,Data21. Means "Data" String is common in all rows.
For ORDER BY ASC it will sort perfectly, For ORDER BY DESC not suitable.
SELECT * FROM table_name ORDER BY LENGTH(column_name), column_name ASC;
I had some good results with
SELECT alphanumeric, integer FROM sorting_test ORDER BY CAST(alphanumeric AS UNSIGNED), alphanumeric ASC
This type of question has been asked previously.
The type of sorting you are talking about is called "Natural Sorting".
The data on which you want to do sort is alphanumeric.
It would be better to create a new column for sorting.
For further help check
natural-sort-in-mysql
If you need to sort an alpha-numeric column that does not have any standard format whatsoever
SELECT * FROM table ORDER BY (name = '0') DESC, (name+0 > 0) DESC, name+0 ASC, name ASC
You can adapt this solution to include support for non-alphanumeric characters if desired using additional logic.
This should sort alphanumeric field like:
1/ Number only, order by 1,2,3,4,5,6,7,8,9,10,11 etc...
2/ Then field with text like: 1foo, 2bar, aaa11aa, aaa22aa, b5452 etc...
SELECT MyField
FROM MyTable
order by
IF( MyField REGEXP '^-?[0-9]+$' = 0,
9999999999 ,
CAST(MyField AS DECIMAL)
), MyField
The query check if the data is a number, if not put it to 9999999999 , then order first on this column, then order on data with text
Good luck!
Instead of trying to write some function and slow down the SELECT query, I thought of another way of doing this...
Create an extra field in your database that holds the result from the following Class and when you insert a new row, run the field value that will be naturally sorted through this class and save its result in the extra field. Then instead of sorting by your original field, sort by the extra field.
String nsFieldVal = new NaturalSortString(getFieldValue(), 4).toString()
The above means:
- Create a NaturalSortString for the String returned from getFieldValue()
- Allow up to 4 bytes to store each character or number (4 bytes = ffff = 65535)
| field(32) | nsfield(161) |
a1 300610001
String sortString = new NaturalSortString(getString(), 4).toString()
import StringUtils;
/**
* Creates a string that allows natural sorting in a SQL database
* eg, 0 1 1a 2 3 3a 10 100 a a1 a1a1 b
*/
public class NaturalSortString {
private String inStr;
private int byteSize;
private StringBuilder out = new StringBuilder();
/**
* A byte stores the hex value (0 to f) of a letter or number.
* Since a letter is two bytes, the minimum byteSize is 2.
*
* 2 bytes = 00 - ff (max number is 255)
* 3 bytes = 000 - fff (max number is 4095)
* 4 bytes = 0000 - ffff (max number is 65535)
*
* For example:
* dog123 = 64,6F,67,7B and thus byteSize >= 2.
* dog280 = 64,6F,67,118 and thus byteSize >= 3.
*
* For example:
* The String, "There are 1000000 spots on a dalmatian" would require a byteSize that can
* store the number '1000000' which in hex is 'f4240' and thus the byteSize must be at least 5
*
* The dbColumn size to store the NaturalSortString is calculated as:
* > originalStringColumnSize x byteSize + 1
* The extra '1' is a marker for String type - Letter, Number, Symbol
* Thus, if the originalStringColumn is varchar(32) and the byteSize is 5:
* > NaturalSortStringColumnSize = 32 x 5 + 1 = varchar(161)
*
* The byteSize must be the same for all NaturalSortStrings created in the same table.
* If you need to change the byteSize (for instance, to accommodate larger numbers), you will
* need to recalculate the NaturalSortString for each existing row using the new byteSize.
*
* #param str String to create a natural sort string from
* #param byteSize Per character storage byte size (minimum 2)
* #throws Exception See the error description thrown
*/
public NaturalSortString(String str, int byteSize) throws Exception {
if (str == null || str.isEmpty()) return;
this.inStr = str;
this.byteSize = Math.max(2, byteSize); // minimum of 2 bytes to hold a character
setStringType();
iterateString();
}
private void setStringType() {
char firstchar = inStr.toLowerCase().subSequence(0, 1).charAt(0);
if (Character.isLetter(firstchar)) // letters third
out.append(3);
else if (Character.isDigit(firstchar)) // numbers second
out.append(2);
else // non-alphanumeric first
out.append(1);
}
private void iterateString() throws Exception {
StringBuilder n = new StringBuilder();
for (char c : inStr.toLowerCase().toCharArray()) { // lowercase for CASE INSENSITIVE sorting
if (Character.isDigit(c)) {
// group numbers
n.append(c);
continue;
}
if (n.length() > 0) {
addInteger(n.toString());
n = new StringBuilder();
}
addCharacter(c);
}
if (n.length() > 0) {
addInteger(n.toString());
}
}
private void addInteger(String s) throws Exception {
int i = Integer.parseInt(s);
if (i >= (Math.pow(16, byteSize)))
throw new Exception("naturalsort_bytesize_exceeded");
out.append(StringUtils.padLeft(Integer.toHexString(i), byteSize));
}
private void addCharacter(char c) {
//TODO: Add rest of accented characters
if (c >= 224 && c <= 229) // set accented a to a
c = 'a';
else if (c >= 232 && c <= 235) // set accented e to e
c = 'e';
else if (c >= 236 && c <= 239) // set accented i to i
c = 'i';
else if (c >= 242 && c <= 246) // set accented o to o
c = 'o';
else if (c >= 249 && c <= 252) // set accented u to u
c = 'u';
else if (c >= 253 && c <= 255) // set accented y to y
c = 'y';
out.append(StringUtils.padLeft(Integer.toHexString(c), byteSize));
}
#Override
public String toString() {
return out.toString();
}
}
For completeness, below is the StringUtils.padLeft method:
public static String padLeft(String s, int n) {
if (n - s.length() == 0) return s;
return String.format("%0" + (n - s.length()) + "d%s", 0, s);
}
The result should come out like the following
-1
-a
0
1
1.0
1.01
1.1.1
1a
1b
9
10
10a
10ab
11
12
12abcd
100
a
a1a1
a1a2
a-1
a-2
áviacion
b
c1
c2
c12
c100
d
d1.1.1
e
MySQL ORDER BY Sorting alphanumeric on correct order
example:
SELECT `alphanumericCol` FROM `tableName` ORDER BY
SUBSTR(`alphanumericCol` FROM 1 FOR 1),
LPAD(lower(`alphanumericCol`), 10,0) ASC
output:
1
2
11
21
100
101
102
104
S-104A
S-105
S-107
S-111
This is from tutorials point
SELECT * FROM yourTableName ORDER BY
SUBSTR(yourColumnName FROM 1 FOR 2),
CAST(SUBSTR(yourColumnName FROM 2) AS UNSIGNED);
it is slightly different from another answer of this thread
For reference, this is the original link
https://www.tutorialspoint.com/mysql-order-by-string-with-numbers
Another point regarding UNSIGNED is written here
https://electrictoolbox.com/mysql-order-string-as-int/
While this has REGEX too
https://www.sitepoint.com/community/t/how-to-sort-text-with-numbers-with-sql/346088/9
SELECT length(actual_project_name),actual_project_name,
SUBSTRING_INDEX(actual_project_name,'-',1) as aaaaaa,
SUBSTRING_INDEX(actual_project_name, '-', -1) as actual_project_number,
concat(SUBSTRING_INDEX(actual_project_name,'-',1),SUBSTRING_INDEX(actual_project_name, '-', -1)) as a
FROM ctts.test22
order by
SUBSTRING_INDEX(actual_project_name,'-',1) asc,cast(SUBSTRING_INDEX(actual_project_name, '-', -1) as unsigned) asc
This is a simple example.
SELECT HEX(some_col) h
FROM some_table
ORDER BY h
order by len(xxxxx),xxxxx
Eg:
SELECT * from customer order by len(xxxxx),xxxxx
Try this For ORDER BY DESC
SELECT * FROM testdata ORDER BY LENGHT(name) DESC, name DESC
SELECT
s.id, s.name, LENGTH(s.name) len, ASCII(s.name) ASCCCI
FROM table_name s
ORDER BY ASCCCI,len,NAME ASC;
Assuming varchar field containing number, decimal, alphanumeric and string, for example :
Let's suppose Column Name is "RandomValues" and Table name is "SortingTest"
A1
120
2.23
3
0
2
Apple
Zebra
Banana
23
86.Akjf9
Abtuo332
66.9
22
ABC
SELECT * FROM SortingTest order by IF( RandomValues REGEXP '^-?[0-9,.]+$' = 0,
9999999999 ,
CAST(RandomValues AS DECIMAL)
), RandomValues
Above query will do sorting on number & decimal values first and after that all alphanumeric values got sorted.
This will always put the values starting with a number first:
ORDER BY my_column REGEXP '^[0-9]' DESC, length(my_column + 0), my_column ";
Works as follows:
Step1 - Is first char a digit? 1 if true, 0 if false, so order by this DESC
Step2 - How many digits is the number? Order by this ASC
Step3 - Order by the field itself
Input:
('100'),
('1'),
('10'),
('0'),
('2'),
('2a'),
('12sdfa'),
('12 sdfa'),
('Bar nah');
Output:
0
1
2
2a
10
12 sdfa
12sdfa
100
Bar nah
Really problematic for my scenario...
select * from table order by lpad(column, 20, 0)
My column is a varchar, but has numeric input (1, 2, 3...) , mixed numeric (1A, 1B, 1C) and too string data (INT, SHIP)