ORDER BY mysql numbers in correct order - mysql

I have a list of data from SQL field. I want to sort by the field ASC but when i do it comes up in the wrong order. I know why it does it, but wondered if there was a solution around this problem. Ive heard of natsort php function, but not investigated it. Is there an easy way?
Academy
Under 10′s Blue
Under 10′s Green
Under 11′s Red
Under 11′s White
Under 13′s Blue
Under 13′s Red
Under 13′s White
Under 14′s Blue
Under 15′s Blue
Under 15′s Red
Under 15′s White
Under 16′s Red
Under 18′s Blue
Under 18′s Red
Under 7′s
Under 8′s Red
Under 9′s Red

There is a very simple approach to sort this list. For all values of Academy starting with Under you perform the sort algorithm based on the following ORDER BY clause:
ORDER BY
REPLACE(Academy,'Under ','') + 0,Academy
The first sort column is based on removing the string 'Under ' and then adding 0. This will force an ordering of the resulting integer.
Here is an example of computing the numeric value by removing 'Under ' first:
mysql> select REPLACE('Under 15\'s Red','Under ','') + 0;
+--------------------------------------------+
| REPLACE('Under 15\'s Red','Under ','') + 0 |
+--------------------------------------------+
| 15 |
+--------------------------------------------+
1 row in set (0.00 sec)
The second sort column will order by the string value of Academy. All 'Under 15's' are grouped together and alphanumerically sorted.
Here is your sample data from the question loaded into a table and sorted:
mysql> use test
Database changed
mysql> drop table if exists under99color;
Query OK, 0 rows affected (0.01 sec)
mysql> create table under99color
-> (academy varchar(30),
-> id int not null auto_increment,
-> primary key (id),
-> index academy (academy)) engine=MyISAM;
Query OK, 0 rows affected (0.04 sec)
mysql> show create table under99color\G
*************************** 1. row ***************************
Table: under99color
Create Table: CREATE TABLE `under99color` (
`academy` varchar(30) DEFAULT NULL,
`id` int(11) NOT NULL AUTO_INCREMENT,
PRIMARY KEY (`id`),
KEY `academy` (`academy`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1
1 row in set (0.00 sec)
mysql> insert into under99color (academy) values
-> ('Under 10\'s Blue'),('Under 10\'s Green'),('Under 11\'s Red'),
-> ('Under 11\'s White'),('Under 13\'s Blue'),('Under 13\'s Red'),
-> ('Under 13\'s White'),('Under 14\'s Blue'),('Under 15\'s Blue'),
-> ('Under 15\'s Red'),('Under 15\'s White'),('Under 16\'s Red'),
-> ('Under 18\'s Blue'),('Under 18\'s Red'),('Under 7\'s'),
-> ('Under 8\'s Red'),('Under 9\`s Red');
Query OK, 17 rows affected (0.00 sec)
Records: 17 Duplicates: 0 Warnings: 0
mysql> select academy from under99color
-> ORDER BY REPLACE(Academy,'Under ','') + 0,Academy;
+------------------+
| academy |
+------------------+
| Under 7's |
| Under 8's Red |
| Under 9`s Red |
| Under 10's Blue |
| Under 10's Green |
| Under 11's Red |
| Under 11's White |
| Under 13's Blue |
| Under 13's Red |
| Under 13's White |
| Under 14's Blue |
| Under 15's Blue |
| Under 15's Red |
| Under 15's White |
| Under 16's Red |
| Under 18's Blue |
| Under 18's Red |
+------------------+
17 rows in set (0.00 sec)
mysql>
Give it a Try !!!

You can add a field to the selection query that uses a CAST to bring it into a numeric. First you'll have to come up with a substring method that will select the number from the string in the first place (perhaps use a Field function on the space and the '). Once you've got it isolated as an integer, sorting at that point should be trivial.
Possible example (pseudo-code - may not work "out of the box"):
SELECT TeamType, CAST(SUBSTRING(TeamType, FIELD(' ', TeamType), FIELD('\'', TeamType) - Field(' ', TeamType)), UNSIGNED) As TeamAge
FROM Teams
ORDER BY TeamAge, TeamType

Your field is string! Thus it's sorting the string values.

It has sorted it alphabetically; you need to parse the output field and sort them based on the number later.

Well, you could store the number and the colour separately (with the option of a blank colour) and then order by number followed by colour. E.g:
SELECT CONCAT('Under ',ageIndex,'\'s ',colour) AS Team FROM Academy
ORDER BY ageIndex, colour
The possible advantage of this, depending on your requirements, is that you can then also run queries on ages and colours separately.

Related

Why is MySQL ignoring the ORDER BY on this condition?

I want to show rows that have updated_at more than 3 hours ago. MySQL seems to be completely ignoring the ORDER BY clause. Any idea why?
Edit: as pointed out by Sebastian, this only occurs in certain timezones, like GMT+5 or GMT+8.
mysql> SET time_zone='+08:00';
Query OK, 0 rows affected (0.00 sec)
mysql> CREATE DATABASE test1; USE test1;
Query OK, 1 row affected (0.01 sec)
Database changed
mysql> CREATE TABLE `boxes` (
-> `box_id` int unsigned NOT NULL AUTO_INCREMENT,
-> `updated_at` timestamp NULL DEFAULT NULL ON UPDATE CURRENT_TIMESTAMP,
-> PRIMARY KEY (`box_id`)
-> ) ENGINE=InnoDB;
Query OK, 0 rows affected (0.01 sec)
mysql> INSERT INTO `boxes` (`box_id`, `updated_at`) VALUES
-> (1, '2020-08-22 05:25:35'),
-> (2, '2020-08-26 18:49:05'),
-> (3, '2020-08-23 03:28:30'),
-> (4, '2020-08-23 03:32:55');
Query OK, 4 rows affected (0.00 sec)
Records: 4 Duplicates: 0 Warnings: 0
mysql> SELECT NOW();
+---------------------+
| NOW() |
+---------------------+
| 2020-08-26 20:49:59 |
+---------------------+
1 row in set (0.00 sec)
mysql> SELECT b.box_id, updated_at, (b.updated_at < NOW() - INTERVAL 3 HOUR) AS more_than_3hr
-> FROM boxes b
-> ORDER BY more_than_3hr DESC;
+--------+---------------------+---------------+
| box_id | updated_at | more_than_3hr |
+--------+---------------------+---------------+
| 1 | 2020-08-22 05:25:35 | 1 |
| 2 | 2020-08-26 18:49:05 | 0 | <--- WHY IS THIS HERE???
| 3 | 2020-08-23 03:28:30 | 1 |
| 4 | 2020-08-23 03:32:55 | 1 |
+--------+---------------------+---------------+
4 rows in set (0.00 sec)
Expectation: the rows with "1" should show up first.
Actual results: ORDER BY is ignored, and the resultset is sorted by primary key
I have a hunch it has something to do with MySQL storing timestamps as UTC and displaying them in the current timezone. My current timezone is GMT+8. However, it still doesn't make sense -- I am sorting the results based on the aliased expression, and the expression's value is clearly shown in the resultset.
MySQL version 8.0.21.
I also tried moving the expression to the ORDER BY clause, and the results are the same.
I don't know why but it compares wrong timezones in the background and thus values at the end are correct, but comparisons are invalid (for specific timezones).
When you query a TIMESTAMP value, MySQL converts the UTC value back to
your connection’s time zone. Note that this conversion does not take
place for other temporal data types such as DATETIME.
https://www.mysqltutorial.org/mysql-timestamp.aspx/
Changing type from TIMESTAMP to DATETIME fixes problem.
Other solution may be casting to the decimal number.
SELECT b.box_id, updated_at, FORMAT((b.updated_at < NOW() - INTERVAL 3 HOUR),0) AS more_than_3hr
FROM boxes b
ORDER BY more_than_3hr DESC;
From the documentation:
https://dev.mysql.com/doc/refman/8.0/en/user-variables.html
HAVING, GROUP BY, and ORDER BY, when referring to a variable that is assigned a value in the select expression list do not work as expected because the expression is evaluated on the client and thus can use stale column values from a previous row.
Basically, you can't use a variable name you created with "AS" in your sorting.
The solution is to use the verbose statement you used for the AS in sorting. Yeah, it's verbose. 🤷‍♂️ It is what it is.

How to extract unique nested variable names out of one string variable?

Case
In our MySql database the data is stored in combined json-strings like this:
| ID | DATA |
| 100 | {var1str: "sometxt", var2double: 0,01, var3integer: 1, var4str: "another text"} |
| 101 | {var3integer: 5, var2double: 2,05, var1str: "txt", var4str: "more text"} |
Problem
Most of the DATA-fields hold over 2500 variables. The order of variables in the DATA-string is random (as shown in above example). Right now we only know how to extract data with the following querie:
select
ID,
json_extract(DATA,'var1str'),
json_extract(DATA,'var2double'),
FROM table
With this querie, only the values of var1str and var2double will be returned as result. Values of variable 3 and 4 are ignored. There is no overview of what possible variables are hiding in the data fields.
With almost 60.000 entries and over 3.000 possible unique variable names, I would like to create a query that loops through all of the 60.000 DATA-fields and extracts every unique variable name that is found in there.
Solution?
The querie I am looking for would give the following result:
var1str
var2double
var3integer
var4str
My knowledge of MySql is very limited. Any direction given to get to this solution is much appreciated.
What version of MySQL are you using?.
From MySQL 8.0.4 and later JSON_TABLE function is supported and can be useful in this case.
mysql> SELECT VERSION();
+-----------+
| VERSION() |
+-----------+
| 8.0.11 |
+-----------+
1 row in set (0.00 sec)
mysql> DROP TABLE IF EXISTS `table`;
Query OK, 0 rows affected (0.09 sec)
mysql> CREATE TABLE IF NOT EXISTS `table` (
-> `ID` BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
-> `DATA` JSON NOT NULL
-> ) AUTO_INCREMENT=100;
Query OK, 0 rows affected (0.00 sec)
mysql> INSERT INTO `table`
-> (`DATA`)
-> VALUES
-> ('{"var1str": "sometxt", "var2double": 0.01, "var3integer": 1, "var4str": "another text"}'),
-> ('{"var3integer": 5, "var2double": 2.05, "var1str": "txt", "var4str": "more text"}');
Query OK, 2 rows affected (0.00 sec)
Records: 2 Duplicates: 0 Warnings: 0
mysql> SELECT
-> DISTINCT `der`.`key`
-> FROM
-> `table`,
-> JSON_TABLE(
-> JSON_KEYS(`DATA`), '$[*]'
-> COLUMNS(
-> `key` VARCHAR(64) PATH "$"
-> )
-> ) `der`;
+-------------+
| key |
+-------------+
| var1str |
| var4str |
| var2double |
| var3integer |
+-------------+
4 rows in set (0.01 sec)
Be aware of the Bug #90610 ERROR 1142 (42000) when using JSON_TABLE.

`UPDATE ... WHERE ... ` multiple rows locking in InnoDB

I'm implementing a custom table-based sequence generator for MySQL database v5.7.16 with InnoDB engine.
The sequence_table looks as follows:
+-------------+-----------+
|sequence_name|next_value |
+-------------+-----------+
| first_seq | 1 |
+-------------+-----------+
| second_seq | 1 |
+-------------+-----------+
sequence_name column is a primary key.
This sequence table contains multiple sequences for different consumers.
I use the following strategy for the sequence updates:
Select current sequence value: select next_val from sequence_table where sequence_name=?.
Add the allocation size to current sequence value.
Update the sequence value if it's current value matches the value selected in the first step: update sequence_table set next_val=? where sequence_name=? and next_val=?.
If the update is successful return the increased sequence value, otherwise repeat the process from step 1.
The documentation contains the following information:
UPDATE ... WHERE ... sets an exclusive next-key lock on every record
the search encounters. However, only an index record lock is required
for statements that lock rows using a unique index to search for a
unique row. 14.5.3 Locks Set by Different SQL Statements in InnoDB
The part in bold is a bit confusing.
As you can see, I match the primary key in the WHERE clause of the UPDATE statement.
Is it possible that the search may encounter more than one record and therefore lock multiple rows in this sequence table?
In other words, will the update in the 3rd step of the algorithm block just one or multiple rows?
You didn't mention what transaction isolation level you're planning to use.
Lets assume you're using repeatable read (in read committed no such a problem should exist)
From here:
For locking reads (SELECT with FOR UPDATE or LOCK IN SHARE MODE),
UPDATE, and DELETE statements, locking depends on whether the
statement uses a unique index with a unique search condition, or a
range-type search condition
and
For a unique index with a unique search condition, InnoDB locks only
the index record found, not the gap before it
So at least in theory it should lock only a single record and no next-key lock will be used.
More quotes from other docs pages to back my thoughts:
innodb-next-key-locks
link
A next-key lock is a combination of a record lock on the index record
and a gap lock on the gap before the index record.
gap locks
link
Gap locking is not needed for statements that lock rows using a unique
index to search for a unique row
Don't grab the sequence numbers inside the main transaction; do it before the START TRANSCTION.
Do the task in a single statement with autocommit=ON.
Both of those lead to it being much faster, less likely to block.
(You code was missing BEGIN/COMMIT and FOR UPDATE. I got rid of those rather than explaining the issues.)
Set up test:
mysql> CREATE TABLE so49197964 (
-> name VARCHAR(22) NOT NULL,
-> next_value INT UNSIGNED NOT NULL,
-> PRIMARY KEY (name)
-> ) ENGINE=InnoDB;
Query OK, 0 rows affected (0.02 sec)
mysql> INSERT INTO so49197964 (name, next_value)
-> VALUES
-> ('first', 1), ('second', 1);
Query OK, 2 rows affected (0.00 sec)
Records: 2 Duplicates: 0 Warnings: 0
mysql> SELECT * FROM so49197964;
+--------+------------+
| name | next_value |
+--------+------------+
| first | 1 |
| second | 1 |
+--------+------------+
2 rows in set (0.00 sec)
Grab 20 nums from 'first' and fetch the starting number:
mysql> UPDATE so49197964
-> SET next_value = LAST_INSERT_ID(next_value) + 20
-> WHERE name = 'first';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0
mysql> SELECT LAST_INSERT_ID();
+------------------+
| LAST_INSERT_ID() |
+------------------+
| 1 |
+------------------+
1 row in set (0.00 sec)
mysql> SELECT * FROM so49197964;
+--------+------------+
| name | next_value |
+--------+------------+
| first | 21 |
| second | 1 |
+--------+------------+
2 rows in set (0.00 sec)
Grab another 20:
mysql> UPDATE so49197964
-> SET next_value = LAST_INSERT_ID(next_value) + 20
-> WHERE name = 'first';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0
mysql> SELECT LAST_INSERT_ID();
+------------------+
| LAST_INSERT_ID() |
+------------------+
| 21 |
+------------------+
1 row in set (0.00 sec)
mysql> SELECT * FROM so49197964;
+--------+------------+
| name | next_value |
+--------+------------+
| first | 41 |
| second | 1 |
+--------+------------+
2 rows in set (0.00 sec)

How to backfill an auto increment field in MySQL

I have a MySQL Database 5.6.32 connected to SuiteCRM. I am using a plugin that allows for the creation of an auto increment field without any coding.
The challenge I'm having is that I have created this auto increment field after records with data are already in our system. I need to update all previous entries with the auto increment values.
When I create a new record the auto increment field works fine, but I need the unique number for all records as this is being used as a Unique Identifier and the default ID in the system is too long for us to use.
The type of auto increment field it created in the MySQL database is as follows:
# Name Type Collation Null Default
10 customer_number_c varchar(80) utf8_general_ci Yes NULL
This is what I have tried so far to try and populate the field:
UPDATE `suitecrm`.`accounts_cstm` SET `customer_number_c` = auto_increment
The result is:
ERROR #1054 - Unknown column 'AUTO_INCNREMENTAL' in 'Field list'
The field already has a default value of NULL in it as well.
MySQL's builtin auto-increment feature only works with columns of integer data types. Yours is varchar(80). I'm not sure why you did that, but I assume it was deliberate.
You could use a session variable to populate the customer number. As you assign values, it will implicitly cast the integer value of the session variable to the string representation.
SET #num := 0;
UPDATE suitecrm.accounts_cstm
SET customer_number_c = (#num := #num + 1)
ORDER BY ...;
You would have to specify some ORDER BY clause to make sure the increasing values get assigned in the order you want them to be.
But you still won't be able to use AUTO_INCREMENT on the customer_number_c column. So your app must generate new customer number values before inserting new rows to this table.
MySQL will retroactively populate existing rows for you if you add an auto_increment primary key. I just validated this with the following test code:
mysql> create table mytable (name varchar(32)) engine=innodb;
Query OK, 0 rows affected (0.04 sec)
mysql> insert into mytable (name) values ('miles'), ('trane'), ('monk');
Query OK, 3 rows affected (0.00 sec)
Records: 3 Duplicates: 0 Warnings: 0
mysql> select * from mytable;
+-------+
| name |
+-------+
| miles |
| trane |
| monk |
+-------+
3 rows in set (0.00 sec)
mysql> alter table mytable add column id int unsigned primary key auto_increment first;
Query OK, 0 rows affected (0.04 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> select * from mytable;
+----+-------+
| id | name |
+----+-------+
| 1 | miles |
| 2 | trane |
| 3 | monk |
+----+-------+
3 rows in set (0.00 sec)

Extracting numerical values from mySQL column string value

I have a "person" column in a mySQL database that represents the age and weight of a person as a string separated by a comma.
Example:
"24,175"
I want to be able to separate and extract those values and cast them as numbers.
Example: turn "24,175" to
24 as age
175 as weight
So that I can write a query similar to the following
SELECT person
FROM TABLE
WHERE age>140 OR weight>1000
I want to be able to check for values that are not possible. i.e age>140 OR weight >1000.
I cannot modify the table/environment I'm working with
I only have access to queries.
I'm thinking about solving it this way
find the index where the comma exists. CHARINDEX(',',person)
Split the string into substrings using LEFT , RIGHT, CAST and CHARINDEX(',',person)
Cast age substring and weight substring to numbers using CAST(age AS INT) CAST(weight AS INT)
SELECT person
FROM TABLE
WHERE CAST(LEFT(person,CHARINDEX(',',person) AS INT)>150 OR CAST(RIGHT(person,CHARINDEX(',',person) AS INT) >1000
If I did anything wrong please correct me.
Are all the functions usable/supported by mySQL? (RIGHT, LEFT, CHARINDEX) Will this work?
Exception: Another value for this column could be "unknown". Will this cause errors if we're trying to check for the index of , if it doesn't exist in the string? Is there a way to include "unknown" cases in the result and have it output a message of "error, person not recognized"
you can also split is with SUBSTR_INDEX like this:
MariaDB [yourschema]> SELECT * FROM spliit;
+----+--------+
| id | d |
+----+--------+
| 1 | 24,175 |
+----+--------+
1 row in set (0.03 sec)
MariaDB [yourschema]> SELECT
-> SUBSTRING_INDEX(d, ',', 1) AS age
-> , SUBSTRING_INDEX(d, ',', -1) AS weight
->
-> FROM spliit;
+------+--------+
| age | weight |
+------+--------+
| 24 | 175 |
+------+--------+
1 row in set (0.00 sec)
MariaDB [yourschema]>
sample
yes, you can direct calculate with it in MySQL
MariaDB [yourschema]> SELECT
-> SUBSTRING_INDEX(d, ',', 1) + 2 AS age
-> , SUBSTRING_INDEX(d, ',', 1) * 12 AS `month`
-> , SUBSTRING_INDEX(d, ',', -1) + 3 AS weight
-> FROM spliit;
+------+-------+--------+
| age | month | weight |
+------+-------+--------+
| 26 | 288 | 178 |
+------+-------+--------+
1 row in set, 1 warning (0.03 sec)
MariaDB [yourschema]>
SELECT person
FROM TABLE
WHERE CAST(LEFT(person,LOCATE(',',person) AS INTEGER)>150 OR CAST(RIGHT(person,(LOCATE(',',person)+1) AS INTEGER) >1000
Instead of Char index use LOCATE im MqSQL
Also note the CAST function
You also can use VIRTUAL PERSITENT COLUMNS that calculate the fields automatis and you can also use a INDEX on each substr / Integer.
sample
MariaDB [yourschema]> CREATE TABLE `splitit` (
-> `id` INT(11) UNSIGNED NOT NULL AUTO_INCREMENT,
-> `d` VARCHAR(32) DEFAULT NULL,
-> age INT(11) AS (SUBSTRING_INDEX(d, ',', 1)) PERSISTENT,
-> weight INT(5) AS (SUBSTRING_INDEX(d, ',', -1)) PERSISTENT,
-> PRIMARY KEY (`id`),
-> INDEX idx_age (age),
-> INDEX idx_weight (weight)
-> ) ENGINE=INNODB DEFAULT CHARSET=utf8;
Query OK, 0 rows affected (0.79 sec)
MariaDB [yourschema]> INSERT INTO splitit (d) VALUES ('11,234'),('2,66'),('5,2');
Query OK, 3 rows affected (0.06 sec)
Records: 3 Duplicates: 0 Warnings: 0
MariaDB [yourschema]> SELECT * FROM splitit;
+----+--------+------+--------+
| id | d | age | weight |
+----+--------+------+--------+
| 1 | 11,234 | 11 | 234 |
| 2 | 2,66 | 2 | 66 |
| 3 | 5,2 | 5 | 2 |
+----+--------+------+--------+
3 rows in set (0.00 sec)
MariaDB [yourschema]>
You can do this all in the where clause:
where substring_index(person, ',', 1) + 0 > 140 or
substring_index(person, ',' -1) + 0 > 1000
Note that the + 0 does an silent conversion to integers. And, substring_index()is much more convenient than the functions in SQL Server.
You can readily incorporate this logic into a view:
create view v_table as
select t.*,
substring_index(person, ',', 1) + 0 as age,
substring_index(person, ',' -1) + 0 as weight
from table t;
If you want to filter out bad values within the view, you can use a MySQL extension and add:
having age > 140 or weight > 1000
after the from clause.