How to extract unique nested variable names out of one string variable? - mysql

Question

Case
In our MySql database the data is stored in combined json-strings like this:
| ID | DATA |
| 100 | {var1str: "sometxt", var2double: 0,01, var3integer: 1, var4str: "another text"} |
| 101 | {var3integer: 5, var2double: 2,05, var1str: "txt", var4str: "more text"} |
Problem
Most of the DATA-fields hold over 2500 variables. The order of variables in the DATA-string is random (as shown in above example). Right now we only know how to extract data with the following querie:
select
ID,
json_extract(DATA,'var1str'),
json_extract(DATA,'var2double'),
FROM table
With this querie, only the values of var1str and var2double will be returned as result. Values of variable 3 and 4 are ignored. There is no overview of what possible variables are hiding in the data fields.
With almost 60.000 entries and over 3.000 possible unique variable names, I would like to create a query that loops through all of the 60.000 DATA-fields and extracts every unique variable name that is found in there.
Solution?
The querie I am looking for would give the following result:
var1str
var2double
var3integer
var4str
My knowledge of MySql is very limited. Any direction given to get to this solution is much appreciated.

Answer 1

What version of MySQL are you using?.
From MySQL 8.0.4 and later JSON_TABLE function is supported and can be useful in this case.
mysql> SELECT VERSION();
+-----------+
| VERSION() |
+-----------+
| 8.0.11 |
+-----------+
1 row in set (0.00 sec)
mysql> DROP TABLE IF EXISTS `table`;
Query OK, 0 rows affected (0.09 sec)
mysql> CREATE TABLE IF NOT EXISTS `table` (
-> `ID` BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
-> `DATA` JSON NOT NULL
-> ) AUTO_INCREMENT=100;
Query OK, 0 rows affected (0.00 sec)
mysql> INSERT INTO `table`
-> (`DATA`)
-> VALUES
-> ('{"var1str": "sometxt", "var2double": 0.01, "var3integer": 1, "var4str": "another text"}'),
-> ('{"var3integer": 5, "var2double": 2.05, "var1str": "txt", "var4str": "more text"}');
Query OK, 2 rows affected (0.00 sec)
Records: 2 Duplicates: 0 Warnings: 0
mysql> SELECT
-> DISTINCT `der`.`key`
-> FROM
-> `table`,
-> JSON_TABLE(
-> JSON_KEYS(`DATA`), '$[*]'
-> COLUMNS(
-> `key` VARCHAR(64) PATH "$"
-> )
-> ) `der`;
+-------------+
| key |
+-------------+
| var1str |
| var4str |
| var2double |
| var3integer |
+-------------+
4 rows in set (0.01 sec)
Be aware of the Bug #90610 ERROR 1142 (42000) when using JSON_TABLE.