Get Json keys in a MySQL for a particular value - mysql

I just installed MySQL 5.7.27 and I would like to use some Json fields so, I created some records, for example this value in a field:
{
"Intitule": {
"name": "Intitule de la formation",
"stats": false,
"is_array": false,
"is_filter": true,
"chart": "pie",
"col": "6"
},
"Fin": {
"name": "Date de fin",
"stats": false,
"is_array": false,
"is_filter": false,
"chart": "pie",
"col": "6"
}
}
I would like to know how can I retrieve keys of elements which contain "is_filter":true, dans the associated "name"
I tried with JSON_SEARCH, JSON_EXTRACT but I think I don't use it well.
For example, for this value, I expect the output will be:
Key: Intitule
Name: Intitule de la formation
Because is_filter is true, but not for "Fin"
Thank you for your help!

Like i said, it can be a challenge parsing JSON in MySQL as you are dealing with text based keys here.
So you would need to use JSON_KEYS() to get those in combination with a number generator a dynamic JSON path is generated to be used in JSON_EXTRACT()
MySQL 's 8 function JSON_TABLE() makes it much much more easy..
Keep in mind this answer is purely meant for educational fun
Bill Karwin 's comment
You should either structure your JSON differently to support the
search you need to do, or else you should not use JSON. Just store the
data in multiple rows, and then SELECT * FROM mytable WHERE is_filter = true -
Query
SELECT
JSON_UNQUOTE(
JSON_EXTRACT(json , CONCAT('$.', SUBSTRING_INDEX(
SUBSTRING_INDEX(json_parsed, ',', number_generator.number)
, ','
, -1
), '.name'))) AS name
FROM (
SELECT
#row := #row + 1 AS number
FROM (
SELECT 0 UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9
) row1
CROSS JOIN (
SELECT 0 UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9
) row2
CROSS JOIN (
SELECT #row := 0
) init_user_params
) AS number_generator
CROSS JOIN (
SELECT
SUBSTRING(json_keys, 2, json_keys_length - 2) AS json_parsed
, json_keys
, json
, JSON_LENGTH(json_keys) AS json_array_length
FROM (
SELECT
JSON_KEYS(record.json) AS json_keys
, json
, LENGTH(JSON_KEYS(record.json)) AS json_keys_length
FROM (
SELECT
'{
"Intitule": {
"name": "Intitule de la formation",
"stats": false,
"is_array": false,
"is_filter": true,
"chart": "pie",
"col": "6"
},
"Fin": {
"name": "Date de fin",
"stats": false,
"is_array": false,
"is_filter": false,
"chart": "pie",
"col": "6"
}
}' AS json
FROM
DUAL
) AS record
) AS json_information
) AS json_init
WHERE
number_generator.number BETWEEN 0 AND json_array_length
AND
JSON_EXTRACT(json , CONCAT('$.', SUBSTRING_INDEX(
SUBSTRING_INDEX(json_parsed, ',', number_generator.number)
, ','
, -1
), '.is_filter')) = true
Result
| name |
| ------------------------ |
| Intitule de la formation |
see demo

This answer is to expand on the comment I made above.
When you store data in non-relational format, you should understand that the structure of the data must serve the queries you need to run. You can't just store any data and expect it to be easy to query.
For example, if you need to search for the subdocument where is_filter is true, then store your JSON like this:
{
"filter": {
"label": "Intitule",
"name": "Intitule de la formation",
"stats": false,
"is_array": false,
"chart": "pie",
"col": "6"
},
"not_filter": {
"label": "Fin",
"name": "Date de fin",
"stats": false,
"is_array": false,
"chart": "pie",
"col": "6"
}
}
If you need to use a variety of queries against the data, then using non-relational data like JSON is probably not as useful as storing the data in a relational manner.
mysql> select * from mytable;
+----------+--------------------------+-------+----------+-----------+-------+------+
| label | name | stats | is_array | is_filter | chart | col |
+----------+--------------------------+-------+----------+-----------+-------+------+
| Intitule | Intitule de la formation | 0 | 0 | 1 | pie | 6 |
| Fin | Date de fin | 0 | 0 | 0 | pie | 6 |
+----------+--------------------------+-------+----------+-----------+-------+------+
Then you can query by any attribute:
SELECT label FROM mytable WHERE is_filter = true;
NoSQL is more flexible than SQL for storing data.
SQL is more flexible than NoSQL for querying data.

Related

Nested JSON in MySQL with filtering

Here's the JSON that I have in the MySQL table:
{
"qmSubsegmentFlags": [
{
"text": "Accuracy",
"id": 1,
"children": [
{
"text": "Mistranslation",
"id": 2,
"children": [
{
"text": "Terminology",
"id": 3
}
]
},
{
"text": "Omission",
"id": 4
},
{
"text": "Addition",
"id": 5
},
{
"text": "Untranslated",
"id": 6
}
]
}
]
}
So you see, id's and text can be in different levels. I only have the value of the id and have to get the text for this with SQL. Below a JSON path that works for the first level, but not for all levels. Neither does it work in MySQL, since MySQL only supports limited filtering.
$.qmSubsegmentFlags[?(#.id==1)].text
A little SQL that goes into the right direction, but also does not solve the issue.
SELECT JSON_EXTRACT(t.qmSubsegmentFlags, '$.qmSubsegmentFlags[*].text') AS lqa FROM translate5.LEK_task AS t where id = 3719;
I used to be a programmer, but after 4 years of project management, my skills are dead. But I need a proof of concept until monday. Any help would be great!
Here's a solution tested on MySQL 8.0.32.
WITH RECURSIVE tree AS (
SELECT j.id, j.text, j.children
FROM LEK_task CROSS JOIN JSON_TABLE(
JSON_EXTRACT(qmSubsegmentFlags, '$.qmSubsegmentFlags'),
'$[*]' COLUMNS (
text TEXT PATH '$.text',
id INT PATH '$.id',
children JSON PATH '$.children'
)
) AS j
WHERE LEK_task.id = 3719
UNION ALL
SELECT j.id, j.text, j.children
FROM tree CROSS JOIN JSON_TABLE(
children,
'$[*]' COLUMNS(
text TEXT PATH '$.text',
id INT PATH '$.id',
children JSON PATH '$.children'
)
) AS j
)
SELECT id, text FROM tree;
Result:
+------+----------------+
| id | text |
+------+----------------+
| 1 | Accuracy |
| 2 | Mistranslation |
| 4 | Omission |
| 5 | Addition |
| 6 | Untranslated |
| 3 | Terminology |
+------+----------------+
You can add a WHERE id = 1 to the last line of the query to get the row for a specific id.

Converting a Dictionary in MySql 8

I have a JSON structure stored in a MySql table. Now, months later, I have a requirement to join to pieces found deep in the bowels of this JSON string.
{
"id": "000689ba-9891-4d82-ad13-a7b96dc08ec4",
"type": "emp",
"firstName": "Brooke",
"facilities": {
​​​​​"0001": [{​​​​​
"id": 1,
"code": "125",
"name": "DSL",
"type": "MGMTSupport",
"payRate": 18}​​​​​],
"0002": [
{
​​​​​"id": 1,
"code": "100",
"name": "Server",
"type": "FOH",
"payRate": 8
}, {
​​​​​"id": 2,
"code": "320",
"name": "NCFOHTrainer",
"type": "NCHourlyTraining",
"payRate": 14.5
}​​​]
},
"permissions": ["read:availability", "..."],
"primaryJobCode": "150",
"primaryPayRate": 9,
"primaryFacility": "0260"
}​
​​​​The big question is: How do I shape this as a query in MySql when the facilities do not follow a single key/value pattern? i.e.: the key to the first entry is the facilityId so I cannot use a path like '$.0001' and the dictionary value is an array so how do I path that correctly?
select id as EmployeeId
, companyId as cpkEmployeeId
, json_table( `data`
, '$.facilities[*]' COLUMNS( facilityId VARCHAR(10) PATH '$.????'
, NESTED PATH '??? $[*] ???' COLUMNS ( code VARCHAR(10) PATH '$.code'
, payRate DECIMAL(8,4) PATH '$.payRate') facilities
from employee
;
Yea - the above does not work. Any help appreciated.
Desired output?
[Other columns from the table] plus facilityId, code & payrate.
A single row in the native table could produce something like:
id | companyId | facilityId | code | payRate
--------+-----------+------------+------+---------
1 | 324337 | 0001 | 125 | 18.0000
1 | 324337 | 0002 | 100 | 8.0000
1 | 324337 | 0002 | 320 | 14.5000
WITH
cte AS (
SELECT test.id,
test.value,
jsontable.facilityId,
CONCAT('$.facilities."', jsontable.facilityId, '"') path
FROM test
CROSS JOIN JSON_TABLE(JSON_KEYS(test.value, '$.facilities'),
'$[*]' COLUMNS (facilityId CHAR(4) PATH '$')) jsontable
)
SELECT cte.id,
cte.facilityId,
jsontable.code,
jsontable.payRate
FROM cte
CROSS JOIN JSON_TABLE(JSON_EXTRACT(cte.value, cte.path),
'$[*]' COLUMNS (code CHAR(3) PATH '$.code',
payRate DECIMAL(6, 4) PATH '$.payRate')) jsontable
https://dbfiddle.uk/?rdbms=mysql_8.0&fiddle=18ab7b6f181b61fb53f88a6de6e049be

MySQL Parse and Split JSON value

I have a column which contains a JSON value of different lengths
["The Cherries:2.50","Draw:3.25","Swansea Jacks:2.87"]
I want to split them and store into a JSON like so:
[
{
name: "The Cherries",
odds: 2.50
},
{
name: "Draw",
odds: 3.25
},
{
name: "Swansea",
odds: 2.87
},
]
What I did right now is looping and splitting them in the UI which to me is quite heavy for the client. I want to parse and split them all in a single query.
If you are running MySQL 8.0, you can use json_table() to split the original arrayto rows, and then build new objects and aggregate them with json_arrayagg().
We need a primary key column (or set of columns) so we can properly aggreate the generated rows, I assumed id:
select
t.id,
json_arrayagg(json_object(
'name', substring(j.val, 1, locate(':', j.val) - 1),
'odds', substring(j.val, locate(':', j.val) + 1)
)) new_js
from mytable t
cross join json_table(t.js, '$[*]' columns (val varchar(500) path '$')) as j
group by t.id
Demo on DB Fiddle
Sample data:
id | js
-: | :-------------------------------------------------------
1 | ["The Cherries:2.50", "Draw:3.25", "Swansea Jacks:2.87"]
Query results:
id | new_js
-: | :----------------------------------------------------------------------------------------------------------------------
1 | [{"name": "The Cherries", "odds": "2.50"}, {"name": "Draw", "odds": "3.25"}, {"name": "Swansea Jacks", "odds": "2.87"}]
You can use json_table to create rows from the json object.
Just replace table_name with your table name and json with the column that contains json
SELECT json_arrayagg(json_object('name',SUBSTRING_INDEX(person, ':', 1) ,'odds',SUBSTRING_INDEX(person, ':', -1) ))
FROM table_name,
JSON_TABLE(json, '$[*]' COLUMNS (person VARCHAR(40) PATH '$') people;
Here is a Db fiddle you can refer
https://dbfiddle.uk/?rdbms=mysql_8.0&fiddle=801de9f067e89a48d45ef9a5bd2d094a

How to select JSON object from JSON array field of mysql by some condition

I have a table with JSON field which contains an array of JSON objects. I need to select objects by some condition.
Create and fill a table:
CREATE TABLE test (
id INT AUTO_INCREMENT PRIMARY KEY,
json_list JSON
);
INSERT INTO test(json_list) VALUES
("{""list"": [{""type"": ""color"", ""value"": ""red""}, {""type"": ""shape"", ""value"": ""oval""}, {""type"": ""color"", ""value"": ""green""}]}"),
("{""list"": [{""type"": ""shape"", ""value"": ""rect""}, {""type"": ""color"", ""value"": ""olive""}]}"),
("{""list"": [{""type"": ""color"", ""value"": ""red""}]}")
;
Now I need to select all objects with type = color from all rows.
I want to see this output:
id extracted_value
1 {"type": "color", "value": "red"}
1 {"type": "color", "value": "green"}
2 {"type": "color", "value": "olive"}
3 {"type": "color", "value": "red"}
It would be good to get this too:
id color
1 red
1 green
2 olive
3 red
I can't change the DB or JSON.
I'm using MySQL 5.7
My current solution
My solution is to cross join the table with some index set and then extract all elements of JSON array.
I don't like it as if possible object count in one array is large it is required to have all indexes till the maximum one. It makes the query slow as it won't stop calculation of JSON value when the end of array is reached.
SELECT
test.id,
JSON_EXTRACT(test.json_list, CONCAT('$.list[', ind.ind, ']')),
ind.ind
FROM
test
CROSS JOIN
(SELECT 0 AS ind UNION ALL SELECT 1 AS ind UNION ALL SELECT 2 AS ind) ind
WHERE
JSON_LENGTH(json_list, "$.list") > ind.ind
AND JSON_EXTRACT(json_list, CONCAT('$.list[', ind.ind, '].type')) = "color";
It is easy to get only values by changing JSON_EXTRACT path. But is it there a better way?
Edits
Added a check for json_list.list length. This filtered out 67% of derived table rows in this case.
SELECT JSON_EXTRACT(json_list, '$.list[*]')
FROM `test`
where JSON_CONTAINS(json_list, '{"type":"color"}', '$.list')
So current best solution is mine:
SELECT
test.id,
JSON_EXTRACT(test.json_list, CONCAT('$.list[', ind.ind, ']')),
ind.ind
FROM
test
CROSS JOIN
(SELECT 0 AS ind UNION ALL SELECT 1 AS ind UNION ALL SELECT 2 AS ind) ind
WHERE
JSON_LENGTH(json_list, "$.list") > ind.ind
AND JSON_EXTRACT(json_list, CONCAT('$.list[', ind.ind, '].type')) = "color";

Postgresql json querying in 9.3+ on nested payloads

Using postgresql 9.3 (and the new json awesomness) if I have a simple table named 'races' with a two column description such as:
race-id integer,
race-data json
And the json is a payload for each race is something like
{ "race-time": some-date,
"runners": [ { "name": "fred","age": 30, "position": 1 },
{ "name": "john","age": 29, "position": 3 },
{ "name": "sam","age": 31, "position": 2 } ],
"prize-money": 200 }
How can I query the table for:
1) Races where sam has come 1st
2) Races where sam has come 1st and john has come 2nd
3) Where the number of runners with age greater than 30 is > 5 and prize-money > 5000
My experimentation (particularly in querying a nested array payload) so far has lead to further normalizing the data, i.e. creating a table called runners just to make such queries. Ideally I'd like to use this new fangled json query awesomeness but I can't seem to make heads or tails of it in respective to the 3 simple queries.
You can unwind json into one record and then do your queries as you want (see json functions):
with cte as (
select
race_id,
json_array_elements(r.race_data->'runners') as d,
(r.race_data->>'prize-money')::int as price_money
from races as r
), cte2 as (
select
race_id, price_money,
max(case when (d->>'position')::int = 1 then d->>'name' end) as name1,
max(case when (d->>'position')::int = 2 then d->>'name' end) as name2,
max(case when (d->>'position')::int = 3 then d->>'name' end) as name3
from cte
group by race_id, price_money
)
select *
from cte2
where name1 = 'sam' and name2 = 'john'
sql fiddle demo
It's a bit complicated because of your JSON structure. I think that if you change your structure a bit, your queries could be much simplier:
{
"race-time": some-date,
"runners":
{
"1": {"name": "fred","age": 30},
"2": {"name": "sam","age": 31},
"3": {"name": "john","age": 29}
},
"prize-money": 200
}
you can use ->> and -> operators or json_extract_path_text function to get data you need and then use it in the where clause:
select *
from races as r
where
r.race_data->'runners'->'1'->>'name' = 'sam';
select *
from races as r
where
json_extract_path_text(r.race_data, 'runners','1','name') = 'sam' and
json_extract_path_text(r.race_data, 'runners','2','name') = 'john';
select *
from races as r
where
(r.race_data->>'prize-money')::int > 100 and
(
select count(*)
from json_each(r.race_data->'runners')
where (value->>'age')::int >= 30
) >= 2
sql fiddle demo