MySQL Parse and Split JSON value - mysql

I have a column which contains a JSON value of different lengths
["The Cherries:2.50","Draw:3.25","Swansea Jacks:2.87"]
I want to split them and store into a JSON like so:
[
{
name: "The Cherries",
odds: 2.50
},
{
name: "Draw",
odds: 3.25
},
{
name: "Swansea",
odds: 2.87
},
]
What I did right now is looping and splitting them in the UI which to me is quite heavy for the client. I want to parse and split them all in a single query.

If you are running MySQL 8.0, you can use json_table() to split the original arrayto rows, and then build new objects and aggregate them with json_arrayagg().
We need a primary key column (or set of columns) so we can properly aggreate the generated rows, I assumed id:
select
t.id,
json_arrayagg(json_object(
'name', substring(j.val, 1, locate(':', j.val) - 1),
'odds', substring(j.val, locate(':', j.val) + 1)
)) new_js
from mytable t
cross join json_table(t.js, '$[*]' columns (val varchar(500) path '$')) as j
group by t.id
Demo on DB Fiddle
Sample data:
id | js
-: | :-------------------------------------------------------
1 | ["The Cherries:2.50", "Draw:3.25", "Swansea Jacks:2.87"]
Query results:
id | new_js
-: | :----------------------------------------------------------------------------------------------------------------------
1 | [{"name": "The Cherries", "odds": "2.50"}, {"name": "Draw", "odds": "3.25"}, {"name": "Swansea Jacks", "odds": "2.87"}]

You can use json_table to create rows from the json object.
Just replace table_name with your table name and json with the column that contains json
SELECT json_arrayagg(json_object('name',SUBSTRING_INDEX(person, ':', 1) ,'odds',SUBSTRING_INDEX(person, ':', -1) ))
FROM table_name,
JSON_TABLE(json, '$[*]' COLUMNS (person VARCHAR(40) PATH '$') people;
Here is a Db fiddle you can refer
https://dbfiddle.uk/?rdbms=mysql_8.0&fiddle=801de9f067e89a48d45ef9a5bd2d094a

Related

How to select JSON object from JSON array field of mysql by some condition

I have a table with JSON field which contains an array of JSON objects. I need to select objects by some condition.
Create and fill a table:
CREATE TABLE test (
id INT AUTO_INCREMENT PRIMARY KEY,
json_list JSON
);
INSERT INTO test(json_list) VALUES
("{""list"": [{""type"": ""color"", ""value"": ""red""}, {""type"": ""shape"", ""value"": ""oval""}, {""type"": ""color"", ""value"": ""green""}]}"),
("{""list"": [{""type"": ""shape"", ""value"": ""rect""}, {""type"": ""color"", ""value"": ""olive""}]}"),
("{""list"": [{""type"": ""color"", ""value"": ""red""}]}")
;
Now I need to select all objects with type = color from all rows.
I want to see this output:
id extracted_value
1 {"type": "color", "value": "red"}
1 {"type": "color", "value": "green"}
2 {"type": "color", "value": "olive"}
3 {"type": "color", "value": "red"}
It would be good to get this too:
id color
1 red
1 green
2 olive
3 red
I can't change the DB or JSON.
I'm using MySQL 5.7
My current solution
My solution is to cross join the table with some index set and then extract all elements of JSON array.
I don't like it as if possible object count in one array is large it is required to have all indexes till the maximum one. It makes the query slow as it won't stop calculation of JSON value when the end of array is reached.
SELECT
test.id,
JSON_EXTRACT(test.json_list, CONCAT('$.list[', ind.ind, ']')),
ind.ind
FROM
test
CROSS JOIN
(SELECT 0 AS ind UNION ALL SELECT 1 AS ind UNION ALL SELECT 2 AS ind) ind
WHERE
JSON_LENGTH(json_list, "$.list") > ind.ind
AND JSON_EXTRACT(json_list, CONCAT('$.list[', ind.ind, '].type')) = "color";
It is easy to get only values by changing JSON_EXTRACT path. But is it there a better way?
Edits
Added a check for json_list.list length. This filtered out 67% of derived table rows in this case.
SELECT JSON_EXTRACT(json_list, '$.list[*]')
FROM `test`
where JSON_CONTAINS(json_list, '{"type":"color"}', '$.list')
So current best solution is mine:
SELECT
test.id,
JSON_EXTRACT(test.json_list, CONCAT('$.list[', ind.ind, ']')),
ind.ind
FROM
test
CROSS JOIN
(SELECT 0 AS ind UNION ALL SELECT 1 AS ind UNION ALL SELECT 2 AS ind) ind
WHERE
JSON_LENGTH(json_list, "$.list") > ind.ind
AND JSON_EXTRACT(json_list, CONCAT('$.list[', ind.ind, '].type')) = "color";

Counting words in JSON array mySQL

I need to count true and false words in a JSON datatype.
I have this JSON in the cell:
{"1": true, "2": false, "3": true}
The number of values may vary. I realize that I can count the total number of values in the array but how can I count true and false separately?
For total count I used JSON_LENGTH()
One option would be using below approach containing JSON_LENGTH(), JSON_EXTRACT() and JSON_SEARCH() functions together even for the version 5.7 (5.7.13+) where an array(js) extracted in the subquery and they're splitted to individual array for each values (true and false) by using JSON_SEARCH() function containing all as the second argument, and then counted by JSON_LENGTH() function :
SELECT ID,
JSON_LENGTH( JSON_SEARCH(js, 'all', 'true') ) AS Cnt_True,
JSON_LENGTH( JSON_SEARCH(js, 'all', 'false') ) AS Cnt_False
FROM ( SELECT *, JSON_EXTRACT(jsdata, '$.*') AS js
FROM tab ) t
provided JSON field has quoted values such as "true" and "false"
JSON_EXTRACT(jsdata, '$.*') still can be used in case that the boolean values are unquoted as in your case. But, this time some string operations would be needed. Here, I preferred using CHAR_LENGTH() function :
SELECT ID,
CHAR_LENGTH(js) - CHAR_LENGTH(REPLACE(js, 'true', SPACE(LENGTH('true')-1)))
AS Cnt_True,
CHAR_LENGTH(js) - CHAR_LENGTH(REPLACE(js, 'false', SPACE(LENGTH('false')-1)))
AS Cnt_False
FROM
( SELECT *, JSON_EXTRACT(jsdata, '$.*') AS js
FROM tab ) t
Demo
Here is one option using json_table(), availabe in MySQL 8.0.
You can first turn each json object to an array of values using path $.*'. Then, you can pass the resulting json array to json_table(), which will put each value on a separate row. The final step is conditional aggregation.
Assuming that the json column is called js, that would be:
select sum(x.val = 'true') cnt_true, sum(x.val = 'false') cnt_false
from mytable t
cross join json_table(js -> '$.*', '$[*]' columns (val varchar(5) path '$')) x
Demo on DB Fiddle
Sample data (I added another row to make this more meaningful):
| js |
| :--------------------------------- |
| {"1": true, "2": false, "3": true} |
| {"bar": false, "foo": true} |
Results:
cnt_true | cnt_false
-------: | --------:
3 | 2

MySQL 8 search JSON key by value in array

I've got MySQL table with JSON field, where I store data in such a format.
{
"fields": {
"1": {
"s": "y"
},
"2": {
"s": "n"
}
}
}
I need to obtain the keys in fields, e.g. 1 or 2 given the value of s.
Example query:
create table mytable ( mycol json );
insert into mytable set mycol = '{"fields": {"1": {"s": "y"},"2": {"s": "n"}}}';
select j.* from mytable, JSON_TABLE(mycol,
'$.fields.*' COLUMNS (
json_key VARCHAR(10) PATH '$',
s VARCHAR(10) PATH '$.s'
)
) AS j where j.s = 'y';
gives:
# json_key, s
null, y
I would expect to get
# json_key, s
1, y
Is it possible to get that data somehow?
I don't need the results in row / table format. I would be happy to get the comma separated list of IDs (json_keys) meeting my criterium.
EDIT:
I was also thinking about getting the paths using JSON_SEARCH and passing that to JSON_EXTRACT, this was achieved here: Combining JSON_SEARCH and JSON_EXTRACT get me: "Invalid JSON path expression."
Unfortunately the difference is that I would need to use JSON_SEARCH in all mode, as I need all results. In such a mode JSON_SEARCH returns list of paths, where as JSON_EXTRACT accepts list of arguments.
Try FOR ORDINALITY (see 12.17.6 JSON Table Functions), this type enumerates rows in the COLUMNS clause:
SELECT
JSON_UNQUOTE(
JSON_EXTRACT(
JSON_KEYS(`mycol` ->> '$.fields'),
CONCAT('$[', `j`.`row` - 1, ']')
)
) `json_key`,
`j`.`s`
FROM
`mytable`,
JSON_TABLE(
`mycol`,
'$.fields.*' COLUMNS (
`row` FOR ORDINALITY,
`s` VARCHAR(10) PATH '$.s'
)
) `j`
WHERE
`j`.`s` = 'y';
See dbfiddle.

Convert flattened key/value table into hierarchical JSON in PostgreSQL

I have a PostgreSQL table with unique key/value pairs, which were originally in a JSON format, but have been normalized and melted:
key | value
-----------------------------
name | Bob
address.city | Vancouver
address.country | Canada
I need to turn this into a hierarchical JSON:
{
"name": "Bob",
"address": {
"city": "Vancouver",
"country": "Canada"
}
}
Is there a way to do this easily within SQL?
jsonb_set() almost does everything for you, but unfortunately it can only create missing leafs (i.e. missing last keys on a path), but not whole missing branches. To overcome this, here is a modified version of it, which can set values on any missing levels:
create function jsonb_set_rec(jsonb, jsonb, text[])
returns jsonb
language sql
as $$
select case
when array_length($3, 1) > 1 and ($1 #> $3[:array_upper($3, 1) - 1]) is null
then jsonb_set_rec($1, jsonb_build_object($3[array_upper($3, 1)], $2), $3[:array_upper($3, 1) - 1])
else jsonb_set($1, $3, $2, true)
end
$$;
Now you only need to apply this function one-by-one to your rows, starting with an empty json object: {}. You can do this with either recursive CTEs:
with recursive props as (
(select distinct on (grp)
pk, grp, jsonb_set_rec('{}', to_jsonb(value), string_to_array(key, '.')) json_object
from eav_tbl
order by grp, pk)
union all
(select distinct on (grp)
eav_tbl.pk, grp, jsonb_set_rec(json_object, to_jsonb(value), string_to_array(key, '.'))
from props
join eav_tbl using (grp)
where eav_tbl.pk > props.pk
order by grp, eav_tbl.pk)
)
select distinct on (grp)
grp, json_object
from props
order by grp, pk desc;
Or, with a custom aggregate defined as:
create aggregate jsonb_set_agg(jsonb, text[]) (
sfunc = jsonb_set_rec,
stype = jsonb,
initcond = '{}'
);
your query could became as simple as:
select grp, jsonb_set_agg(to_jsonb(value), string_to_array(key, '.'))
from eav_tbl
group by grp;
https://rextester.com/TULNU73750
There are no ready to use tools for this. The function generates a hierarchical json object based on a path:
create or replace function jsonb_build_object_from_path(path text, value text)
returns jsonb language plpgsql as $$
declare
obj jsonb;
keys text[] := string_to_array(path, '.');
level int := cardinality(keys);
begin
obj := jsonb_build_object(keys[level], value);
while level > 1 loop
level := level- 1;
obj := jsonb_build_object(keys[level], obj);
end loop;
return obj;
end $$;
You also need the aggregate function jsonb_merge_agg(jsonb) described in this answer. The query:
with my_table (path, value) as (
values
('name', 'Bob'),
('address.city', 'Vancouver'),
('address.country', 'Canada'),
('first.second.third', 'value')
)
select jsonb_merge_agg(jsonb_build_object_from_path(path, value))
from my_table;
gives this object:
{
"name": "Bob",
"first":
{
"second":
{
"third": "value"
}
},
"address":
{
"city": "Vancouver",
"country": "Canada"
}
}
The function do not recognize json arrays.
I can't really think of something simpler, although I think there should be an easier way.
I assume there is some additional column that can be used to bring the keys that belong to one "person" together, I used p_id for that in my example.
select p_id,
jsonb_object_agg(k, case level when 1 then v -> k else v end)
from (
select p_id,
elements[1] k,
jsonb_object_agg(case cardinality(elements) when 1 then ky else elements[2] end, value) v,
max(cardinality(elements)) as level
from (
select p_id,
"key" as ky,
string_to_array("key", '.') as elements, value
from kv
) t1
group by p_id, k
) t2
group by p_id;
The innermost query just converts the dot notation to an array for easier access later.
The next level then builds JSON objects depending on the "key". For the "single level" keys, it just uses key/value, for the others it uses the second element + the value and then aggregates those that belong together.
The second query level returns the following:
p_id | k | v | level
-----+---------+--------------------------------------------+------
1 | address | {"city": "Vancouver", "country": "Canada"} | 2
1 | name | {"name": "Bob"} | 1
2 | address | {"city": "Munich", "country": "Germany"} | 2
2 | name | {"name": "John"} | 1
The aggregation done in the second step, leaves one level too much for the "single element" keys, and that's what we need level for.
If that distinction wasn't made, the final aggregation would return {"name": {"name": "Bob"}, "address": {"city": "Vancouver", "country": "Canada"}} instead of the wanted: {"name": "Bob", "address": {"city": "Vancouver", "country": "Canada"}}.
The expression case level when 1 then v -> k else v end essentially turns {"name": "Bob"} back to "Bob".
So, with the following sample data:
create table kv (p_id integer, "key" text, value text);
insert into kv
values
(1, 'name','Bob'),
(1, 'address.city','Vancouver'),
(1, 'address.country','Canada'),
(2, 'name','John'),
(2, 'address.city','Munich'),
(2, 'address.country','Germany');
then query returns:
p_id | jsonb_object_agg
-----+-----------------------------------------------------------------------
1 | {"name": "Bob", "address": {"city": "Vancouver", "country": "Canada"}}
2 | {"name": "John", "address": {"city": "Munich", "country": "Germany"}}
Online example: https://rextester.com/SJOTCD7977
create table kv (key text, value text);
insert into kv
values
('name','Bob'),
('address.city','Vancouver'),
('address.country','Canada'),
('name','John'),
('address.city','Munich'),
('address.country','Germany');
create view v_kv as select row_number() over() as nRec, key, value from kv;
create view v_datos as
select k1.nrec, k1.value as name, k2.value as address_city, k3.value as address_country
from v_kv k1 inner join v_kv k2 on (k1.nrec + 1 = k2.nrec)
inner join v_kv k3 on ((k1.nrec + 2= k3.nrec) and (k2.nrec + 1 = k3.nrec))
where mod(k1.nrec, 3) = 1;
select json_agg(json_build_object('name',name, 'address', json_build_object('city',address_city, 'country', address_country)))
from v_datos;

query to Extract from Json in Postgres

I've a json object in my postgres db, which looks like as given below
{"Actor":[{"personName":"Shashi Kapoor","characterName":"Prem"},{"personName":"Sharmila Tagore","characterName":"Preeti"},{"personName":"Shatrughan Sinha","characterName":"Dr. Amar"]}
Edited (from editor: left the original because it is an invalid json, in my edit I fixed it)
{
"Actor":[
{
"personName":"Shashi Kapoor",
"characterName":"Prem"
},
{
"personName":"Sharmila Tagore",
"characterName":"Preeti"
},
{
"personName":"Shatrughan Sinha",
"characterName":"Dr. Amar"
}
]
}
the name of the column be xyz and I've a corresponding content_id.
I need to retrieve content_ids that have Actor & personName = Sharmila Tagore.
I tried many queries, among those these two where very possible query to get but still i didn't get.
SELECT content_id
FROM content_table
WHERE cast_and_crew #>> '{Actor,personName}' = '"C. R. Simha"'
.
SELECT cast_and_crew ->> 'content_id' AS content_id
FROM content_table
WHERE cast_and_crew ->> 'Actor' -> 'personName' = 'C. R. Simha'
You should use jsonb_array_elements() to search in a nested jsonb array:
select content_id, value
from content_table,
lateral jsonb_array_elements(cast_and_crew->'Actor');
content_id | value
------------+-----------------------------------------------------------------
1 | {"personName": "Shashi Kapoor", "characterName": "Prem"}
1 | {"personName": "Sharmila Tagore", "characterName": "Preeti"}
1 | {"personName": "Shatrughan Sinha", "characterName": "Dr. Amar"}
(3 rows)
Column value is of the type jsonb so you can use ->> operator for it:
select content_id, value
from content_table,
lateral jsonb_array_elements(cast_and_crew->'Actor')
where value->>'personName' = 'Sharmila Tagore';
content_id | value
------------+--------------------------------------------------------------
1 | {"personName": "Sharmila Tagore", "characterName": "Preeti"}
(1 row)
Note, if you are using json (not jsonb) use json_array_elements() of course.