Background
I have a MySQL table with a json field. That field stores a array of json objects.
The order is not always the same, so i need to get the path to a key for updating operations.
Entity | ID, jsonField |
Entity | 1, [{clazz:'health', hp:'100'},{...},{...}] // Health the first index
Entity | 2, [{...},{...},{clazz:'health', hp:'25'}] // Health at the last index
The question
How do i get the path to the .hp field for every single entity in order to update its value ? Or... a bit more precious, how do we set the .hp field to, lets say 100, in every entity jsonField array, regardless of its position ?
You can use a recursive CTE to find the index for the hp key in the array (if it exists), and then use that result to make the update:
with recursive cte(id, js, ind, f) as (
select e.id, e.jsonfield, 0, json_extract(e.jsonfield, '$[0].hp') is not null from entities e
union all
select c.id, c.js, c.ind+1, json_extract(c.js, concat('$[', c.ind+1, '].hp')) is not null from cte c where not c.f and c.ind+1 < json_length(c.js)
),
inds(id, ind) as (select id, ind from cte where f)
update entities e join inds i on e.id = i.id set e.jsonfield = json_set(e.jsonfield, concat('$[', i.ind, '].hp'), '100');
select * from entites;
Output:
entity
id
jsonfield
entity
1
[{"hp": "100", "clazz": "health"}, {"x": "1"}, {"y": "2"}]
entity
2
[{"x": "1"}, {"y": "2"}, {"hp": "100", "clazz": "health"}]
Related
I have a MySQL 8.0.22 JSON column containing objects with keys that aren't known in advance:
'{"x": 1, "y": 2, "z": 3}'
'{"e": 4, "k": 5}'
I want to use JSON_TABLE to expand these values into multiple rows containing key value pairs:
key
value
x
1
y
2
z
3
e
4
k
5
The difficulty of course is that the keys aren't known a priori. The best thing I've come up with is...
SET #json_doc = '{"x": 1, "y": 2, "z": 3}';
SELECT a.seq, b.k, a.v
FROM
JSON_TABLE(
#json_doc,
"$.*"
COLUMNS(
seq FOR ordinality,
v INT PATH "$"
)
) AS a,
JSON_TABLE(
JSON_KEYS(#json_doc),
"$[*]"
COLUMNS(
seq FOR ordinality,
k CHAR(1) PATH "$"
)
) AS b
WHERE a.seq = b.seq;
This feels strange because it uses two JSON_TABLE calls, does a cross join on the values and keys, then keeps the ones that align. I'd like to find a simpler query like this...
SELECT a.seq, b.k, a.v
FROM
JSON_TABLE(
#json_doc,
"$.*"
COLUMNS(
seq FOR ordinality,
k CHAR(1) PATH "?" -- <-- what do I put here to find each key?
v INT PATH "$"
)
) AS a,
I know this problem can probably be solved with CTEs or a numbers table and JSON_EXTRACT. But, I'd like to find something performant and readable if possible.
You can use enumarete by using ROW_NUMBER() window function while determining the key values through use of JSON_KEYS(), and then extract the respective keys by using JSON_EXTRACT() from the arrays we got such as
WITH k AS
(
SELECT *,
ROW_NUMBER() OVER(PARTITION BY `jsdata` ORDER BY value DESC) AS rn,
JSON_KEYS(`jsdata`) AS jk
FROM `tab` AS t
JOIN JSON_TABLE(`jsdata`,'$.*' COLUMNS (value INT PATH '$')) j
)
SELECT JSON_UNQUOTE(JSON_EXTRACT(jk, CONCAT('$[',rn-1,']'))) AS "key",
value
FROM k
or use the following query as being more straightforward
SELECT JSON_UNQUOTE(
JSON_EXTRACT(JSON_KEYS(`jsdata`),
CONCAT('$[',
ROW_NUMBER() OVER(PARTITION BY `jsdata` ORDER BY value DESC)-1,
']'))
) AS "key", value
FROM `tab` AS t
JOIN JSON_TABLE(`jsdata`,'$.*' COLUMNS (value INT PATH '$')) j
Demo
Try to do JSON_EXTRACT directly after you got the JSON_KEYS as rows:
WITH j AS (
SELECT CAST('{"a": 1, "b": "-1", "c": null}' AS JSON) o UNION ALL
SELECT CAST('{"x": 2, "y": "-2", "z": null}' AS JSON)
)
SELECT k, JSON_EXTRACT(j.o, CONCAT('$."', jt.k, '"')) v
FROM j
, JSON_TABLE(JSON_KEYS(o), '$[*]' COLUMNS (k VARCHAR(200) PATH '$')) jt;
The answer by Barbaros can solve your problem with the demo data you provided, but it may not get what you want if your json objects have same value under different keys.
I have a column which contains a JSON value of different lengths
["The Cherries:2.50","Draw:3.25","Swansea Jacks:2.87"]
I want to split them and store into a JSON like so:
[
{
name: "The Cherries",
odds: 2.50
},
{
name: "Draw",
odds: 3.25
},
{
name: "Swansea",
odds: 2.87
},
]
What I did right now is looping and splitting them in the UI which to me is quite heavy for the client. I want to parse and split them all in a single query.
If you are running MySQL 8.0, you can use json_table() to split the original arrayto rows, and then build new objects and aggregate them with json_arrayagg().
We need a primary key column (or set of columns) so we can properly aggreate the generated rows, I assumed id:
select
t.id,
json_arrayagg(json_object(
'name', substring(j.val, 1, locate(':', j.val) - 1),
'odds', substring(j.val, locate(':', j.val) + 1)
)) new_js
from mytable t
cross join json_table(t.js, '$[*]' columns (val varchar(500) path '$')) as j
group by t.id
Demo on DB Fiddle
Sample data:
id | js
-: | :-------------------------------------------------------
1 | ["The Cherries:2.50", "Draw:3.25", "Swansea Jacks:2.87"]
Query results:
id | new_js
-: | :----------------------------------------------------------------------------------------------------------------------
1 | [{"name": "The Cherries", "odds": "2.50"}, {"name": "Draw", "odds": "3.25"}, {"name": "Swansea Jacks", "odds": "2.87"}]
You can use json_table to create rows from the json object.
Just replace table_name with your table name and json with the column that contains json
SELECT json_arrayagg(json_object('name',SUBSTRING_INDEX(person, ':', 1) ,'odds',SUBSTRING_INDEX(person, ':', -1) ))
FROM table_name,
JSON_TABLE(json, '$[*]' COLUMNS (person VARCHAR(40) PATH '$') people;
Here is a Db fiddle you can refer
https://dbfiddle.uk/?rdbms=mysql_8.0&fiddle=801de9f067e89a48d45ef9a5bd2d094a
I have a table with JSON field which contains an array of JSON objects. I need to select objects by some condition.
Create and fill a table:
CREATE TABLE test (
id INT AUTO_INCREMENT PRIMARY KEY,
json_list JSON
);
INSERT INTO test(json_list) VALUES
("{""list"": [{""type"": ""color"", ""value"": ""red""}, {""type"": ""shape"", ""value"": ""oval""}, {""type"": ""color"", ""value"": ""green""}]}"),
("{""list"": [{""type"": ""shape"", ""value"": ""rect""}, {""type"": ""color"", ""value"": ""olive""}]}"),
("{""list"": [{""type"": ""color"", ""value"": ""red""}]}")
;
Now I need to select all objects with type = color from all rows.
I want to see this output:
id extracted_value
1 {"type": "color", "value": "red"}
1 {"type": "color", "value": "green"}
2 {"type": "color", "value": "olive"}
3 {"type": "color", "value": "red"}
It would be good to get this too:
id color
1 red
1 green
2 olive
3 red
I can't change the DB or JSON.
I'm using MySQL 5.7
My current solution
My solution is to cross join the table with some index set and then extract all elements of JSON array.
I don't like it as if possible object count in one array is large it is required to have all indexes till the maximum one. It makes the query slow as it won't stop calculation of JSON value when the end of array is reached.
SELECT
test.id,
JSON_EXTRACT(test.json_list, CONCAT('$.list[', ind.ind, ']')),
ind.ind
FROM
test
CROSS JOIN
(SELECT 0 AS ind UNION ALL SELECT 1 AS ind UNION ALL SELECT 2 AS ind) ind
WHERE
JSON_LENGTH(json_list, "$.list") > ind.ind
AND JSON_EXTRACT(json_list, CONCAT('$.list[', ind.ind, '].type')) = "color";
It is easy to get only values by changing JSON_EXTRACT path. But is it there a better way?
Edits
Added a check for json_list.list length. This filtered out 67% of derived table rows in this case.
SELECT JSON_EXTRACT(json_list, '$.list[*]')
FROM `test`
where JSON_CONTAINS(json_list, '{"type":"color"}', '$.list')
So current best solution is mine:
SELECT
test.id,
JSON_EXTRACT(test.json_list, CONCAT('$.list[', ind.ind, ']')),
ind.ind
FROM
test
CROSS JOIN
(SELECT 0 AS ind UNION ALL SELECT 1 AS ind UNION ALL SELECT 2 AS ind) ind
WHERE
JSON_LENGTH(json_list, "$.list") > ind.ind
AND JSON_EXTRACT(json_list, CONCAT('$.list[', ind.ind, '].type')) = "color";
I have a table that describes a list of products made for a given order, in this every row on this table has to have a an id of the product and the reason why it was purchased,
I would like to build a json response that amounts to an array of locations the ordered product is for, within that, an array of unique product codes and with that an array of reasons why that unique product was ordered.
I've only been able to get the topmost part of my query defined but the join and sub-select nature of the request is actually getting me in a bit of a fiddle. Is this kind of thing actually possible in plpgsql?
Additionally I'd like to join product_order.reason on product_order_reason.id and retrieve the longform_text inside the table associated with this row but I figure the bigger thing is to get the return at all and that's where i've been left stumped.
product
id |name |cost |cost_rate|
--------|------------------------|------|---------|
WALLC |Wall Clock | 15.00|SINGLE |
MIRR |Mirror | 25.00|SINGLE |
KEY |Door Keys | 5.00|SINGLE |
KEYFOB |Key Fob | 40.00|SINGLE |
product_order
product_id|quantity|location |quote_detail_quote_id |is_primary_order|reason|
----------|--------|----------|------------------------------------|----------------|------|
MIRR | 2|floor_0 |C7D33FED-CB15-5796-DC7D-A7BCEA8923C5|true | 1|
KEYF | 3|floor_0 |C7D33FED-CB15-5796-DC7D-A7BCEA8923C5|true | 2|
WALLC | 3|floor_1 |C7D33FED-CB15-5796-DC7D-A7BCEA8923C5|true | 1|
WALLC | 3|floor_1 |C7D33FED-CB15-5796-DC7D-A7BCEA8923C5|true | 3|
product_order_reason
------------------------------------------------
id (varchar, pk) | shortform_text(varchar) | longform_text(varchar)
------------------------------------------------
id|shortform_text |longform_text |
--|-------------------------------------|-----------------------------------------------------|
1|Employee Room |Standard employee room with no window |
2|Meeting Room |Standard Meeting Room |
3|Mirror |Additional Mirror Request |
create
or replace
function get_breakdown_v1_0_0(p_quote_id character varying,
p_location character varying,
p_product_code character varying) returns json language plpgsql as $function$ declare row_count smallint := 0;
begin
raise notice 'Location: %',
p_location;
raise notice 'Product: %',
p_product_code;
-- Perform santiy check on quote_id so that the json does not include a null result.
select
count(*) into
strict row_count
from
quote_detail
where
quote_id = p_quote_id;
if row_count = 0 then raise 'Quote ID % not found',
p_quote_id
using ERRCODE = '02000';
-- SQL standard no_data
elseif row_count > 1 then raise 'Too many rows returned for ID %',
p_quote_id
using ERRCODE = 'P0003';
-- PL/pgSQL too_many_rows
end if;
-- Returns an object comprised of unique values for locations, where not null and their associated products
return (
select
jsonb_build_object ('locations',jsonb_agg( jsonb_build_object( 'area', location, 'items', items)))
from
(
select
location,
jsonb_agg(jsonb_build_object ('code', product_id, 'reasons', reason)) as items
from
product_order
where
(quote_detail_quote_id = p_quote_id)
and (location = p_location
or p_location is null)
and (product_id = p_product_code
or p_product_code is null)
group by
location) a );
end $function$ ;
Desired response;
{
"area": "floor_0",
"items": [
{
"code": "WALLC",
"reasons": [
{
"quantity": 2,
"reason_code": "Standard Employee Room"
},
{
"quantity": 2,
"reason_code": "Standard Cubicle"
}
]
},
{
"code": "MIRR",
"reasons": [
{
"quantity": 3,
"reason_code": "Meeting Room"
}
]
}
]
}]
Alright I think I have something for you. The idea is to build one of the arrays at a time and carry the necessary remaining info to the outer queries for further array building. You can add your constraints for quote_detail_quote_id , location, and product_id to the innermost query's WHERE clause.
SQLFiddle to show it in action.
This may take some studying:
SELECT json_build_object('area', t3.location, 'items', t3.code_json)
FROM
(
SELECT t2.location
, array_to_json(array_agg(jsonb_build_object('code', t2.product_id, 'reasons', t2.qty_reason_json))) AS code_json
FROM
(
SELECT t.location
, t.product_id
, array_to_json(array_agg(jsonb_build_object('quantity', t.quantity, 'reason_code', t.longform_text))) AS qty_reason_json
FROM
(
SELECT po.product_id
, po.quantity
, po.location
, po.reason
, por.longform_text
FROM product_order po
JOIN product_order_reason por ON (por.id = po.reason)
WHERE quote_detail_quote_id = 'C7D33FED'
) t
GROUP BY t.location, t.product_id
) t2
GROUP BY t2.location
) t3
;
I have a PostgreSQL table with unique key/value pairs, which were originally in a JSON format, but have been normalized and melted:
key | value
-----------------------------
name | Bob
address.city | Vancouver
address.country | Canada
I need to turn this into a hierarchical JSON:
{
"name": "Bob",
"address": {
"city": "Vancouver",
"country": "Canada"
}
}
Is there a way to do this easily within SQL?
jsonb_set() almost does everything for you, but unfortunately it can only create missing leafs (i.e. missing last keys on a path), but not whole missing branches. To overcome this, here is a modified version of it, which can set values on any missing levels:
create function jsonb_set_rec(jsonb, jsonb, text[])
returns jsonb
language sql
as $$
select case
when array_length($3, 1) > 1 and ($1 #> $3[:array_upper($3, 1) - 1]) is null
then jsonb_set_rec($1, jsonb_build_object($3[array_upper($3, 1)], $2), $3[:array_upper($3, 1) - 1])
else jsonb_set($1, $3, $2, true)
end
$$;
Now you only need to apply this function one-by-one to your rows, starting with an empty json object: {}. You can do this with either recursive CTEs:
with recursive props as (
(select distinct on (grp)
pk, grp, jsonb_set_rec('{}', to_jsonb(value), string_to_array(key, '.')) json_object
from eav_tbl
order by grp, pk)
union all
(select distinct on (grp)
eav_tbl.pk, grp, jsonb_set_rec(json_object, to_jsonb(value), string_to_array(key, '.'))
from props
join eav_tbl using (grp)
where eav_tbl.pk > props.pk
order by grp, eav_tbl.pk)
)
select distinct on (grp)
grp, json_object
from props
order by grp, pk desc;
Or, with a custom aggregate defined as:
create aggregate jsonb_set_agg(jsonb, text[]) (
sfunc = jsonb_set_rec,
stype = jsonb,
initcond = '{}'
);
your query could became as simple as:
select grp, jsonb_set_agg(to_jsonb(value), string_to_array(key, '.'))
from eav_tbl
group by grp;
https://rextester.com/TULNU73750
There are no ready to use tools for this. The function generates a hierarchical json object based on a path:
create or replace function jsonb_build_object_from_path(path text, value text)
returns jsonb language plpgsql as $$
declare
obj jsonb;
keys text[] := string_to_array(path, '.');
level int := cardinality(keys);
begin
obj := jsonb_build_object(keys[level], value);
while level > 1 loop
level := level- 1;
obj := jsonb_build_object(keys[level], obj);
end loop;
return obj;
end $$;
You also need the aggregate function jsonb_merge_agg(jsonb) described in this answer. The query:
with my_table (path, value) as (
values
('name', 'Bob'),
('address.city', 'Vancouver'),
('address.country', 'Canada'),
('first.second.third', 'value')
)
select jsonb_merge_agg(jsonb_build_object_from_path(path, value))
from my_table;
gives this object:
{
"name": "Bob",
"first":
{
"second":
{
"third": "value"
}
},
"address":
{
"city": "Vancouver",
"country": "Canada"
}
}
The function do not recognize json arrays.
I can't really think of something simpler, although I think there should be an easier way.
I assume there is some additional column that can be used to bring the keys that belong to one "person" together, I used p_id for that in my example.
select p_id,
jsonb_object_agg(k, case level when 1 then v -> k else v end)
from (
select p_id,
elements[1] k,
jsonb_object_agg(case cardinality(elements) when 1 then ky else elements[2] end, value) v,
max(cardinality(elements)) as level
from (
select p_id,
"key" as ky,
string_to_array("key", '.') as elements, value
from kv
) t1
group by p_id, k
) t2
group by p_id;
The innermost query just converts the dot notation to an array for easier access later.
The next level then builds JSON objects depending on the "key". For the "single level" keys, it just uses key/value, for the others it uses the second element + the value and then aggregates those that belong together.
The second query level returns the following:
p_id | k | v | level
-----+---------+--------------------------------------------+------
1 | address | {"city": "Vancouver", "country": "Canada"} | 2
1 | name | {"name": "Bob"} | 1
2 | address | {"city": "Munich", "country": "Germany"} | 2
2 | name | {"name": "John"} | 1
The aggregation done in the second step, leaves one level too much for the "single element" keys, and that's what we need level for.
If that distinction wasn't made, the final aggregation would return {"name": {"name": "Bob"}, "address": {"city": "Vancouver", "country": "Canada"}} instead of the wanted: {"name": "Bob", "address": {"city": "Vancouver", "country": "Canada"}}.
The expression case level when 1 then v -> k else v end essentially turns {"name": "Bob"} back to "Bob".
So, with the following sample data:
create table kv (p_id integer, "key" text, value text);
insert into kv
values
(1, 'name','Bob'),
(1, 'address.city','Vancouver'),
(1, 'address.country','Canada'),
(2, 'name','John'),
(2, 'address.city','Munich'),
(2, 'address.country','Germany');
then query returns:
p_id | jsonb_object_agg
-----+-----------------------------------------------------------------------
1 | {"name": "Bob", "address": {"city": "Vancouver", "country": "Canada"}}
2 | {"name": "John", "address": {"city": "Munich", "country": "Germany"}}
Online example: https://rextester.com/SJOTCD7977
create table kv (key text, value text);
insert into kv
values
('name','Bob'),
('address.city','Vancouver'),
('address.country','Canada'),
('name','John'),
('address.city','Munich'),
('address.country','Germany');
create view v_kv as select row_number() over() as nRec, key, value from kv;
create view v_datos as
select k1.nrec, k1.value as name, k2.value as address_city, k3.value as address_country
from v_kv k1 inner join v_kv k2 on (k1.nrec + 1 = k2.nrec)
inner join v_kv k3 on ((k1.nrec + 2= k3.nrec) and (k2.nrec + 1 = k3.nrec))
where mod(k1.nrec, 3) = 1;
select json_agg(json_build_object('name',name, 'address', json_build_object('city',address_city, 'country', address_country)))
from v_datos;