Postgresql json querying in 9.3+ on nested payloads - json

Using postgresql 9.3 (and the new json awesomness) if I have a simple table named 'races' with a two column description such as:
race-id integer,
race-data json
And the json is a payload for each race is something like
{ "race-time": some-date,
"runners": [ { "name": "fred","age": 30, "position": 1 },
{ "name": "john","age": 29, "position": 3 },
{ "name": "sam","age": 31, "position": 2 } ],
"prize-money": 200 }
How can I query the table for:
1) Races where sam has come 1st
2) Races where sam has come 1st and john has come 2nd
3) Where the number of runners with age greater than 30 is > 5 and prize-money > 5000
My experimentation (particularly in querying a nested array payload) so far has lead to further normalizing the data, i.e. creating a table called runners just to make such queries. Ideally I'd like to use this new fangled json query awesomeness but I can't seem to make heads or tails of it in respective to the 3 simple queries.

You can unwind json into one record and then do your queries as you want (see json functions):
with cte as (
select
race_id,
json_array_elements(r.race_data->'runners') as d,
(r.race_data->>'prize-money')::int as price_money
from races as r
), cte2 as (
select
race_id, price_money,
max(case when (d->>'position')::int = 1 then d->>'name' end) as name1,
max(case when (d->>'position')::int = 2 then d->>'name' end) as name2,
max(case when (d->>'position')::int = 3 then d->>'name' end) as name3
from cte
group by race_id, price_money
)
select *
from cte2
where name1 = 'sam' and name2 = 'john'
sql fiddle demo
It's a bit complicated because of your JSON structure. I think that if you change your structure a bit, your queries could be much simplier:
{
"race-time": some-date,
"runners":
{
"1": {"name": "fred","age": 30},
"2": {"name": "sam","age": 31},
"3": {"name": "john","age": 29}
},
"prize-money": 200
}
you can use ->> and -> operators or json_extract_path_text function to get data you need and then use it in the where clause:
select *
from races as r
where
r.race_data->'runners'->'1'->>'name' = 'sam';
select *
from races as r
where
json_extract_path_text(r.race_data, 'runners','1','name') = 'sam' and
json_extract_path_text(r.race_data, 'runners','2','name') = 'john';
select *
from races as r
where
(r.race_data->>'prize-money')::int > 100 and
(
select count(*)
from json_each(r.race_data->'runners')
where (value->>'age')::int >= 30
) >= 2
sql fiddle demo

Related

JSON EXTRACT SUM returning 0 instead of correct value

I'm trying to sum the contents of a json array in a mysql database, below is the JSON format and the query I'm running. Is there something wrong with it?
// Options JSON Format:
[
{
"optionId": 1,
"optionName": "With Meat",
"optionPrice": 2000
},
{
"optionId": 2,
"optionName": "With Veggies",
"optionPrice": 0
}
]
// Query:
SELECT id, SUM(options->'$[*].optionPrice') FROM table_order_items GROUP BY id;
The result is 0, when it should be 2000
While this query:
SELECT id, options->'$[*].optionPrice' FROM table_order_items;
correctly returns [2000,0]
You need the function JSON_TABLE() to extract the prices:
SELECT t.id,
SUM(j.price) AS total
FROM table_order_items t
JOIN JSON_TABLE(
t.options,
'$[*]' COLUMNS(price INT PATH '$.optionPrice')
) j
GROUP BY t.id;
See the demo.

Sql Query Json Array items by Value

I have searched and can't seem to find somewhere doing exactly what I am trying.
I have a json similar to as follows in multiple rows in my database:
{
"date": "0001-01-01T00:00:00",
"details": {
"detail": [
{
"item": "11",
"value": "xt"
},
{
"item": "12",
"value": "xy"
},
{
"item": "13",
"value": "xz"
},
{
"item": "14",
"value": "zz"
}
]
}
}
I want to do sql that does this:
select ID
jsonColumn.value where item=11 as X
jsonColumn.value where item=12 as Y
from tbl
So I have results like this
----------------------
|ID |X |Y |
----------------------
|1 |xt |xy |
----------------------
I have tried using JSONVALUE but I seem to need to do it by the array item number like this:
'$.details.detail[3].value'
which doesn't really work
I have also tried this:
SELECT id, x.item, x.value
FROM
tbl F
CROSS APPLY (select *
FROM OPENJSON(F.Json,'$.details.detail')
CROSS APPLY OPENJSON(value)
WITH (item NVARCHAR(25) '$.item',
value NVARCHAR(max) '$.value') As x
where F.ID=55
Which I can use to print out all the items and values but then I'd have to query each separately again.
Is there a way of combining the two in to one big query that won't be completely inefficient?
Seems what you want is a pivot. I personally use conditional aggregation over the far more restrictive PIVOT operator. The JSON you supplied was invalid, so I took some liberties correcting it in my sandbox environment:
SELECT --ID,
MAX(CASE d.item WHEN 11 THEN d.[value] END) AS X,
MAX(CASE d.item WHEN 12 THEN d.[value] END) AS Y
FROM (VALUES(#JSON))V(J) --Your Table
CROSS APPLY OPENJSON(V.J,'$.details')
WITH (detail nvarchar(MAX) AS JSON ) OJ
CROSS APPLY OPENJSON(OJ.detail)
WITH(item int,
[value] nvarchar(2)) d;
If you are using this against a table, and not limiting the data to a single row, you'll need to also add a GROUP BY clause on the relevant columns (ID?).

How to select JSON object from JSON array field of mysql by some condition

I have a table with JSON field which contains an array of JSON objects. I need to select objects by some condition.
Create and fill a table:
CREATE TABLE test (
id INT AUTO_INCREMENT PRIMARY KEY,
json_list JSON
);
INSERT INTO test(json_list) VALUES
("{""list"": [{""type"": ""color"", ""value"": ""red""}, {""type"": ""shape"", ""value"": ""oval""}, {""type"": ""color"", ""value"": ""green""}]}"),
("{""list"": [{""type"": ""shape"", ""value"": ""rect""}, {""type"": ""color"", ""value"": ""olive""}]}"),
("{""list"": [{""type"": ""color"", ""value"": ""red""}]}")
;
Now I need to select all objects with type = color from all rows.
I want to see this output:
id extracted_value
1 {"type": "color", "value": "red"}
1 {"type": "color", "value": "green"}
2 {"type": "color", "value": "olive"}
3 {"type": "color", "value": "red"}
It would be good to get this too:
id color
1 red
1 green
2 olive
3 red
I can't change the DB or JSON.
I'm using MySQL 5.7
My current solution
My solution is to cross join the table with some index set and then extract all elements of JSON array.
I don't like it as if possible object count in one array is large it is required to have all indexes till the maximum one. It makes the query slow as it won't stop calculation of JSON value when the end of array is reached.
SELECT
test.id,
JSON_EXTRACT(test.json_list, CONCAT('$.list[', ind.ind, ']')),
ind.ind
FROM
test
CROSS JOIN
(SELECT 0 AS ind UNION ALL SELECT 1 AS ind UNION ALL SELECT 2 AS ind) ind
WHERE
JSON_LENGTH(json_list, "$.list") > ind.ind
AND JSON_EXTRACT(json_list, CONCAT('$.list[', ind.ind, '].type')) = "color";
It is easy to get only values by changing JSON_EXTRACT path. But is it there a better way?
Edits
Added a check for json_list.list length. This filtered out 67% of derived table rows in this case.
SELECT JSON_EXTRACT(json_list, '$.list[*]')
FROM `test`
where JSON_CONTAINS(json_list, '{"type":"color"}', '$.list')
So current best solution is mine:
SELECT
test.id,
JSON_EXTRACT(test.json_list, CONCAT('$.list[', ind.ind, ']')),
ind.ind
FROM
test
CROSS JOIN
(SELECT 0 AS ind UNION ALL SELECT 1 AS ind UNION ALL SELECT 2 AS ind) ind
WHERE
JSON_LENGTH(json_list, "$.list") > ind.ind
AND JSON_EXTRACT(json_list, CONCAT('$.list[', ind.ind, '].type')) = "color";

Get a record count into root JSON path without it repeating?

Gleaning several articles online, including this one with a CTE, and this one WITHOUT a CTE, I have been successful in getting the data I need, including a count of the results. However, I need this count to be in a specific place in the JSON object... Basically, I know how to get a rowset into a specific JSON structure with FOR JSON PATH, ROOT ('data'), etc.
However, I do not know how to get the "recordsFiltered" into the root of my JSON output. This count is is derived using COUNT(*) OVER () AS recordsFiltered
Basically, I need my structure to look like this (see below)... How do I get "recordsFiltered" into the root $. of the JSON result without it repeating a billion times under the "data":[] section?
The best idea I can come up with is to create a temporary table, and then use that to structure the JSON. But, I want to do it the fancy SQL way, if one exists, using SELECT statements or CTEs where applicable.
{
"draw": 1,
"recordsTotal": 57,
"recordsFiltered": 57, // <<<--- need records filtered HERE
"data": [
{
"DT_RowId": "row_3",
"recordsFiltered": "69,420", // <<<---- NOT HERE!!!
"first_name": "Angelica",
"last_name": "Ramos",
"position": "System Architect",
"office": "London",
"start_date": "9th Oct 09",
"salary": "$2,875"
},
...
]
}
Here is the example SQL code:
SELECT
COUNT(*) OVER () AS recordsFiltered,
id,
a,
b
FROM t1
WHERE
(#Search IS NULL OR
id LIKE '%'+#Search+'%' OR
a LIKE '%'+#Search+'%' OR
b LIKE '%'+#Search+'%')
ORDER BY
CASE
WHEN #SortDir = 'ASC' THEN
CASE #SortCol
WHEN 0 THEN id
WHEN 1 THEN a
WHEN 2 THEN b
END
END desc,
CASE
WHEN #SortDir = 'desc' THEN
CASE #SortCol
WHEN 0 THEN id
WHEN 1 THEN a
WHEN 2 THEN b
END
END DESC
OFFSET #DisplayStart ROWS
FETCH NEXT #DisplayLength ROWS ONLY
for json path, root ('data')
Looks like you need to generate your table results, then use two (or more?) sub-queries
Here's a simplified example:
declare #tbl table (ID int identity, Col1 varchar(50), Col2 int)
insert into #tbl (Col1, Col2) values ('A',1),('B',2),('C',3)
select
(select count(1) from #tbl) as 'total',
(select * from #tbl for json path) as 'data'
for json path
produces:
[
{
"total": 3,
"data": [
{
"ID": 1,
"Col1": "A",
"Col2": 1
},
{
"ID": 2,
"Col1": "B",
"Col2": 2
},
{
"ID": 3,
"Col1": "C",
"Col2": 3
}
]
}
]
Without knowing the rest of your code/schema, here's my guess at your needed query:
select
*
into
#MyTable
from
t1
WHERE
(#Search IS NULL OR
id LIKE '%'+#Search+'%' OR
a LIKE '%'+#Search+'%' OR
b LIKE '%'+#Search+'%')
select
(select count(*) from #MyTable) as recordsFiltered,
(
select
id,
a,
b
from
#MyTable
ORDER BY
CASE
WHEN #SortDir = 'ASC' THEN
CASE #SortCol
WHEN 0 THEN id
WHEN 1 THEN a
WHEN 2 THEN b
END
END desc,
CASE
WHEN #SortDir = 'desc' THEN
CASE #SortCol
WHEN 0 THEN id
WHEN 1 THEN a
WHEN 2 THEN b
END
END DESC
OFFSET #DisplayStart ROWS
FETCH NEXT #DisplayLength ROWS ONLY
for json path
) as [data]
for json path
Using a CTE:
with cte as ()
select
*
from
t1
WHERE
(#Search IS NULL OR
id LIKE '%'+#Search+'%' OR
a LIKE '%'+#Search+'%' OR
b LIKE '%'+#Search+'%')
)
select
(select count(*) from cte) as recordsFiltered,
(
select
id,
a,
b
from
cte
ORDER BY
CASE
WHEN #SortDir = 'ASC' THEN
CASE #SortCol
WHEN 0 THEN id
WHEN 1 THEN a
WHEN 2 THEN b
END
END desc,
CASE
WHEN #SortDir = 'desc' THEN
CASE #SortCol
WHEN 0 THEN id
WHEN 1 THEN a
WHEN 2 THEN b
END
END DESC
OFFSET #DisplayStart ROWS
FETCH NEXT #DisplayLength ROWS ONLY
for json path
) as [data]
for json path

Convert flattened key/value table into hierarchical JSON in PostgreSQL

I have a PostgreSQL table with unique key/value pairs, which were originally in a JSON format, but have been normalized and melted:
key | value
-----------------------------
name | Bob
address.city | Vancouver
address.country | Canada
I need to turn this into a hierarchical JSON:
{
"name": "Bob",
"address": {
"city": "Vancouver",
"country": "Canada"
}
}
Is there a way to do this easily within SQL?
jsonb_set() almost does everything for you, but unfortunately it can only create missing leafs (i.e. missing last keys on a path), but not whole missing branches. To overcome this, here is a modified version of it, which can set values on any missing levels:
create function jsonb_set_rec(jsonb, jsonb, text[])
returns jsonb
language sql
as $$
select case
when array_length($3, 1) > 1 and ($1 #> $3[:array_upper($3, 1) - 1]) is null
then jsonb_set_rec($1, jsonb_build_object($3[array_upper($3, 1)], $2), $3[:array_upper($3, 1) - 1])
else jsonb_set($1, $3, $2, true)
end
$$;
Now you only need to apply this function one-by-one to your rows, starting with an empty json object: {}. You can do this with either recursive CTEs:
with recursive props as (
(select distinct on (grp)
pk, grp, jsonb_set_rec('{}', to_jsonb(value), string_to_array(key, '.')) json_object
from eav_tbl
order by grp, pk)
union all
(select distinct on (grp)
eav_tbl.pk, grp, jsonb_set_rec(json_object, to_jsonb(value), string_to_array(key, '.'))
from props
join eav_tbl using (grp)
where eav_tbl.pk > props.pk
order by grp, eav_tbl.pk)
)
select distinct on (grp)
grp, json_object
from props
order by grp, pk desc;
Or, with a custom aggregate defined as:
create aggregate jsonb_set_agg(jsonb, text[]) (
sfunc = jsonb_set_rec,
stype = jsonb,
initcond = '{}'
);
your query could became as simple as:
select grp, jsonb_set_agg(to_jsonb(value), string_to_array(key, '.'))
from eav_tbl
group by grp;
https://rextester.com/TULNU73750
There are no ready to use tools for this. The function generates a hierarchical json object based on a path:
create or replace function jsonb_build_object_from_path(path text, value text)
returns jsonb language plpgsql as $$
declare
obj jsonb;
keys text[] := string_to_array(path, '.');
level int := cardinality(keys);
begin
obj := jsonb_build_object(keys[level], value);
while level > 1 loop
level := level- 1;
obj := jsonb_build_object(keys[level], obj);
end loop;
return obj;
end $$;
You also need the aggregate function jsonb_merge_agg(jsonb) described in this answer. The query:
with my_table (path, value) as (
values
('name', 'Bob'),
('address.city', 'Vancouver'),
('address.country', 'Canada'),
('first.second.third', 'value')
)
select jsonb_merge_agg(jsonb_build_object_from_path(path, value))
from my_table;
gives this object:
{
"name": "Bob",
"first":
{
"second":
{
"third": "value"
}
},
"address":
{
"city": "Vancouver",
"country": "Canada"
}
}
The function do not recognize json arrays.
I can't really think of something simpler, although I think there should be an easier way.
I assume there is some additional column that can be used to bring the keys that belong to one "person" together, I used p_id for that in my example.
select p_id,
jsonb_object_agg(k, case level when 1 then v -> k else v end)
from (
select p_id,
elements[1] k,
jsonb_object_agg(case cardinality(elements) when 1 then ky else elements[2] end, value) v,
max(cardinality(elements)) as level
from (
select p_id,
"key" as ky,
string_to_array("key", '.') as elements, value
from kv
) t1
group by p_id, k
) t2
group by p_id;
The innermost query just converts the dot notation to an array for easier access later.
The next level then builds JSON objects depending on the "key". For the "single level" keys, it just uses key/value, for the others it uses the second element + the value and then aggregates those that belong together.
The second query level returns the following:
p_id | k | v | level
-----+---------+--------------------------------------------+------
1 | address | {"city": "Vancouver", "country": "Canada"} | 2
1 | name | {"name": "Bob"} | 1
2 | address | {"city": "Munich", "country": "Germany"} | 2
2 | name | {"name": "John"} | 1
The aggregation done in the second step, leaves one level too much for the "single element" keys, and that's what we need level for.
If that distinction wasn't made, the final aggregation would return {"name": {"name": "Bob"}, "address": {"city": "Vancouver", "country": "Canada"}} instead of the wanted: {"name": "Bob", "address": {"city": "Vancouver", "country": "Canada"}}.
The expression case level when 1 then v -> k else v end essentially turns {"name": "Bob"} back to "Bob".
So, with the following sample data:
create table kv (p_id integer, "key" text, value text);
insert into kv
values
(1, 'name','Bob'),
(1, 'address.city','Vancouver'),
(1, 'address.country','Canada'),
(2, 'name','John'),
(2, 'address.city','Munich'),
(2, 'address.country','Germany');
then query returns:
p_id | jsonb_object_agg
-----+-----------------------------------------------------------------------
1 | {"name": "Bob", "address": {"city": "Vancouver", "country": "Canada"}}
2 | {"name": "John", "address": {"city": "Munich", "country": "Germany"}}
Online example: https://rextester.com/SJOTCD7977
create table kv (key text, value text);
insert into kv
values
('name','Bob'),
('address.city','Vancouver'),
('address.country','Canada'),
('name','John'),
('address.city','Munich'),
('address.country','Germany');
create view v_kv as select row_number() over() as nRec, key, value from kv;
create view v_datos as
select k1.nrec, k1.value as name, k2.value as address_city, k3.value as address_country
from v_kv k1 inner join v_kv k2 on (k1.nrec + 1 = k2.nrec)
inner join v_kv k3 on ((k1.nrec + 2= k3.nrec) and (k2.nrec + 1 = k3.nrec))
where mod(k1.nrec, 3) = 1;
select json_agg(json_build_object('name',name, 'address', json_build_object('city',address_city, 'country', address_country)))
from v_datos;