Oracle: How to load hierachical JSON data into related tables - json

I have got JSON data in a JSON column of an Oracle database. The JSON contains arrays and I should load the data into tables by means of PL/SQL. E. g.
{
"A1": "V1",
"A2": true,
"A3": 42,
"A4": [
{
"B1": "Q1",
"B4": [
{
"C1": "R1",
"C2": false
},
{
"C1": "R2",
"C2": false
}
]
},
{
"B1": "Q2",
"B4": [
{
"C1": "R3",
"C2": false
},
{
"C1": "R4",
"C2": true
}
]
}
]
}
{
"A1": "V2",
"A2": false,
"A3": 42,
"A4": [
{
"B1": "T1",
"B4": [
{
"C1": "S1",
"C2": false
},
{
"C1": "S2",
"C2": false
}
]
},
{
"B1": "T2",
"B4": [
{
"C1": "S3",
"C2": false
},
{
"C1": "S4",
"C2": true
}
]
}
]
}
The data should be loaded to three tables.
---
A
---
| 0..1
|
| 1..n
---
B (no (business) key available)
---
| 0..1
|
| 1..n
---
C
---
ID
A1
A2
A3
1
V1
true
42
2
V2
false
42
ID
ID A
B1
1
1
Q1
2
1
Q2
3
2
T1
4
2
T2
ID
ID B
C1
C2
1
1
R1
false
2
1
R2
false
3
1
R3
false
4
1
R4
true
5
2
S1
false
6
2
S2
false
7
2
S3
false
8
2
S4
true
I am trying to use JSON_TABLE to flatten the data for an implicit cursor of a for loop. I did so by using NESTED PATH. Now imagine we are in the for loop, i.e. for every "record" in C there is a record in the cursor/loop. While the data of C is "as is", the data of B gets mimeographed, and the data of A likewise, i.e.
A1
A2
A3
B1
C1
C2
V1
true
42
Q1
R1
false
V1
true
42
Q1
R2
false
V1
true
42
Q2
R3
false
V1
true
42
Q2
R4
true
V2
false
42
T1
S1
false
V2
false
42
T1
S2
false
V2
false
42
T2
S3
false
V2
false
42
T2
S4
true
My initial idea was to insert from the "A" data into table "A" only the first occurrence of a A-tuple saving the written ID by RETURNING .. INTO, insert the first occurrence of a B-tuple into "B" saving the written ID likewise and finally insert the C-tuple into "C". The problem here is to detect the change of the A-tuple respectively of the B-tuple. "A" should not be too much of a difficulty as it has a (business) key I also can preserve for checks. However, B has no (business) key.
My idea with B was to use an unknown, maybe inexistent pseudo column JSON_TABLE/NESTED PATH provides, that tells e.g. to which expression of "B" the current "C" belongs to. In fact, it would be a key for "B". Yet, I have not found anything in the Oracle docs and on the internet. Do you happen to know such a column?
As an alternative, I think about not using the NESTED PATH for "C" but return the array as JSON column and afterwards to somehow create an inner for loop, but I do not know how to use a JSON typed PL/SQL variable to extract an array of objects as nested table of some sort to be able to use as for loop cursor. Do you happen to know examples for such?
Is there a way, I miss completely?
Kind regards
Thiemo

Related

how to extract and modify inner array objects with parent object data in jq

We are tying to format a json similar to this:
[
{"id": 1,
"type": "A",
"changes": [
{"id": 12},
{"id": 13}
],
"wanted_key": "good",
"unwanted_key": "aaa"
},
{"id": 2,
"type": "A",
"unwanted_key": "aaa"
},
{"id": 3,
"type": "B",
"changes": [
{"id": 31},
{"id": 32}
],
"unwanted_key": "aaa",
"unwanted_key2": "aaa"
},
{"id": 4,
"type": "B",
"unwanted_key3": "aaa"
},
null,
null,
{"id": 7}
]
into something like this:
[
{
"id": 1,
"type": "A",
"wanted_key": true # every record must have this key/value
},
{
"id": 12, # note: this was in the "changes" property of record id 1
"type": "A", # type should be the same type than record id 1
"wanted_key": true
},
{
"id": 13, # note: this was in the "changes" property of record id 1
"type": "A", # type should be the same type than record id 1
"wanted_key": true
},
{
"id": 2,
"type": "A",
"wanted_key": true
},
{
"id": 3,
"type": "B",
"wanted_key": true
},
{
"id": 31, # note: this was in the "changes" property of record id 3
"type": "B", # type should be the same type than record id 3
"wanted_key": true
},
{
"id": 32, # note: this was in the "changes" property of record id 3
"type": "B", # type should be the same type than record id 3
"wanted_key": true
},
{
"id": 4,
"type": "B",
"wanted_key": true
},
{
"id": 7,
"type": "UNKN", # records without a type should have this type
"wanted_key": true
}
]
So far, I've been able to:
remove null records
obtain the keys we need with their default
give records without a type a default type
What we are missing:
from records having a changes key, create new records with the type of their parent record
join all records in a single array
Unfortunately we are not entirely sure how to proceed... Any help would be appreciated.
So far our jq goes like this:
del(..|nulls) | map({id, type: (.type // "UNKN"), wanted_key: (true)}) | del(..|nulls)
Here's our test code:
https://jqplay.org/s/eLAWwP1ha8P
The following should work:
map(select(values))
| map(., .type as $type | (.changes[]? + {$type}))
| map({id, type: (.type // "UNKN"), wanted_key: true})
Only select non-null values
Return the original items followed by their inner changes array (+ outer type)
Extract 3 properties for output
Multiple map calls can usually be combined, so this becomes:
map(
select(values)
| ., (.type as $type | (.changes[]? + {$type}))
| {id, type: (.type // "UNKN"), wanted_key: true}
)
Another option without variables:
map(
select(values)
| ., .changes[]? + {type}
| {id, type: (.type // "UNKN"), wanted_key: true}
)
# or:
map(select(values))
| map(., .changes[]? + {type})
| map({id, type: (.type // "UNKN"), wanted_key: true})
or even with a separate normalization step for the unknown type:
map(select(values))
| map(.type //= "UNKN")
| map(., .changes[]? + {type})
| map({id, type, wanted_key: true})
# condensed to a single line:
map(select(values) | .type //= "UNKN" | ., .changes[]? + {type} | {id, type, wanted_key: true})
Explanation:
Select only non-null values from the array
If type is not set, create the property with value "UNKN"
Produce the original array items, followed by their nested changes elements extended with the parent type
Reshape objects to only contain properties id, type, and wanted_key.
Here's one way:
map(
select(values)
| (.type // "UNKN") as $type
| ., .changes[]?
| {id, $type, wanted_key: true}
)
[
{
"id": 1,
"type": "A",
"wanted_key": true
},
{
"id": 12,
"type": "A",
"wanted_key": true
},
{
"id": 13,
"type": "A",
"wanted_key": true
},
{
"id": 2,
"type": "A",
"wanted_key": true
},
{
"id": 3,
"type": "B",
"wanted_key": true
},
{
"id": 31,
"type": "B",
"wanted_key": true
},
{
"id": 32,
"type": "B",
"wanted_key": true
},
{
"id": 4,
"type": "B",
"wanted_key": true
},
{
"id": 7,
"type": "UNKN",
"wanted_key": true
}
]
Demo
Something like below should work
map(
select(type == "object") |
( {id}, {id : ( .changes[]? .id )} ) +
{ type: (.type // "UNKN"), wanted_key: true }
)
jq play - demo

How to flatten json into table

I have just realised on my AWS Aurora postgres cluster having functions with temp_tables are not friendly with read replicas. I need to do a re-write (using CTEs) - anyway.... How do I take a json object with arrays nested and flatten them to a table like so:
{
"data": [
{
"groupName": "TeamA",
"groupCode": "12",
"subGroupCodes": [
"11"
]
},
{
"groupName": "TeamB",
"groupCode": "13",
"subGroupCodes": [
"15", "22"
]
}
]
}
I would like the output table to be:
groupName groupCode subGroupCodes
TeamA 12 11
TeamB 13 15
TeamB 13 22
I know I can get most of the way there with:
SELECT j."groupCode" as int, j."groupName" as pupilgroup_name
FROM json_to_recordset(p_in_filters->'data') j ("groupName" varchar(50), "groupCode" int)
But I just need to get the subGroupCodes as well but unpacking the array and joining to the correct parent groupCodes.
You need to first unnest the array, and then another unnest to get the subgroup codes:
with data (j) as (
values ('{
"data": [
{
"groupName": "TeamA",
"groupCode": "12",
"subGroupCodes": [
"11"
]
},
{
"groupName": "TeamB",
"groupCode": "13",
"subGroupCodes": [
"15", "22"
]
}
]
}'::jsonb)
)
select e ->> 'groupName' as group_name,
e ->> 'groupCode' as code,
sg.*
from data d
cross join lateral jsonb_array_elements(d.j -> 'data') as e(g)
cross join lateral jsonb_array_elements_text(g -> 'subGroupCodes') as sg(subgroup_code)

How can I use MariaDB to pull values from multiple objects from a JSON array?

In MariaDB 10.2.19, I have a table named forms with a column template which always contains a JSON array of objects. Some of these objects will have properties I want to return: name (should always be present), rule, and parameters. How can I return just these three properties from the entire array, but only for objects on which rule is present?
A sample array (formatted for easier viewing):
[{
"label": "Employed?",
"class": "select",
"name": "employed",
"parameters": "Yes",
"rule": "in"
},
{
"label": "Breed of dog?",
"class": "select",
"name": "breed",
"parameters": "spaniel, collie, mix",
"rule": "in"
},
{
"label": "Number",
"class": "text",
"name": "breed"
}]
If you are using MySQL 8.0.4 or later one way is using JSON_TABLE:
mysql> SELECT * FROM foo;
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| data |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| [{"name": "employed", "rule": "in", "class": "select", "label": "Employed?", "parameters": "Yes"}, {"name": "breed", "rule": "in", "class": "select", "label": "Breed of dog?", "parameters": "spaniel, collie, mix"}, {"name": "breed", "class": "text", "label": "Number"}] |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0,00 sec)
mysql> SELECT name, parameters
FROM foo,
JSON_TABLE (
foo.data,
"$[*]" COLUMNS (
name VARCHAR(100) PATH "$.name",
rule VARCHAR(100) PATH "$.rule",
parameters VARCHAR(100) PATH "$.parameters")
) AS t
WHERE rule IS NOT NULL;
+----------+----------------------+
| name | parameters |
+----------+----------------------+
| employed | Yes |
| breed | spaniel, collie, mix |
+----------+----------------------+
2 rows in set (0,00 sec)
https://dev.mysql.com/doc/refman/8.0/en/json-table-functions.html
You can accomplish this by using a dedicated numbers_table table:
SELECT
JSON_VALUE(f.template,CONCAT('$[',n.number,'].name')) AS `name`,
JSON_VALUE(f.template,CONCAT('$[',n.number,'].rule')) AS `rule`,
JSON_VALUE(f.template,CONCAT('$[',n.number,'].parameters')) AS `parameters`
FROM forms AS f
JOIN numbers_table AS n
WHERE
n.number < JSON_LENGTH(f.template) AND
JSON_VALUE(f.template,CONCAT('$[',n.number,'].rule')) IS NOT NULL;
The numbers_table table contains a single column called number which starts at 0 and can be as long as your use cases require (I have 1000 values 0 to 999). It is very useful for extracting each element of a JSON_ARRAY field into its own row.
The first WHERE condition makes sure we only use as many numbers as there are elements in the JSON_ARRAY (template in your case).
The second WHERE condition eliminates the ones that don't have a rule as per your use case.

Postgres - Convert adjacency list to nested JSON object

I have a table with this data in Postgres and I am having a hard time to convert this in to a JSON object.
node_id parent_node name
------- ----------- ----
1 node1
2 1 node2
3 1 node3
4 2 node4
5 2 node5
6 2 node6
7 3 node7
8 3 node8
How do I convert it like this?
{
name: 'node1'
childs: [
{
name: 'node2',
childs: [
{
name: 'node4',
childs: []
},
{
name: 'node5',
childs: []
},
{
name: 'node6',
childs: []
}
]
},
...
]
}
Any suggestion will help. Thanks
Using WITH RECURSIVE (https://www.postgresql.org/docs/current/static/queries-with.html) and JSON Functions (https://www.postgresql.org/docs/current/static/functions-json.html) I build this solution:
db<>fiddle
The core functionality:
WITH RECURSIVE tree(node_id, ancestor, child, path, json) AS (
SELECT
t1.node_id,
NULL::int,
t2.node_id,
'{children}'::text[] ||
(row_number() OVER (PARTITION BY t1.node_id ORDER BY t2.node_id) - 1)::text,-- C
jsonb_build_object('name', t2.name, 'children', array_to_json(ARRAY[]::int[])) -- B
FROM test t1
LEFT JOIN test t2 ON t1.node_id = t2.parent_node -- A
WHERE t1.parent_node IS NULL
UNION
SELECT
t1.node_id,
t1.parent_node,
t2.node_id,
tree.path || '{children}' || (row_number() OVER (PARTITION BY t1.node_id ORDER BY t2.node_id) - 1)::text,
jsonb_build_object('name', t2.name, 'children', array_to_json(ARRAY[]::int[]))
FROM test t1
LEFT JOIN test t2 ON t1.node_id = t2.parent_node
INNER JOIN tree ON (t1.node_id = tree.child)
WHERE t1.parent_node = tree.node_id -- D
)
SELECT -- E
child as node_id, path, json
FROM tree
WHERE child IS NOT NULL ORDER BY path
Every WITH RECURSIVE contains a start SELECT and a recursion part (the second SELECT) combined by a UNION.
A: Joining the table agains itself for finding the children of a node_id.
B: Building the json object for the child which can be inserted into its parent
C: Building the path where the child object has to be inserted (from root). The window function row_number() (https://www.postgresql.org/docs/current/static/tutorial-window.html) generates the index of the child within the children array of the parent.
D: The recursion part works as the initial part with one difference: It's not searching for the root element but for the element which has the parent node of the last recursion.
E: Executing the recursion and filtering all elements without any children gives this result:
node_id path json
2 children,0 {"name": "node2", "children": []}
4 children,0,children,0 {"name": "node4", "children": []}
5 children,0,children,1 {"name": "node5", "children": []}
6 children,0,children,2 {"name": "node6", "children": []}
3 children,1 {"name": "node3", "children": []}
7 children,1,children,0 {"name": "node7", "children": []}
8 children,1,children,1 {"name": "node8", "children": []}
Though I found no way to add all children elements in the recursion (the origin json is no global variable; so it always knows the changes of the direct ancestors, not their siblings), I had to iterate the rows in a seconds step.
That's why I build the function. In there I can do the iteration for a global variable. With the function jsonb_insert I am inserting all calculated elements into a root json object - using the calculated path.
CREATE OR REPLACE FUNCTION json_tree() RETURNS jsonb AS $$
DECLARE
_json_output jsonb;
_temprow record;
BEGIN
SELECT
jsonb_build_object('name', name, 'children', array_to_json(ARRAY[]::int[]))
INTO _json_output
FROM test
WHERE parent_node IS NULL;
FOR _temprow IN
/* Query above */
LOOP
SELECT jsonb_insert(_json_output, _temprow.path, _temprow.json) INTO _json_output;
END LOOP;
RETURN _json_output;
END;
$$ LANGUAGE plpgsql;
Last step is calling the function and make the JSON more readable (jsonb_pretty())
{
"name": "node1",
"children": [{
"name": "node2",
"children": [{
"name": "node4",
"children": []
},
{
"name": "node5",
"children": []
},
{
"name": "node6",
"children": []
}]
},
{
"name": "node3",
"children": [{
"name": "node7",
"children": []
},
{
"name": "node8",
"children": []
}]
}]
}
I am sure it is possible to optimize the query but for a sketch it works.

Create a json object using array of json objects

In postgres Say I have schema as such:
table item {
type varchar(40)
entity_id bigint
entity_type varchar(40)
user_id bigint
}
And I want to query the table to get the info like this:
{
"typeA": {
"count": 3,
"me": true
},
"typeC": {
"count": 3,
"me": false
},
"typeE": {
"count": 3,
"me": false
},
"typeR": {
"count": 3,
"me": true
}
}
From a query where the main data is this:
SELECT ARRAY_AGG(x)
FROM
(
SELECT type,
count(*),
(CASE
WHEN (SELECT id
FROM items as i
WHERE i.entity_type = 'sometype'
AND i.entity_id = 234
AND i.user_id = 32
AND i.type = items.type) is not null
THEN true
ELSE false
END) AS me
FROM items
WHERE items.entity_type = 'sometype'
AND items.entity_id = 234
GROUP BY type
) as x
This returns an array of the info i need type count and me. But I need it formatted like above versus:
[
{
"type": "typeA",
"count": 3,
"me": true
},
{
"type": "typeC",
"count": 3,
"me": false
},
{
"type": "typeE",
"count": 3,
"me": false
},
{
"type": "typeR",
"count": 3,
"me": true
}
]
Which is the current way it is formatted. Have been unable to find a way to build the json object I need. I was able to get three json objects that are like that But I need the three nested in one object.
Not exactly what you want, but from PostgreSQL - Aggregate Functions, I would guess, you can try json_object_agg(name, value), e.g.
SELECT JSON_OBJECT_AGG(type, x)
FROM
(
SELECT type,
count(*),
(CASE
WHEN (SELECT id
FROM items as i
WHERE i.entity_type = 'sometype'
AND i.entity_id = 234
AND i.user_id = 32
AND i.type = items.type) is not null
THEN true
ELSE false
END) AS me
FROM items
WHERE items.entity_type = 'sometype'
AND items.entity_id = 234
GROUP BY type, me
) as x