how to extract and zip values from json with SQL, mysql - mysql

I struggle to extract the required values from the JSON column in MySQL.
Imagine the variable:
SET #j = '[{"a": 1, "b": 2},{"a": 3, "b": 4, "c": {"d": 4}}, {"a": 5}]';
and I need to extract and return a new JSON array containing only selected key-value pairs, eg.: only a's and b's and if the required key does not exist I want to return null for that key:
[{"a": 1, "b": 2},{"a": 3, "b": 4},{"a": 5, "b": null}]
thanks

Here's a solution using MySQL 8.0's JSON_TABLE() function:
mysql> SET #j = '[{"a": 1, "b": 2},{"a": 3, "b": 4, "c": {"d": 4}}, {"a": 5}]';
Query OK, 0 rows affected (0.00 sec)
mysql> select j.* from json_table(#j, '$[*]'
columns (a int path '$.a', b int path '$.b')) as j;
+------+------+
| a | b |
+------+------+
| 1 | 2 |
| 3 | 4 |
| 5 | NULL |
+------+------+
3 rows in set (0.00 sec)
Based on this result, we can build a new JSON array of objects:
mysql> select json_arrayagg(json_object('a', j.a, 'b', j.b)) as new
from json_table(#j, '$[*]'
columns (a int path '$.a', b int path '$.b')) as j;
+-----------------------------------------------------------+
| new |
+-----------------------------------------------------------+
| [{"a": 1, "b": 2}, {"a": 3, "b": 4}, {"a": 5, "b": null}] |
+-----------------------------------------------------------+
If you are using MySQL 5.x, I would recommend just fetch the whole JSON document to a client application where you can deserialize it into data structures to manipulate in code.

Related

MySQL: Extract all Keys from a nested JSON String

In addition to this answer is it possible to extract nested keys in a simple way?
Example:
{
"a": value,
"b": {
"c": value
"d": {
"e": value
}
}
}
Expected output: ['a', 'b.c', 'b.d.e']
What I have tried:
SELECT
f.`id` AS `field_name`
FROM table t,
JSON_TABLE(
JSON_KEYS(t.`column`, '$.b'),
'$[*]' COLUMNS(
`id` VARCHAR(191) PATH '$'
)
) AS t
but that would only show me one of the nested keys and skip the outer
SELECT JSON_SEARCH(val, 'all', 'value') result
FROM test;
| result |
| :-------------------------- |
| ["$.a", "$.b.c", "$.b.d.e"] |
db<>fiddle here

How to check if json keys are from an allowed set in PostgreSQL?

'{"a":1, "b":2, "c":3}'::jsonb ?& array['a', 'b'] checks if right hand side, an array of text, is contained by the top-level keys of the left hand side.
How can I check the opposite? That is, whether the keys of the json string are from a known set of texts.
You can get the keys of your object with json(b)_object_keys and collect them into an array. Then use the array contains operator:
array(SELECT jsonb_object_keys('{"a":1, "b":2, "c":3}'::jsonb)) <# array['a', 'b']
Alternatively, you could use a subquery on the keys and conjugate tests against the array (equivalent to the NOT EXISTS check by #GMB):
SELECT bool_and(k = ANY(array['a', 'b'])) FROM jsonb_object_keys(object) as k
(online demo)
You can use jsonb_object_keys() and ANY like so:
select 1
from jsonb_object_keys(t.js) o(x)
where not x = ANY(ar)
Here is how to use this in a query:
with t as (
select '{"a":1, "b":2, "c":3}'::jsonb js, array['a', 'b', 'c'] ar
union all select '{"a":1, "b":2, "z":3}'::jsonb js, array['a', 'b'] ar
)
select
js,
ar,
not exists(
select 1
from jsonb_object_keys(t.js) o(x)
where not x = ANY(ar)
) res
from t
Yields:
js | ar | res
:----------------------- | :------ | :-----
{"a": 1, "b": 2, "c": 3} | {a,b,c} | true
{"a": 1, "b": 2, "z": 3} | {a,b} | false

How to search a json array for some items with a certain key?

Table: some_table, Records:
id|data (json array of json object)
1|[{"a": 1, "b": 2}, {"a": 3, "b": 4}]
2|[{"a": 5, "b": 6}, {"a": 3, "b": 4}]
Would u show me the SQL (MySQL), to find any record whose data has a item (with a key "a" and value 5). Only record #2 will be found. I tried the following SQL but failed, because I'm new to use SQL for JSON.
select * from some_table where json_contains(data, '5', '$.a') = 1;
select id, data
from
(
select 1 as id, '[{"a": 1, "b": 2}, {"a": 3, "b": 4}]' as data
union select 2 as id, '[{"a": 5, "b": 6}, {"a": 3, "b": 4}]' as data
) as tmp
where json_contains(data -> '$[*].a','5')
You have two fundamental flaws in your current SQL statement:
For checking if keys exist in a JSON document, you'll want to utilize json_contains_path() as opposed to json_contains(), which will allow you to specify a specific key path without actually having to provide a value for which you're looking for said key.
The path you're leveraging to look for key a doesn't respect that that key would be contained within an array structure - you need to modify your path to look everywhere in the document for a key a. (H/T to MySQL :: MySQL 8.0 Reference Manual :: 11.6 The JSON Data Type)
With this knowledge, your query would now look like this:
select * from some_table where json_contains_path(data, 'one', '$**.a')
which, utilizing your sample dataset, returns:
+----+--------------------------------------+
| id | data |
+----+--------------------------------------+
| 1 | [{"a": 1, "b": 2}, {"a": 3, "b": 4}] |
| 2 | [{"a": 5, "b": 6}, {"a": 3, "b": 4}] |
+----+--------------------------------------+
DB Fiddle

Updateing mysql json field in array

Having this json:
{
f1: "abc",
f2: [
{id: 1, val:"a"},
{id: 2, val:"b"},
{id: 3, val:"c"}
],
f3: [
"a",
"b",
"c"
]
}
Update:
As an example:
SELECT JSON_SEARCH(
'{"f1": "abc", "f2": [{"id": "1", "val":"a"},{"id": "2", "2":"b"},{"id": 3, "val":"2"}], "f3": ["a","b","c"]}',
'all', '1', null, '$.f2[*].id');
returns the needed path for f2->id==1
I can then use
select json_set(
'{"f1": "abc", "f2": [{"id": "1", "val":"a"},{"id": "2", "2":"b"},{"id": 3, "val":"2"}], "f3": ["a","b","c"]}',
'$.f2[0].val', 'd');
to update the data.
but
SELECT JSON_SEARCH(
'{"f1": "abc", "f2": [{"id": 1, "val":"a"},{"id": "2", "2":"b"},{"id": 3, "val":"2"}], "f3": ["a","b","c"]}',
'all', '1', null, '$.f2[*].id');
won't find f2->id==1 . Neither does
SELECT JSON_SEARCH(
'{"f1": "abc", "f2": [{"id": 1, "val":"a"},{"id": "2", "2":"b"},{"id": 3, "val":"2"}], "f3": ["a","b","c"]}',
'all', 1, null, '$.f2[*].id');
=========
update 2:
I will just save the IDs as string... But right now i have onother problem:
SELECT JSON_SEARCH(
'{"mm": [{"id":"1","field":"test","value":33}]}',
'one', '1', null, '$.mm[*].id') as path;
is working
SELECT * FROM document_data where document_id=5;
update document_data set data=JSON_SET(data, '$.mm', json_array()) where document_id=5;
update document_data set data=JSON_ARRAY_APPEND(data, '$.mm', '{"id":"1","field":"test","value":33}') where document_id=5;
SELECT JSON_SEARCH(data, 'one', '1', null, '$.mm[*].id') as path from document_data where id='5';
is not working. Seems to be the quoting.
Can someone help?
==========
how can i update f2 where id==2 using JSON_SET?
Tried everything but i can't seem to figure out how to do it.
Many thanks
Rene
JSON_SEARCH, by design, seems to search only strings, see WL#7909: Server side JSON functions :: JSON_SEARCH.
One option, very unintuitive is to use something like (be careful with performance problems):
mysql> SELECT VERSION();
+-----------+
| VERSION() |
+-----------+
| 5.7.20 |
+-----------+
1 row in set (0.00 sec)
mysql> SET #`json` := '{
'> "f1": "abc",
'> "f2": [
'> {"id": 1, "val": "a"},
'> {"id": 2, "val": "b"},
'> {"id": 3, "val": "c"}
'> ],
'> "f3": ["a", "b", "c"]
'> }',
-> #`value` := 2,
-> #`base_path` := '$.f2';
Query OK, 0 rows affected (0.00 sec)
mysql> SELECT JSON_SEARCH(
-> REPLACE(
-> REPLACE(
-> REPLACE(
-> JSON_EXTRACT(#`json`, CONCAT(#`base_path`, '[*].id')),
-> ', ', '","'),
-> '[', '["'),
-> ']', '"]'),
-> 'one', #`value`) INTO #`path`;
Query OK, 1 row affected (0.00 sec)
mysql> SELECT #`path`;
+---------+
| #`path` |
+---------+
| "$[1]" |
+---------+
1 row in set (0.00 sec)
mysql> SELECT
-> CONCAT(
-> REPLACE(
-> JSON_UNQUOTE(#`path`),
-> '$',
-> #`base_path`
-> ),
-> '.val') INTO #`path`;
Query OK, 1 row affected (0.00 sec)
mysql> SELECT #`path`;
+-------------+
| #`path` |
+-------------+
| $.f2[1].val |
+-------------+
1 row in set (0.00 sec)
mysql> SELECT JSON_SET(#`json`, #`path`, 'd');
+-------------------------------------------------------------------------------------------------------------------+
| JSON_SET(#`json`, #`path`, 'd') |
+-------------------------------------------------------------------------------------------------------------------+
| {"f1": "abc", "f2": [{"id": 1, "val": "a"}, {"id": 2, "val": "d"}, {"id": 3, "val": "c"}], "f3": ["a", "b", "c"]} |
+-------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
See db-fiddle.

MySQL JSON extract a single value by id

I have a database with a JSON array in column extra_fields. I need to extract one value by id.
The JSONs looks something like this, though the number of objects is random for each row:
[
{"id":"1","value":"48768"},
{"id":"2","value" ["String","http:someurl"]},
{"id":"5","value":"somevalue"},
{"id":"6","value":""},
{"id":"8","value":"Op-Ed"},
{"id":"9","value":"8111,13498,15408"},
{"id":"10","value":"30"},
{"id":"11","value":"This is the target string"}
]
I can extract an array of ids with:
SELECT extra_fields->>"$[*].id" FROM esqt7_k2_items;
I can extract an array of values with:
SELECT extra_fields->>"$[*].value" FROM esqt7_k2_items;
I can extract a single value at the nth zero-ordered object position with:
SELECT extra_fields->>"$[2].value" FROM esqt7_k2_items;
But the problem is that there's a variable number of objects and I specifically need the value of object id = 11. JSON_EXTRACT doesn't seem to support filtering, as every variation of a JSONPATH seems to fail. $..[?(#.id=11)].value works as a JSONPATH, but this (and many variants) fails:
SELECT extra_fields->"$.[?(#.id=11)].value" FROM esqt7_k2_items; #FAILS
This is a migration project and the JSON stuff is just how Joomla chose to implement their extra fields. Kind of a pain if you ask me.
One option you can consider (be careful about performance issues):
mysql> SELECT VERSION();
+-----------+
| VERSION() |
+-----------+
| 5.7.19 |
+-----------+
1 row in set (0.00 sec)
mysql> DROP TABLE IF EXISTS `esqt7_k2_items`;
Query OK, 0 rows affected (0.00 sec)
mysql> CREATE TABLE IF NOT EXISTS `esqt7_k2_items` (
-> `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY,
-> `extra_fields` JSON NOT NULL
-> );
Query OK, 0 rows affected (0.00 sec)
mysql> INSERT INTO `esqt7_k2_items`
-> (`extra_fields`)
-> VALUES
-> ('
'> [
'> {"id": "1", "value": "48768"},
'> {"id": "2", "value": ["String","http:someurl"]},
'> {"id": "5", "value": "somevalue"},
'> {"id": "6", "value": ""},
'> {"id": "8", "value": "Op-Ed"},
'> {"id": "9", "value": "8111,13498,15408"},
'> {"id": "10", "value": "30"},
'> {"id": "11", "value": "This is the target string"}
'> ]
'> '),
-> ('
'> [
'> {"id": "1", "value": ""},
'> {"id": "9", "value": "ONE This is the target string"}
'> ]
'> '),
-> ('
'> [
'> {"id": "6", "value": ""},
'> {"id": "11", "value": "TWO This is the target string"}
'> ]
'> ');
Query OK, 3 rows affected (0.00 sec)
Records: 3 Duplicates: 0 Warnings: 0
mysql> SET #`search_id` := '11';
Query OK, 0 rows affected (0.00 sec)
mysql> SELECT `der`.`id`, `der`.`value`
-> FROM (
-> SELECT
-> `id`,
-> JSON_UNQUOTE(
-> JSON_EXTRACT(`extra_fields`,
-> CONCAT((SELECT
-> JSON_UNQUOTE(
-> JSON_SEARCH(`extra_fields` ->> '$[*].id', 'one', #`search_id`)
-> )
-> FROM `esqt7_k2_items` `esqt7_k2_items_in`
-> WHERE `esqt7_k2_items_out`.`id` = `esqt7_k2_items_in`.`id`
-> ), ".value"
-> )
-> )
-> ) `value`
-> FROM `esqt7_k2_items` `esqt7_k2_items_out`
-> ) `der`
-> WHERE `der`.`value` IS NOT NULL;
+----+-------------------------------+
| id | value |
+----+-------------------------------+
| 1 | This is the target string |
| 3 | TWO This is the target string |
+----+-------------------------------+
2 rows in set (0.00 sec)
See db-fiddle.
Simpliest way i found to get
the value of object id = 11
its by finding the right "index" of the needed object, doing a bit of PHP.
Firstly we get all objects Ids, by your simple request:
> SELECT extra_fields->>"$[*].id" FROM esqt7_k2_items;
then in PHP
$rightIndex = array_search($ids, '11');
and then you get the right object by passing this $rightIndex :
SELECT extra_fields->>"$[$rightIndex].value" FROM esqt7_k2_items;
Simple as hell! The same thing you can do anywhere else, for example, when you need to UPDATE an object in array, or its value/values, etc..
That is, Cheers!