I have a table with a JSON column, let's call it json_data and the column contents look like...
[{ "data": { ... }, "name": "name_1" }, { "data": { ... }, "name": "name_2" }]
[{ "data": { ... }, "name": "name_2" }]
[{ "data": { ... }, "name": "name_3" }, { "data": { ... }, "name": "name_5" }]
[{ "data": { ... }, "name": "name_4" }]
...and I am looking to get back
["name_1", "name_2", "name_3", "name_4", "name_5"]
or something of that sort. I can use JSON_EXTRACT to quite easily get each row's set of name fields...
SELECT JSON_EXTRACT(json_data, "$**.name") FROM my_table;
...so now I have rows each containing a single array of comma separated strings, and can use GROUP_CONCAT to merge them...
SELECT REPLACE(REPLACE(GROUP_CONCAT(names SEPARATOR ','), '[', ''), ']', '')
FROM (
SELECT JSON_EXTRACT(json_data, '$**.name') as names
FROM my_table
WHERE json_data <> '' -- exclude empty entries
LIMIT 10) x -- test on sample size as the table is quite large
ORDER BY NULL; -- get names from all rows
...at this point I have all the data that I want in a single row as comma separated strings...
"name_1","name_2","name_2","name_3","name_4","name_5"
except that it has duplicates (MANY of them).
It seems like it should be easy to take this and run distinct on it, but so far I cannot figure out how to split the string into all the elements and perform distinct. SUBSTRING_INDEX seems to be what I need, but that only gets individual elements... any help is appreciated!
In modern versions of MySQL (>= 8.0.4) the query would be relatively simple:
SELECT
GROUP_CONCAT(
DISTINCT JSON_QUOTE(`der`.`names`)
) `names`
FROM
`my_table`,
JSON_TABLE(`my_table`.`json_data`,
'$[*]' COLUMNS(
`names` VARCHAR(10) PATH '$.name'
)
) `der`
ORDER BY
`names`;
See db-fiddle.
However, in older versions, it's not that simple, one option may be to use a temporary table and a prepared statement:
SET #`ddl` := CONCAT('INSERT INTO `my_table` VALUES ',
(SELECT
GROUP_CONCAT(
REPLACE(
REPLACE(
REPLACE(
`json_data` -> '$**.name',
'[', '('),
']', ')'),
',', '),(')
)
FROM
`my_table`
)
);
See Rextester.
In any case, remember 5.1.7 Server System Variables::group_concat_max_len.
Related
In MySQL, I am trying to create a JSON_OBJECT from rows of data containing key-value pairs.
Here is the sample data:
CREATE TABLE TempValuePair( MyKey VARCHAR(64), MyValue VARCHAR(64) );
INSERT INTO TempValuePair VALUE
('Country', 'Argentina'),
('Capital', 'Buenos Aires'),
('Population', 45810000 );
The following statement seems to return an argument that conforms to the JSON_OBJECT requirements:
SELECT GROUP_CONCAT(
CONCAT( '\'', MyKey, '\',\'', Myvalue, '\'' )
ORDER BY MyKey
)
FROM TempValuePair;
However, the following statement fails:
SELECT GROUP_CONCAT(
CONCAT( '\'', MyKey, '\',\'', Myvalue, '\'' )
ORDER BY MyKey
)
FROM TempValuePair
);
Any advice about what I am doing wrong would be greatly appreciated. Thanks!
You seem to want json_objectagg, which is available in MySQL since version 5.7. The function aggregates key/value pairs from multiple rows into a single JSON object:
select json_objectagg(mykey, myvalue) as js from TempValuePair;
Yields:
{"Capital": "Buenos Aires", "Country": "Argentina", "Population": "45810000"}
Demo on DB Fiddle
I'd like to convert result table to JSON array in MySQL using preferably only plain MySQL commands. For example with query
SELECT name, phone FROM person;
| name | phone |
| Jack | 12345 |
| John | 23455 |
the expected JSON output would be
[
{
"name": "Jack",
"phone": 12345
},
{
"name": "John",
"phone": 23455
}
]
Is there way to do that in plain MySQL?
EDIT:
There are some answers how to do this with e.g. MySQL and PHP, but I couldn't find pure MySQL solution.
New solution:
Built using Your great comments, thanks!
SELECT JSON_ARRAYAGG(JSON_OBJECT('name', name, 'phone', phone)) from Person;
Old solution:
With help from #Schwern I managed to put up this query, which seems to work!
SELECT CONCAT(
'[',
GROUP_CONCAT(JSON_OBJECT('name', name, 'phone', phone)),
']'
)
FROM person;
You can use json_object to get rows as JSON objects.
SELECT json_object('name', name, 'phone', phone)
FROM person;
This won't put them in an array, or put commas between them. You'll have to do that in the code which is fetching them.
If you're stuck on MySQL 5.6 like me, try this:
SELECT
CONCAT(
'[',
GROUP_CONCAT(
CONCAT(
'{"name":"', name, '"',
',"phone":"', phone, '"}'
)
),
']'
) as json
FROM person
There are two "group by" functions for JSON called json_arrayagg, json_objectagg.
This problem can be solved with:
SELECT json_arrayagg(
json_merge(
json_object('name', name),
json_object('phone', phone)
)
) FROM person;
This requires MySQL 5.7+.
If you need a nested JSON Array Object, you can join JSON_OBJECT with json_arrayagg as below:
{
"nome": "Moon",
"resumo": "This is a resume.",
"dt_inicial": "2018-09-01",
"v.dt_final": null,
"data": [
{
"unidade": "unit_1",
"id_unidade": 9310
},
{
"unidade": "unit_2",
"id_unidade": 11290
},
{
"unidade": "unit_3",
"id_unidade": 13544
},
{
"unidade": "unit_4",
"id_unidade": 13608
}
]
}
You can also do it like this:
CREATE DEFINER=`root`#`localhost` PROCEDURE `get_lst_caso`(
IN `codigo` int,
IN `cod_base` int)
BEGIN
DECLARE json TEXT DEFAULT '';
SELECT JSON_OBJECT(
'nome', v.nome,
'dt_inicial', v.dt_inicial,
'v.dt_final', v.dt_final,
'resumo', v.resumo,
'data', ( select json_arrayagg(json_object(
'id_unidade',`tb_unidades`.`id_unidade`,
'unidade',`tb_unidades`.`unidade`))
from tb_caso_unidade
INNER JOIN tb_unidades ON tb_caso_unidade.cod_unidade = tb_unidades.id_unidade
WHERE tb_caso_unidade.cod_caso = codigo)
) INTO json
FROM v_caso AS v
WHERE v.codigo = codigo and v.cod_base = cod_base;
SELECT json;
END
For most situations, I use DataGreap, but for big tables, it is not work.
My GIST shell script
If I have two json arrays of strings in mysql, is there a native(or not native) way to merge these two arrays into one with unique strings?
If I try json_merge I get the following result with duplicates:
set #array1 =JSON_EXTRACT('["apple","pear","banana"]', '$');
set #array2 =JSON_EXTRACT('["pear","banana","apple","kiwi"]', '$');
select json_merge(#array1,#array2);
> ["apple", "pear", "banana", "pear", "banana", "apple", "kiwi"]
And If is try json_merge_preserve gives me the same result:
set #array1 =JSON_EXTRACT('["apple","pear","banana"]', '$');
set #array2 =JSON_EXTRACT('["pear","banana","apple","kiwi"]', '$');
select json_merge_preserve(#array1,#array2);
> ["apple", "pear", "banana", "pear", "banana", "apple", "kiwi"]
Is there a function that will return the unique array?
["apple", "banana", "pear", "kiwi"]
Edit: json_merge_patch doesn't work because it only replaces the first array with the second:
set #array1 =JSON_EXTRACT('["apple","grape","banana"]', '$');
set #array2 =JSON_EXTRACT('["pear","banana","apple","kiwi"]', '$');
select json_merge_patch(#array1,#array2);
> ["pear", "banana", "apple", "kiwi"]
In this case I lose "grape". I believe that the logic in patch is 0 : 'val', 1:'val2' merge with 0:val3 then 0 : 'val3', 1:'val2'
If the question still lives, here's a simple solution using MySQL 8.0's JSON_TABLE.
set #a1 ='["apple","grape","banana","banana","pear"]';
set #a2 ='["pear","banana","apple","kiwi","banana","apple"]';
select fruit
from json_table(
json_merge_preserve(#a1, #a2),
'$[*]' columns (
fruit varchar(255) path '$'
)
) as fruits
group by fruit; # get distinct values
# gives
apple
grape
banana
pear
kiwi
To get a one-line response, we have to drop group by and get a bit more creative.
Unfortunately, JSON_ARRAYAGG doesn't support distinct directive, so we'll have to use GROUP_CONCAT:
select group_concat(distinct fruit)
from json_table(
json_merge_preserve(#a1, #a2),
'$[*]' columns (
fruit varchar(255) path '$'
)
) as fruits;
# without group by directive!
# gives: apple,banana,grape,kiwi,peas
To get a proper json array on-line response, we just play around with CONCATs:
select cast(
concat('["', group_concat(distinct fruit separator '", "'), '"]')
as json)
...
# gives: ["apple", "banana", "grape", "kiwi", "pear"]
EDIT:
I've found out a proper JSON_ARRAYAGG solution using one more nested virtual table to group results in.
select json_arrayagg(fruit)
from (
select fruit
from json_table(
json_merge_preserve(#a1, #a2),
'$[*]' columns (
fruit varchar(255) path '$'
)
) as fruits
group by fruit -- group here!
) as unique_fruits;
Read my Best Practices for using MySQL as JSON storage :)
After too much thinking, and thanks to #lefred. I found a hack that can accomplish this.
This is way too hacky, but i will publish it while someone else comes with a better implementation or the mysql guys make a proper function for this.
First, we replace the string strategically to create a json object instead of an array.
Then, we use json_merge_path and finally we use json_keys to obtain an array :V
set #array1 ='["apple","grape","banana","banana","pear"]';
set #array2 ='["pear","banana","apple","kiwi","banana","apple"]';
set #aux1 = REPLACE(REPLACE(REPLACE(#array1, ',', ' : "1", '), ']', ' : "1" }'), '[', '{');
set #aux2 = REPLACE(REPLACE(REPLACE(#array2, ',', ' : "1", '), ']', ' : "1" }'), '[', '{');
select #aux1, #aux2;
select json_keys(json_merge_patch(json_extract(#aux1, '$'),json_extract(#aux2,'$')))
> ["kiwi", "pear", "apple", "grape", "banana"]
Use SELECT DISTINCT:
set #array1 =JSON_EXTRACT('["apple","grape","banana"]', '$');
set #array2 =JSON_EXTRACT('["pear","banana","apple","kiwi"]', '$');
select json_arrayagg(fruit) from (
select
distinct fruit
from json_table(
json_merge_preserve(#array1, #array2),
'$[*]' columns (fruit varchar(255) path '$')
) fruits
) f;
According to the documentation, json_merge_preserve preserves duplicates. Also, if you are using MySQL 8.0.3 or over, json_merge is deprecated and json_merge_preserve should be used. I think that you need to use JSON_MERGE_PATCH.
More details here https://database.guide/json_merge_preserve-merge-multiple-json-documents-in-mysql/
I have a table with about 900,000 rows. One of the columns has a 7 character value that I need to search for client-side validation of user input. I'm currently using ajax as the user types, but most of the users can out-run the ajax round trips and end up having to wait until the all the validation calls return. So I want to shift the wait time to the initial load of the app and take advantage of browser caching. So I'll bundle minify and gzip the json file with webpack. I'll probably make it an entry that I can then require/ensure as the app loads.
To make the validation super fast on the client side I want to produce a json file that has a single json structure with the first two characters of the 7 character column as the key to the object with an array of all values that start with the first two characters as an array in the value for said key (see example below). I can then use indexOf to find the value within this segmented list and it will be very quick.
As mentioned above I'm currently using ajax as the user types.
I'm not going to show my code because it's too complex. But I basically keep track of the pending ajax requests and when the request that started with the last value the user entered returns (the value currently sitting in the text box), then I can show the user if the entry exists or not. That way if the request return out of order, I'm not giving false positives.
I'm using SQL Server 2016 so I want to use for json to produce my desired output. But here's what I want to produce:
{
"00": [ "0000001", "0000002", ... ],
"10": [ "1000000", "1000001", ... ],
//...
"99": [ "9900000", "9900001", ...]
}
So far I have been unable to figure out how to use substring( mySevenDigitCol, 1, 2 ) as the key in the json object.
I'm not sure if this can be done using FOR JSON AUTO (["0000001", "0000002", ... ] is the difficult part), but next approach based on string manipulation is one possible solution to your problem:
Input:
CREATE TABLE #Data (
SevenDigitColumn varchar(7)
)
INSERT INTO #Data
(SevenDigitColumn)
VALUES
('0000001'),
('0000002'),
('0000003'),
('0000004'),
('0000005'),
('1000001'),
('1000002'),
('9900001'),
('9900002')
T-SQL:
;WITH JsonData AS (
SELECT
SUBSTRING(dat.SevenDigitColumn, 1, 2) AS [Key],
agg.StringAgg AS [Values]
FROM #Data dat
CROSS APPLY (
SELECT STUFF(
(
SELECT CONCAT(',"', SevenDigitColumn, '"')
FROM #Data
WHERE SUBSTRING(SevenDigitColumn, 1, 2) = SUBSTRING(dat.SevenDigitColumn, 1, 2)
FOR XML PATH('')
), 1, 1, '') AS StringAgg
) agg
GROUP BY SUBSTRING(dat.SevenDigitColumn, 1, 2), agg.StringAgg
)
SELECT CONCAT(
'{',
STUFF(
(
SELECT CONCAT(',"', [Key], '": [', [Values], ']')
FROM JsonData
FOR XML PATH('')
), 1, 1, ''),
'}')
Output:
{"00": ["0000001","0000002","0000003","0000004","0000005"],"10": ["1000001","1000002"],"99": ["9900001","9900002"]}
Notes:
With SQL Server 2017+ you can use STRING_AGG() function:
SELECT
CONCAT(
'{',
STRING_AGG(KeyValue, ','),
'}'
)
FROM (
SELECT CONCAT(
'"',
SUBSTRING(dat.SevenDigitColumn, 1, 2),
'": [',
STRING_AGG('"' + SevenDigitColumn + '"', ','),
']'
) AS KeyValue
FROM #Data dat
GROUP BY SUBSTRING(dat.SevenDigitColumn, 1, 2)
) JsonData
Notes:
If your column data is not only digits, you should use STRING_ESCAPE() with 'json' as second parameter to escape special characters.
I'd like to convert result table to JSON array in MySQL using preferably only plain MySQL commands. For example with query
SELECT name, phone FROM person;
| name | phone |
| Jack | 12345 |
| John | 23455 |
the expected JSON output would be
[
{
"name": "Jack",
"phone": 12345
},
{
"name": "John",
"phone": 23455
}
]
Is there way to do that in plain MySQL?
EDIT:
There are some answers how to do this with e.g. MySQL and PHP, but I couldn't find pure MySQL solution.
New solution:
Built using Your great comments, thanks!
SELECT JSON_ARRAYAGG(JSON_OBJECT('name', name, 'phone', phone)) from Person;
Old solution:
With help from #Schwern I managed to put up this query, which seems to work!
SELECT CONCAT(
'[',
GROUP_CONCAT(JSON_OBJECT('name', name, 'phone', phone)),
']'
)
FROM person;
You can use json_object to get rows as JSON objects.
SELECT json_object('name', name, 'phone', phone)
FROM person;
This won't put them in an array, or put commas between them. You'll have to do that in the code which is fetching them.
If you're stuck on MySQL 5.6 like me, try this:
SELECT
CONCAT(
'[',
GROUP_CONCAT(
CONCAT(
'{"name":"', name, '"',
',"phone":"', phone, '"}'
)
),
']'
) as json
FROM person
There are two "group by" functions for JSON called json_arrayagg, json_objectagg.
This problem can be solved with:
SELECT json_arrayagg(
json_merge(
json_object('name', name),
json_object('phone', phone)
)
) FROM person;
This requires MySQL 5.7+.
If you need a nested JSON Array Object, you can join JSON_OBJECT with json_arrayagg as below:
{
"nome": "Moon",
"resumo": "This is a resume.",
"dt_inicial": "2018-09-01",
"v.dt_final": null,
"data": [
{
"unidade": "unit_1",
"id_unidade": 9310
},
{
"unidade": "unit_2",
"id_unidade": 11290
},
{
"unidade": "unit_3",
"id_unidade": 13544
},
{
"unidade": "unit_4",
"id_unidade": 13608
}
]
}
You can also do it like this:
CREATE DEFINER=`root`#`localhost` PROCEDURE `get_lst_caso`(
IN `codigo` int,
IN `cod_base` int)
BEGIN
DECLARE json TEXT DEFAULT '';
SELECT JSON_OBJECT(
'nome', v.nome,
'dt_inicial', v.dt_inicial,
'v.dt_final', v.dt_final,
'resumo', v.resumo,
'data', ( select json_arrayagg(json_object(
'id_unidade',`tb_unidades`.`id_unidade`,
'unidade',`tb_unidades`.`unidade`))
from tb_caso_unidade
INNER JOIN tb_unidades ON tb_caso_unidade.cod_unidade = tb_unidades.id_unidade
WHERE tb_caso_unidade.cod_caso = codigo)
) INTO json
FROM v_caso AS v
WHERE v.codigo = codigo and v.cod_base = cod_base;
SELECT json;
END
For most situations, I use DataGreap, but for big tables, it is not work.
My GIST shell script