How to extract data from Mysql column stored in serialized JSON column? - mysql

I have column in mysql DB which contains serialized data (used PHP JSON encode to serialize them).
Here is an example:
{"name":"Group ltd","email":"support#domain.org","auth":"Andrey Ucholnik"}
Is there a built in function in mysql to extract these values without PHP ?
I mean to build query that will unserialize data.
Of course its possible to use combination of LOCATE and SUBSTR function to do that but I prefer something built in if possible.

There is no build in function in MySQL but by using small code in PHP you can easily do it as under:
<?php
$json = '{{"generated": "2010-02-26T22:26:03.156866 blahblahblah ": null, "thumbnail_url": "http://thumbs.mochiads.com/c/g/tetword-pro/_thumb_100x100.jpg", "screen4_url": "http://thumbs.mochiads.com/c/g/tetword-pro/screen4.png", "leaderboard_enabled": true, "resolution": "600x550", "width": 600}]}}';
$out = json_decode($json, true);
foreach($out["games"] as $game) {
$name = addslashes($game[name]);
$description = addslashes($game[description]);
//Here you can use your mysql_insert code
mysql_query("INSERT INTO games (name, description) VALUES('$name', '$description')") or die (mysql_error());
}
?>

There are no built-in MySQL functions to work with JSON, but here is a very simple stored function to extract values from JSON:
DELIMITER $$
CREATE FUNCTION JSON_EXTRACT(json TEXT, name CHAR(64))
RETURNS CHAR(64) DETERMINISTIC
BEGIN
SET #namePos = LOCATE(name, json);
IF #namePos = 0 THEN RETURN ''; END IF;
SET #valuePos = LOCATE(':', json, #namePos) + 1;
IF SUBSTR(json, #valuePos, 1) = '"' THEN
SET #valuePos = #valuePos + 1;
RETURN SUBSTR(json, #valuePos, LOCATE('"', json, #valuePos) - #valuePos);
ELSE
SET #valueBegin = TRIM(SUBSTR(json, #valuePos));
SET #delim1 = LOCATE(' ', #valueBegin); SET #delim1 = IF(#delim1 = 0, 64, #delim1);
SET #delim2 = LOCATE(',', #valueBegin); SET #delim2 = IF(#delim2 = 0, 64, #delim2);
SET #delim3 = LOCATE('}', #valueBegin); SET #delim3 = IF(#delim3 = 0, 64, #delim3);
RETURN LEFT(#valueBegin, LEAST(#delim1, #delim2, #delim3) - 1);
END IF;
END$$
Usage example:
SELECT JSON_EXTRACT('{"a":"aa","b" : 1, "c": 3}', 'b') AS test;
Note that the function has many limitations. For example, it doesn't handle nested classes and key names shouldn't be included in the values.

Yes , you can definitely to it using JSON_EXTRACT() function in mysql.
lets take a table that contains JSON (table client_services here) :
+-----+-----------+--------------------------------------+
| id | client_id | service_values |
+-----+-----------+------------+-------------------------+
| 100 | 1000 | { "quota": 1,"data_transfer":160000} |
| 101 | 1000 | { "quota": 2,"data_transfer":800000} |
| 102 | 1000 | { "quota": 3,"data_transfer":70000} |
| 103 | 1001 | { "quota": 1,"data_transfer":97000} |
| 104 | 1001 | { "quota": 2,"data_transfer":1760} |
| 105 | 1002 | { "quota": 2,"data_transfer":1060} |
+-----+-----------+--------------------------------------+
To Select each JSON fields , run this query :
SELECT
id, client_id,
json_extract(service_values, '$.quota') AS quota,
json_extract(service_values, '$.data_transfer') AS data_transfer
FROM client_services;
So the output will be :
+-----+-----------+----------------------+
| id | client_id | quota | data_transfer|
+-----+-----------+----------------------+
| 100 | 1000 | 1 | 160000 |
| 101 | 1000 | 2 | 800000 |
| 102 | 1000 | 3 | 70000 |
| 103 | 1001 | 1 | 97000 |
| 104 | 1001 | 2 | 1760 |
| 105 | 1002 | 2 | 1060 |
+-----+-----------+----------------------+
Hope this helps!

Related

MySQL many-many JSON aggregation merging duplicate keys

I'm having trouble returning a JSON representation of a many-many join. My plan was to encode the columns returned using the following JSON format
{
"dog": [
"duke"
],
"location": [
"home",
"scotland"
]
}
This format would handle duplicate keys by aggregating the results in a JSON array, howver all of my attempts at aggregating this structure so far have just removed duplicates, so the arrays only ever have a single element.
Tables
Here is a simplified table structure I've made for the purposes of explaining this query.
media
| media_id | sha256 | filepath |
| 1 | 33327AD02AD09523C66668C7674748701104CE7A9976BC3ED8BA836C74443DBC | /photos/cat.jpeg |
| 2 | 323b5e69e72ba980cd4accbdbb59c5061f28acc7c0963fee893c9a40db929070 | /photos/dog.jpeg |
| 3 | B986620404660DCA7B3DEC4EFB2DE80C0548AB0DE243B6D59DA445DE2841E474 | /photos/dog2.jpeg |
| 4 | 1be439dd87cd87087a425c760d6d8edc484f126b5447beb2203d21e09e2a8f11 | /photos/balloon.jpeg |
media_metdata_labels_has_media (for many-many joins)
| media_metadata_labels_label_id | media_media_id |
| 1 | 1 |
| 2 | 1 |
| 3 | 1 |
| 1 | 2 |
| 4 | 2 |
| 5 | 2 |
| 1 | 3 |
| 6 | 3 |
| 7 | 3 |
| 8 | 4 |
| 9 | 4 |
media_metadata_labels
| label_id | label_key | label_value |
| 2 | cat | lily |
| 4 | dog | duke |
| 6 | dog | rex |
| 1 | pet size | small |
| 3 | location | home |
| 7 | location | park |
| 8 | location | scotland |
| 9 | location | sky |
| 5 | location | studio |
My current attempt
My latest attempt at querying this data uses JSON_MERGE_PRESERVE with two arguments, the first is just an empty JSON object and the second is an invalid JSON document. It's invalid because there are duplicate keys, but I was hoping that JSON_MERGE_PRESERVE would merge them. It turns out JSON_MERGE_PRESERVE will only merge duplicates if they're not in the same JSON argument.
For example, this won't merge two keys
SET #key_one = '{}';
SET #key_two = '{"location": ["home"], "location": ["scotland"]}';
SELECT JSON_MERGE_PRESERVE(#key_one, #key_two);
-- returns {"location": ["scotland"]}
but this will
SET #key_one = '{"location": ["home"] }';
SET #key_two = '{"location": ["scotland"]}';
SELECT JSON_MERGE_PRESERVE(#key_one, #key_two);
-- returns {"location": ["home", "scotland"]}
So anyway, here's my current attempt
SELECT
m.media_id,
m.filepath,
JSON_MERGE_PRESERVE(
'{}',
CAST(
CONCAT(
'{',
GROUP_CONCAT(CONCAT('"', l.label_key, '":["', l.label_value, '"]')),
'}'
)
AS JSON)
)
as labels
FROM media AS m
LEFT JOIN media_metadata_labels_has_media AS lm ON lm.media_media_id = m.media_id
LEFT JOIN media_metadata_labels AS l ON l.label_id = lm.media_metadata_labels_label_id
GROUP BY m.media_id, m.filepath
-- HAVING JSON_CONTAINS(labels, '"location"', CONCAT('$.', '"home"')); -- this would let me filter on labels one they're in the correct JSON format
After trying different combinations of JSON_MERGE, JSON_OBJECTAGG, JSON_ARRAYAGG, CONCAT and GROUP_CONCAT this still leaves me scratching my head.
Disclaimer: Since posting this question I've started using mariadb instead of oracle MySQL. The function below should work for MySQL too, but in case it doesn't then any changes required will likely be small syntax fixes.
I solved this by creating a custom aggregation function
DELIMITER //
CREATE AGGREGATE FUNCTION JSON_LABELAGG (
json_key TEXT,
json_value TEXT
) RETURNS JSON
BEGIN
DECLARE complete_json JSON DEFAULT '{}';
DECLARE current_jsonpath TEXT;
DECLARE current_jsonpath_value_type TEXT;
DECLARE current_jsonpath_value JSON;
DECLARE CONTINUE HANDLER FOR NOT FOUND RETURN complete_json;
main_loop: LOOP
FETCH GROUP NEXT ROW;
SET current_jsonpath = CONCAT('$.', json_key); -- the jsonpath to our json_key
SET current_jsonpath_value_type = JSON_TYPE(JSON_EXTRACT(complete_json, current_jsonpath)); -- the json object type at the current path
SET current_jsonpath_value = JSON_QUERY(complete_json, current_jsonpath); -- the json value at the current path
-- if this is the first label value with this key then place it in a new array
IF (ISNULL(current_jsonpath_value_type)) THEN
SET complete_json = JSON_INSERT(complete_json, current_jsonpath, JSON_ARRAY(json_value));
ITERATE main_loop;
END IF;
-- confirm that an array is at this jsonpath, otherwise that's an exception
CASE current_jsonpath_value_type
WHEN 'ARRAY' THEN
-- check if our json_value is already within the array and don't push a duplicate if it is
IF (ISNULL(JSON_SEARCH(JSON_EXTRACT(complete_json, current_jsonpath), "one", json_value))) THEN
SET complete_json = JSON_ARRAY_APPEND(complete_json, current_jsonpath, json_value);
END IF;
ITERATE main_loop;
ELSE
SIGNAL SQLSTATE '45000'
SET MESSAGE_TEXT = 'Expected JSON label object to be an array';
END CASE;
END LOOP;
RETURN complete_json;
END //
DELIMITER ;
and editing my query to use it
SELECT
m.media_id,
m.filepath,
JSON_LABELAGG(l.label_key, l.label_value) as labels
FROM media AS m
LEFT JOIN media_metadata_labels_has_media AS lm ON lm.media_media_id = m.media_id
LEFT JOIN media_metadata_labels AS l ON l.label_id = lm.media_metadata_labels_label_id
GROUP BY m.media_id, m.filepath

MySQL - Search JSON data column

I have a MySQL database column that contains JSON array encoded strings. I would like to search the JSON array where the "Elapsed" value is greater than a particular number and return the corresponding TaskID value of the object the value was found. I have been attempting to use combinations of the JSON_SEARCH, JSON_CONTAINS, and JSON_EXTRACT functions but I am not getting the desired results.
[
{
"TaskID": "TAS00000012344",
"Elapsed": "25"
},
{
"TaskID": "TAS00000012345",
"Elapsed": "30"
},
{
"TaskID": "TAS00000012346",
"Elapsed": "35"
},
{
"TaskID": "TAS00000012347",
"Elapsed": "40"
}
]
Referencing the JSON above, if I search for "Elapsed" > "30" then 2 records would return
'TAS00000012346'
'TAS00000012347'
I am using MySQL version 5.7.11 and new to querying json data. Any help would be appreciated. thanks
With MySQL pre-8.0, there is no easy way to turn a JSON array to a recordset (ie, function JSON_TABLE() is not yet available).
So, one way or another, we need to manually iterate through the array to extract the relevant pieces of data (using JSON_EXTRACT()). Here is a solution that uses an inline query to generate a list of numbers ; another classic approchach is to use a number tables.
Assuming a table called mytable with a column called js holding the JSON content:
SELECT
JSON_EXTRACT(js, CONCAT('$[', n.idx, '].TaskID')) TaskID,
JSON_EXTRACT(js, CONCAT('$[', n.idx, '].Elapsed')) Elapsed
FROM mytable t
CROSS JOIN (
SELECT 0 idx
UNION ALL SELECT 1
UNION ALL SELECT 2
UNION ALL SELECT 3
) n
WHERE JSON_EXTRACT(js, CONCAT('$[', n.idx, '].Elapsed')) * 1.0 > 30
NB: in the WHERE clause, the * 1.0 operation is there to force the conversion to a number.
Demo on DB Fiddle with your sample data:
| TaskID | Elapsed |
| -------------- | ------- |
| TAS00000012346 | 35 |
| TAS00000012347 | 40 |
Yes , you can definitely to it using JSON_EXTRACT() function in mysql.
lets take a table that contains JSON (table client_services here) :
+-----+-----------+--------------------------------------+
| id | client_id | service_values |
+-----+-----------+------------+-------------------------+
| 100 | 1000 | { "quota": 1,"data_transfer":160000} |
| 101 | 1000 | { "quota": 2,"data_transfer":800000} |
| 102 | 1000 | { "quota": 3,"data_transfer":70000} |
| 103 | 1001 | { "quota": 1,"data_transfer":97000} |
| 104 | 1001 | { "quota": 2,"data_transfer":1760} |
| 105 | 1002 | { "quota": 2,"data_transfer":1060} |
+-----+-----------+--------------------------------------+
And now lets say we want client_id for all who have quota>1 , then use this query :
SELECT
id,client_id,
JSON_EXTRACT(service_values, '$.quota') AS quota
FROM client_services
WHERE JSON_EXTRACT(service_values, '$.quota') > 1;
And hence it will result into :
+-----+-----------+-------+
| id | client_id | quota |
+-----+-----------+--------
| 101 | 1000 | 2 |
| 102 | 1000 | 3 |
| 104 | 1001 | 2 |
| 105 | 1002 | 2 |
+-----+-----------+-------+
hope this helps!

how to tokenize string in record to be row

I have the data in a database table named term like this :
---------------------------------------
id_term | keyword |
------- | -----------------------------
1 | how to make search engine |
2 | application engineering |
3 | android application example |
--------------------------------------
then I want it to be like this table :
----------------------------------
| id_term | keyword |
----------------------------------
1 | how |
1 | to |
1 | Make |
1 | search |
1 | engine |
2 | application |
2 | engineering |
3 | example |
3 | application |
3 | android |
----------------------------------
I've tried googling to find references to split the string, but still have not found the appropriate expectations. In an experiment that I've done using substring_index results I could actually like this:
---------------------------------------
id_term | keyword |
------- | -------------------------------
1 | how to make search engine |
1 | how to make search engine |
1 | how to make search engine |
--------------------------------------
there anything you can help me or has the solution of my problem? mysql code that I try something like this:
select term.id_kata, SUBSTRING_INDEX (SUBSTRING_INDEX (term.keyword, ',', term.id_kata), ',', -1) keyword term from inner join keyword_doc on CHAR_LENGTH (term.keyword) -CHAR_LENGTH (REPLACE (term.keyword , ',', ''))> = term.id_kata-1 ORDER BY `term`.`keyword` DESC
I've tried googling for approximately 5 hours to find a solution, but have not found until I was confused to be asked where. there any ideas or can help provide a solution?
The solution for it problem is please take 'BETWEEN' in SQL SYNTAX. this code 100% work for it problem :
<?php
#connection
$dbhost = 'localhost';
$dbuser = 'root';
$dbpass = 'pasword';
$dbname = 'yourdatabasename';
error_reporting(E_ALL ^ E_DEPRECATED);
mysql_connect($dbhost,$dbuser,$dbpass) or die(mysqli_error('cannot connect to the server'));
mysql_select_db($dbname) or die(mysqli('database selection problem'));
$frequency = array();
$datastring = 'SELECT id_term,keyword FROM data_token WHERE id_term BETWEEN 1 AND 3';
mysql_select_db('yourdatabasename');
$calldata = mysql_query($datastring);
while($takedata = mysql_fetch_array($calldata,MYSQL_ASSOC))
{
$array = explode("\n",$takedata['keyword']);
foreach ($array as $index => $keyword)
{
if(! array_key_exists($keyword,$frequency))
{
$frequency[$keyword] = 1;
}
else
{
$frequency[$keyword] = $frequency[$keyword] + 1;
}
}
$document = $takedata['id_term'];
foreach ($frequency as $term => $tf)
{
$sqlInput = "INSERT INTO yourtablename (id_term,keyword,frequency) VALUES ('$dokumen','{$term}', {$tf})";
mysql_query($sqlInput);
}
}
?>

Extract data from json inside mysql field

I've got a a table with rows, and one of the rows has a field with data like this
{"name":"Richard","lastname":null,"city":"Olavarria","cityId":null}
And i want to select all the distinct "city" values i've got. Only using mysql server.
Is it possible? I'm trying with something like this
SELECT id FROM table_name WHERE field_name REGEXP '"key_name":"([^"]*)key_word([^"]*)"';
But i can't make the regexp work
Thanks in advance
MySQL has got support for JSON in version 5.7.7
http://mysqlserverteam.com/json-labs-release-native-json-data-type-and-binary-format/
You will be able to use the jsn_extract function to efficiently parse your JSON string.
If you have an older version and you want to solve it purely in mysql then I am afraid you have to treat it as a string and cut the value out of it (just normal string functions or use regular expressions)
This is not elegant but it will work
http://sqlfiddle.com/#!9/97cfd/14
SELECT
DISTINCT(substring(jsonfield, locate('"city":',jsonfield)+8,
locate('","', jsonfield, locate('"city":',jsonfield))-locate('"city":',jsonfield)-8)
)
FROM
ForgeRock
I have wrapped this into a stored function for those constrained to MySQL <5.7.7:
CREATE FUNCTION `json_extract_string`(
p_json text,
p_key text
) RETURNS varchar(40) CHARSET latin1
BEGIN
SET #pattern = CONCAT('"', p_key, '":"');
SET #start_i = LOCATE(#pattern, p_json) + CHAR_LENGTH(#pattern);
if #start_i = CHAR_LENGTH(#pattern) then
SET #end_i = 0;
else
SET #end_i = LOCATE('"', p_json, #start_i) - #start_i;
end if;
RETURN SUBSTR(p_json, #start_i, #end_i);
END
Note this only works with string values but is a bit more robust than #DmitryK's answer, in that it returns an empty string if the key is not found and the key can be anywhere in the JSON string.
Yes , you can definitely to it using JSON_EXTRACT() function in mysql.
lets take a table that contains JSON (table client_services here) :
+-----+-----------+--------------------------------------+
| id | client_id | service_values |
+-----+-----------+------------+-------------------------+
| 100 | 1000 | { "quota": 1,"data_transfer":160000} |
| 101 | 1000 | { "quota": 2,"data_transfer":800000} |
| 102 | 1000 | { "quota": 3,"data_transfer":70000} |
| 103 | 1001 | { "quota": 1,"data_transfer":97000} |
| 104 | 1001 | { "quota": 2,"data_transfer":1760} |
| 105 | 1002 | { "quota": 2,"data_transfer":1060} |
+-----+-----------+--------------------------------------+
To Select each JSON fields , run this query :
SELECT
id, client_id,
json_extract(service_values, '$.quota') AS quota,
json_extract(service_values, '$.data_transfer') AS data_transfer
FROM client_services;
So the output will be :
+-----+-----------+----------------------+
| id | client_id | quota | data_transfer|
+-----+-----------+----------------------+
| 100 | 1000 | 1 | 160000 |
| 101 | 1000 | 2 | 800000 |
| 102 | 1000 | 3 | 70000 |
| 103 | 1001 | 1 | 97000 |
| 104 | 1001 | 2 | 1760 |
| 105 | 1002 | 2 | 1060 |
+-----+-----------+----------------------+
NOW, if you want lets say DISTINCT quota , then run this query :
SELECT
distinct( JSON_EXTRACT(service_values, '$.quota')) AS quota
FROM client_services;
So this will result into your desired output :
+-------+
| quota |
+-------+
| 1 |
| 2 |
| 3 |
+-------+
hope this helps!
See MariaDB's Dynamic Columns.
Also, search this forum for [mysql] [json]; the topic has been discussed often.
This may be a little late, but the accepted answer didn't work for me. I used SUBSTRING_INDEX to achieve the desired result.
SELECT
ID, SUBSTRING_INDEX(SUBSTRING_INDEX(JSON, '"mykey" : "', -1), '",', 1) MYKEY
FROM MY_TABLE;
Hope this helps.

Count and select all dates for a specific field in MySQL

i have a data format like this:
+----+--------+---------------------+
| ID | utente | data |
+----+--------+---------------------+
| 1 | Man1 | 2014-02-10 12:12:00 |
+----+--------+---------------------+
| 2 | Women1 | 2015-02-10 12:12:00 |
+----+--------+---------------------+
| 3 | Man2 | 2016-02-10 12:12:00 |
+----+--------+---------------------+
| 4 | Women1 | 2014-03-10 12:12:00 |
+----+--------+---------------------+
| 5 | Man1 | 2014-04-10 12:12:00 |
+----+--------+---------------------+
| 6 | Women1 | 2014-02-10 12:12:00 |
+----+--------+---------------------+
I want to make a report that organise the ouptout in way like this:
+---------+--------+-------+---------------------+---------------------+---------------------+
| IDs | utente | count | data1 | data2 | data3 |
+---------+--------+-------+---------------------+---------------------+---------------------+
| 1, 5 | Man1 | 2 | 2014-02-10 12:12:00 | 2014-04-10 12:12:00 | |
+---------+--------+-------+---------------------+---------------------+---------------------+
| 2, 4, 6 | Women1 | 3 | 2015-02-10 12:12:00 | 2014-03-10 12:12:00 | 2014-05-10 12:12:00 |
+---------+--------+-------+---------------------+---------------------+---------------------+
All the row thath include the same user (utente) more than one time will be included in one row with all the dates and the count of records.
Thanks
While it's certainly possible to write a query that returns the data in the format you want, I would suggest you to use a GROUP BY query and two GROUP_CONCAT aggregate functions:
SELECT
GROUP_CONCAT(ID) as IDs,
utente,
COUNT(*) as cnt,
GROUP_CONCAT(data ORDER BY data) AS Dates
FROM
tablename
GROUP BY
utente
then at the application level you can split your Dates field to multiple columns.
Looks like a fairly standard "Breaking" report, complicated only by the fact that your dates extend horizontally instead of down...
SELECT * FROM t ORDER BY utente, data
$lastutente = $lastdata = '';
echo "<table>\n";
while ($row = fetch()) {
if ($lastutente != $row['utente']) {
if ($lastutente != '') {
/****
* THIS SECTION REF'D BELOW
***/
echo "<td>$cnt</td>\n";
foreach ($datelst[] as $d)
echo "<td>$row[data]</td>\n";
for ($i = count($datelst); $i < $NumberOfDateCells; $i++)
echo "<td> </td>\n";
echo "</tr>\n";
/****
* END OF SECTION REF'D BELOW
***/
}
echo "<tr><td>$row[utente]</td>\n"; // start a new row - you probably want to print other stuff too
$datelst = array();
$cnt = 0;
}
if ($lastdata != $row['data']) {
datelst[] = $row['data'];
}
$cnt += $row['cnt']; // or $cnt++ if it's one per row
}
print the end of the last row - see SECTION REF'D ABOVE
echo "</table>\n";
You could add a GROUP BY utente, data to your query above to put a little more load on mysql and a little less on your code - then you should have SUM(cnt) as cnt or COUNT(*) as cnt.