MySQL query/Search Experiment - mysql

Question

Puzzle puzzle, riddle me functional (MySQL query/Search Experiment)
Stored Table
--------------------------------------------
| id | namespace | key | value |
--------------------------------------------
| 1 | page | item.id | test1 |
| 1 | page | content.title | page2 |
| 1 | trigger | tag | val1 |
| 2 | page | item.id | t1 |
| 2 | page | content.title | page3 |
| 2 | trigger | tag | val2 |
| 2 | oddball | num | in |
| 3 | truck | plate | 12345 |
--------------------------------------------
Search parameter: "page" can be anywhere but not in id
Desired Request output:
---------------------------------------------------------------------
|id | page.item.id | page.content.title | trigger.tag | oddball.num |
---------------------------------------------------------------------
|1 | test1 | page2 | val1 | NULL |
|2 | t1 | page3 | val2 | in |
---------------------------------------------------------------------
Hints:
ok solution: Solution with backend language (ex: php) + SQL queries
better solution: Solution with stored procedures
best solution: Solution with single SQL query, (pivot table?, temporary table?)
Fastest solution wins! (50 bounty points)
Cheers!
Goal is to have dynamic columns from agregated rows.

Answer 1

To get it working as pivot table you must run two queries:
Get the columns to be used
select distinct concat(namespace,'.',`key`) as `column`,
namespace,`key` from your_table;
+--------------------+-----------+---------------+
| column | namespace | key |
+--------------------+-----------+---------------+
| page.item.id | page | item.id |
| page.content.title | page | content.title |
| trigger.tag | trigger | tag |
| oddball.num | oddball | num |
| truck.plate | truck | plate |
+--------------------+-----------+---------------+
Combine with unique ids and get the each value as sub-query, to prevent sub-query more than one result it must contain aggregate function, I used max().
I created a stored procedure:
DELIMITER $$
DROP PROCEDURE IF EXISTS `get_pivot_table`$$
CREATE PROCEDURE `get_pivot_table`()
BEGIN
declare done int default 0;
declare v_sql text;
declare v_column varchar(100);
declare v_namespace varchar(100);
declare v_key varchar(100);
-- (1) getting the columns with this cursor
declare c_columns cursor for
select distinct concat(namespace,'.',`key`) as `column`
, namespace
,`key`
from your_table;
declare continue handler for not found set done = 1;
open c_columns;
-- (2) now creating the sub-queries based on cursor results
set v_sql = "select p.id ";
read_loop: loop
fetch c_columns into v_column, v_namespace, v_key;
if done then
leave read_loop;
end if;
set v_sql = concat(v_sql,", (select max(t.`value`) from your_table t
where t.id = p.id
and t.namespace = '", v_namespace ,"'
and t.`key` = '", v_key ,"') as `", v_column,"` ");
end loop;
close c_columns;
-- now run the entire query
set #sql = concat(v_sql," from (select distinct id from your_table) as p");
prepare stmt1 from #sql;
execute stmt1;
deallocate prepare stmt1;
END$$
DELIMITER ;
Then you can call the stored procedure:
mysql> call get_pivot_table();
+------+--------------+--------------------+-------------+-------------+-------------+
| id | page.item.id | page.content.title | trigger.tag | oddball.num | truck.plate |
+------+--------------+--------------------+-------------+-------------+-------------+
| 1 | test1 | page2 | val1 | NULL | NULL |
| 2 | t1 | page3 | val2 | in | NULL |
| 3 | NULL | NULL | NULL | NULL | 12345 |
+------+--------------+--------------------+-------------+-------------+-------------+
3 rows in set (0.00 sec)
The speed of that query will depend on the indexes of your_table and the amount of data.
It is based on An approach to mysql dynamic cross reference article.

Answer 2

Here's my solution using pivot table. Not in a single query though...
USE tempdb
GO
CREATE TABLE _temp ([id] int, [namespace] varchar(20), [key] varchar(20), [value] varchar(20))
INSERT INTO _temp VALUES (1, 'page', 'content.title', 'page2')
INSERT INTO _temp VALUES (1, 'page', 'item.id', 'test1')
INSERT INTO _temp VALUES(1, 'trigger', 'tag', 'val1')
INSERT INTO _temp VALUES (2, 'oddball', 'num', 'in')
INSERT INTO _temp VALUES (2, 'page', 'content.title', 'page3')
INSERT INTO _temp VALUES (2, 'page', 'item.id', 't1')
INSERT INTO _temp VALUES (2, 'trigger', 'tag', 'val2')
INSERT INTO _temp VALUES (3, 'truck', 'plate', '12345')
DECLARE #param AS varchar(15)
SET #param = 'page'
DECLARE #c AS nvarchar(100)
DECLARE #sql AS nvarchar(max)
SELECT #c =
ISNULL(
#c + ',[' + c + ']',
'[' + c + ']'
)
FROM (SELECT DISTINCT [namespace] + '.' + [key] AS c FROM _temp WHERE id IN (SELECT id FROM _temp WHERE ISNULL([namespace], '') + ISNULL([key], '') + ISNULL([value], '') LIKE '%' + #param + '%') ) AS col
SET #sql = N'
SELECT *
FROM
(
SELECT id,
namespace + ''.'' + [key] AS [column],
value
FROM _temp
WHERE id IN (SELECT id FROM _temp WHERE ISNULL([namespace], '''') + ISNULL([key], '''') + ISNULL([value], '''') LIKE ''%' + #param + '%'')
) AS src
PIVOT
(
MAX(value)
FOR [column]
IN (' + #c + ')
) AS piv'
EXECUTE (#sql)
DROP TABLE _temp

Answer 3

The nature of pivot tables in SQL is that it takes two queries.
The first to discover the set of distinct values and build a dynamic SQL query with one column per distinct value.
The second query to run the the dynamic query to get the pivot table result.
The reason for this is that SQL requires that you define the select-list columns before it accesses any data. There is no SQL query that can dynamically expand the columns of the select-list based on the distinct data values it discovers as it scans the table.
In other words: you can't pivot in a single SQL query.
Even in SQL implementations that have a built-in PIVOT operation, like Microsoft SQL Server, you still have to name the columns in the query syntax before you run it. Which means you need to know the distinct values you want to represent in the columns before that.
You would discover the distinct values with a simple query like this:
SELECT DISTINCT namespace, `key` FROM NoOneEverNamesTheirTableInSqlQuestions;
Then use the result of that to build a dynamic SQL query.
$sql = "SELECT DISTINCT namespace, `key` FROM NoOneEverNamesTheirTableInSqlQuestions";
$stmt = $pdo->query($sql);
$results = $stmt->fetchAll(PDO::FETCH_ASSOC);
$select_list = [];
foreach ($results as $row) {
$select_list[] = sprintf(
"MAX(CASE WHEN namespace=%s AND `key`=%s THEN value END) AS `%s.%s`",
$pdo->quote($row['namespace']), $pdo->quote($row['key']),
$row['namespace'], $row['key']);
}
$dynamic_sql = sprintf(
"SELECT id, %s FROM NoOneEverNamesTheirTableInSqlQuestions GROUP BY id",
implode(', ', $select_list));
You could also use SQL to do both at the same time, by returning the result of the first query in the form of a new SQL query to do the actual pivot.
SELECT CONCAT('SELECT id, ', GROUP_CONCAT(DISTINCT CONCAT(
'MAX(CASE WHEN namespace=', QUOTE(namespace), ' AND `key`=', QUOTE(`key`),
' THEN value END) AS `', CONCAT_WS('.', namespace, `key`), '`')),
' FROM NoOneEverNamesTheirTableInSqlQuestions GROUP BY id;') AS _sql
FROM NoOneEverNamesTheirTableInSqlQuestions;
The output of the query above is the real dynamic SQL for the pivot query, with each respective column of the select-list populated:
SELECT id,
MAX(CASE WHEN namespace='page' AND `key`='content.title' THEN value END) AS `page.content.title`,
MAX(CASE WHEN namespace='page' AND `key`='item.id' THEN value END) AS `page.item.id`,
MAX(CASE WHEN namespace='trigger' AND `key`='tag' THEN value END) AS `trigger.tag`,
MAX(CASE WHEN namespace='oddball' AND `key`='num' THEN value END) AS `oddball.num`,
MAX(CASE WHEN namespace='truck' AND `key`='plate' THEN value END) AS `truck.plate`
FROM NoOneEverNamesTheirTableInSqlQuestions GROUP BY id;
Then you run the dynamic query and you get the result you asked for:
+----+--------------------+--------------+-------------+-------------+-------------+
| id | page.content.title | page.item.id | trigger.tag | oddball.num | truck.plate |
+----+--------------------+--------------+-------------+-------------+-------------+
| 1 | page2 | test1 | val1 | NULL | NULL |
| 2 | page3 | t1 | val2 | in | NULL |
| 3 | NULL | NULL | NULL | NULL | 12345 |
+----+--------------------+--------------+-------------+-------------+-------------+
Here's both steps implemented as a MySQL stored procedure:
DELIMITER ;;
CREATE PROCEDURE PivotProc()
BEGIN
SELECT CONCAT('SELECT id, ', GROUP_CONCAT(DISTINCT CONCAT(
'MAX(CASE WHEN namespace=', QUOTE(namespace), ' AND `key`=', QUOTE(`key`),
' THEN value END) AS `', CONCAT_WS('.', namespace, `key`), '`')),
' FROM NoOneEverNamesTheirTableInSqlQuestions GROUP BY id;') AS _sql
FROM NoOneEverNamesTheirTableInSqlQuestions
INTO #sql;
PREPARE stmt FROM #sql;
EXECUTE stmt;
END;;
So what's the alternative if you don't want to run two queries?
The alternative is to run a single simple query to fetch the data as it exists in the database, with multiple rows per id. Then fix it up by post-processing it your application.
$sql = "SELECT id, namespace, `key`, value FROM NoOneEverNamesTheirTableInSqlQuestions";
$stmt = $pdo->query($sql);
$results = $stmt->fetchAll(PDO::FETCH_ASSOC);
$pivot_results = [];
foreach ($results as $row) {
if (!array_key_exists($row['id'], $pivot_results)) {
$pivot_results[$row['id']] = ['id' = $row['id']];
}
$field = sprintf("%s.%s", $row['namespace'], $row['key']);
$pivot_results[$row['id']][$field] = $row['value'];
}
Once you're done post-processing, you'll have a hash array with one row per id, each pointing to a hash array of fields indexed as the namespace.key names you described.