MySQL / MariaDB: Historization - How do it better - mysql

Question

I have exports of person data which I would like to import into a table considering historization.
I wrote single sql-steps but two questions arises:
1. There is a step where I got a unexpected date
2: I would like to avoid manually submitting some steps and using stored procedure
The tables are:
Table to be filled considering historization:
CREATE TABLE person (
id INTEGER DEFAULT NULL
, name VARCHAR(50) DEFAULT NULL
, effective_dt DATE DEFAULT NULL
, expiry_dt DATE DEFAULT NULL
);
Table with person data to be imported:
CREATE TABLE person_stg (
id INTEGER DEFAULT NULL
, name VARCHAR(50) DEFAULT NULL
, export_dt DATE DEFAULT NULL
, import_flag TINYINT DEFAULT 0
);
-- Several exports which has to be imported
INSERT INTO person_stg (id, name, export_dt) VALUES
(1,'Jonn' , '2000-01-01')
, (2,'Marry' , '2000-01-01')
, (1,'John' , '2000-01-05')
, (2,'Marry' , '2000-01-06')
, (2,'Mary' , '2000-01-10')
, (3,'Samuel', '2000-01-10')
, (2,'Maria' , '2000-01-15')
;
The following first step (1) populates the table person with the first state of the person:
INSERT INTO person
SELECT a.id, a.name, a.export_dt, '9999-12-31' expiry_dt
FROM person_stg a
LEFT JOIN person_stg b
ON a.id = b.id
AND a.export_dt > b.export_dt
WHERE b.id IS NULL
;
SELECT * FROM person ORDER BY id, effective_dt;
+----+--------+--------------+------------+
| id | name | effective_dt | expiry_dt |
+----+--------+--------------+------------+
| 1 | Jonn | 2000-01-01 | 9999-12-31 |
| 2 | Marry | 2000-01-01 | 9999-12-31 |
| 3 | Samuel | 2000-01-10 | 9999-12-31 |
+----+--------+--------------+------------+
Step (2) changes the expiry date:
-- (2) Update expiry_dt where changes happened
UPDATE
person a
, person_stg b
SET a.expiry_dt = SUBDATE(b.export_dt,1)
WHERE a.id = b.id
AND a.name <> b.name
AND a.expiry_dt = '9999-12-31'
AND b.export_dt = (SELECT MIN(b.export_dt)
FROM person_stg c
WHERE b.id = c.id
AND c.import_flag = 0
)
;
SELECT * FROM person ORDER BY id, effective_dt;
+----+--------+--------------+------------+
| id | name | effective_dt | expiry_dt |
+----+--------+--------------+------------+
| 1 | Jonn | 2000-01-01 | 2000-01-04 |
| 2 | Marry | 2000-01-01 | 2000-01-09 |
| 3 | Samuel | 2000-01-10 | 9999-12-31 |
+----+--------+--------------+------------+
The third step (3) inserts the second status of person data:
-- (3) Insert new exports which has changes
INSERT INTO person
SELECT a.id, a.name, a.export_dt, '9999-12-31' expiry_dt
FROM person_stg a
INNER JOIN person b
ON a.id = b.id
AND b.expiry_dt = SUBDATE(a.export_dt,1)
AND a.export_dt > b.effective_dt
AND a.import_flag = 0
;
SELECT * FROM person ORDER BY id, effective_dt;
+----+--------+--------------+------------+
| id | name | effective_dt | expiry_dt |
+----+--------+--------------+------------+
| 1 | Jonn | 2000-01-01 | 2000-01-04 |
| 1 | John | 2000-01-05 | 9999-12-31 |
| 2 | Marry | 2000-01-01 | 2000-01-09 |
| 2 | Mary | 2000-01-10 | 9999-12-31 |
| 3 | Samuel | 2000-01-10 | 9999-12-31 |
+----+--------+--------------+------------+
And the last step (4) defines on person_stg which record was inserted:
-- (4) Define imported records
UPDATE
person_stg a
, person b
SET import_flag = 1
WHERE a.id = b.id
AND a.export_dt = b.effective_dt
;
So far, so good. If I repeat step (2) I got the following table:
+----+--------+--------------+------------+
| id | name | effective_dt | expiry_dt |
+----+--------+--------------+------------+
| 1 | Jonn | 2000-01-01 | 2000-01-04 |
| 1 | John | 2000-01-05 | 9999-12-31 |
| 2 | Marry | 2000-01-01 | 2000-01-09 |
| 2 | Mary | 2000-01-10 | 1999-12-31 | <--- ??? Should be 2000-01-14
| 3 | Samuel | 2000-01-10 | 9999-12-31 |
+----+--------+--------------+------------+
Mary/2000-01-10 got expiry_dt 1999-12-31 instead of 2000-01-14. I don't understand how this can happened.
So, my questions are:
(1a) Why this update of the expiry date gives this strange date?
(1b) Is there maybe a better code then (2)?
(2) How can I repeat steps (2) until (4) automatically? I need only some hints for a stored procedure.
-- (4) Define imported records
UPDATE
person_stg a
, person b
SET import_flag = 1
WHERE a.id = b.id
AND a.export_dt = b.effective_dt
;

Answer 1

If I understand what you want to do, you don't need a multi-step process. You are just looking for the "end date" for each record. Here is a method that uses correlated subqueries:
SELECT p.*, export_dt as effdate,
COALESCE((SELECT export_dt - interval 1 day
FROM person_stg p2
WHERE p2.id = p.id AND
p2.export_dt > p.export_dt
ORDER BY p2.export_dt
LIMIT 1
), '9999-12-31') as enddate
FROM person_stg p;
You can also do something using variables.
I'm not sure if this answers your question, because it replaces the whole process with a simpler query.

Answer 2

I found a solution using cursor which I never used before. First I made a stored procedure (SP) sp_add_record which update, insert new status or insert a new element given id and export_dt from patient_stg. This stored procedure was then used using SP with cursor (curs_add_records):
CALL curs_add_records();
SELECT * FROM person;
+----+--------+--------------+------------+
| id | name | effective_dt | expiry_dt |
+----+--------+--------------+------------+
| 1 | Jonn | 2000-01-01 | 2000-01-04 |
| 2 | Marry | 2000-01-01 | 2000-01-09 |
| 1 | John | 2000-01-05 | 9999-12-31 |
| 2 | Mary | 2000-01-10 | 2000-01-14 |
| 3 | Samuel | 2000-01-10 | 9999-12-31 |
| 2 | Maria | 2000-01-15 | 9999-12-31 |
+----+--------+--------------+------------+
The advantage of this procedure is that I can load table with the same code independently if it is an inital load (population load) or incremental.
Literatur I used:
Djoni Damrawikarte: Dimensional Data Warehousing with MySQL (DWH issues)
Ben Forta: MariaDB Crash Course (SP issues)
What follows are the SP I used.
PS: Was it appropriate to answer to my own question?
DELIMITER //
DROP PROCEDURE IF EXISTS sp_add_record //
CREATE PROCEDURE sp_add_record(
IN p_id INTEGER
, IN p_export_dt DATE
)
BEGIN
-- Change expiry_dt
UPDATE
person p
, person_stg s
SET p.expiry_dt = SUBDATE(p_export_dt,1)
WHERE p.id = s.id
AND p.id = p_id
AND s.export_dt = p_export_dt
AND p.effective_dt <= p_export_dt
AND ( p.name <> s.name )
AND p.expiry_dt = '9999-12-31'
;
-- Add new status
INSERT INTO person
SELECT s.id, s.name, s.export_dt, '9999-12-31' expiry_dt
FROM
person p
, person_stg s
WHERE p.id = s.id
AND p.id = p_id
AND s.export_dt = p_export_dt
AND ( p.name <> s.name )
-- does a entry exists with new expiry_dt?
AND EXISTS (SELECT *
FROM person p2
WHERE p2.id = p.id
AND p.expiry_dt = SUBDATE(p_export_dt,1)
)
-- entry with open expiry_dt not should not exist
AND NOT EXISTS (SELECT *
FROM person p3
WHERE p3.id = p.id
AND p3.expiry_dt = '9999-12-31'
)
;
-- Add new id
INSERT INTO person
SELECT s.id, s.name, s.export_dt, '9999-12-31' expiry_dt
FROM person_stg s
WHERE s.export_dt = p_export_dt
AND s.id = p_id
-- Add new id from stage if it does not exist in person
AND s.id NOT IN (SELECT p3.id
FROM person p3
WHERE p3.id = s.id
AND p3.expiry_dt = '9999-12-31'
)
;
END
//
DELIMITER ;
DELIMITER //
DROP PROCEDURE IF EXISTS curs_add_records //
CREATE PROCEDURE curs_add_records()
BEGIN
-- Local variables
DECLARE done BOOLEAN DEFAULT 0;
DECLARE p_id INTEGER;
DECLARE p_export_dt DATE;
-- Cursor
DECLARE c1 CURSOR
FOR
SELECT id, export_dt
FROM person_stg
ORDER BY export_dt, id
;
-- Declare continue handler
DECLARE CONTINUE HANDLER FOR SQLSTATE '02000' SET done=1;
-- Open cursor
OPEN c1;
-- Loop through all rows
REPEAT
-- Get record
FETCH c1 INTO p_id, p_export_dt;
-- Call add record procedure
CALL sp_add_record(p_id,p_export_dt);
-- End of loop
UNTIL done END REPEAT;
-- Close cursor
CLOSE c1;
END;
//
DELIMITER ;