Recursive update of start-end date to avoid overlaps - mysql

Question

I am dealing with a huge database in MySQL about Italian working contracts (number of rows about 20 million). Each row of my core table represents a specific signed contract for a worker with a specific employer. In order to reconstruct the work history of each worker, when I indexed the table in the import process, I have ordered workers by their identification code and the starting date of each contract. Then, each row has its own progressive ID but at the same time, I have added two fields to each row one referring to the previous ID, the other to the following one. These two fields are effectively not null only if the previous or the subsequent ID refers to the same worker.
I have made a small example of how my data looks like here (alternatively, in the following script I have created a small reproducible example).
How the history of a worker may look like
How it should change at the end
My current task is to calculate the effective number of days worked by each individual on my table. Nonetheless, data are undoubtedly characterized by huge overlapping. After all, each individual may have several overlapping contracts. For example, a contract started on date 01/01/2010 and ended on date 01/01/2012 may be followed by several other shorter contracts started later on by ending before the date 01/01/2012. Therefore, if I count the number of days effectively worked by this individual, I may have a double counting. For this reason, I want to rearrange contracts by changing their end date in order to obtain subsequent nonoverlapping contracts. The only possible overlap could be of one day.
I have made a graphical example of how the working history of an individual may look like and how I want to re-arrange it in the following two images.
Since I cannot modify the starting date of each contract/row, I wanted to work on the ending date of each contract by modifying it according to the previous contract.
I worked by following these steps:
If the ending date of the previous contract is greater than the end of the current contract (of each row), I modified the ending date placing it equal to the end date of the previous one.
Since I do not know how many contracts are actually overlapping (each contract if mliked to the previous one and the following one but there may be an overlapping contract further in the past), I decided to iterate this process by the maximum number of contract that an individual may have in my table. With this procedure, I substantially extend the overlapping time up to the case where this overlapping ceases to occur. For example, the end date of contract n.3 of the example would extend to contract n.4, n.5, and n.6. At the end of this iterative procedure, they will all have the same ending date equal today 12.
Once finished this procedure I modified the end date of each contract by placing it equal to the starting date of the following one if there is overlapping.
Here below you can find the code I used for this procedure.
-- My example table (data_example.csv on GitHub)
drop table if exists mytable;
create table mytable
(
id INT,
WORKER_ID INT not null,
EMPLOYER_ID INT not null,
dt_start date not null, -- Contract start date
dt_end date, -- Contract end date
id_prev INT, -- ID of previous contract
dt_start_prev date, -- Start date of previous contract
dt_end_prev date, -- End date of previous contract
id_next INT, -- ID of next contract
dt_start_next date, -- Start date of next contract
dt_end_next date, -- End date of next contract
primary key(id)
);
insert into mytable
(id, WORKER_ID, EMPLOYER_ID, dt_start, dt_end,
id_prev, dt_start_prev, dt_end_prev,
id_next, dt_start_next, dt_end_next)
values
(1, 5157, 3384722, '2012-01-01', '2012-01-03', NULL, NULL, NULL, 2, '2012-01-02', '2012-01-04'),
(2, 5157, 3384722, '2012-01-02', '2012-01-04', 1, '2012-01-01', '2012-01-03', 3, '2012-01-04', '2012-01-12'),
(3, 5157, 96120, '2012-01-04', '2012-01-12', 2, '2012-01-02', '2012-01-04', 4, '2012-01-07', '2012-01-08'),
(4, 5157, 3384722, '2012-01-07', '2012-01-08', 3, '2012-01-04', '2012-01-12', 5, '2012-01-08', '2012-01-10'),
(5, 5157, 3384722, '2012-01-08', '2012-01-10', 4, '2012-01-07', '2012-01-08', 6, '2012-01-10', '2012-01-11'),
(6, 5157, 3954093, '2012-01-10', '2012-01-11', 5, '2012-01-08', '2012-01-10', 7, '2012-01-12', '2012-01-15'),
(7, 5157, 3384722, '2012-01-12', '2012-01-15', 6, '2012-01-10', '2012-01-11', 8, '2012-01-14', '2012-01-16'),
(8, 5157, 3954093, '2012-01-14', '2012-01-16', 7, '2012-01-12', '2012-01-15', 9, '2012-01-14', '2012-01-14'),
(9, 5157, 3384722, '2012-01-14', '2012-01-14', 8, '2012-01-14', '2012-01-16', 10, '2012-01-14', '2012-01-20'),
(10, 5157, 96120, '2012-01-14', '2012-01-20', 9, '2012-01-14', '2012-01-14', NULL, NULL, NULL),
(11, 5990, 1940957, '2012-01-01', '2012-01-30', NULL, NULL, NULL, 12, '2012-02-01', '2012-02-15'),
(12, 5990, 4822105, '2012-02-01', '2012-02-15', 11, '2012-01-01', '2012-01-30', 13, '2012-02-10', '2012-02-10'),
(13, 5990, 1940957, '2012-02-10', '2012-02-10', 12, '2012-02-01', '2012-02-15', 14, '2012-02-16', '2012-02-20'),
(14, 5990, 1940957, '2012-02-16', '2012-02-20', 13, '2012-02-10', '2012-02-10', 15, '2012-02-17', '2012-02-28'),
(15, 5990, 4822105, '2012-02-17', '2012-02-28', 14, '2012-02-16', '2012-02-20', NULL, NULL, NULL);
-- The following table counts the number of contracts for each individual
-- I will use it the determine the maximum number of contract per worker
drop table if exists max_act;
create table max_act
as select WORKER_ID, count(*) n
from mytable
group by WORKER_ID;
set SQL_SAFE_UPDATES = 0;
-- Here I create the procedure
drop procedure if exists doiterate;
delimiter //
create procedure doiterate()
begin
declare total INT unsigned DEFAULT 0;
-- The number of iterations is equal to the maximum value in the table 'max_act'
while total <= (select MAX(n) from max_act) do
-- If the end date of the previous contract is greater than the end of the current contract
-- the procedure sets the end date equal to the end date of the previous contract
update mytable a
set a.dt_end =
case
when a.dt_end is NOT null and a.dt_end_prev > a.dt_end then a.dt_end_prev
else a.dt_end end
;
-- Here I update in each row the end date of the previous contract
update mytable a
left outer join mytable p on a.id_prev = p.id
set a.dt_end_prev =
case
when a.dt_end_prev is NOT null and a.dt_end_prev != p.dt_end then p.dt_end
else a.dt_end_prev end
;
set total = total + 1;
end while;
end//
delimiter ;
CALL doiterate();
-- Here I set the end date of each contract equal to the beginning of the next one if there is overlapping
update mytable a
set a.dt_end =
case
when a.dt_end is NOT null and a.dt_start_next < a.dt_end then a.dt_start_next
else a.dt_end end
;
set SQL_SAFE_UPDATES = 1;
However, I think this procedure is all but optimal. I have estimated it would take me days until it ends. I would really appreciate it if someone may give me some hints on how to handle this issue. Thank you in advance.

Answer 1

As already stated in one comment, I tried the use of both LAG() and LEAD() functions to concatenate in chronological order all contracts by individual. However, the procedure - maybe my fault - results to be even slower.
Therefore, I simply decided to run the procedure only on those workers only on those workers who actually had at least two overlapping contracts, maybe not the best solution (for sure not in term of coding) but at least I was able to perform the procedure (it took me more or less 1 day and half).
-- Here I am identifying contracts with an overlapping previous contract
alter table mytable add column flag_overlap INT default 0;
update mytable set flag_overlap = 1 where dt_end is NOT null and dt_end_prev > dt_end;
-- Creating a table with only those workers with at least two overlapping contracts
drop table if exists mytable_id;
create table mytable_id as select WORKER_ID
from mytable where flag_overlap = 1
group by WORKER_ID;
-- This is my table of interests with all the contracts for those workers identified in the previous step
drop table if exists mytable_mod;
create table mytable_mod
as select *
from mytable a
inner join mytable_id b on a.WORKER_ID = b.WORKER_ID
order by WORKER_ID , dt_start;
alter table mytable_mod add unique index idx_ord_id(id);
-- The rest of the code is the same as the one posted in this question,
-- simply I referred to the table 'mytable_mod' and no longer to 'mytable'.
-- [...]
-- At the end I updated the 'revised' end date of my original table 'mytable'
UPDATE mytable a
left outer join mytable_mod b on a.ord_all = b.ord_all
set
a.dt_end = b.dt_end ,
a.dt_end_next = b.dt_end_next ,
a.dt_end_prev = b.dt_end_prev
;