Calculate churn rate using a query instead of a stored procedure - mysql

Question

I am writing queries for some KPIs (Key Performance Indicators) to track user engagement. One such KPI is "Churn Rate", which I am calculating for a given month by:
Churn rate = (Total users deleted in month)/(Total users on the 1st of month)
I am using a users table with the following columns:
created_at, deleted_at
My process is to get all relevant months of user activity (in this case, based on "created_at" column, since we are getting several new users per month. We also have an activity log table which might technically be more accurate to use but doesn't go back as far) and then loop over them in a stored procedure. For each month, I'm calculating who was deleted that month and who was active on the first of that month (created on or before the 1st of the month and either not deleted or deleted after the first of that month). Then I'm dividing them to find churn rate and inserting into a temporary table. Here is my stored procedure:
DROP PROCEDURE ChurnRate;
DELIMITER $$
CREATE PROCEDURE ChurnRate()
BEGIN
DECLARE start_date DATETIME;
DECLARE end_date DATETIME;
DECLARE cur_date DATETIME;
DECLARE current_month VARCHAR(255);
DECLARE end_month VARCHAR(255);
DECLARE deleted_count BIGINT;
DECLARE active_user_count BIGINT;
DECLARE churn_rate FLOAT;
SELECT created_at FROM users ORDER BY created_at ASC LIMIT 1 INTO start_date;
SELECT created_at FROM users ORDER BY created_at DESC LIMIT 1 INTO end_date;
SET cur_date = start_date;
SET current_month = SUBSTR(cur_date,1,7);
SET end_month = SUBSTR(end_date,1,7);
DROP TEMPORARY TABLE IF EXISTS churn_table;
CREATE TEMPORARY TABLE churn_table
(
user_month VARCHAR(255),
deleted_count BIGINT,
active_user_count BIGINT,
churn_rate FLOAT
);
loop_label: LOOP
SELECT COUNT(U.id) FROM users AS U WHERE SUBSTR(U.deleted_at,1,7) = current_month INTO deleted_count;
SELECT COUNT(U.id) FROM users AS U
WHERE (U.deleted_at >= DATE_ADD(DATE_ADD(LAST_DAY(cur_date),INTERVAL 1 DAY),INTERVAL -1 MONTH) OR U.deleted_at IS NULL)
AND SUBSTR(U.created_at,1,7) <= current_month
INTO active_user_count;
INSERT INTO churn_table (user_month, deleted_count, active_user_count, churn_rate) VALUES (current_month, deleted_count, active_user_count, (deleted_count/active_user_count));
SET cur_date = DATE_ADD(cur_date, INTERVAL 1 MONTH);
SET current_month = SUBSTR(cur_date,1,7);
IF current_month <= end_month THEN
ITERATE loop_label;
END IF;
LEAVE loop_label;
END LOOP;
SELECT * FROM churn_table;
END$$
DELIMITER ;
CALL ChurnRate();
Here is a sample of some data that was produced:
user_month
churn_rate_percentage
2019-12
0
2020-01
0.0396982
2020-02
0
2020-03
0
2020-04
0
2020-05
0.112116
2020-06
0.59691
2020-07
0.26689
2020-08
0.144374
2020-09
0.141767
2020-10
0.125
2020-11
0.272904
2020-12
0.14937
My problem is this: I am using an API that requires this to be a select query. I have previously tried writing select queries for this, but they have been flawed. Grouping by "deleted_at" will not work because we will not show months for which no users have been deleted. Grouping by "created_at" and using subqueries ends up being extremely slow, as we have about 50k users. Is there a clean, efficient way to write this as a select query without affecting performance?
If there is not, I will have to write a chron to run this procedure and export the data.
Thank you

Answer 1

You shouldn't use loops in SQL that is often an indication you are doing something wrong.
Here is how to do this in a single query:
-- recursive CTE to create list of months of interest
with RECURSIVE base_months(d,y,m) AS
(
SELECT DateSerial(Year(min(create_at)), Month(min(create_at)), "1"),
min(create_at) , year(min(create_at)) , month(min(create_at))
FROM users
UNION ALL
SELECT data_add(d INTERVAL 1 MONTH) , year(data_add(d INTERVAL 1 MONTH)) , month(data_add(d INTERVAL 1 MONTH))
FROM base_months
WHERE YEAR(d) <= YEAR(CURDATE()) && MONTH(d) <= MONTH(CURDATE())
)
select
b.y as year,
b.m as month,
count(u.created_at) as total_user
sum(case when month(u.deleted_at) = b.m and year(u.deleted_at = b.y) then 1 else 0 end) as left_this_month
from base_months b
-- for each month join to the users table
join user u on u.created_at < b.d and (u.deleted_at > b.d or u.deleted_at is null)
group by b.y, b.m
If this isn't clear, first we use a recursive CTE to get all the months and years of interest -- you could do a non-recursive query on the table with a group by if only want to include create date months that are in the table -- but I think that would give you interesting results since months that don't have anyone created in that month would not be included.
Then I join that back to the users table with filters on the join to only include the rows we want to count for the given year and month. We use group by and aggregation functions to find the results.

Answer 2

Looping is likely to be terribly slow.
Is this how you decide if a user exists on Nov 1, 2020?
WHERE created_at < '2020-11'
AND deleted_at > '2020-11'
Hence, a COUNT(*) with that test would give that count?
For deletions for that month:
WHERE LEFT(deleted_at, 7) = '2020-11'
Putting those together into a single query or all months:
SELECT LEFT(created_at, 7) AS yyyymm,
( SELECT COUNT(*)
FROM users
WHERE created_at < yyyymm
AND deleted_at > yyyymm
) AS new_users,
( SELECT COUNT(*)
FROM users
WHERE deleted_at >= yyyymm
AND deleted_at < CONCAT(yyyymm, '-01')
) AS deleted_users
FROM users
GROUP BY yyyymm
ORDER BY yyyymm
That gives you 3 columns; check it out. To get the churn:
SELECT LEFT(created_at, 7) AS yyyymm,
( SELECT ... ) / ( SELECT ... ) AS churn
FROM users
GROUP BY yyyymm
ORDER BY yyyymm