ClickHouse deduplication/upsert with different functions per column - duplicates

Question

I have a ClickHouse table which looks like this:
CREATE TABLE test
(
id Int,
property_id Int,
created_at DateTime('UTC'),
modified_at DateTime('UTC'),
data Int,
json_str Nullable(String)
) ENGINE = MergeTree()
PARTITION BY toYYYYMM(created_at)
ORDER BY (property_id, created_at);
When inserting new rows, I want to update (upsert) existing rows with matching id and property_id according to these rules:
created_at: Keep the earliest
modified_at: Keep the latest
data: Keep the value of the row with the latest modified_at
json_str: Ideally, deep merge json objects (stored as strings) of all matching rows
I did quite a bit of research and tried setting up a deduplication pipeline, using a source table, a destination table (ENGINE = AggregatingMergeTree) and a materialized view (using minState, maxState, argMaxState) but I couldn't figure it out so far. I'm running into errors related to primary key, partitioning, wrong aggregation functions, etc. Even a setup without merging json_str would be very helpful.

Answer 1

After a lot of trial and error, I found a solution (ignoring json_str for now):
-- Source table with duplicates
DROP TABLE IF EXISTS ingest;
CREATE TABLE ingest
(
id Int,
property_id Int,
created_at DateTime('UTC'), -- Should be preserved
modified_at DateTime('UTC'), -- Should be updated
data Int -- Should be updated
) ENGINE = MergeTree
ORDER BY (property_id, created_at);
-- Destination table without duplicates
DROP TABLE IF EXISTS dedup;
CREATE TABLE dedup
(
id Int,
property_id Int,
created_at_state AggregateFunction(min, DateTime),
modified_at_state AggregateFunction(max, DateTime),
data_state AggregateFunction(argMax, Int, DateTime)
) ENGINE = SummingMergeTree
ORDER BY (property_id, id);
-- Transformation pipeline
DROP VIEW IF EXISTS pipeline;
CREATE MATERIALIZED VIEW pipeline TO dedup
AS SELECT
id,
property_id,
minState(created_at) AS created_at_state,
maxState(modified_at) AS modified_at_state,
argMaxState(data, modified_at) AS data_state
FROM ingest
GROUP BY property_id, id;
-- Insert data with a duplicate
INSERT INTO ingest (id, property_id, created_at, modified_at, data)
VALUES (1, 100, '2022-01-01 08:00:00', '2022-01-01 08:00:00', 2000),
(1, 100, '2022-01-01 08:01:00', '2022-01-01 08:01:00', 3000),
(2, 100, '2022-01-01 08:00:00', '2022-01-01 08:00:00', 4000),
(3, 200, '2022-01-01 08:05:00', '2022-01-01 08:05:00', 5000);
-- Query deduplicated table with merge functions
SELECT id,
property_id,
toDateTime(minMerge(created_at_state), 'UTC') AS created_at,
toDateTime(maxMerge(modified_at_state), 'UTC') AS modified_at,
argMaxMerge(data_state) AS data
FROM dedup
GROUP BY property_id, id
ORDER BY id, property_id;
id
property_id
created_at
modified_at
data
1
100
2022-01-01T08:00Z
2022-01-01T08:01Z
3000
2
100
2022-01-01T08:00Z
2022-01-01T08:00Z
4000
3
200
2022-01-01T08:05Z
2022-01-01T08:05Z
5000