ClickHouse - Merge similar entries into a new one - duplicates

Question

This is my current (simplified) model in ClickHouse:
Credential table
user: String
domain: String
password: String
leak: UInt64
The field leak is a reference to the id of a leak.
Leak table
id: UInt64
name: String
desc: String
date: String
I'm using the MergeTree engine for Credential and Log for Leak.
Sometimes, two sames credentials are inserted, so I have for example:
| user | domain | password | 0 |
| user | domain | password | 1 |
I would like to turn this into:
| user | domain | password | [0, 1] |
I've read about ReplacingMergeTree but I did not find another way to specify some rules of rewriting/deleting than the version parameter.
My problem is I have billions of entries and it sounds crazy to process the data before insertion. Even if ClickHouse's response times are incredibly low, it seems ridiculous to check if every new credential I'm trying to insert is already in there, and if it is, update its data, right ?
I'm trying to reduce disk storage as much as possible, but it looks hard to balance it with keeping these incredible response times. I'm listening to your ideas.

Answer 1

There are number of ways to deal with this.
You can create a Materialized Table along with your actual table.
You can use ReplacingMergeTree.
You can mutate(ALTER UPDATE) existing data.
Use your table as is but query it using groupArray to get the result you want.
Here I demonstrate 4th solution. You just insert your data in MergeTree as you always do. But when you query it, you use groupArray to roll the desired column to be an array based on other columns.
Let's say your table is something like this:
SELECT *
FROM mem
┌─a─────┬─b───────┬─c─────────┬─d─┐
│ user1 │ domain1 │ password1 │ 0 │
│ user1 │ domain1 │ password1 │ 1 │
│ user2 │ domain2 │ password2 │ 0 │
│ user2 │ domain2 │ password2 │ 2 │
└───────┴─────────┴───────────┴───┘
You use groupArray to solve your problem:
SELECT a, b, c, groupArray(d)
FROM mem
GROUP BY a, b, c
┌─a─────┬─b───────┬─c─────────┬─groupArray(d)─┐
│ user2 │ domain2 │ password2 │ [0,2] │
│ user1 │ domain1 │ password1 │ [0,1] │
└───────┴─────────┴───────────┴───────────────┘