How can I consume message from kafka in order? - mysql

Question

Background
A producer produces some data and send to Kafka in order, like:
{uuid: 123 status: 1}
{uuid: 123 status: 3}
status 1 means begin
status 3 means succeed
I use sarama.NewConsumerGroup(xx, xx, config).Consume(xx, xx, myhandler) to consume with the code:
func (h myhandler) ConsumeClaim(sess sarama.ConsumerGroupSession, claim sarama.ConsumerGroupClaim) error {
for msg := range claim.Messages() {
key := fmt.Sprintf("%q-%d-%d", msg.Topic, msg.Partition, msg.Offset)
_, err := rdb.RedisClient.Get(h.ctx, key).Result()
if err == redis.Nil {
msgQueue <- msg.Value
sess.MarkMessage(msg, "")
rdb.RedisClient.Set(h.ctx, key, none, 12*time.Hour)
} else if err != nil {
log.Errorln("RedisClient get key error : ", err)
return err
} else {
continue
}
}
return nil
}
msgQueue := make(chan interface{}, 1000)
And then I decode the value in msgQueue to a struct and insert a record into mysql.
Question
Normally, final data status is '3', but I find that sometimes it is '1'
And I find the message order in channel msgQueue is not fixed.
So how can I ensure the final status of data is 3 ?
How to fix
I have provided a method that is not good enough to see how it can be optimized.
conn := &gorm.DB{}
data := &Log{}
if data.Status != 1 {
conn = conn.Clauses(clause.OnConflict{
Columns: []clause.Column{{Name: "uuid"}},
DoUpdates: clause.AssignmentColumns([]string{"status"}),
})
}
conn.Create(data)
return conn.Error
And mysql has a unique constraint index for uuid.
When the data order is {uuid: 123 status: 1},
{uuid: 123 status: 3}, It's right.
When the data order is {uuid: 123 status: 3},
{uuid: 123 status: 1}, the final status is also right, but it will return error Error 1062: Duplicate entry '123' for key 'unique_index_uuid'.
It's not beautiful. So how can I optimize or are there other ways to do it?

Answer 1

That depends on the topic partitions. Kafka does not provide ordering guarantees within a topic, only within a partition.
In other words, if you sent a message A, then message B to partition 0, then the order will be that: first A, then B. But if they end up on different partitions it can happen that B is written to its partitions, before A is written to its.
Here's a quote from Confluent's web site:
Kafka only provides a total order over records within a partition, not between different partitions in a topic. Per-partition ordering combined with the ability to partition data by key is sufficient for most applications. However, if you require a total order over records this can be achieved with a topic that has only one partition, though this will mean only one consumer process per consumer group.
Link