Is it possible for Multiple Consumers to consume same message on a Pulsar Topic? - language-agnostic

Let's say for that one has a Pulsar Producer for a persistent topic topic1 (namespace and tenant are not relevant for the question).
And let's say we have multiple consumers for the same topic (topic1) with different subscription names.
Is it possible to configure the consumers to receive same message? So for example if message msg1 is sent to the topic both consumer1 and consumer2 receive this message?
Both consumers and producer are written in Java but programming language is not important.
Clarification
The current behavior observed with multiple subscriptions on the same topic with multiple subscriptions is that each of the subscribers do not receive all messages that have been published to the topic. I need to receive all messages from the topic.

Yes, you just need to use multiple subscriptions on the topic.
I wrote a blog post on this exact topic: "Subscriptions: Multiple Groups of Consumers on a Pulsar Topic".

Yes. It is possible for multiple consumers to receive the same copy of messages from a topic. Subscription determines how messages are delivered to consumers. What you need is each consumer has own Exclusive subscription. Here is the code example in Java.
String topicName = "your-tenant/namespace/topic1"
Consumer consumer1 = client.newConsumer()
.topic(topicName)
.subscriptionName("my-subscription1")
.subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
.subscriptionType(SubscriptionType.Exclusive)
.subscribe();
// create another consumer
Consumer consumer2 = client.newConsumer()
.topic(topicName)
.subscriptionName("my-subscription2")
.subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
.subscriptionType(SubscriptionType.Exclusive)
.subscribe();
// two consumers receive messages in alternate from the same topic
while (true) {
Message msgFromConsumer1 = consumer1.receive();
Message msgFromConsumer2 = consumer2.receive();
}

Related

Openshift AMQ6 - message order - queue

I use AMQ 6 (ActiveMQ) on OpenShift, and I use a queue with re-delivery with exponentialBackoff (set in connection query params).
When I have one consumer and two messages and the first message gets processed by my single consumer and does NOT get an ACK...
Will the broker deliver the 2nd message to the single consumer?
Or will the broker wait for the re-delivery to preserve message order.
This documentation states:
...Typically a consumer handles redelivery so that it can maintain message order while a message appears as inflight on the broker. ...
I don't want to have my consumer wait for re-delivery. It should consume other messages. Can I do this without multiple consumers? If so, how?
Note: In my connection query params I don't have the ActiveMQ exclusive consumer set.
I have read the Connection Configuration URI docs, but jms.nonBlockingRedelivery isn't mentioned there.
Can the resource adapter use it by query param?
If you set jms.nonBlockingRedelivery=true on your client's connection URL then messages will be delivered to your consumer while others are in the process of redelivery. This is false by default.

Google Pubsub - Receive delivery attempt for push subscription

I have a Google cloud function that is being triggered by a Pubsub push subscription.
I wish to know the current delivery attempt of the given message.
In pull subscription it works by setting dead letter topic, however I am not able to get the delivery attempt in a push subscription message attributes. Tried to configure a dead letter topic and delivery_attempt attribute is not in message attributes.
Is there a way to get the delivery attempt parameter in a push subscription?
For push subscriptions, use deliveryAttempt, not delivery_attempt. The documentation calls out this here:
When Pub/Sub forwards undeliverable messages from a push subscription, every message you receive from the subscription includes the deliveryAttempt field.
I am not sure it is possible to get that data explicitly... but I can suggest an idea of a workaround for consideration.
Every time a cloud function is invoked, you can (in your code) create/update a firestore document with some unique id (either a message event id, or some unique business related identifier). That document may have attributes, one of which - is the time and/or the attempt number.
in java you can use a static method of Subscriber class.
for a PubsubMessage message;
com.google.cloud.pubsub.v1.Subscriber.getDeliveryAttempt(message);
As de documentation sais Returns the delivery attempt count for a received PubsubMessage.
Integer maxAttemps = 5;
private void killMessage(Event<String> event, Exception en,
BasicAcknowledgeablePubsubMessage acknowledgeable) {
Integer attemps = Subscriber.getDeliveryAttempt(acknowledgeable.getPubsubMessage());
if(attemps.intValue() < maxAttemps) {
acknowledgeable.nack();
return;
}
sendToErrorQueue(event, en, acknowledgeable);
}
https://googleapis.dev/java/google-cloud-pubsub/latest/com/google/cloud/pubsub/v1/Subscriber.html

NATS - just one subscriber to take action for published event in a microservicearchitecture

I'm new to NATS and have read all the examples for:
https://nats.io/documentation/concepts/nats-messaging/
I'm in Microservciearchitecture where in microservice-Y (MSY) need to store some information published from other microservice-X (MSX) I have 2-10 instances of MSY so when changes are made in MSX and MSX-instance publishes event I want that only 1 instance of MSY should save information so not all of them save the same data.
I have read Request-Repy:
https://nats.io/documentation/concepts/nats-req-rep/
but there seems that all of instances receives message (and will handle it) even if it is point-to-point and reply is handled just for the one instance that is quickest to reply
Is this correct or have I missunderstood example?
If I only need that 1 instance of MSY should handle given message (store data in db) what can I do to acheve this?
Use queue groups. If you have multiple subscriptions on the same subject with the same queue group, only one of the members of the group will receive the message.
Check this out: https://nats.io/documentation/concepts/nats-queueing/

Sending messages to unknown users in XMPP MultiUserChat with OMEMO encryption

I'm trying to build a MUC platform with OMEMO encryption to be used on an Android app. I'm using Ejabberd (v17.11) and Smack library (v4.2.1).
MUC rooms are persistent and they allow_subscription for Muc/Sub support, for offline messages.
When a client starts a new room as 'owner' and adds 'member's from his roster; all the clients join the room and everything seem fine.
The owner can send messages to the group and it is delivered to every member.
But when a group member tries to send a message, if he doesn't 'know' all the members of the group (if they are not in his roster) he cannot send message!
I found out that, when sending OMEMO message to the group, it is necessary to encrypt the message separately for every member and this necessitates getting every members' devicelist!
On this step:
mOmemoManager.encrypt(muc, msgBody);
It tries to encrypt for all recipients but fetching an unknown user's device list causes crash.
SENT:
<iq to='unknownuser#server.com' id='141' type='get'><query xmlns='http://jabber.org/protocol/disco#info' node='eu.siacs.conversations.axolotl.devicelist'></query></iq>
RECV:
<iq xml:lang='en' to='myuser#server.com/mobile' from='unknownuser#server.com' type='error' id='141'><query node='eu.siacs.conversations.axolotl.devicelist' xmlns='http://jabber.org/protocol/disco#info'/><error code='407' type='auth'><subscription-required xmlns='urn:ietf:params:xml:ns:xmpp-stanzas'/><text xml:lang='en' xmlns='urn:ietf:params:xml:ns:xmpp-stanzas'>Not subscribed</text></error></iq>
Smack's error:
Could not fetch device list of unknownuser#server.com: .....
XMPPError: subscription-required - auth
So, how I can solve this?
In chat groups every user don't have to meet each other. Should I add everyone to everyone's roster when they come together on a group?
Or is this something about room affiliations or roles?
Or something about Ejabberd configuration?
[{title,<<"groupName">>},
{allow_query_users,true},
{allow_private_messages,true},
{allow_private_messages_from_visitors,anyone},
{allow_visitor_status,true},
{allow_visitor_nickchange,true},
{public,false},
{public_list,false},
{persistent,true},
{moderated,true},
{members_by_default,true},
{members_only,true},
{allow_user_invites,true},
{anonymous,false},
{logging,false},
{allow_voice_requests,true},
{allow_subscription,true},
{mam,true},
{presence_broadcast,[moderator,participant,visitor]},
{voice_request_min_interval,1800},
{vcard,<<>>},
{captcha_whitelist,[]},
{affiliations,[{{<<"user1">>,<<"server.com">>,<<>>},{member,<<>>}}, {{<<"user2">>,<<"server.com">>,<<>>},{owner,<<>>}}, {{<<"user3">>,<<"server.com">>,<<>>},{member,<<>>}}]},
{subject,[]},
{subject_author,<<>>}]
OMEMO requires the sender to be able to read the recipients OMEMO pubsub node. This is necessary to fetch the recipients preKey bundle to build a session.
Modern implementations configure the OMEMO pubsub node to be public - there is also an ejabberd config to force this for legacy clients.
Most client implementations of OMEMO therefore restrict the encryption feature to be available in private group chats only. In this scenario all contacts are in each others rosters and have access to each others OMEMO nodes. Encrypting publicly available group chats doesn't make sense anyways as an attacker could simply join the chat in order to read messages.
If you want to configure your OMEMO pubsub nodes to be public from within your client, take a look at how smack-openpgp does change to access model.

Is my logic for a .tell ( notification ) function on my IRC bot sound?

I have a bot in node.js and I'm wondering what logic I should use for a .tell function. The purpose of the function will be to give a message to users who are offline. So, if john was offline on freenode.net/#foo and I typed .tell john your mother was great last night, I want the bot to store this message and display it whenever john joins the channel.
If john is already in the channel, I want the bot to immediately just display the message.
I am already using mongodb as a database to store information, so I can probably store all my messages in there. Here's the logic I had in mind so far:
1. After user A types .tell john your mother was great, a listener will pickup the pattern /\.tell ([\w-]+) (.*)/ and store a variable for the message ( var msg, the user who posted the message ( var from ), and the name of the user to be messaged ( var to ). I will also have a global variable signifying the channel name and network of the server.
2. I'll store a row in mongodb such as:
to from message network channel delivered time
--------------------------------------------------------------------------------
john meder your mother was great freenode.net #foobar 0 (TS)
last night
3. Setup a listener for anyone joining a channel since the bot can live in multiple channels under one network, and detect if there any messages that are not delivered and if there are, attempt to match the to and network and channel to the stored messages, and if row(s) are found, display them.
This sounds ok, but can anyone review it and maybe offer advice? I'll be forced to have a listener listen for anyone entering any channel, right? There's no alternative to that?
This seems all fine. I don't see how you could not have a listener.
One thing I'd add is maybe a hash in the bot to keep track of who has pending messages. This way you won't be querying MongoDB every time someone joins a channel.
I.e.
messages = {};
messages['freenode.net'] = {};
messages['freenode.net']['john'] = 1;
Might be overkill, but if your monitoring a channel that has many JOINs, you might find yourself querying MongoDB a lot.
My two cents.
Christian