Subscriber never gets the entire queue - message-queue

Question

we are working with PubSub to integrate several systems with each other. Some systems may push data to PubSub as JSON, while others can pull that data and use it. (Note: we have to pull from PubSub instead of push to the app due to other restrictions with the receiving application) Every pulling application gets it's own subscriber to each topic.
I have noticed that the PubSub pull does not get all data currently in the queue if it is triggered too frequently. The problem originally occurred in a Java Spring App with the respective library, but the gcloud command in the cloud console exhibits the same behaviour, so I am just going to use that example. I removed the ack-ids and borders to make it fit this window. Note how I don't use the '--auto-ack' flag, so the queue should stay the same, no other system is pulling from that subscriber.
First pull (complete content):
max_binnewies#cloudshell:~ $ gcloud pubsub subscriptions pull testSubscriber --limit=100
│ DATA │ MESSAGE_ID │
│ 4 - FOUR │ 189640873208084 │
│ 5 - FIVE │ 189636274179799 │
│ 2 - TWO │ 189638666587304 │
│ 3 - THREE │ 189627470480903 │
│ 1 - ONE │ 189639207684195 │
Second pull (only one):
max_binnewies#cloudshell:~ $ gcloud pubsub subscriptions pull testSubscriber --limit=100
│ DATA │ MESSAGE_ID │
│ 1 - ONE │ 189639207684195 │
Third pull (two different ones):
max_binnewies#cloudshell:~ $ gcloud pubsub subscriptions pull testSubscriber --limit=100
│ DATA │ MESSAGE_ID │
│ 4 - FOUR │ 189640873208084 │
│ 5 - FIVE │ 189636274179799 │
Fourth pull (first one again):
max_binnewies#cloudshell:~ $ gcloud pubsub subscriptions pull testSubscriber --limit=100
│ DATA │ MESSAGE_ID │
│ 1 - ONE │ 189639207684195 │
That behaviour is confusing to me. Is that normal PubSub behaviour or am I doing something wrong? The only thing I found is this link where it says that PubSub uses load balancing for the pull method:
https://cloud.google.com/pubsub/docs/subscriber
So I think that the subscriber thinks multiple clients are subscribing to it and spreads out the data if calls come in too quickly. Is that correct? What exactly is happening here?
If I wait a little while, I get more data again, but I never seem to be getting everything even if I wait five minutes... It is very confusing.
Can that cause a problem for the consuming application? How do I make sure all the data arrives at the receiving application even if it pulls very frequently? Is there a way to turn this off?

Answer 1

There are a couple of things the result in you not receiving all messages every time:
With pull requests, there is no guarantee that all messages will be returned in a particular request, even if there are fewer messages available than max messages. This is because Pub/Sub tries to balance returning more messages with minimizing end-to-end latency.
Messages have an ack deadline, which is specified on subscription creation time (and defaults to 10 seconds). What this means is that when you pull messages and don't ack or nack them, they will not be redelivered for the period of the ack deadline, basically giving the process that pulled the messages a lease on them. If you want messages to be redelivered immediately, you'd need to nack them if you are using the Java client library (the preferred way to interact with Cloud Pub/Sub) or you need to send a ModifyAckDeadline request with the ack_deadline_seconds set to 0.