Is it possible to use badger as PubSub. The requirements are:
- Messages are published to a topic
- Subcribe to a topic. Can resume susbcription from a given topic message offset
We are thinking of creating keys with prefix as topic name and the key suffix as incrementing offsets(integers). Then we can use Badger Subscribe API on a topic key prefix to listen to all new changes on a topic prefix.
We also need ability to start pubsub subscription from an existing topic offset. For this we are thinking of using iterator to resume from an existing offset. So, when a new pubsub subscription starts with a given offset, we can call db.Subscribe() to see all new changes followed by tx.iterate() to play all existing changes from the given offset. We have a way to effectively de-duplicate keys from the two streams(iterator and subscribe).
Few questions:
- How many subscriptions and iterators can Badger support at a time? We are looking at around 5K total subscriptions and 5K iterators(the iterators will be only used and close during pubsub susbcription resume from offset) which can have overlapping prefixes.
- With 5K subscriptions concurrently running, can badger support 5-10K/sec key inserts with up to 500 KB key value pair size? Key size would be around 128 bytes.
- Does badger subscription provide replaying the key-values or calling the callback in same order as the changes occur. Ordering guarantees and no missing event is something we really care about.
Thanks,
Abhishek