How to subscribe to dgraph changes?

There are 3 related questions inline below:

We are trying to listen to changes to dgraph data and then do something with that data (e.g. replicate it elsewhere in a different serialization format). We were told that some people have tried listening to the badger files. Question 1: Is this a good approach?

If we want to copy 100% of dgraph data, no more, and no less then I imagine we’d want to deploy exactly 1 listener per shard, regardless of number of replicas. Question 2: Is that right?

To accomplish the above, we tried using the go lib github.com/dgraph-io/badger/v2 , but we couldn’t get Subscribe to work. Our subscription callback is never called and this panic is never hit when we add to or remove from the database: https://gist.github.com/chrishiestand/8bdcfed48d11c581629ad74f91dd6b82#file-dgraph-badger-subscription-poc-go-L27
Question 3: Any idea what we’re doing wrong?

program output:

2020/03/03 14:03:43 starting
badger 2020/03/03 14:03:43 INFO: All 0 tables opened in 0s
badger 2020/03/03 14:03:43 INFO: Replaying file id: 0 at offset: 0
badger 2020/03/03 14:03:43 INFO: Replay took: 30.650425ms
badger 2020/03/03 14:03:43 DEBUG: Value log discard stats empty

So the database is opened, but the subscription callback never gets called

1 Like

Hey @chrishiestand,

Our subscription callback is never called and this panic is never hit when we add to or remove from the database: https://gist.github.com/chrishiestand/8bdcfed48d11c581629ad74f91dd6b82#file-dgraph-badger-subscription-poc-go-L27
Question 3: Any idea what we’re doing wrong?

Badger subscription works only for new updates. I made a few changes to your script and it seems to work https://gist.github.com/jarifibrahim/cd8f64de07d5a690cc3bbab66777b57d

1 Like

Thanks so much for the response! I can verify your approach, which is something! I knew that subscription only worked for new updates - those were happening out of band (which is why in my gist there was a channel blocking main() from returning). But I would like to listen to all changes to the database. The current v2 documentation says that this is possible:

You can use an empty prefix to monitor all changes to the DB

https://pkg.go.dev/github.com/dgraph-io/badger/v2?tab=doc#DB.Subscribe

Is that wrong? If it’s right, how does one specify an empty prefix? I tried no argument, nil, and []byte(nil) and more, but nothing seemed to work.

@chrishiestand there was an issue with the Subscribe API and because of it empty prefixes were not working. I’ve raised https://github.com/dgraph-io/badger/pull/1246 to fix this.

1 Like

Thanks so much! Looking forward to this fix getting merged.