Support logical replication / change data capture (CDC) with Kafka

Moved from GitHub dgraph/2122

Posted by fritzblue:

The following workflow is becoming more and more common in streaming architectures:

If Dgraph is to be used as a primary data store in a production system, it must support logical replication / change data capture.

Is this currently possible? If not, is it planned for the future?

pawanrawal commented :

It’s not possible right now but we can do something like this in the future. Some users have asked for subscriptions. This would be a similar feature I suppose.

manishrjain commented :

We could potentially support this at Badger level first, and then expose it via Dgraph.

https://github.com/dgraph-io/badger/issues/684

campoy commented :

After chatting with @gitlw, it seems like this could be implemented relatively easily on top on our current plans for datacenter to datacenter replication, which already uses Kafka.

Adding it to our 1.X plans!

fritzblue commented :

For an example of a distributed DB that’s doing this in what I would argue is the Right Way™, see CockroachDB Redirecting…

Is there any progress on this?

Hi @lemonwx, Our team has prioritized CDC and Kafka integration in Q1 2021. This should be available in our next major release currently scheduled for March 2021.

1 Like

Hey guys, please follow this thread for further discussion around CDC Dgraph Enterprise: Change Data Capture (CDC).

Thanks.