Dgraph Enterprise: Change Data Capture (CDC)

Users Dgraph
1

What we will subscribe to?

The leader of each group will be streaming out CDC updates. Later on, once we have namespaces(from multi tenancy), we will make it listen to updates accordingly.

We have made raft logs readable for CDC. Hence each raft entry would be read and sent as CDC event if necessary. This ensures all the guarantees we needed for CDC.

Solution:

  1. CDC will be managed via config flags.
  2. CDC job will read the raft logs and decide which events to send based on the type of operation.
  3. Initially only Kafka will be provided as a sink. All alpha nodes are required to specify with the Kafka sink config, for CDC to work efficiently.
  4. Events will be sent out via leader node only.
  5. Events will be sent to Kafka topics named as dgraph-cdc. Later with multi-tenancy, events will be distributed based on namespaces over the partitions.

Handling Drop operations:

There are 4 kinds of drop operations:

  1. Drop All
  2. Drop Data
  3. Drop Attribute
  4. Drop Type

Drop Event will be sent for each of these cases.

Sample a DROP ALL event would look like this:

{"meta":{"commit_ts":13},"type":"drop","event":{"operation":"all"}}

Mutation Events

There will be 2 kinds of events; Set and Delete that we will be emitting. Other DBs do support another operation Update that would be costly for us to support because of extra lookup needed for that. We don’t intend to support that.

Raft entries will have information about Predicate, EntityId, Value and DataType. Information extracted from these entries would go into Set and Delete field of JSON below.

A Set mutation event updating counter.val to 10 would look like this:

{"meta":{"commit_ts":29},"type":"mutation","event":{"operation":"set","uid":3,"attr":"counter.val","value":10,"value_type":"int"}}

Similarly, a sample Delete mutation event of removing all values for predicate Author.name for a node with uid=7 would look like this:

{"meta":{"commit_ts":44},"type":"mutation","event":{"operation":"del","uid":7,"attr":"Author.name","value":"_STAR_ALL","value_type":"default"}}

Integration With Sink Clients

Initially, only two sink clients will be available i.e. Kafka and File.

  • Kafka will adhere to the TLS encryption as well as sasl authentication. All the CDC events will be generated over the default topic and events will be distributed based on namespace over the partitions.
  • File-based sinks are just dumping events into a file. This is generally useful in the case of testing.

Important things to note

  1. If the sink is failing or down, clients will not lose events. But it will lead to an increase in the raft files because CDC jobs read the raft logs and since no events are getting published, raft files are not getting cleared. Hence it becomes necessary to manage the cluster properly in case of failure events.
  2. In case of node crashes or leadership changes there might be duplicated events but no loss of events.
  3. In the case of live loader, there will be events for each predicate.
  4. In the case of an old cluster, CDC events will be sent from the last raft index available.
  5. In the case of bulk loader, there will be no events.
  6. Value for password fields will not be sent out in CDC events.

Limitations

  1. CDC is not supported in case of ludicrous mode. We are working on fixing this. We will update the timelines for that later.
  2. Schema updates will not available over CDC at this moment. We are working on fixing this. We will update the timelines for that later.

Important: This is an enterprise feature.

3 Likes