Mongo-dgraph data synchronization


(jihantoro) #1

Hi, I have list of user and article save in my mongo database, and now i’m trying to use DGraph as database that will serving content for showing in Search, Ontology, Relation, Chart etc in user frontend

but how i can keep data update between 2 database ?

I imagine to create microservice that write into DGraph in near real-time update, thank you


(Michel Conrado) #2

IMHO It would be hard. You would have to create some sort of bridge between Dgraph and MongoDB. To synchronize and avoid duplicates.

It is best to migrate once for all. You can export a complete dump in JSON (containing the relations) and if the relations are very complex, You would have to take more steps. This will depend a lot on your context. Database migration is very complicated indeed.

Worse still is when the idea is to create a link/bridge between two DBs with very different concepts. Not impossible. But hard to maintain tho.

If you still want to use both as you mentioned, maybe create a GraphQL server that writes in both DBs, but there are no warranties doing that. You can do it, but it is much better to adopt a solution only.

Unless your application is not copying (broadcasting) between different DB’s. And its microservices are independent. So you could for sure have multiple DB’s (even using GraphQL - which I think would be very good for microservices) in a microservices context. Which isn’t your case, only a possibility.

Cheers.


(Aman Mangal) #3

As Michel said, it’s a very complex problem to have both databases consistent. Simply because they store data differently, they could have different information (i.e. computed and hence, stored different information). One standard technique to do this is to create a WAL of changes (updates, deletion) first and then stream them to both the databases. You will have to also make sure that each change you apply is idempotent and produces the exact same results every time. At Dgraph, we are working on streaming the bin logs (delta change logs) which you could potentially use as the WAL and send those to mongodb to keep them consistent… I am not sure whether mongodb has similar WAL already available that you could stream to Dgraph.


(jihantoro) #4

umm yeah i forgot to tell you, the mongo database is CREATE & READ only database