What is best way to batch mutate and avoid node duplication?

Users
1

Hi, I am new to dgraph database. ver 1.0.4. I do love it.
Sorry about my language since not native speaker.

I have many relations stored in MongoDB (1billion docs in 100collections), like

a->b
c->b
a->d
c->e
...

Both source node and target node has its own unique id, and the edge between two unique id nodes is unique.
I want to push these relations into Dgraph.
After read some documents and posts, I realized that (maybe wrong):

  • I can’t insert node with customId by go client
  • I can insert many nodes in one transaction by same blank node name and transaction will deal with node duplication . But if nodes are too many, I have to use multiple transactions.
  • I have to query before write to do upsert operation.

I don’t know what is best way to do this job in efficiently. If I query before write, should I use a transaction do mutate one doc by one doc, or should I use a redis like cache to accelerate it? and when nodes gets more and more, will query operation in Dgraph keep in O(1) time?

2

Since you are doing an upsert, you would have to query first and then mutate. You can batch these mutates and don’t have to do them one by one.

Yeah, query operation would still be O(1) since it uses indexing which makes it fast.

3

OK, the query should be outside or inside of mutate transaction? Do I have to query one by one or maybe I can batch query too? Should I process like this:

-> create Data object 
   -> get distinct [node orignal_id] 
         -> Query for [uid] by each node orignal id 
         -> ...(rest query to get uid if exists)
            -> set [Uid] to orignal [data object]
               -> create a Txn for mutate
                    -> Mutate(obj)
                      -> ...(mutate to insert rest obj)
                        -> Txn.Commit()
4

@pawan I can’t do batch in single json, right? If I use object array without Uid and Marshal it, dgraph will generate uid for every nodes. What should I do? Is it possible to set blank identifier for data object in go?

5

Oh, solved. uid = "_:XX". Having a code example for this will be better.

6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.