Dgraph slows down progressively

Users Dgraph
1

Hello,

I just realised one of my DB regular cleaning operation was taking more and more time recently.

It consists of removing approx. 1700 edges on a predicate having @count and @reverse indexes, then re-adding them.

Recently it took 1 minute each which seemed very slow to me.

I tried to export my base, stop dgraph, remove DB files, start dgraph, import my base and finally try again my operations, and they took 5 et 9 seconds!

What is happening in Dgraph database to get progressively slower?

Dgraph version: 1.2.1, CPU: i7-7700HQ CPU @ 2.80GHz, RAM: 16Gb DDR3

1 Like
2

I added around 100k edges and I tried removing and adding 1700 edges on a predicate with @reverse and @count indexes.
It’s taking around 14sec. Could you try running this at your end and let me know how much time it takes?

func TestCountIndexThroughput(t *testing.T) {
	total := 100000
	numUIDs := uint64(total)

	dg, err := testutil.DgraphClientWithGroot(testutil.SockAddr)
	if err != nil {
		t.Fatalf("Error while getting a dgraph client: %v", err)
	}

	if err := testutil.AssignUids(uint64(total * 10)); err != nil {
		t.Fatalf("error in assigning UIDs :: %v", err)
	}

	rdfs := make([]string, 0)
	for uid := 1; uid < int(numUIDs); uid++ {
		rdf := fmt.Sprintf("<%v> <friend> <%v> .\n", uid, uid+1)
		if err := testutil.RetryMutation(dg, &api.Mutation{
			CommitNow: false,
			SetNquads: []byte(rdf),
		}); err != nil {
			t.Fatalf("error in mutation :: %v", err)
		}
		rdfs = append(rdfs, rdf)
	}

	start := time.Now()
	for idx, rdf := range rdfs {
		if err := testutil.RetryMutation(dg, &api.Mutation{
			CommitNow: false,
			DelNquads: []byte(rdf),
		}); err != nil {
			t.Fatalf("error in mutation :: %v", err)
		}
		if idx > 1700 {
			break
		}
	}

	for idx, rdf := range rdfs {
		if err := testutil.RetryMutation(dg, &api.Mutation{
			CommitNow: false,
			SetNquads: []byte(rdf),
		}); err != nil {
			t.Fatalf("error in mutation :: %v", err)
		}
		if idx > 1700 {
			break
		}
	}
	elapsed := time.Since(start)
	log.Printf("Time: %s", elapsed)
}
3

Hello, where do I find your testutil package, and how should I use it?

4

I wrote this code in dgraph/query/mutation_test.go but you can use the dgo(GitHub - dgraph-io/dgo: Official Dgraph Go client) client as well.

5

Hello,
I don’t have the slowdown anymore. When I reset my test database I now use a bulk import, instead of a clear alter & live import. It’s like if the clear op doesn’t clear everything. I’m also on 1.2.2 now.
Thank you anyway.