Dgraph server error: Couldn't take snapshot

Users
1

2018/03/13 03:33:59 draft.go:653: Couldn’t take snapshot, txn watermark: [3811], applied watermark: [28390]
Will this error affect data ingestion?

2

Yeah, if snapshot can’t be done for long it will make the server go out of memory very quickly. Are you using the Go client to load the data?

3

yes. and the dgraph image is dgraph/dgraph:v1.0.4

4

Ok, we’d need to replicate this and see why snapshot isn’t happening. What is your general setup like and how do you load this data?

5

the cluster consists of 5 dgraph zeros and 30 servers

dgraph zero --replicas 3 --my=$(hostname -f):5080 --peer dgraph-zero-0.dgraph-zero.default.svc.cluster.local:5080 --idx $idx --rebalance_interval 60m0s

dgraph server --my=$(hostname -f):7090 \
              --memory_mb 6000 \
              --zero dgraph-zero-0.dgraph-zero.default.svc.cluster.local:5080 \
              --idx $idx \
              --port_offset $offset \
              --debugmode \
              --query_edge_limit 18446744073709551615

we load the data via Go client with transaction batch size set to 100,000.

6

I fixed a small issue yesterday. So maybe try with dgraph-io/dgraph:master and see if it still happens. I will spin up a Kubernetes cluster soon and try to load data using dgraph live to see if I can replicate this issue.

7

Had the same problem today with a single 32GB instance

8

using 1.0.4

 server.go:409: Received query: schema {}
2018/03/15 14:59:34 groups.go:316: Asking if I can serve tablet for: _share_hash_
2018/03/15 14:59:37 draft.go:653: Couldn't take snapshot, txn watermark: [1239490], applied watermark: [1872343]
2018/03/15 14:59:41 server.go:409: Received query: {
  me(func: has(username)) @filter(has(pagerank)) {
    count(uid)
  }
}
2018/03/15 15:00:07 draft.go:653: Couldn't take snapshot, txn watermark: [1239490], applied watermark: [1872343]
9

ok now it works…

Writing snapshot at index: 1872369, applied mark: 1872379
2018/03/15 15:00:37 wal.go:118: Writing snapshot to WAL, metadata: {ConfState:{Nodes:[1] XXX_unrecognized:[]} Index:1872369 Term:13 XXX_unrecognized:[]}, len(data): 27
10

The warning can be there temporarily. We will change the logs to mention it as a warning which is logged only in debug mode. It is only an issue if it remains for a long time.

11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.