Dgraph server error: Couldn't take snapshot

2018/03/13 03:33:59 draft.go:653: Couldn’t take snapshot, txn watermark: [3811], applied watermark: [28390]
Will this error affect data ingestion?

Yeah, if snapshot can’t be done for long it will make the server go out of memory very quickly. Are you using the Go client to load the data?

yes. and the dgraph image is dgraph/dgraph:v1.0.4

Ok, we’d need to replicate this and see why snapshot isn’t happening. What is your general setup like and how do you load this data?

the cluster consists of 5 dgraph zeros and 30 servers

dgraph zero --replicas 3 --my=$(hostname -f):5080 --peer dgraph-zero-0.dgraph-zero.default.svc.cluster.local:5080 --idx $idx --rebalance_interval 60m0s
dgraph server --my=$(hostname -f):7090 \
              --memory_mb 6000 \
              --zero dgraph-zero-0.dgraph-zero.default.svc.cluster.local:5080 \
              --idx $idx \
              --port_offset $offset \
              --debugmode \
              --query_edge_limit 18446744073709551615

we load the data via Go client with transaction batch size set to 100,000.

I fixed a small issue yesterday. So maybe try with dgraph-io/dgraph:master and see if it still happens. I will spin up a Kubernetes cluster soon and try to load data using dgraph live to see if I can replicate this issue.

Had the same problem today with a single 32GB instance

using 1.0.4

 server.go:409: Received query: schema {}
2018/03/15 14:59:34 groups.go:316: Asking if I can serve tablet for: _share_hash_
2018/03/15 14:59:37 draft.go:653: Couldn't take snapshot, txn watermark: [1239490], applied watermark: [1872343]
2018/03/15 14:59:41 server.go:409: Received query: {
  me(func: has(username)) @filter(has(pagerank)) {
2018/03/15 15:00:07 draft.go:653: Couldn't take snapshot, txn watermark: [1239490], applied watermark: [1872343]

ok now it works…

Writing snapshot at index: 1872369, applied mark: 1872379
2018/03/15 15:00:37 wal.go:118: Writing snapshot to WAL, metadata: {ConfState:{Nodes:[1] XXX_unrecognized:[]} Index:1872369 Term:13 XXX_unrecognized:[]}, len(data): 27

The warning can be there temporarily. We will change the logs to mention it as a warning which is logged only in debug mode. It is only an issue if it remains for a long time.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.