Corrupted database

Users Dgraph
1

What version of Dgraph are you using?

v21.03

What is the hardware spec (RAM, OS)?

AWS EC2 m6a.4xlarge, Linux

What happened?

The database was suddenly unavailable. After restart of the whole cluster (docker-compose) I got these logs

graphAlpha_1  | W0817 11:37:36.004553      17 mvcc.go:148] Error Key with size 171798 exceeded 65000 limit. Key:
graphAlpha_1  | 00000000  00 00 00 00 00 00 00 00  00 00 05 74 69 74 6c 65  |...........title|
graphAlpha_1  | 00000010  02 01 67 66 67 67 68 64  66 67 68 64 69 66 67 6a  |..gfgghdfghdifgj|
graphAlpha_1  | 00000020  64 68 66 67 6a 6b 64 68  67 6b 6a 68 66 6a 6b 67  |dhfgjkdhgkjhfjkg|
graphAlpha_1  | 00000030  68 6a 64 6b 66 6b 67 6a  68 64 66 6a 67 68 6a 6b  |hjdkfkgjhdfjghjk|
graphAlpha_1  | 00000040  6b 6a 6b 66 67 68 6b 64  6a 66 67 6b 6a 64 66 67  |kjkfghkdjfgkjdfg|
graphAlpha_1  | 00000050  68 6b 6a 64 66 68 67 6a  6b 64 68 66 67 6a 6b 68  |hkjdfhgjkdhfgjkh|
graphAlpha_1  | 00000060  64 66 6a 67 6b 68 64 6a  66 67 68 6a 6b 64 66 68  |dfjgkhdjfghjkdfh|
...

Continues to print a lot of hex code.

This happened in production. Starting the cluster with a new volume works. We needed to go back to a backup.

We’re trying to find out what the problem is. Any ideas? Thank you.

2

This issue seems related to this fix fix(mutation): validate mutation before applying it by mangalaman93 · Pull Request #8623 · dgraph-io/dgraph · GitHub. Essentially, a mutation has a much larger value and which leads to a much larger index key. This PR disallows such mutations.

1 Like
3

Thanks so much for your response. That makes a lot of sense. We’ll update to v23 asap.

Just to get a better understanding: There was a value (so not the key itself) that exceeded a certain limit? And second: Could seeing the word “title” in the error logs be an indicator that it was the “title” key?

Thanks!

4

First: The value in the data becomes the key for the index which eventually leads to the problem. And for second, that seems like a reasonable guess. Predicates are usually in the beginning of the key.

1 Like