How to clean up w directory logs

Issues Dgraph
,
1

Moved from GitHub dgraph/5266

Posted by jameswangcnbj:

i have 3 node in 3 machine, p directory have 4GB data,but w directory hava 30GB log,how can i clean up w directory,if i del the data of w directory,the cluster will crash down ?how can i clean up it? thanks

2

JimWen commented :

Same problem.

What version of Dgraph are you using?

Dgraph version : v1.2.2
Dgraph SHA-256 : 86e27921d3a53dbc4faba78e9c7d7879ce8bafd6006c9fd4b77220535b8ae1c2
Commit SHA-1 : 21cc8936
Commit timestamp : 2020-03-19 12:13:11 -0700
Branch : HEAD
Go version : go1.13.5

Have you tried reproducing the issue with the latest release?

Only with latest 1.X version

What is the hardware spec (RAM, OS)?

128G mem & 1.8T SSD

Steps to reproduce the issue (command/config used to run Dgraph).

Move data with “/moveTablet?tablet=name&group=2”, and the w directory of group 2 master alpha node becomes very big.

Expected behaviour and actual result.

After restarting the alpha node, cluster is block and the log is as followings. The cluster becomes avariable after serveral hours’s replay and is block again if I restart the alpha node.

W0422 08:54:30.342161 2721 draft.go:1014] Raft.Ready took too long to process: Timer Total: 310ms. Breakdown: [{proposals 310ms} {disk 0s} {advance 0s}] Num entries: 0. MustSync: false
W0422 08:54:31.479896 2721 draft.go:1014] Raft.Ready took too long to process: Timer Total: 221ms. Breakdown: [{proposals 221ms} {disk 0s} {advance 0s}] Num entries: 0. MustSync: false
W0422 08:54:40.714056 2721 draft.go:1014] Raft.Ready took too long to process: Timer Total: 582ms. Breakdown: [{proposals 582ms} {disk 0s} {advance 0s}] Num entries: 0. MustSync: false
W0422 08:54:41.280721 2721 draft.go:1014] Raft.Ready took too long to process: Timer Total: 207ms. Breakdown: [{proposals 207ms} {disk 0s} {advance 0s}] Num entries: 0. MustSync: false
E0422 08:54:53.147889 2721 groups.go:969] While proposing delta with MaxAssigned: 146060036 and num txns: 487052. Error=Server overloaded with pending proposals. Please retry later. Retrying…
E0422 08:55:22.084565 2721 groups.go:969] While proposing delta with MaxAssigned: 146060036 and num txns: 487052. Error=Server overloaded with pending proposals. Please retry later. Retrying…
E0422 08:55:51.648740 2721 groups.go:969] While proposing delta with MaxAssigned: 146060036 and num txns: 487052. Error=Server overloaded with pending proposals. Please retry later. Retrying…
E0422 08:56:23.024916 2721 groups.go:969] While proposing delta with MaxAssigned: 146060036 and num txns: 487052. Error=Server overloaded with pending proposals. Please retry later. Retrying…

Wait for your help, thank you!!!

3

jarifibrahim commented :

The value log should be cleared up automatically by the GC. We run GC for w and p directory periodically dgraph/server_state.go at 21cc8936de30df433f6ccf01f8eaf1415fc0b391 · dgraph-io/dgraph · GitHub

One possibly could be that there’s not enough stale data for GC to reclaim space. The badger GC (which Dgraph uses) performs sampling on the file to determine if the file has enough stale data.

4

ashish-goswami commented :

@jameswangcnbj also please run badger info(badger/info.go at master · dgraph-io/badger · GitHub) command on your w directory.

5

OmarAyo commented :

Hi @jameswangcnbj,

Did you have the chance to read the previous messages ? Did you find them useful ?

Best,