Dgrpah alpha is getting crash after deleting predicate node

Issues Dgraph
1

case : While deleting schema for a node having index type: exact, using ratel ,alpha got crashed & that predicate is not deleted and whenever I am restarting alpha it’s giving me below error.

logs

Raft.Ready took too long to process: Timer Total: 2m14.714s. Breakdown: [{proposals 2m14.713s} {disk 0s} {advance 0s}] Num entries: 0. MustSync: false
Apr 06 08:29:10 de-optisam-dgraph-1 dgraph[5468]: W0406 08:29:10.524827    5478 draft.go:1183] Raft.Ready took too long to process: Timer Total: 823ms. Breakdown: [{proposals 823ms} {disk 0s} {advance 0s}] Num entries: 0. MustSync: false
Apr 06 08:30:01 de-optisam-dgraph-1 dgraph[5468]: fatal error: runtime: out of memory

Here is my system configuration:

version: 20.0.7
ram : 8GB
cpu: 4
my alpha and zero are running on the same server
total records : 200K-300K
2

You had OOM. For sure it isn’t related to that action you mentioned only.

hum, that’s curious. When you kill the instance and bring it back again in general it frees RAM. It should not go OOM again.

Do the following:

  1. Kill the instances and start again.
  2. Leave it for some time, 15 minutes maybe.
  3. Export your DB and upgrade.

We don’t have this version. Did you meant v20.07.0?

3

Hi,

yes my bad, I am using the 20.07.0 version.

I have already done the above 2 steps that you mentioned but in a few min, less than 15 mins alpha again crashed, as it stuck in a loop.

here are more logs, might help.

Apr 06 07:43:23 de-optisam-dgraph-1 dgraph[3956]: W0406 07:43:23.864740    3966 draft.go:1183] Raft.Ready took too long to process: Timer Total: 1.272s. Breakdown: [{proposals 1.272s} {disk 0s} {advance 0s}] Num entries: 0. MustSync: false
Apr 06 07:44:14 de-optisam-dgraph-1 dgraph[3956]: I0406 07:44:14.450691    3966 draft.go:1466] Skipping snapshot at index: 402598. Insufficient discard entries: 1. MinPendingStartTs: 474323

No membership update for 10s. Closing connection to Zero.
Apr 06 07:46:23 de-optisam-dgraph-1 dgraph[3956]: fatal error: runtime: out of memory
Apr 06 07:46:23 de-optisam-dgraph-1 dgraph[3956]: runtime stack:
Apr 06 07:46:23 de-optisam-dgraph-1 dgraph[3956]: runtime.throw(0x195a44b, 0x16)

I didn’t understand the 3rd point,do I need to upgrade my dgrpah version, like this is bug existing in this version (20.07.0).

For the system configuration (as mentioned earlier), do I need to change anything in that?

4

Can you share some profiling? Retrieving Debug Information - Howto (CPU, and RAM)

Also, would be possible to share the p directory? if it’s not sensitive data.

In general, can happened to have a fix in the latest versions. As this is an old one and we had several fixes since then. Upgrading can fix the problem.