V1.0.7-rc4 - Dgraph server stuck on acting to abort

Users Dgraph
1

Using k8s HA config setup. Dgraph recovered on deleting that dgraph server pod.

2018/08/07 03:48:52 draft.go:411: Creating snapshot at index: 56869. ReadTs: 63563.
2018/08/07 03:54:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 03:54:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 03:55:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 03:55:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 03:56:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 03:56:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 03:57:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 03:57:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 03:58:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 03:58:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 03:59:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 03:59:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 04:00:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 04:00:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 04:01:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 04:01:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 04:02:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 04:02:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 04:03:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 04:03:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 04:04:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 04:04:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 04:05:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 04:05:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 04:06:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 04:06:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 04:07:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 04:07:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 04:08:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 04:08:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 04:09:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 04:09:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 04:10:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 04:10:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 04:11:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 04:11:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 04:12:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 04:12:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 04:13:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 04:13:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 04:14:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 04:14:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 04:15:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 04:15:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 04:16:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 04:16:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 04:17:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 04:17:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 04:18:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 04:18:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 04:19:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 04:19:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 04:20:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 04:20:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 04:21:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 04:21:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 04:22:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 04:22:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 04:23:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 04:23:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 04:24:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 04:24:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 04:25:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 04:25:52 draft.go:750: TryAbort 6 txns with start ts. Error:
2018/08/07 04:26:52 draft.go:773: Found 6 old transactions. Acting to abort them.
2018/08/07 04:26:52 draft.go:750: TryAbort 6 txns with start ts. Error:

2

Hi Coolguru,

Thanks for your report.

So “Stuck”, you mean you can not use Dgraph any more?
Was the Pod deleted during a job?
Was it exactly the Pod that was working (connected to your application) or was it a worker in idle?
The Pod’s volume was deleted to during this?

Cheers.

3

We saw this issue when dgraph mutation latency was really high and client was getting context canceled error. Pod was connected to our application. As far as we know, pod volume was not deleted during this process.

4

Can you share spec of your K8s machine?

This thing of 6 old txns is not so important, if there is something missing just make another mutation.

5

ec2 m5.xlarge instance. Please let me know if there id any other info you need. thanks!

6

What was the error? It just shows empty.