# Unrecoverable Assert failed error in a container of a Kubernetes Pod **URL:** \ **Category:** Users\ **Created:** [December 10, 2017, 11:03pm UTC](https://discuss.dgraph.io/t/unrecoverable-assert-failed-error-in-a-container-of-a-kubernetes-pod/2012 "2017-12-10T23:03:27Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![jzhu077](https://avatars.discourse-cdn.com/v4/letter/j/da6949/32.png) [@jzhu077](https://discuss.dgraph.io/u/jzhu077)\ **Post date:** [December 10, 2017, 11:03pm UTC](https://discuss.dgraph.io/t/unrecoverable-assert-failed-error-in-a-container-of-a-kubernetes-pod/2012/1 "2017-12-10T23:03:27Z")
I setup a dgraph (v0.9.3) cluster with 3 zero instances and 10 servers running in a single pod of Kubernetes. One of the server is having an “Assert failed” error This particular container cannot restart properly and always runs into the same error. Maybe it’s reassigning the predicates when a mutation was received? as it happens ~10mins after ingesting starts. Is there a way to get around this issue? ---
**Author:** ![janardhan](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/janardhan/32/759_2.png) [@janardhan](https://discuss.dgraph.io/u/janardhan)\ **Post date:** [December 11, 2017, 12:39am UTC](https://discuss.dgraph.io/t/unrecoverable-assert-failed-error-in-a-container-of-a-kubernetes-pod/2012/2 "2017-12-11T00:39:33Z")
@jzhu077: Did you notice this error before restart also? ---
**Author:** ![jzhu077](https://avatars.discourse-cdn.com/v4/letter/j/da6949/32.png) [@jzhu077](https://discuss.dgraph.io/u/jzhu077)\ **Post date:** [December 11, 2017, 12:51am UTC](https://discuss.dgraph.io/t/unrecoverable-assert-failed-error-in-a-container-of-a-kubernetes-pod/2012/3 "2017-12-11T00:51:22Z")
yes, it ran into this error then crashed. ---
**Author:** ![janardhan](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/janardhan/32/759_2.png) [@janardhan](https://discuss.dgraph.io/u/janardhan)\ **Post date:** [December 11, 2017, 3:43am UTC](https://discuss.dgraph.io/t/unrecoverable-assert-failed-error-in-a-container-of-a-kubernetes-pod/2012/4 "2017-12-11T03:43:08Z")
Do you have the logs from zero before the crash. ---
**Author:** ![janardhan](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/janardhan/32/759_2.png) [@janardhan](https://discuss.dgraph.io/u/janardhan)\ **Post date:** [December 11, 2017, 7:05am UTC](https://discuss.dgraph.io/t/unrecoverable-assert-failed-error-in-a-container-of-a-kubernetes-pod/2012/5 "2017-12-11T07:05:57Z")
Can you please try the same on latest master. ---
**Author:** ![jzhu077](https://avatars.discourse-cdn.com/v4/letter/j/da6949/32.png) [@jzhu077](https://discuss.dgraph.io/u/jzhu077)\ **Post date:** [December 11, 2017, 9:07pm UTC](https://discuss.dgraph.io/t/unrecoverable-assert-failed-error-in-a-container-of-a-kubernetes-pod/2012/6 "2017-12-11T21:07:48Z")
I don’t have the logs from zero before the crash since I have reset it, and extend the rebalancing time so I can carry on with my testing. I will try the same on latest master after I finish the test. ---
**Author:** ![jzhu077](https://avatars.discourse-cdn.com/v4/letter/j/da6949/32.png) [@jzhu077](https://discuss.dgraph.io/u/jzhu077)\ **Post date:** [December 13, 2017, 8:21pm UTC](https://discuss.dgraph.io/t/unrecoverable-assert-failed-error-in-a-container-of-a-kubernetes-pod/2012/7 "2017-12-13T20:21:44Z")
it’s strange that only 1 of the 10 dgraph instances has this problem. dgraph zero log ```auto Groups sorted by size: [{gid:20 size:4554185} {gid:26 size:4657067} {gid:21 size:5071232} {gid:18 size:5787171} {gid:11 size:6685647} {gid:27 size:6792231} {gid:29 size:7302747} {gid:23 size:7506995} {gid:15 size:15740063} {gid:1 size:17982928}] 2017/12/13 08:15:32 tablet.go:170: size_diff 13428743 2017/12/13 08:15:32 tablet.go:87: Going to move predicate _dummy_ from 1 to 20 2017/12/13 08:15:32 node.go:162: SENDING: MsgApp 1-->3 2017/12/13 08:15:32 node.go:485: RECEIVED: MsgAppResp 3-->1 2017/12/13 08:15:32 node.go:162: SENDING: MsgApp 1-->3 2017/12/13 08:15:32 node.go:485: RECEIVED: MsgAppResp 3-->1 2017/12/13 08:15:32 node.go:485: RECEIVED: MsgReadIndex 3-->1 2017/12/13 08:15:32 node.go:485: RECEIVED: MsgReadIndex 3-->1 2017/12/13 08:15:32 node.go:162: SENDING: MsgReadIndexResp 1-->3 2017/12/13 08:15:32 node.go:162: SENDING: MsgReadIndexResp 1-->3 2017/12/13 08:15:32 node.go:162: SENDING: MsgApp 1-->3 2017/12/13 08:15:32 node.go:485: RECEIVED: MsgAppResp 3-->1 2017/12/13 08:15:32 node.go:162: SENDING: MsgApp 1-->3 2017/12/13 08:15:32 tablet.go:91: Error while trying to move predicate _dummy_ from 1 to 20: rpc error: code = Unavailable desc = transport is closing 2017/12/13 08:15:32 node.go:485: RECEIVED: MsgAppResp 3-->1 2017/12/13 08:15:32 node.go:485: RECEIVED: MsgReadIndex 3-->1 2017/12/13 08:15:32 node.go:485: RECEIVED: MsgReadIndex 3-->1 2017/12/13 08:15:32 node.go:162: SENDING: MsgReadIndexResp 1-->3 2017/12/13 08:15:32 node.go:162: SENDING: MsgReadIndexResp 1-->3 2017/12/13 08:15:33 pool.go:168: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure 2017/12/13 08:15:35 zero.go:293: Got connection request: id:1 addr:"localhost:7092" 2017/12/13 08:15:35 zero.go:389: Connected 2017/12/13 08:15:38 node.go:485: RECEIVED: MsgReadIndex 3-->1 2017/12/13 08:15:38 node.go:162: SENDING: MsgReadIndexResp 1-->3 2017/12/13 08:15:42 oracle.go:372: No healthy connection found to leader of group 1 2017/12/13 08:15:43 pool.go:168: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure 2017/12/13 08:15:50 zero.go:293: Got connection request: id:1 addr:"localhost:7092" 2017/12/13 08:15:50 zero.go:389: Connected 2017/12/13 08:15:52 oracle.go:372: No healthy connection found to leader of group 1 2017/12/13 08:15:53 pool.go:168: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure 2017/12/13 08:15:57 node.go:485: RECEIVED: MsgReadIndex 3-->1 2017/12/13 08:15:57 node.go:162: SENDING: MsgReadIndexResp 1-->3 2017/12/13 08:16:02 oracle.go:372: No healthy connection found to leader of group 1 2017/12/13 08:16:03 pool.go:168: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure ...(a lot of repetitive logs) 2017/12/13 20:10:02 oracle.go:372: No healthy connection found to leader of group 1 2017/12/13 20:10:03 pool.go:168: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure 2017/12/13 20:10:12 oracle.go:372: No healthy connection found to leader of group 1 2017/12/13 20:10:13 pool.go:168: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure 2017/12/13 20:10:22 oracle.go:372: No healthy connection found to leader of group 1 2017/12/13 20:10:23 pool.go:168: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure 2017/12/13 20:10:32 oracle.go:372: No healthy connection found to leader of group 1 2017/12/13 20:10:33 pool.go:168: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure 2017/12/13 20:10:38 node.go:485: RECEIVED: MsgReadIndex 3-->1 2017/12/13 20:10:38 node.go:162: SENDING: MsgReadIndexResp 1-->3 2017/12/13 20:10:42 oracle.go:372: No healthy connection found to leader of group 1 2017/12/13 20:10:43 pool.go:168: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure 2017/12/13 20:10:52 oracle.go:372: No healthy connection found to leader of group 1 2017/12/13 20:10:53 pool.go:168: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure 2017/12/13 20:10:57 node.go:485: RECEIVED: MsgReadIndex 3-->1 2017/12/13 20:10:57 node.go:162: SENDING: MsgReadIndexResp 1-->3 2017/12/13 20:11:00 zero.go:293: Got connection request: id:1 addr:"localhost:7092" 2017/12/13 20:11:00 zero.go:389: Connected 2017/12/13 20:11:02 oracle.go:372: No healthy connection found to leader of group 1 2017/12/13 20:11:03 pool.go:168: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure 2017/12/13 20:11:12 oracle.go:372: No healthy connection found to leader of group 1 2017/12/13 20:11:13 pool.go:168: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure 2017/12/13 20:11:22 oracle.go:372: No healthy connection found to leader of group 1 2017/12/13 20:11:23 pool.go:168: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure 2017/12/13 20:11:32 oracle.go:372: No healthy connection found to leader of group 1 2017/12/13 20:11:33 pool.go:168: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure 2017/12/13 20:11:38 node.go:485: RECEIVED: MsgReadIndex 3-->1 2017/12/13 20:11:38 node.go:162: SENDING: MsgReadIndexResp 1-->3 2017/12/13 20:11:42 oracle.go:372: No healthy connection found to leader of group 1 2017/12/13 20:11:43 pool.go:168: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure 2017/12/13 20:11:52 oracle.go:372: No healthy connection found to leader of group 1 2017/12/13 20:11:53 pool.go:168: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure 2017/12/13 20:11:57 node.go:485: RECEIVED: MsgReadIndex 3-->1 2017/12/13 20:11:57 node.go:162: SENDING: MsgReadIndexResp 1-->3 2017/12/13 20:12:02 oracle.go:372: No healthy connection found to leader of group 1 2017/12/13 20:12:03 pool.go:168: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure 2017/12/13 20:12:12 oracle.go:372: No healthy connection found to leader of group 1 2017/12/13 20:12:13 pool.go:168: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure 2017/12/13 20:12:22 oracle.go:372: No healthy connection found to leader of group 1 2017/12/13 20:12:23 pool.go:168: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure 2017/12/13 20:12:32 oracle.go:372: No healthy connection found to leader of group 1 2017/12/13 20:12:33 pool.go:168: Echo error from localhost:7092. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure ``` ---
**Author:** ![janardhan](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/janardhan/32/759_2.png) [@janardhan](https://discuss.dgraph.io/u/janardhan)\ **Post date:** [December 13, 2017, 10:35pm UTC](https://discuss.dgraph.io/t/unrecoverable-assert-failed-error-in-a-container-of-a-kubernetes-pod/2012/8 "2017-12-13T22:35:15Z")
Did you try on latest Master branch, the issue shouldn’t occur on master ---
**Author:** ![jzhu077](https://avatars.discourse-cdn.com/v4/letter/j/da6949/32.png) [@jzhu077](https://discuss.dgraph.io/u/jzhu077)\ **Post date:** [December 14, 2017, 3:12am UTC](https://discuss.dgraph.io/t/unrecoverable-assert-failed-error-in-a-container-of-a-kubernetes-pod/2012/9 "2017-12-14T03:12:19Z")
My mistake, I didn’t use the correct commit. cheers ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex007/uploads/dgraph/original/2X/2/2b38fdece2abe5814f2e51bee69d1e67fc304b0a.png) [@system](https://discuss.dgraph.io/u/system)\ **Post date:** [January 13, 2018, 3:12am UTC](https://discuss.dgraph.io/t/unrecoverable-assert-failed-error-in-a-container-of-a-kubernetes-pod/2012/10 "2018-01-13T03:12:24Z")
Closed on January 13, 2018, 3:12am UTC This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.