In my case, query func: eq(name,"foobar") returns empty result in dgraph server 2,5,and 10, when server-10’s group peers 9 and 11 are working correctly.
In my understanding, all dgraph server within a group should have the same copy of the database. Is it possible when one database is corrupted but it doesn’t get synced to it’s peers?
RAFT ensures that data is replicated across a group so this shouldn’t happen ideally. This seems like a bug which I believe should be related to predicate move.
Could you provide some more information about your setup and the workload so that we can replicate and fix this?
Optionally, is it possible for you to share the logs from dgraph zero to see if predicate move happened and the p directory of a server which returns an empty result? Also, share the output of /state endpoint.
Setup:
5 zeros and 30 servers divided into 10 groups.
{
"counter":"62528",
"groups":{
"1":{
"members":{
"10":{
"id":"10",
"groupId":1,
"addr":"dgraph-server-0.dgraph-server.default.svc.cluster.local:7090",
"leader":true,
"lastUpdate":"1519342798"
},
"11":{
"id":"11",
"groupId":1,
"addr":"dgraph-server-1.dgraph-server.default.svc.cluster.local:7090"
},
"12":{
"id":"12",
"groupId":1,
"addr":"dgraph-server-2.dgraph-server.default.svc.cluster.local:7090"
}
},
"tablets":{
"___commonkind":{
"groupId":1,
"predicate":"___commonkind",
"space":"214503983"
},
"bool_prop":{
"groupId":1,
"predicate":"bool_prop",
"space":"48753475"
},
"dummy":{
"groupId":1,
"predicate":"dummy",
"space":"17"
},
"float_prop":{
"groupId":1,
"predicate":"float_prop",
"space":"52753475"
},
"host":{
"groupId":1,
"predicate":"host",
"space":"1817"
},
"int_prop":{
"groupId":1,
"predicate":"int_prop",
"space":"51753475"
},
"name":{
"groupId":1,
"predicate":"name",
"space":"73683180"
},
"string_prop":{
"groupId":1,
"predicate":"string_prop",
"space":"51753516"
}
}
},
"2":{
"members":{
"13":{
"id":"13",
"groupId":2,
"addr":"dgraph-server-3.dgraph-server.default.svc.cluster.local:7090",
"leader":true,
"lastUpdate":"1519342890"
},
"14":{
"id":"14",
"groupId":2,
"addr":"dgraph-server-4.dgraph-server.default.svc.cluster.local:7090"
},
"15":{
"id":"15",
"groupId":2,
"addr":"dgraph-server-5.dgraph-server.default.svc.cluster.local:7090"
}
},
"tablets":{
"start_column":{
"groupId":2,
"predicate":"start_column",
"space":"133146604"
},
"username":{
"groupId":2,
"predicate":"username"
}
}
},
"3":{
"members":{
"16":{
"id":"16",
"groupId":3,
"addr":"dgraph-server-6.dgraph-server.default.svc.cluster.local:7090",
"lastUpdate":"1519352232"
},
"17":{
"id":"17",
"groupId":3,
"addr":"dgraph-server-7.dgraph-server.default.svc.cluster.local:7090",
"lastUpdate":"1519352523"
},
"18":{
"id":"18",
"groupId":3,
"addr":"dgraph-server-8.dgraph-server.default.svc.cluster.local:7090",
"leader":true,
"lastUpdate":"1519354344"
}
},
"tablets":{
"chan_dir":{
"groupId":3,
"predicate":"chan_dir",
"space":"95601"
},
"end_column":{
"groupId":3,
"predicate":"end_column",
"space":"124432549"
},
"end_line":{
"groupId":3,
"predicate":"end_line",
"space":"122097964"
},
"expr":{
"groupId":3,
"predicate":"expr",
"space":"35700237"
},
"obj_kind":{
"groupId":3,
"predicate":"obj_kind",
"space":"20799913"
},
"pos":{
"groupId":3,
"predicate":"pos",
"space":"97862369"
},
"private":{
"groupId":3,
"predicate":"private",
"space":"38944731"
},
"public":{
"groupId":3,
"predicate":"public",
"space":"38740569"
},
"token":{
"groupId":3,
"predicate":"token",
"space":"16293230"
},
"type":{
"groupId":3,
"predicate":"type",
"space":"41719043"
},
"value":{
"groupId":3,
"predicate":"value",
"space":"49463444"
}
}
},
"4":{
"members":{
"19":{
"id":"19",
"groupId":4,
"addr":"dgraph-server-9.dgraph-server.default.svc.cluster.local:7090",
"leader":true,
"lastUpdate":"1519343018"
},
"20":{
"id":"20",
"groupId":4,
"addr":"dgraph-server-10.dgraph-server.default.svc.cluster.local:7090"
},
"21":{
"id":"21",
"groupId":4,
"addr":"dgraph-server-11.dgraph-server.default.svc.cluster.local:7090"
}
},
"tablets":{
"start_line":{
"groupId":4,
"predicate":"start_line",
"space":"136645002"
}
}
},
"5":{
"members":{
"22":{
"id":"22",
"groupId":5,
"addr":"dgraph-server-12.dgraph-server.default.svc.cluster.local:7090",
"leader":true,
"lastUpdate":"1519343083"
},
"23":{
"id":"23",
"groupId":5,
"addr":"dgraph-server-13.dgraph-server.default.svc.cluster.local:7090"
},
"24":{
"id":"24",
"groupId":5,
"addr":"dgraph-server-14.dgraph-server.default.svc.cluster.local:7090"
}
},
"tablets":{
"___child":{
"groupId":5,
"predicate":"___child",
"space":"196454197"
}
}
},
"6":{
"members":{
"25":{
"id":"25",
"groupId":6,
"addr":"dgraph-server-15.dgraph-server.default.svc.cluster.local:7090",
"leader":true,
"lastUpdate":"1519343158"
},
"26":{
"id":"26",
"groupId":6,
"addr":"dgraph-server-16.dgraph-server.default.svc.cluster.local:7090"
},
"27":{
"id":"27",
"groupId":6,
"addr":"dgraph-server-17.dgraph-server.default.svc.cluster.local:7090"
}
},
"tablets":{
"alias":{
"groupId":6,
"predicate":"alias",
"space":"11828"
},
"bool":{
"groupId":6,
"predicate":"bool",
"space":"396115"
},
"owner":{
"groupId":6,
"predicate":"owner",
"space":"1762"
},
"password":{
"groupId":6,
"predicate":"password"
}
}
},
"7":{
"members":{
"28":{
"id":"28",
"groupId":7,
"addr":"dgraph-server-18.dgraph-server.default.svc.cluster.local:7090",
"leader":true,
"lastUpdate":"1519343191"
},
"29":{
"id":"29",
"groupId":7,
"addr":"dgraph-server-19.dgraph-server.default.svc.cluster.local:7090"
},
"30":{
"id":"30",
"groupId":7,
"addr":"dgraph-server-20.dgraph-server.default.svc.cluster.local:7090"
}
},
"tablets":{
"byRef":{
"groupId":7,
"predicate":"byRef",
"space":"321856"
},
"kind":{
"groupId":7,
"predicate":"kind",
"space":"196398"
},
"start_offset":{
"groupId":7,
"predicate":"start_offset",
"space":"126268410"
},
"static":{
"groupId":7,
"predicate":"static",
"space":"483"
},
"unpack":{
"groupId":7,
"predicate":"unpack",
"space":"204930"
},
"variadic":{
"groupId":7,
"predicate":"variadic",
"space":"33157"
}
}
},
"8":{
"members":{
"31":{
"id":"31",
"groupId":8,
"addr":"dgraph-server-21.dgraph-server.default.svc.cluster.local:7090",
"leader":true,
"lastUpdate":"1519343312"
},
"32":{
"id":"32",
"groupId":8,
"addr":"dgraph-server-22.dgraph-server.default.svc.cluster.local:7090"
},
"33":{
"id":"33",
"groupId":8,
"addr":"dgraph-server-23.dgraph-server.default.svc.cluster.local:7090"
}
},
"tablets":{
"___kind":{
"groupId":8,
"predicate":"___kind",
"force":true,
"space":"210409460"
},
"___older_sibling":{
"groupId":8,
"predicate":"___older_sibling",
"space":"175596542"
},
"___orderable":{
"groupId":8,
"predicate":"___orderable",
"force":true,
"space":"192925461"
},
"decl":{
"groupId":8,
"predicate":"decl",
"space":"414721"
},
"filename":{
"groupId":8,
"predicate":"filename",
"space":"169362111"
},
"flags":{
"groupId":8,
"predicate":"flags",
"space":"89640"
},
"sha":{
"groupId":8,
"predicate":"sha",
"space":"2725"
},
"stmt":{
"groupId":8,
"predicate":"stmt",
"space":"2159713"
}
}
},
"9":{
"members":{
"34":{
"id":"34",
"groupId":9,
"addr":"dgraph-server-24.dgraph-server.default.svc.cluster.local:7090",
"leader":true,
"lastUpdate":"1519343366"
},
"35":{
"id":"35",
"groupId":9,
"addr":"dgraph-server-25.dgraph-server.default.svc.cluster.local:7090"
},
"36":{
"id":"36",
"groupId":9,
"addr":"dgraph-server-26.dgraph-server.default.svc.cluster.local:7090"
}
},
"tablets":{
"end_offset":{
"groupId":9,
"predicate":"end_offset",
"space":"130070852"
},
"var":{
"groupId":9,
"predicate":"var",
"space":"292"
}
}
},
"10":{
"members":{
"37":{
"id":"37",
"groupId":10,
"addr":"dgraph-server-27.dgraph-server.default.svc.cluster.local:7090",
"leader":true,
"lastUpdate":"1519343433"
},
"38":{
"id":"38",
"groupId":10,
"addr":"dgraph-server-28.dgraph-server.default.svc.cluster.local:7090"
},
"39":{
"id":"39",
"groupId":10,
"addr":"dgraph-server-29.dgraph-server.default.svc.cluster.local:7090"
}
},
"tablets":{
"_predicate_":{
"groupId":10,
"predicate":"_predicate_",
"space":"962104735"
}
}
}
},
"zeros":{
"1":{
"id":"1",
"addr":"dgraph-zero-0.dgraph-zero.default.svc.cluster.local:5080",
"leader":true
},
"2":{
"id":"2",
"addr":"dgraph-zero-1.dgraph-zero.default.svc.cluster.local:5080"
},
"3":{
"id":"3",
"addr":"dgraph-zero-2.dgraph-zero.default.svc.cluster.local:5080"
},
"4":{
"id":"4",
"addr":"dgraph-zero-3.dgraph-zero.default.svc.cluster.local:5080"
},
"5":{
"id":"5",
"addr":"dgraph-zero-4.dgraph-zero.default.svc.cluster.local:5080"
}
},
"maxTxnTs":"60000"
}
Groups sorted by size: [{gid:6 size:397417} {gid:7 size:126271520} {gid:9 size:130070852} {gid:2 size:133146604} {gid:4 size:136645002} {gid:5 size:196454197} {gid:1 size:493202649} {gid:3 size:586149650} {gid:8 size:750870319} {gid:10 size:962104735}]
2018/02/28 10:39:56 tablet.go:178: size_diff 961707318
2018/02/28 10:39:56 tablet.go:178: size_diff 750472902
2018/02/28 10:39:56 tablet.go:71: Going to move predicate ___kind from 8 to 6
2018/02/28 10:39:56 tablet.go:64: Error while trying to move predicate ___kind from 8 to 6: rpc error: code = Unknown desc = Conflicts with pending transaction. Please abort.
2018/02/28 11:39:56 tablet.go:173:
Groups sorted by size: [{gid:6 size:397417} {gid:7 size:126271520} {gid:9 size:130070852} {gid:2 size:133146604} {gid:4 size:136645002} {gid:5 size:196454197} {gid:1 size:493202649} {gid:3 size:586149650} {gid:8 size:750870319} {gid:10 size:962104735}]
2018/02/28 11:39:56 tablet.go:178: size_diff 961707318
2018/02/28 11:39:56 tablet.go:178: size_diff 750472902
2018/02/28 11:39:56 tablet.go:71: Going to move predicate ___kind from 8 to 6
2018/02/28 11:39:56 tablet.go:64: Error while trying to move predicate ___kind from 8 to 6: rpc error: code = Unknown desc = Conflicts with pending transaction. Please abort.
2018/02/28 12:39:56 tablet.go:173:
Groups sorted by size: [{gid:6 size:397417} {gid:7 size:126271520} {gid:9 size:130070852} {gid:2 size:133146604} {gid:4 size:136645002} {gid:5 size:196454197} {gid:1 size:493202649} {gid:3 size:586149650} {gid:8 size:750870319} {gid:10 size:962104735}]
2018/02/28 12:39:56 tablet.go:178: size_diff 961707318
2018/02/28 12:39:56 tablet.go:178: size_diff 750472902
2018/02/28 12:39:56 tablet.go:71: Going to move predicate ___kind from 8 to 6
2018/02/28 12:39:56 tablet.go:64: Error while trying to move predicate ___kind from 8 to 6: rpc error: code = Unknown desc = Conflicts with pending transaction. Please abort.
2018/02/28 13:39:56 tablet.go:173:
Groups sorted by size: [{gid:6 size:397417} {gid:7 size:126271520} {gid:9 size:130070852} {gid:2 size:133146604} {gid:4 size:136645002} {gid:5 size:196454197} {gid:1 size:493202649} {gid:3 size:586149650} {gid:8 size:750870319} {gid:10 size:962104735}]
2018/02/28 13:39:56 tablet.go:178: size_diff 961707318
2018/02/28 13:39:56 tablet.go:178: size_diff 750472902
2018/02/28 13:39:56 tablet.go:71: Going to move predicate ___kind from 8 to 6
2018/02/28 13:39:56 tablet.go:64: Error while trying to move predicate ___kind from 8 to 6: rpc error: code = Unknown desc = Conflicts with pending transaction. Please abort.
2018/02/28 14:39:56 tablet.go:173:
Groups sorted by size: [{gid:6 size:397417} {gid:7 size:126271520} {gid:9 size:130070852} {gid:2 size:133146604} {gid:4 size:136645002} {gid:5 size:196454197} {gid:1 size:493202649} {gid:3 size:586149650} {gid:8 size:750870319} {gid:10 size:962104735}]
2018/02/28 14:39:56 tablet.go:178: size_diff 961707318
2018/02/28 14:39:56 tablet.go:178: size_diff 750472902
2018/02/28 14:39:56 tablet.go:71: Going to move predicate ___kind from 8 to 6
2018/02/28 14:39:56 tablet.go:64: Error while trying to move predicate ___kind from 8 to 6: rpc error: code = Unknown desc = Conflicts with pending transaction. Please abort.
2018/02/28 15:39:56 tablet.go:173:
Groups sorted by size: [{gid:6 size:397417} {gid:7 size:126271520} {gid:9 size:130070852} {gid:2 size:133146604} {gid:4 size:136645002} {gid:5 size:196454197} {gid:1 size:493202649} {gid:3 size:586149650} {gid:8 size:750870319} {gid:10 size:962104735}]
2018/02/28 15:39:56 tablet.go:178: size_diff 961707318
2018/02/28 15:39:56 tablet.go:178: size_diff 750472902
2018/02/28 15:39:56 tablet.go:71: Going to move predicate ___kind from 8 to 6
2018/02/28 15:39:56 tablet.go:64: Error while trying to move predicate ___kind from 8 to 6: rpc error: code = Unknown desc = Conflicts with pending transaction. Please abort.
2018/02/28 16:39:56 tablet.go:173:
Groups sorted by size: [{gid:6 size:397417} {gid:7 size:126271520} {gid:9 size:130070852} {gid:2 size:133146604} {gid:4 size:136645002} {gid:5 size:196454197} {gid:1 size:493202649} {gid:3 size:586149650} {gid:8 size:750870319} {gid:10 size:962104735}]
2018/02/28 16:39:56 tablet.go:178: size_diff 961707318
2018/02/28 16:39:56 tablet.go:178: size_diff 750472902
2018/02/28 16:39:56 tablet.go:71: Going to move predicate ___kind from 8 to 6
2018/02/28 16:39:56 tablet.go:64: Error while trying to move predicate ___kind from 8 to 6: rpc error: code = Unknown desc = Conflicts with pending transaction. Please abort.
2018/02/28 17:39:56 tablet.go:173:
Groups sorted by size: [{gid:6 size:397417} {gid:7 size:126271520} {gid:9 size:130070852} {gid:2 size:133146604} {gid:4 size:136645002} {gid:5 size:196454197} {gid:1 size:493202649} {gid:3 size:586149650} {gid:8 size:750870319} {gid:10 size:962104735}]
2018/02/28 17:39:56 tablet.go:178: size_diff 961707318
2018/02/28 17:39:56 tablet.go:178: size_diff 750472902
2018/02/28 17:39:56 tablet.go:71: Going to move predicate ___kind from 8 to 6
2018/02/28 17:39:56 tablet.go:64: Error while trying to move predicate ___kind from 8 to 6: rpc error: code = Unknown desc = Conflicts with pending transaction. Please abort.
2018/02/28 18:39:56 tablet.go:173:
Groups sorted by size: [{gid:6 size:397417} {gid:7 size:126271520} {gid:9 size:130070852} {gid:2 size:133146604} {gid:4 size:136645002} {gid:5 size:196454197} {gid:1 size:493202649} {gid:3 size:586149650} {gid:8 size:750870319} {gid:10 size:962104735}]
2018/02/28 18:39:56 tablet.go:178: size_diff 961707318
2018/02/28 18:39:56 tablet.go:178: size_diff 750472902
2018/02/28 18:39:56 tablet.go:71: Going to move predicate ___kind from 8 to 6
2018/02/28 18:39:56 tablet.go:64: Error while trying to move predicate ___kind from 8 to 6: rpc error: code = Unknown desc = Conflicts with pending transaction. Please abort.
2018/02/28 19:39:56 tablet.go:173:
Groups sorted by size: [{gid:6 size:397417} {gid:7 size:126271520} {gid:9 size:130070852} {gid:2 size:133146604} {gid:4 size:136645002} {gid:5 size:196454197} {gid:1 size:493202649} {gid:3 size:586149650} {gid:8 size:750870319} {gid:10 size:962104735}]
2018/02/28 19:39:56 tablet.go:178: size_diff 961707318
2018/02/28 19:39:56 tablet.go:178: size_diff 750472902
2018/02/28 19:39:56 tablet.go:71: Going to move predicate ___kind from 8 to 6
2018/02/28 19:39:56 tablet.go:64: Error while trying to move predicate ___kind from 8 to 6: rpc error: code = Unknown desc = Conflicts with pending transaction. Please abort.
2018/02/28 20:29:23 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 20:29:23 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 20:29:23 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 20:29:23 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 20:29:23 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 20:29:23 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 20:29:23 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 20:29:23 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 20:29:23 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 20:29:23 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 20:29:23 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 20:29:23 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 20:29:23 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 20:29:23 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 20:29:23 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 20:39:56 tablet.go:173:
Groups sorted by size: [{gid:6 size:397417} {gid:7 size:126271520} {gid:9 size:130070852} {gid:2 size:133146604} {gid:4 size:136645002} {gid:5 size:196454197} {gid:1 size:493202649} {gid:3 size:586149650} {gid:8 size:750870319} {gid:10 size:962104735}]
2018/02/28 20:39:56 tablet.go:178: size_diff 961707318
2018/02/28 20:39:56 tablet.go:178: size_diff 750472902
2018/02/28 20:39:56 tablet.go:71: Going to move predicate ___kind from 8 to 6
2018/02/28 20:39:56 tablet.go:64: Error while trying to move predicate ___kind from 8 to 6: rpc error: code = Unknown desc = Conflicts with pending transaction. Please abort.
2018/02/28 20:54:04 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 20:54:04 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 20:54:04 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 20:54:04 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 20:54:04 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 20:54:04 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 20:54:04 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 20:54:04 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 20:54:10 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 20:54:10 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 20:54:10 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 20:54:10 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 20:54:10 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:10:44 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:10:44 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:10:44 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:10:44 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:10:44 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:10:44 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:10:44 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:10:44 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:11:00 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:11:00 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:11:00 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:11:00 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:11:00 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:11:00 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:31:31 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:31:31 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:31:31 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:31:31 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:31:31 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:31:31 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:31:31 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:31:31 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:31:38 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:31:38 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:31:38 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:31:38 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:31:38 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:37:43 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:37:43 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:37:43 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:37:43 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:37:43 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:37:43 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:37:43 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:37:43 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:39:56 tablet.go:173:
Groups sorted by size: [{gid:6 size:397877} {gid:7 size:127025234} {gid:9 size:130071144} {gid:2 size:133146604} {gid:4 size:136645002} {gid:5 size:196454197} {gid:1 size:493202938} {gid:3 size:586149650} {gid:8 size:750960373} {gid:10 size:962112545}]
2018/02/28 21:39:56 tablet.go:178: size_diff 961714668
2018/02/28 21:39:56 tablet.go:71: Going to move predicate alias from 10 to 6
2018/02/28 21:57:18 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:57:18 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:57:18 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:57:18 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:57:18 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:57:18 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 21:57:18 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 22:11:50 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 22:11:50 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 22:11:50 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 22:11:50 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 22:11:50 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 22:11:50 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 22:11:50 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 22:12:56 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 22:12:56 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 22:12:56 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 22:12:56 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 22:12:56 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 22:12:56 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 22:12:56 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 22:20:51 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 22:20:51 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 22:20:51 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 22:20:51 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 22:20:51 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 22:20:51 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 22:20:51 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 22:20:51 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 22:20:51 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 22:20:51 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 22:20:51 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 22:20:51 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 22:20:51 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 22:39:56 tablet.go:173:
Groups sorted by size: [{gid:6 size:409705} {gid:7 size:127025234} {gid:9 size:130071144} {gid:2 size:133146604} {gid:4 size:136645002} {gid:5 size:196454197} {gid:1 size:493202938} {gid:3 size:586149650} {gid:8 size:750960373} {gid:10 size:962104735}]
2018/02/28 22:39:56 tablet.go:178: size_diff 961695030
2018/02/28 22:39:56 tablet.go:178: size_diff 750550668
2018/02/28 22:39:56 tablet.go:71: Going to move predicate ___kind from 8 to 6
2018/02/28 22:39:56 tablet.go:64: Error while trying to move predicate ___kind from 8 to 6: rpc error: code = Unknown desc = Conflicts with pending transaction. Please abort.
2018/02/28 23:20:19 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 23:20:19 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 23:20:19 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 23:20:19 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 23:20:19 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 23:20:19 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 23:20:19 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 23:21:19 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 23:21:19 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 23:21:19 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 23:21:19 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 23:21:19 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 23:21:19 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 23:21:19 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 23:21:19 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 23:21:19 raft.go:533: While applying proposal: Tablet is already being served
2018/02/28 23:21:19 raft.go:533: While applying proposal: Tablet is already being served
the log is quite long not sure which section I should share. I have run the test recently, it behaviours the same during the above logging period.
What I am interested in knowing is whether the predicate move for ___kind was completed successfully ever. Did the group for it change or is still served by the same group it was initially assigned to.
No, the predicate move for ___kind has never completed, it always hit the error Conflicts with pending transaction. Please abort. previously it was Going to move predicate ___kind from 8 to 4 failed since 23th Feb and starting Going to move predicate ___kind from 8 to 6 since 27th Feb and yet has completed.
Looks like the predicate ___kind is stuck in group 8 for some unknown reason.
Predicate ___kind isn’t causing any issue at the moment. I can query it every time, do you think it is related to the misbehaviour of other predicates like name in this case?
Sorry, I got confused with the other post. How about name, was it moved?
No, it hasn’t shown up in the dgraph-zero log.
All the nodes serving name, where they brought up at the same time? Do you know if some of them were down in between?
yes, they were brought up at the same time by a statefulset and none of them has restarted since then. After reset the schema a few time they seem to work just fine now. It’s very strange.
I have a wild guess that indexing silently failed on some nodes after RAFT proposal was done. I will grok the code to see how we handle that scenario. Can you check if dgraph servers printed some logs about any Error while adding index mutation?
No, I haven’t seen the error log in the faulty dgraph server pods
The issue is back again, this time it is affecting more dgraph servers and more predicates.
Ok, I am investigating this. How do you load you live load your data into Dgraph, which client do you use? Can you use the dgraph/dgraph:master image if not already doing so?
live load. We use the grpc endpoint to execute the transaction (batch size 100,000). We are using our fork to ingest data, because we are still using custom UID (switching to XID soon). After the switching is completed and few flags PR are merged we can use dgraph/dgraph:master.
None of the changes we made should affect the dgraph query as the empty result issue only found in a few dgraph servers.
I think you already told me that this not related to predicate move as name was never moved. Is it possible for you to share p directory from a server which returns successfully and another from a server which doesn’t?
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.