I0115 04:32:53.988778 16 draft.go:606] Creating snapshot at Index: 88828543, ReadTs: 101723401 I0115 04:34:53.128355 16 log.go:34] Block cache metrics: hit: 48352315 miss: 22752462 keys-added: 8809950 keys-updated: 106843 keys-evicted: 7528178 cost-added: 38036154531 cost-evicted: 32518129795 sets-dropped: 0 sets-rejected: 3665648 gets-dropped: 57280 gets-kept: 44622720 gets-total: 71104777 hit-ratio: 0.68 I0115 04:34:53.470987 16 draft.go:606] Creating snapshot at Index: 88843774, ReadTs: 101751731 I0115 04:36:53.433496 16 draft.go:606] Creating snapshot at Index: 88860531, ReadTs: 101783464 I0115 04:38:53.476303 16 draft.go:606] Creating snapshot at Index: 88875230, ReadTs: 101808887 I0115 04:39:53.132060 16 log.go:34] Block cache metrics: hit: 48729517 miss: 22860707 keys-added: 8903233 keys-updated: 112070 keys-evicted: 7621107 cost-added: 38436414392 cost-evicted: 32918390274 sets-dropped: 0 sets-rejected: 3675383 gets-dropped: 60416 gets-kept: 44992192 gets-total: 71590224 hit-ratio: 0.68 I0115 04:41:53.290466 16 draft.go:606] Creating snapshot at Index: 88885507, ReadTs: 101823407 I0115 04:44:53.132091 16 log.go:34] Block cache metrics: hit: 48788940 miss: 22875217 keys-added: 8915454 keys-updated: 112595 keys-evicted: 7633287 cost-added: 38488869752 cost-evicted: 32970845201 sets-dropped: 0 sets-rejected: 3677147 gets-dropped: 60416 gets-kept: 45040704 gets-total: 71664157 hit-ratio: 0.68 I0115 04:46:53.734516 16 draft.go:606] Creating snapshot at Index: 88895787, ReadTs: 101840727 I0115 04:49:53.128132 16 log.go:34] Block cache metrics: hit: 48897064 miss: 22904526 keys-added: 8939411 keys-updated: 113977 keys-evicted: 7657145 cost-added: 38591619974 cost-evicted: 33073594951 sets-dropped: 0 sets-rejected: 3681117 gets-dropped: 63040 gets-kept: 45147200 gets-total: 71801590 hit-ratio: 0.68 I0115 04:51:53.638156 16 draft.go:606] Creating snapshot at Index: 88913314, ReadTs: 101872755 I0115 04:54:53.128215 16 log.go:34] Block cache metrics: hit: 49094831 miss: 22906770 keys-added: 8940961 keys-updated: 114336 keys-evicted: 7658676 cost-added: 38598211569 cost-evicted: 33080185314 sets-dropped: 0 sets-rejected: 3681452 gets-dropped: 63040 gets-kept: 45262656 gets-total: 72001601 hit-ratio: 0.68 I0115 04:59:53.165073 16 log.go:34] Block cache metrics: hit: 49292869 miss: 22908442 keys-added: 8942199 keys-updated: 114529 keys-evicted: 7659894 cost-added: 38603452377 cost-evicted: 33085430191 sets-dropped: 0 sets-rejected: 3681693 gets-dropped: 63040 gets-kept: 45396480 gets-total: 72201311 hit-ratio: 0.68 W0115 05:01:07.553939 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 1.067s. Breakdown: [{disk 1.063s} {proposals 4ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:01:11.963037 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 773ms. Breakdown: [{disk 372ms} {proposals 0s} {advance 0s}] Num entries: 0. MustSync: false W0115 05:01:21.312065 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 669ms. Breakdown: [{disk 456ms} {proposals 0s} {advance 0s}] Num entries: 0. MustSync: false W0115 05:01:53.792344 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 995ms. Breakdown: [{disk 994ms} {advance 1ms} {proposals 0s}] Num entries: 1. MustSync: true E0115 05:01:54.183543 16 groups.go:937] Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader I0115 05:01:55.299609 16 groups.go:893] Leader idx=0x11 of group=3 is connecting to Zero for txn updates I0115 05:01:55.331723 16 groups.go:905] Got Zero leader: 192.168.3.12:5080 W0115 05:01:56.959485 16 server_state.go:205] Error while retrieving timestamps: rpc error: code = Unknown desc = Assigning IDs is only allowed on leader. with delay: 10ms. Will retry... E0115 05:01:57.017707 16 groups.go:937] Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader I0115 05:01:57.821979 16 groups.go:893] Leader idx=0x11 of group=3 is connecting to Zero for txn updates I0115 05:01:57.823954 16 groups.go:905] Got Zero leader: 192.168.3.11:5080 W0115 05:02:01.933965 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 835ms. Breakdown: [{proposals 705ms} {disk 1ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:02:04.359761 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 972ms. Breakdown: [{proposals 815ms} {advance 133ms} {disk 20ms}] Num entries: 0. MustSync: false I0115 05:02:50.586866 16 log.go:34] 11 [logterm: 7, index: 88920945, vote: 11] cast MsgPreVote for 7 [logterm: 7, index: 88920945] at term 7 I0115 05:02:50.827620 16 log.go:34] 11 [term: 7] received a MsgVote message with higher term from 7 [term: 8] I0115 05:02:50.827734 16 log.go:34] 11 became follower at term 8 I0115 05:02:50.827766 16 log.go:34] 11 [logterm: 7, index: 88920945, vote: 0] cast MsgVote for 7 [logterm: 7, index: 88920945] at term 8 I0115 05:02:50.827795 16 log.go:34] raft.node: 11 lost leader 11 at term 8 I0115 05:02:50.827809 16 log.go:34] 11 [logterm: 7, index: 88920945, vote: 7] rejected MsgPreVote from 8 [logterm: 7, index: 88920945] at term 8 W0115 05:02:50.585870 16 groups.go:869] No membership update for 10s. Closing connection to Zero. W0115 05:02:50.586226 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 40.329s. Breakdown: [{disk 40.329s} {proposals 0s} {advance 0s}] Num entries: 0. MustSync: false I0115 05:02:50.837232 16 log.go:34] 11 [logterm: 7, index: 88920945, vote: 7] rejected MsgVote from 8 [logterm: 7, index: 88920945] at term 8 I0115 05:02:50.837311 16 log.go:34] 11 [logterm: 7, index: 88920945, vote: 7] cast MsgPreVote for 8 [logterm: 7, index: 88920945] at term 8 I0115 05:02:50.837336 16 log.go:34] 11 [term: 8] received a MsgAppResp message with higher term from 7 [term: 9] I0115 05:02:50.837401 16 log.go:34] 11 became follower at term 9 I0115 05:02:50.839005 16 groups.go:807] Got address of a Zero leader: 192.168.3.11:5080 I0115 05:02:50.839350 16 groups.go:821] Starting a new membership stream receive from 192.168.3.11:5080. E0115 05:02:50.836872 16 groups.go:829] Unable to sync memberships. Error: rpc error: code = Canceled desc = context canceled. State: I0115 05:02:50.929081 16 groups.go:838] Received first state update from Zero: counter:36743255 groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101868742 checksum:18290414373100373824 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101884062 checksum:2757678515920796499 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101872755 checksum:287855597640554238 > > zeros: > zeros: > zeros: > maxLeaseId:737090000 maxTxnTs:101900000 maxRaftId:17 removed: removed: removed: removed: removed: removed: removed: removed: removed: cid:"1582cefb-03ec-4014-b336-a4857ccc1344" license: I0115 05:02:50.934823 16 log.go:34] 11 [logterm: 7, index: 88920945, vote: 0] cast MsgVote for 8 [logterm: 7, index: 88920945] at term 9 I0115 05:02:50.986236 16 log.go:34] raft.node: 11 elected leader 8 at term 9 E0115 05:02:57.851066 16 groups.go:1000] No longer the leader of group 3. Exiting E0115 05:02:57.851212 16 groups.go:937] Error in oracle delta stream. Error: rpc error: code = Canceled desc = context canceled I0115 05:04:53.176769 16 log.go:34] Block cache metrics: hit: 49357512 miss: 22908926 keys-added: 8942542 keys-updated: 114597 keys-evicted: 7660230 cost-added: 38604902650 cost-evicted: 33086877796 sets-dropped: 0 sets-rejected: 3681766 gets-dropped: 63040 gets-kept: 45449152 gets-total: 72266438 hit-ratio: 0.68 W0115 05:05:45.993845 16 groups.go:869] No membership update for 10s. Closing connection to Zero. E0115 05:06:16.362005 16 groups.go:773] While sending membership update: rpc error: code = DeadlineExceeded desc = context deadline exceeded W0115 05:06:46.276630 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 29.982s. Breakdown: [{disk 10.568s} {proposals 6.922s} {advance 0s}] Num entries: 0. MustSync: false I0115 05:07:05.831906 16 groups.go:807] Got address of a Zero leader: 192.168.3.11:5080 W0115 05:07:05.985084 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 3.379s. Breakdown: [{proposals 1.699s} {advance 1.65s} {disk 0s}] Num entries: 0. MustSync: false W0115 05:09:17.622240 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 11.85s. Breakdown: [{proposals 11.841s} {disk 0s} {advance 0s}] Num entries: 0. MustSync: false W0115 05:10:40.531906 16 groups.go:869] No membership update for 10s. Closing connection to Zero. I0115 05:10:42.535046 16 groups.go:821] Starting a new membership stream receive from 192.168.3.11:5080. I0115 05:10:38.874861 16 log.go:34] Block cache metrics: hit: 49357547 miss: 22908929 keys-added: 8942544 keys-updated: 114597 keys-evicted: 7660232 cost-added: 38604911321 cost-evicted: 33086886312 sets-dropped: 0 sets-rejected: 3681767 gets-dropped: 63040 gets-kept: 45449152 gets-total: 72266476 hit-ratio: 0.68 W0115 05:12:49.342913 16 raft_server.go:239] Error while raft.Step from 0x8: context deadline exceeded. Closing RaftMessage stream. I0115 05:12:49.350625 16 groups.go:838] Received first state update from Zero: counter:36743384 groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101868742 checksum:18290414373100373824 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101884062 checksum:2757678515920796499 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101872755 checksum:287855597640554238 > > zeros: > zeros: > zeros: > maxLeaseId:737100000 maxTxnTs:101900000 maxRaftId:17 removed: removed: removed: removed: removed: removed: removed: removed: removed: cid:"1582cefb-03ec-4014-b336-a4857ccc1344" license: W0115 05:14:54.084952 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 2m7.01s. Breakdown: [{disk 2m7.006s} {proposals 4ms} {advance 0s}] Num entries: 0. MustSync: false I0115 05:15:00.285574 16 groups.go:807] Got address of a Zero leader: 192.168.3.11:5080 W0115 05:16:11.441430 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 18.909s. Breakdown: [{disk 0s} {proposals 0s} {advance 0s}] Num entries: 0. MustSync: false I0115 05:16:44.453818 16 groups.go:821] Starting a new membership stream receive from 192.168.3.11:5080. W0115 05:17:03.612728 16 groups.go:869] No membership update for 10s. Closing connection to Zero. W0115 05:21:22.043555 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 4m18.597s. Breakdown: [{disk 3m26.944s} {proposals 26.012s} {advance 10.578s}] Num entries: 0. MustSync: false I0115 05:21:22.048331 16 groups.go:807] Got address of a Zero leader: 192.168.3.11:5080 I0115 05:21:22.048514 16 groups.go:821] Starting a new membership stream receive from 192.168.3.11:5080. I0115 05:21:22.067696 16 groups.go:838] Received first state update from Zero: counter:36743384 groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101868742 checksum:18290414373100373824 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101884062 checksum:2757678515920796499 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101872755 checksum:287855597640554238 > > zeros: > zeros: > zeros: > maxLeaseId:737100000 maxTxnTs:101900000 maxRaftId:17 removed: removed: removed: removed: removed: removed: removed: removed: removed: cid:"1582cefb-03ec-4014-b336-a4857ccc1344" license: I0115 05:20:48.684774 16 groups.go:838] Received first state update from Zero: counter:36743384 groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101868742 checksum:18290414373100373824 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101884062 checksum:2757678515920796499 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101872755 checksum:287855597640554238 > > zeros: > zeros: > zeros: > maxLeaseId:737100000 maxTxnTs:101900000 maxRaftId:17 removed: removed: removed: removed: removed: removed: removed: removed: removed: cid:"1582cefb-03ec-4014-b336-a4857ccc1344" license: I0115 05:21:22.024500 16 log.go:34] Block cache metrics: hit: 49357547 miss: 22908929 keys-added: 8942544 keys-updated: 114597 keys-evicted: 7660232 cost-added: 38604911321 cost-evicted: 33086886312 sets-dropped: 0 sets-rejected: 3681767 gets-dropped: 63040 gets-kept: 45449152 gets-total: 72266476 hit-ratio: 0.68 W0115 05:21:22.814060 16 server_state.go:205] Error while retrieving timestamps: rpc error: code = DeadlineExceeded desc = context deadline exceeded with delay: 10ms. Will retry... W0115 05:21:24.029161 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 922ms. Breakdown: [{advance 6ms} {disk 0s} {proposals 0s}] Num entries: 0. MustSync: false W0115 05:22:35.084131 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 749ms. Breakdown: [{proposals 647ms} {disk 101ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:23:43.698443 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 582ms. Breakdown: [{proposals 567ms} {disk 0s} {advance 0s}] Num entries: 0. MustSync: false W0115 05:23:50.898900 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 3.163s. Breakdown: [{proposals 1.17s} {advance 223ms} {disk 21ms}] Num entries: 0. MustSync: false W0115 05:24:00.537641 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 9.334s. Breakdown: [{disk 5.551s} {advance 1.022s} {proposals 921ms}] Num entries: 0. MustSync: false W0115 05:24:11.999113 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 1.36s. Breakdown: [{disk 1.193s} {proposals 2ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:24:14.840905 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 859ms. Breakdown: [{proposals 847ms} {disk 12ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:24:26.927457 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 755ms. Breakdown: [{proposals 341ms} {disk 168ms} {advance 58ms}] Num entries: 0. MustSync: false W0115 05:24:29.475420 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 1.575s. Breakdown: [{proposals 837ms} {disk 739ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:24:32.405517 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 754ms. Breakdown: [{disk 147ms} {proposals 30ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:24:37.416357 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 621ms. Breakdown: [{proposals 394ms} {disk 226ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:24:40.583693 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 1.01s. Breakdown: [{disk 716ms} {proposals 284ms} {advance 0s}] Num entries: 1. MustSync: true W0115 05:24:44.314053 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 821ms. Breakdown: [{proposals 460ms} {disk 10ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:24:46.207925 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 1.269s. Breakdown: [{advance 680ms} {disk 555ms} {proposals 11ms}] Num entries: 0. MustSync: false W0115 05:24:53.107659 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 1.791s. Breakdown: [{disk 361ms} {proposals 159ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:24:57.098458 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 1.63s. Breakdown: [{disk 1.628s} {proposals 2ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:25:01.648698 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 558ms. Breakdown: [{proposals 518ms} {disk 32ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:26:00.091128 16 groups.go:869] No membership update for 10s. Closing connection to Zero. I0115 05:26:00.387557 16 log.go:34] Block cache metrics: hit: 49360284 miss: 22908990 keys-added: 8942595 keys-updated: 114599 keys-evicted: 7660283 cost-added: 38605129290 cost-evicted: 33087106411 sets-dropped: 0 sets-rejected: 3681775 gets-dropped: 63040 gets-kept: 45449984 gets-total: 72269274 hit-ratio: 0.68 I0115 05:26:48.285912 16 groups.go:807] Got address of a Zero leader: 192.168.3.11:5080 E0115 05:27:33.817794 16 groups.go:773] While sending membership update: rpc error: code = Unimplemented desc = grpc: Decompressor is not installed for grpc-encoding "snappy" W0115 05:27:45.897247 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 5.167s. Breakdown: [{proposals 5.068s} {disk 68ms} {advance 0s}] Num entries: 0. MustSync: false I0115 05:27:47.500759 16 groups.go:821] Starting a new membership stream receive from 192.168.3.11:5080. I0115 05:27:47.861353 16 groups.go:838] Received first state update from Zero: counter:36743390 groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101868742 checksum:18290414373100373824 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101884062 checksum:2757678515920796499 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101872755 checksum:287855597640554238 > > zeros: > zeros: > zeros: > maxLeaseId:737100000 maxTxnTs:101900000 maxRaftId:17 removed: removed: removed: removed: removed: removed: removed: removed: removed: cid:"1582cefb-03ec-4014-b336-a4857ccc1344" license: W0115 05:27:47.942800 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 892ms. Breakdown: [{proposals 885ms} {advance 6ms} {disk 2ms}] Num entries: 6. MustSync: true W0115 05:28:26.533257 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 782ms. Breakdown: [{disk 620ms} {proposals 7ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:29:52.131373 16 groups.go:869] No membership update for 10s. Closing connection to Zero. W0115 05:29:58.765713 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 35.051s. Breakdown: [{disk 8.349s} {advance 5.892s} {proposals 313ms}] Num entries: 0. MustSync: false W0115 05:30:02.562874 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 2.31s. Breakdown: [{proposals 2.068s} {advance 237ms} {disk 0s}] Num entries: 0. MustSync: false I0115 05:30:03.276070 16 groups.go:807] Got address of a Zero leader: 192.168.3.11:5080 I0115 05:30:03.276521 16 groups.go:821] Starting a new membership stream receive from 192.168.3.11:5080. I0115 05:30:04.181040 16 groups.go:838] Received first state update from Zero: counter:36743390 groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101868742 checksum:18290414373100373824 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101884062 checksum:2757678515920796499 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101872755 checksum:287855597640554238 > > zeros: > zeros: > zeros: > maxLeaseId:737100000 maxTxnTs:101900000 maxRaftId:17 removed: removed: removed: removed: removed: removed: removed: removed: removed: cid:"1582cefb-03ec-4014-b336-a4857ccc1344" license: W0115 05:30:49.390830 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 1.086s. Breakdown: [{proposals 312ms} {disk 167ms} {advance 122ms}] Num entries: 0. MustSync: false W0115 05:30:51.292815 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 763ms. Breakdown: [{disk 671ms} {proposals 80ms} {advance 12ms}] Num entries: 0. MustSync: false W0115 05:30:54.362752 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 1.425s. Breakdown: [{disk 772ms} {proposals 652ms} {advance 0s}] Num entries: 0. MustSync: false I0115 05:30:53.912681 16 log.go:34] Block cache metrics: hit: 49360309 miss: 22909031 keys-added: 8942627 keys-updated: 114599 keys-evicted: 7660314 cost-added: 38605265303 cost-evicted: 33087240064 sets-dropped: 0 sets-rejected: 3681784 gets-dropped: 63040 gets-kept: 45449984 gets-total: 72269340 hit-ratio: 0.68 W0115 05:30:55.538526 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 526ms. Breakdown: [{disk 364ms} {proposals 0s} {advance 0s}] Num entries: 0. MustSync: false W0115 05:31:00.852333 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 814ms. Breakdown: [{disk 807ms} {proposals 0s} {advance 0s}] Num entries: 0. MustSync: false W0115 05:32:00.762062 16 groups.go:869] No membership update for 10s. Closing connection to Zero. I0115 05:32:15.984891 16 groups.go:807] Got address of a Zero leader: 192.168.3.11:5080 E0115 05:32:15.987193 16 groups.go:773] While sending membership update: rpc error: code = DeadlineExceeded desc = context deadline exceeded I0115 05:32:15.991723 16 groups.go:821] Starting a new membership stream receive from 192.168.3.11:5080. W0115 05:32:15.440120 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 32.057s. Breakdown: [{proposals 11.232s} {disk 6.233s} {advance 0s}] Num entries: 1. MustSync: true I0115 05:32:16.249036 16 groups.go:838] Received first state update from Zero: counter:36743390 groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101868742 checksum:18290414373100373824 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101884062 checksum:2757678515920796499 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101872755 checksum:287855597640554238 > > zeros: > zeros: > zeros: > maxLeaseId:737100000 maxTxnTs:101900000 maxRaftId:17 removed: removed: removed: removed: removed: removed: removed: removed: removed: cid:"1582cefb-03ec-4014-b336-a4857ccc1344" license: W0115 05:32:40.135135 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 2.765s. Breakdown: [{disk 1.266s} {proposals 0s} {advance 0s}] Num entries: 0. MustSync: false W0115 05:32:55.847713 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 921ms. Breakdown: [{disk 592ms} {proposals 0s} {advance 0s}] Num entries: 0. MustSync: false W0115 05:33:17.071782 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 3.389s. Breakdown: [{proposals 1.852s} {advance 109ms} {disk 14ms}] Num entries: 0. MustSync: false W0115 05:33:32.077349 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 8.499s. Breakdown: [{proposals 5.199s} {disk 560ms} {advance 1ms}] Num entries: 0. MustSync: false E0115 05:33:36.721363 16 groups.go:773] While sending membership update: rpc error: code = DeadlineExceeded desc = context deadline exceeded W0115 05:33:39.412071 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 7.334s. Breakdown: [{disk 3.825s} {proposals 3.505s} {advance 4ms}] Num entries: 1. MustSync: true W0115 05:33:54.043603 16 pool.go:267] Connection lost with 192.168.3.12:7085. Error: rpc error: code = Unavailable desc = transport is closing I0115 05:35:53.128119 16 log.go:34] Block cache metrics: hit: 49360326 miss: 22909044 keys-added: 8942637 keys-updated: 114600 keys-evicted: 7660324 cost-added: 38605308840 cost-evicted: 33087282928 sets-dropped: 0 sets-rejected: 3681786 gets-dropped: 63040 gets-kept: 45449984 gets-total: 72269370 hit-ratio: 0.68 I0115 05:40:53.128347 16 log.go:34] Block cache metrics: hit: 49360326 miss: 22909044 keys-added: 8942637 keys-updated: 114600 keys-evicted: 7660324 cost-added: 38605308840 cost-evicted: 33087282928 sets-dropped: 0 sets-rejected: 3681786 gets-dropped: 63040 gets-kept: 45449984 gets-total: 72269370 hit-ratio: 0.68 I0115 05:44:43.993481 16 run.go:744] Caught Ctrl-C. Terminating now (this may take a few seconds)... E0115 05:44:43.993548 16 run.go:393] GRPC listener canceled: accept tcp [::]:9087: use of closed network connection E0115 05:44:43.993766 16 server.go:24] error from cmux serve: accept tcp [::]:8087: use of closed network connection E0115 05:44:43.993791 16 server.go:66] Stopped taking more http(s) requests. Err: mux: listener closed I0115 05:44:43.993828 16 run.go:788] GRPC and HTTP stopped. I0115 05:44:43.993845 16 worker.go:120] Stopping group... I0115 05:44:43.993892 16 server.go:70] All http(s) requests finished. I0115 05:44:43.993930 16 groups.go:739] Closing sendMembershipUpdates I0115 05:44:43.993947 16 groups.go:786] Closing receiveMembershipUpdates I0115 05:44:43.993966 16 groups.go:885] Closing processOracleDeltaStream I0115 05:44:43.993982 16 worker.go:124] Updating RAFT state before shutting down... E0115 05:44:43.993987 16 groups.go:829] Unable to sync memberships. Error: rpc error: code = Canceled desc = context canceled. State: E0115 05:44:43.994023 16 groups.go:1143] Error during SubscribeForUpdates for prefix "\x00\x00\vdgraph.cors\x00": while receiving from stream: rpc error: code = Canceled desc = context canceled. closer err: context canceled I0115 05:44:43.994072 16 groups.go:1112] SubscribeForUpdates closing for prefix: "\x00\x00\vdgraph.cors\x00" I0115 05:44:43.994213 16 worker.go:129] Stopping node... I0115 05:44:43.994260 16 draft.go:207] Stopped all ongoing registered tasks. I0115 05:44:43.994260 16 draft.go:1021] Stopping node.Run I0115 05:44:43.994303 16 draft.go:124] Operation completed with id: opRollup I0115 05:44:43.994347 16 draft.go:1095] Raft node done. I0115 05:44:43.994371 16 worker.go:132] Stopping worker server... I0115 05:44:43.995000 16 run.go:795] worker stopped. E0115 05:44:43.995091 16 groups.go:1143] Error during SubscribeForUpdates for prefix "\x00\x00\x15dgraph.graphql.schema\x00": while receiving from stream: rpc error: code = Canceled desc = context canceled. closer err: context canceled I0115 05:44:43.995128 16 groups.go:1112] SubscribeForUpdates closing for prefix: "\x00\x00\x15dgraph.graphql.schema\x00" I0115 05:44:43.995144 16 run.go:798] adminCloser closed. I0115 05:44:43.995159 16 log.go:34] Lifetime L0 stalled for: 0s I0115 05:44:43.995176 16 server_state.go:162] Exiting fillTimestampRequests W0115 05:44:44.292103 16 pool.go:267] Connection lost with 192.168.3.12:7086. Error: rpc error: code = Unavailable desc = transport is closing I0115 05:44:45.111346 16 log.go:34] Level 0 [ ]: NumTables: 04. Size: 87 MiB of 0 B. Score: 0.00->0.00 Target FileSize: 64 MiB Level 1 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 Target FileSize: 2.0 MiB Level 2 [B]: NumTables: 05. Size: 9.7 MiB of 10 MiB. Score: 0.00->0.00 Target FileSize: 2.0 MiB Level 3 [ ]: NumTables: 23. Size: 48 MiB of 49 MiB. Score: 0.00->0.00 Target FileSize: 4.0 MiB Level 4 [ ]: NumTables: 85. Size: 481 MiB of 488 MiB. Score: 0.00->0.00 Target FileSize: 8.0 MiB Level 5 [ ]: NumTables: 186. Size: 1.5 GiB of 4.8 GiB. Score: 0.00->0.00 Target FileSize: 16 MiB Level 6 [ ]: NumTables: 799. Size: 48 GiB of 48 GiB. Score: 0.00->0.00 Target FileSize: 32 MiB Level Done I0115 05:44:47.335089 16 run.go:802] worker.State disposed. I0115 05:44:47.335125 16 run.go:805] updaters closed. I0115 05:44:47.335138 16 run.go:807] Server shutdown. Bye! [Sentry] 2021/01/15 05:44:47 Buffer flushed successfully. [Decoder]: Using assembly version of decoder Page Size: 4096 [Sentry] 2021/01/15 05:47:14 Integration installed: ContextifyFrames [Sentry] 2021/01/15 05:47:14 Integration installed: Environment [Sentry] 2021/01/15 05:47:14 Integration installed: Modules [Sentry] 2021/01/15 05:47:14 Integration installed: IgnoreErrors [Decoder]: Using assembly version of decoder Page Size: 4096 [Sentry] 2021/01/15 05:47:14 Integration installed: ContextifyFrames [Sentry] 2021/01/15 05:47:14 Integration installed: Environment [Sentry] 2021/01/15 05:47:14 Integration installed: Modules [Sentry] 2021/01/15 05:47:14 Integration installed: IgnoreErrors I0115 05:47:14.750479 19 sentry_integration.go:48] This instance of Dgraph will send anonymous reports of panics back to Dgraph Labs via Sentry. No confidential information is sent. These reports help improve Dgraph. To opt-out, restart your instance with the --enable_sentry=false flag. For more info, see https://dgraph.io/docs/howto/#data-handling. W0115 05:47:14.765932 19 run.go:573] --lru_mb is deprecated, use --cache_mb instead I0115 05:47:14.767375 19 util_ee.go:126] KeyReader instantiated of type I0115 05:47:14.999349 19 init.go:107]