I0115 01:34:53.128682 16 log.go:34] Block cache metrics: hit: 44266825 miss: 22116337 keys-added: 8589989 keys-updated: 50801 keys-evicted: 7349075 cost-added: 37097005378 cost-evicted: 31746782836 sets-dropped: 0 sets-rejected: 3305527 gets-dropped: 45824 gets-kept: 41540864 gets-total: 66383162 hit-ratio: 0.67 I0115 01:34:53.497335 16 log.go:34] Processed 0 entries in 0 loops I0115 01:34:53.497386 16 log.go:34] Total entries: 3285. Moved: 0 I0115 01:34:53.497393 16 log.go:34] Removing fid: 2118 I0115 01:34:53.672483 16 log.go:34] Found value log max discard fid: 2110 discard: 1073823051 I0115 01:34:53.672513 16 log.go:34] Rewriting fid: 2110 I0115 01:34:59.155270 16 log.go:34] Processed 0 entries in 0 loops I0115 01:34:59.155334 16 log.go:34] Total entries: 3288. Moved: 0 I0115 01:34:59.155344 16 log.go:34] Removing fid: 2110 I0115 01:34:59.347903 16 log.go:34] Found value log max discard fid: 2106 discard: 1073822561 I0115 01:34:59.347940 16 log.go:34] Rewriting fid: 2106 I0115 01:35:04.709496 16 log.go:34] Processed 0 entries in 0 loops I0115 01:35:04.709543 16 log.go:34] Total entries: 3286. Moved: 0 I0115 01:35:04.709557 16 log.go:34] Removing fid: 2106 I0115 01:35:04.892720 16 log.go:34] Found value log max discard fid: 2108 discard: 1073818886 I0115 01:35:04.892778 16 log.go:34] Rewriting fid: 2108 I0115 01:35:10.296762 16 log.go:34] Processed 0 entries in 0 loops I0115 01:35:10.296792 16 log.go:34] Total entries: 3278. Moved: 0 I0115 01:35:10.296797 16 log.go:34] Removing fid: 2108 I0115 01:35:10.477432 16 log.go:34] Found value log max discard fid: 2093 discard: 1073803092 I0115 01:35:10.477473 16 log.go:34] Rewriting fid: 2093 I0115 01:35:15.744482 16 log.go:34] Processed 0 entries in 0 loops I0115 01:35:15.744515 16 log.go:34] Total entries: 3285. Moved: 0 I0115 01:35:15.744522 16 log.go:34] Removing fid: 2093 I0115 01:35:15.941600 16 log.go:34] Found value log max discard fid: 2123 discard: 1073795734 I0115 01:35:15.941652 16 log.go:34] Rewriting fid: 2123 I0115 01:35:21.379997 16 log.go:34] Processed 4 entries in 1 loops I0115 01:35:21.380019 16 log.go:34] Total entries: 3293. Moved: 4 I0115 01:35:21.380023 16 log.go:34] Removing fid: 2123 I0115 01:35:21.566565 16 log.go:34] Found value log max discard fid: 2111 discard: 1073774725 I0115 01:35:21.566627 16 log.go:34] Rewriting fid: 2111 I0115 01:35:26.954622 16 log.go:34] Processed 0 entries in 0 loops I0115 01:35:26.954651 16 log.go:34] Total entries: 3264. Moved: 0 I0115 01:35:26.954657 16 log.go:34] Removing fid: 2111 I0115 01:35:27.129284 16 log.go:34] Found value log max discard fid: 2122 discard: 1073771929 I0115 01:35:27.129321 16 log.go:34] Rewriting fid: 2122 I0115 01:35:32.513404 16 log.go:34] Processed 0 entries in 0 loops I0115 01:35:32.513429 16 log.go:34] Total entries: 3282. Moved: 0 I0115 01:35:32.513434 16 log.go:34] Removing fid: 2122 I0115 01:35:32.693392 16 log.go:34] Found value log max discard fid: 2084 discard: 1073767697 I0115 01:35:32.693424 16 log.go:34] Rewriting fid: 2084 I0115 01:35:37.953807 16 log.go:34] Processed 0 entries in 0 loops I0115 01:35:37.953847 16 log.go:34] Total entries: 3287. Moved: 0 I0115 01:35:37.953861 16 log.go:34] Removing fid: 2084 I0115 01:35:38.143139 16 log.go:34] Found value log max discard fid: 2102 discard: 1073752208 I0115 01:35:38.143191 16 log.go:34] Rewriting fid: 2102 I0115 01:35:43.489982 16 log.go:34] Processed 0 entries in 0 loops I0115 01:35:43.490008 16 log.go:34] Total entries: 3280. Moved: 0 I0115 01:35:43.490016 16 log.go:34] Removing fid: 2102 I0115 01:35:43.667831 16 log.go:34] Found value log max discard fid: 2092 discard: 1073745817 I0115 01:35:43.667858 16 log.go:34] Rewriting fid: 2092 I0115 01:35:49.176800 16 log.go:34] Processed 0 entries in 0 loops I0115 01:35:49.176838 16 log.go:34] Total entries: 3285. Moved: 0 I0115 01:35:49.176846 16 log.go:34] Removing fid: 2092 I0115 01:35:49.400492 16 log.go:34] Found value log max discard fid: 2078 discard: 1073744639 I0115 01:35:49.400569 16 log.go:34] Rewriting fid: 2078 I0115 01:35:54.722892 16 log.go:34] Processed 0 entries in 0 loops I0115 01:35:54.722921 16 log.go:34] Total entries: 3274. Moved: 0 I0115 01:35:54.722928 16 log.go:34] Removing fid: 2078 I0115 01:35:54.916600 16 log.go:34] Found value log max discard fid: 2088 discard: 1073743019 I0115 01:35:54.916632 16 log.go:34] Rewriting fid: 2088 I0115 01:36:00.207200 16 log.go:34] Processed 1 entries in 1 loops I0115 01:36:00.207229 16 log.go:34] Total entries: 3286. Moved: 1 I0115 01:36:00.207237 16 log.go:34] Removing fid: 2088 I0115 01:36:00.380826 16 log.go:34] Found value log max discard fid: 2089 discard: 1073466898 I0115 01:36:00.380880 16 log.go:34] Rewriting fid: 2089 I0115 01:36:05.704771 16 log.go:34] Processed 134 entries in 1 loops I0115 01:36:05.704806 16 log.go:34] Total entries: 3413. Moved: 134 I0115 01:36:05.704811 16 log.go:34] Removing fid: 2089 I0115 01:39:53.128221 16 log.go:34] Block cache metrics: hit: 44396129 miss: 22116409 keys-added: 8590056 keys-updated: 50806 keys-evicted: 7349075 cost-added: 37097290956 cost-evicted: 31746782836 sets-dropped: 0 sets-rejected: 3305527 gets-dropped: 45824 gets-kept: 41665344 gets-total: 66512538 hit-ratio: 0.67 I0115 01:44:53.128195 16 log.go:34] Block cache metrics: hit: 44396169 miss: 22116442 keys-added: 8590086 keys-updated: 50809 keys-evicted: 7349075 cost-added: 37097420914 cost-evicted: 31746782836 sets-dropped: 0 sets-rejected: 3305527 gets-dropped: 45824 gets-kept: 41665344 gets-total: 66512611 hit-ratio: 0.67 I0115 01:49:53.128207 16 log.go:34] Block cache metrics: hit: 44396211 miss: 22116465 keys-added: 8590108 keys-updated: 50810 keys-evicted: 7349075 cost-added: 37097516185 cost-evicted: 31746782836 sets-dropped: 0 sets-rejected: 3305527 gets-dropped: 45824 gets-kept: 41665344 gets-total: 66512676 hit-ratio: 0.67 I0115 01:54:53.128127 16 log.go:34] Block cache metrics: hit: 44396248 miss: 22116485 keys-added: 8590127 keys-updated: 50811 keys-evicted: 7349075 cost-added: 37097598464 cost-evicted: 31746782836 sets-dropped: 0 sets-rejected: 3305527 gets-dropped: 45824 gets-kept: 41665344 gets-total: 66512733 hit-ratio: 0.67 I0115 01:59:53.128229 16 log.go:34] Block cache metrics: hit: 44396307 miss: 22116551 keys-added: 8590192 keys-updated: 50812 keys-evicted: 7349075 cost-added: 37097876517 cost-evicted: 31746782836 sets-dropped: 0 sets-rejected: 3305527 gets-dropped: 45824 gets-kept: 41665344 gets-total: 66512858 hit-ratio: 0.67 I0115 02:04:53.128143 16 log.go:34] Block cache metrics: hit: 44396316 miss: 22116566 keys-added: 8590207 keys-updated: 50812 keys-evicted: 7349075 cost-added: 37097940402 cost-evicted: 31746782836 sets-dropped: 0 sets-rejected: 3305527 gets-dropped: 45824 gets-kept: 41665344 gets-total: 66512882 hit-ratio: 0.67 I0115 02:09:53.128176 16 log.go:34] Block cache metrics: hit: 44396368 miss: 22116605 keys-added: 8590244 keys-updated: 50814 keys-evicted: 7349075 cost-added: 37098100202 cost-evicted: 31746782836 sets-dropped: 0 sets-rejected: 3305527 gets-dropped: 45824 gets-kept: 41665344 gets-total: 66512973 hit-ratio: 0.67 I0115 02:14:53.128165 16 log.go:34] Block cache metrics: hit: 44396442 miss: 22116663 keys-added: 8590302 keys-updated: 50814 keys-evicted: 7349075 cost-added: 37098351210 cost-evicted: 31746782836 sets-dropped: 0 sets-rejected: 3305527 gets-dropped: 45824 gets-kept: 41665344 gets-total: 66513105 hit-ratio: 0.67 I0115 02:19:53.128141 16 log.go:34] Block cache metrics: hit: 44396462 miss: 22116703 keys-added: 8590340 keys-updated: 50816 keys-evicted: 7349075 cost-added: 37098514984 cost-evicted: 31746782836 sets-dropped: 0 sets-rejected: 3305527 gets-dropped: 45824 gets-kept: 41665344 gets-total: 66513165 hit-ratio: 0.67 I0115 02:24:53.128167 16 log.go:34] Block cache metrics: hit: 44396496 miss: 22116744 keys-added: 8590381 keys-updated: 50816 keys-evicted: 7349075 cost-added: 37098691250 cost-evicted: 31746782836 sets-dropped: 0 sets-rejected: 3305527 gets-dropped: 45824 gets-kept: 41665344 gets-total: 66513240 hit-ratio: 0.67 I0115 02:29:53.128124 16 log.go:34] Block cache metrics: hit: 44396523 miss: 22116782 keys-added: 8590419 keys-updated: 50816 keys-evicted: 7349075 cost-added: 37098855184 cost-evicted: 31746782836 sets-dropped: 0 sets-rejected: 3305527 gets-dropped: 45824 gets-kept: 41665344 gets-total: 66513305 hit-ratio: 0.67 I0115 02:34:53.128130 16 log.go:34] Block cache metrics: hit: 44396555 miss: 22116822 keys-added: 8590459 keys-updated: 50816 keys-evicted: 7349075 cost-added: 37099026728 cost-evicted: 31746782836 sets-dropped: 0 sets-rejected: 3305527 gets-dropped: 45824 gets-kept: 41665344 gets-total: 66513377 hit-ratio: 0.67 I0115 02:39:53.128129 16 log.go:34] Block cache metrics: hit: 44396627 miss: 22116875 keys-added: 8590510 keys-updated: 50818 keys-evicted: 7349075 cost-added: 37099246622 cost-evicted: 31746782836 sets-dropped: 0 sets-rejected: 3305527 gets-dropped: 45824 gets-kept: 41665344 gets-total: 66513502 hit-ratio: 0.67 I0115 02:41:53.404697 16 draft.go:606] Creating snapshot at Index: 88223031, ReadTs: 100806534 I0115 02:44:53.129567 16 log.go:34] Block cache metrics: hit: 44566310 miss: 22125165 keys-added: 8597399 keys-updated: 52219 keys-evicted: 7349075 cost-added: 37128652376 cost-evicted: 31746782836 sets-dropped: 0 sets-rejected: 3305527 gets-dropped: 45824 gets-kept: 41745472 gets-total: 66691475 hit-ratio: 0.67 I0115 02:49:53.128387 16 log.go:34] Block cache metrics: hit: 44792117 miss: 22135895 keys-added: 8606642 keys-updated: 53706 keys-evicted: 7349075 cost-added: 37168055480 cost-evicted: 31746782836 sets-dropped: 0 sets-rejected: 3305527 gets-dropped: 45824 gets-kept: 41890368 gets-total: 66928012 hit-ratio: 0.67 I0115 02:53:53.440651 16 draft.go:606] Creating snapshot at Index: 88245739, ReadTs: 100839527 I0115 02:54:53.170409 16 log.go:34] Block cache metrics: hit: 44955842 miss: 22142556 keys-added: 8612468 keys-updated: 54541 keys-evicted: 7349075 cost-added: 37192844289 cost-evicted: 31746782836 sets-dropped: 0 sets-rejected: 3305527 gets-dropped: 45824 gets-kept: 41978816 gets-total: 67098398 hit-ratio: 0.67 I0115 02:54:53.590594 16 draft.go:606] Creating snapshot at Index: 88265157, ReadTs: 100870800 I0115 02:55:53.527532 16 draft.go:606] Creating snapshot at Index: 88290741, ReadTs: 100908751 I0115 02:56:53.532200 16 draft.go:606] Creating snapshot at Index: 88315465, ReadTs: 100944623 I0115 02:57:53.387055 16 draft.go:606] Creating snapshot at Index: 88330761, ReadTs: 100966311 I0115 02:58:53.429377 16 draft.go:606] Creating snapshot at Index: 88357215, ReadTs: 101006516 I0115 02:59:53.128120 16 log.go:34] Block cache metrics: hit: 45078844 miss: 22150164 keys-added: 8618471 keys-updated: 56145 keys-evicted: 7349075 cost-added: 37218436287 cost-evicted: 31746782836 sets-dropped: 0 sets-rejected: 3305527 gets-dropped: 45824 gets-kept: 42034176 gets-total: 67229008 hit-ratio: 0.67 I0115 02:59:53.334331 16 draft.go:606] Creating snapshot at Index: 88370386, ReadTs: 101026623 I0115 03:00:53.637454 16 draft.go:606] Creating snapshot at Index: 88388042, ReadTs: 101051774 I0115 03:01:53.572096 16 draft.go:606] Creating snapshot at Index: 88399286, ReadTs: 101069296 I0115 03:02:53.427998 16 draft.go:606] Creating snapshot at Index: 88411146, ReadTs: 101088228 I0115 03:04:53.128925 16 log.go:34] Block cache metrics: hit: 45252700 miss: 22160418 keys-added: 8627137 keys-updated: 57733 keys-evicted: 7349075 cost-added: 37255264189 cost-evicted: 31746782836 sets-dropped: 0 sets-rejected: 3305527 gets-dropped: 45824 gets-kept: 42129280 gets-total: 67413118 hit-ratio: 0.67 I0115 03:04:53.485186 16 draft.go:606] Creating snapshot at Index: 88435102, ReadTs: 101124604 I0115 03:06:53.384178 16 draft.go:606] Creating snapshot at Index: 88457434, ReadTs: 101159336 I0115 03:08:53.371929 16 draft.go:606] Creating snapshot at Index: 88473682, ReadTs: 101184263 I0115 03:09:53.128172 16 log.go:34] Block cache metrics: hit: 45404929 miss: 22193607 keys-added: 8634064 keys-updated: 59551 keys-evicted: 7353719 cost-added: 37284784595 cost-evicted: 31766761556 sets-dropped: 0 sets-rejected: 3329972 gets-dropped: 45824 gets-kept: 42223296 gets-total: 67598536 hit-ratio: 0.67 I0115 03:09:53.399160 16 draft.go:606] Creating snapshot at Index: 88486982, ReadTs: 101204402 I0115 03:11:53.377115 16 draft.go:606] Creating snapshot at Index: 88503999, ReadTs: 101228894 I0115 03:12:53.346916 16 draft.go:606] Creating snapshot at Index: 88516319, ReadTs: 101246516 I0115 03:13:53.375790 16 draft.go:606] Creating snapshot at Index: 88535960, ReadTs: 101277805 I0115 03:14:53.128405 16 log.go:34] Block cache metrics: hit: 45539476 miss: 22234101 keys-added: 8640060 keys-updated: 61125 keys-evicted: 7359667 cost-added: 37310372527 cost-evicted: 31792347449 sets-dropped: 0 sets-rejected: 3362896 gets-dropped: 45824 gets-kept: 42314944 gets-total: 67773577 hit-ratio: 0.67 I0115 03:15:53.367469 16 draft.go:606] Creating snapshot at Index: 88551021, ReadTs: 101300028 I0115 03:17:53.331069 16 draft.go:606] Creating snapshot at Index: 88570260, ReadTs: 101327026 I0115 03:19:53.128086 16 log.go:34] Block cache metrics: hit: 45696536 miss: 22286544 keys-added: 8648051 keys-updated: 63125 keys-evicted: 7367601 cost-added: 37344505351 cost-evicted: 31826481988 sets-dropped: 0 sets-rejected: 3405348 gets-dropped: 45824 gets-kept: 42435456 gets-total: 67983080 hit-ratio: 0.67 I0115 03:19:53.323805 16 draft.go:606] Creating snapshot at Index: 88582806, ReadTs: 101344704 I0115 03:21:53.494820 16 draft.go:606] Creating snapshot at Index: 88600875, ReadTs: 101370356 I0115 03:24:53.128112 16 log.go:34] Block cache metrics: hit: 45848920 miss: 22339125 keys-added: 8655644 keys-updated: 64977 keys-evicted: 7375130 cost-added: 37376898162 cost-evicted: 31858873296 sets-dropped: 0 sets-rejected: 3448484 gets-dropped: 45824 gets-kept: 42542656 gets-total: 68188045 hit-ratio: 0.67 I0115 03:24:53.375610 16 draft.go:606] Creating snapshot at Index: 88615996, ReadTs: 101391100 I0115 03:27:53.306681 16 draft.go:606] Creating snapshot at Index: 88629569, ReadTs: 101410147 I0115 03:29:53.155999 16 log.go:34] Block cache metrics: hit: 46002262 miss: 22396428 keys-added: 8662739 keys-updated: 66819 keys-evicted: 7382164 cost-added: 37407169406 cost-evicted: 31889145333 sets-dropped: 0 sets-rejected: 3496850 gets-dropped: 45824 gets-kept: 42663360 gets-total: 68398702 hit-ratio: 0.67 I0115 03:30:53.298254 16 draft.go:606] Creating snapshot at Index: 88642957, ReadTs: 101429551 I0115 03:33:53.454292 16 draft.go:606] Creating snapshot at Index: 88663139, ReadTs: 101458219 I0115 03:34:53.128868 16 log.go:34] Block cache metrics: hit: 46144414 miss: 22453034 keys-added: 8669277 keys-updated: 68437 keys-evicted: 7388654 cost-added: 37435093504 cost-evicted: 31917069572 sets-dropped: 0 sets-rejected: 3545300 gets-dropped: 45824 gets-kept: 42774912 gets-total: 68597448 hit-ratio: 0.67 I0115 03:35:53.289378 16 draft.go:606] Creating snapshot at Index: 88673668, ReadTs: 101471351 I0115 03:38:53.287903 16 draft.go:606] Creating snapshot at Index: 88684410, ReadTs: 101483808 I0115 03:39:53.128136 16 log.go:34] Block cache metrics: hit: 46266336 miss: 22500891 keys-added: 8674710 keys-updated: 69494 keys-evicted: 7394039 cost-added: 37458273489 cost-evicted: 31940248684 sets-dropped: 0 sets-rejected: 3586667 gets-dropped: 45824 gets-kept: 42877760 gets-total: 68767227 hit-ratio: 0.67 I0115 03:44:53.128155 16 log.go:34] Block cache metrics: hit: 46266381 miss: 22500950 keys-added: 8674712 keys-updated: 69494 keys-evicted: 7394041 cost-added: 37458282253 cost-evicted: 31940257388 sets-dropped: 0 sets-rejected: 3586724 gets-dropped: 45824 gets-kept: 42877824 gets-total: 68767331 hit-ratio: 0.67 I0115 03:49:53.128145 16 log.go:34] Block cache metrics: hit: 46266413 miss: 22501044 keys-added: 8674723 keys-updated: 69495 keys-evicted: 7394052 cost-added: 37458330264 cost-evicted: 31940304782 sets-dropped: 0 sets-rejected: 3586806 gets-dropped: 45824 gets-kept: 42877888 gets-total: 68767457 hit-ratio: 0.67 I0115 03:54:53.173555 16 log.go:34] Block cache metrics: hit: 46282058 miss: 22509713 keys-added: 8675726 keys-updated: 70181 keys-evicted: 7395047 cost-added: 37462616425 cost-evicted: 31944591446 sets-dropped: 0 sets-rejected: 3593779 gets-dropped: 45824 gets-kept: 42892928 gets-total: 68791771 hit-ratio: 0.67 I0115 03:56:53.829772 16 draft.go:606] Creating snapshot at Index: 88702000, ReadTs: 101517647 I0115 03:58:53.312888 16 draft.go:606] Creating snapshot at Index: 88716977, ReadTs: 101546616 I0115 03:59:53.128129 16 log.go:34] Block cache metrics: hit: 46762512 miss: 22614733 keys-added: 8699602 keys-updated: 83721 keys-evicted: 7418682 cost-added: 37564419492 cost-evicted: 32046393168 sets-dropped: 0 sets-rejected: 3661390 gets-dropped: 50624 gets-kept: 43355328 gets-total: 69377245 hit-ratio: 0.67 I0115 03:59:53.404732 16 draft.go:606] Creating snapshot at Index: 88728792, ReadTs: 101570279 I0115 04:01:53.386972 16 draft.go:606] Creating snapshot at Index: 88745680, ReadTs: 101601722 I0115 04:03:53.311571 16 draft.go:606] Creating snapshot at Index: 88757310, ReadTs: 101621209 I0115 04:04:53.128132 16 log.go:34] Block cache metrics: hit: 47184437 miss: 22643464 keys-added: 8719882 keys-updated: 91762 keys-evicted: 7438730 cost-added: 37650783807 cost-evicted: 32132761615 sets-dropped: 0 sets-rejected: 3661800 gets-dropped: 51392 gets-kept: 43686656 gets-total: 69827901 hit-ratio: 0.68 I0115 04:07:53.512353 16 draft.go:606] Creating snapshot at Index: 88769239, ReadTs: 101635889 I0115 04:09:53.128242 16 log.go:34] Block cache metrics: hit: 47405524 miss: 22655361 keys-added: 8730020 keys-updated: 93244 keys-evicted: 7448808 cost-added: 37694201545 cost-evicted: 32176175801 sets-dropped: 0 sets-rejected: 3662076 gets-dropped: 51392 gets-kept: 43838400 gets-total: 70060885 hit-ratio: 0.68 I0115 04:11:53.301323 16 draft.go:606] Creating snapshot at Index: 88782111, ReadTs: 101652432 I0115 04:14:53.128076 16 log.go:34] Block cache metrics: hit: 47622060 miss: 22665919 keys-added: 8738822 keys-updated: 94737 keys-evicted: 7457514 cost-added: 37731709198 cost-evicted: 32213682931 sets-dropped: 0 sets-rejected: 3662339 gets-dropped: 51392 gets-kept: 43988288 gets-total: 70287979 hit-ratio: 0.68 I0115 04:15:53.295956 16 draft.go:606] Creating snapshot at Index: 88792653, ReadTs: 101664589 I0115 04:19:53.128149 16 log.go:34] Block cache metrics: hit: 47843088 miss: 22676514 keys-added: 8747812 keys-updated: 96036 keys-evicted: 7466403 cost-added: 37770006768 cost-evicted: 32251983182 sets-dropped: 0 sets-rejected: 3662645 gets-dropped: 51392 gets-kept: 44161792 gets-total: 70519602 hit-ratio: 0.68 I0115 04:19:53.892900 16 draft.go:606] Creating snapshot at Index: 88803859, ReadTs: 101677604 I0115 04:24:53.128251 16 log.go:34] Block cache metrics: hit: 47863409 miss: 22677317 keys-added: 8748511 keys-updated: 96107 keys-evicted: 7467096 cost-added: 37772989560 cost-evicted: 32254966509 sets-dropped: 0 sets-rejected: 3662678 gets-dropped: 51392 gets-kept: 44179968 gets-total: 70540726 hit-ratio: 0.68 I0115 04:29:53.128505 16 log.go:34] Block cache metrics: hit: 47900009 miss: 22680390 keys-added: 8750336 keys-updated: 97287 keys-evicted: 7468897 cost-added: 37780755082 cost-evicted: 32262729512 sets-dropped: 0 sets-rejected: 3662746 gets-dropped: 51904 gets-kept: 44209600 gets-total: 70580399 hit-ratio: 0.68 I0115 04:30:53.396196 16 draft.go:606] Creating snapshot at Index: 88814019, ReadTs: 101695846 I0115 04:32:53.988778 16 draft.go:606] Creating snapshot at Index: 88828543, ReadTs: 101723401 I0115 04:34:53.128355 16 log.go:34] Block cache metrics: hit: 48352315 miss: 22752462 keys-added: 8809950 keys-updated: 106843 keys-evicted: 7528178 cost-added: 38036154531 cost-evicted: 32518129795 sets-dropped: 0 sets-rejected: 3665648 gets-dropped: 57280 gets-kept: 44622720 gets-total: 71104777 hit-ratio: 0.68 I0115 04:34:53.470987 16 draft.go:606] Creating snapshot at Index: 88843774, ReadTs: 101751731 I0115 04:36:53.433496 16 draft.go:606] Creating snapshot at Index: 88860531, ReadTs: 101783464 I0115 04:38:53.476303 16 draft.go:606] Creating snapshot at Index: 88875230, ReadTs: 101808887 I0115 04:39:53.132060 16 log.go:34] Block cache metrics: hit: 48729517 miss: 22860707 keys-added: 8903233 keys-updated: 112070 keys-evicted: 7621107 cost-added: 38436414392 cost-evicted: 32918390274 sets-dropped: 0 sets-rejected: 3675383 gets-dropped: 60416 gets-kept: 44992192 gets-total: 71590224 hit-ratio: 0.68 I0115 04:41:53.290466 16 draft.go:606] Creating snapshot at Index: 88885507, ReadTs: 101823407 I0115 04:44:53.132091 16 log.go:34] Block cache metrics: hit: 48788940 miss: 22875217 keys-added: 8915454 keys-updated: 112595 keys-evicted: 7633287 cost-added: 38488869752 cost-evicted: 32970845201 sets-dropped: 0 sets-rejected: 3677147 gets-dropped: 60416 gets-kept: 45040704 gets-total: 71664157 hit-ratio: 0.68 I0115 04:46:53.734516 16 draft.go:606] Creating snapshot at Index: 88895787, ReadTs: 101840727 I0115 04:49:53.128132 16 log.go:34] Block cache metrics: hit: 48897064 miss: 22904526 keys-added: 8939411 keys-updated: 113977 keys-evicted: 7657145 cost-added: 38591619974 cost-evicted: 33073594951 sets-dropped: 0 sets-rejected: 3681117 gets-dropped: 63040 gets-kept: 45147200 gets-total: 71801590 hit-ratio: 0.68 I0115 04:51:53.638156 16 draft.go:606] Creating snapshot at Index: 88913314, ReadTs: 101872755 I0115 04:54:53.128215 16 log.go:34] Block cache metrics: hit: 49094831 miss: 22906770 keys-added: 8940961 keys-updated: 114336 keys-evicted: 7658676 cost-added: 38598211569 cost-evicted: 33080185314 sets-dropped: 0 sets-rejected: 3681452 gets-dropped: 63040 gets-kept: 45262656 gets-total: 72001601 hit-ratio: 0.68 I0115 04:59:53.165073 16 log.go:34] Block cache metrics: hit: 49292869 miss: 22908442 keys-added: 8942199 keys-updated: 114529 keys-evicted: 7659894 cost-added: 38603452377 cost-evicted: 33085430191 sets-dropped: 0 sets-rejected: 3681693 gets-dropped: 63040 gets-kept: 45396480 gets-total: 72201311 hit-ratio: 0.68 W0115 05:01:07.553939 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 1.067s. Breakdown: [{disk 1.063s} {proposals 4ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:01:11.963037 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 773ms. Breakdown: [{disk 372ms} {proposals 0s} {advance 0s}] Num entries: 0. MustSync: false W0115 05:01:21.312065 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 669ms. Breakdown: [{disk 456ms} {proposals 0s} {advance 0s}] Num entries: 0. MustSync: false W0115 05:01:53.792344 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 995ms. Breakdown: [{disk 994ms} {advance 1ms} {proposals 0s}] Num entries: 1. MustSync: true E0115 05:01:54.183543 16 groups.go:937] Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader I0115 05:01:55.299609 16 groups.go:893] Leader idx=0x11 of group=3 is connecting to Zero for txn updates I0115 05:01:55.331723 16 groups.go:905] Got Zero leader: 192.168.3.12:5080 W0115 05:01:56.959485 16 server_state.go:205] Error while retrieving timestamps: rpc error: code = Unknown desc = Assigning IDs is only allowed on leader. with delay: 10ms. Will retry... E0115 05:01:57.017707 16 groups.go:937] Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader I0115 05:01:57.821979 16 groups.go:893] Leader idx=0x11 of group=3 is connecting to Zero for txn updates I0115 05:01:57.823954 16 groups.go:905] Got Zero leader: 192.168.3.11:5080 W0115 05:02:01.933965 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 835ms. Breakdown: [{proposals 705ms} {disk 1ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:02:04.359761 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 972ms. Breakdown: [{proposals 815ms} {advance 133ms} {disk 20ms}] Num entries: 0. MustSync: false I0115 05:02:50.586866 16 log.go:34] 11 [logterm: 7, index: 88920945, vote: 11] cast MsgPreVote for 7 [logterm: 7, index: 88920945] at term 7 I0115 05:02:50.827620 16 log.go:34] 11 [term: 7] received a MsgVote message with higher term from 7 [term: 8] I0115 05:02:50.827734 16 log.go:34] 11 became follower at term 8 I0115 05:02:50.827766 16 log.go:34] 11 [logterm: 7, index: 88920945, vote: 0] cast MsgVote for 7 [logterm: 7, index: 88920945] at term 8 I0115 05:02:50.827795 16 log.go:34] raft.node: 11 lost leader 11 at term 8 I0115 05:02:50.827809 16 log.go:34] 11 [logterm: 7, index: 88920945, vote: 7] rejected MsgPreVote from 8 [logterm: 7, index: 88920945] at term 8 W0115 05:02:50.585870 16 groups.go:869] No membership update for 10s. Closing connection to Zero. W0115 05:02:50.586226 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 40.329s. Breakdown: [{disk 40.329s} {proposals 0s} {advance 0s}] Num entries: 0. MustSync: false I0115 05:02:50.837232 16 log.go:34] 11 [logterm: 7, index: 88920945, vote: 7] rejected MsgVote from 8 [logterm: 7, index: 88920945] at term 8 I0115 05:02:50.837311 16 log.go:34] 11 [logterm: 7, index: 88920945, vote: 7] cast MsgPreVote for 8 [logterm: 7, index: 88920945] at term 8 I0115 05:02:50.837336 16 log.go:34] 11 [term: 8] received a MsgAppResp message with higher term from 7 [term: 9] I0115 05:02:50.837401 16 log.go:34] 11 became follower at term 9 I0115 05:02:50.839005 16 groups.go:807] Got address of a Zero leader: 192.168.3.11:5080 I0115 05:02:50.839350 16 groups.go:821] Starting a new membership stream receive from 192.168.3.11:5080. E0115 05:02:50.836872 16 groups.go:829] Unable to sync memberships. Error: rpc error: code = Canceled desc = context canceled. State: I0115 05:02:50.929081 16 groups.go:838] Received first state update from Zero: counter:36743255 groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101868742 checksum:18290414373100373824 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101884062 checksum:2757678515920796499 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101872755 checksum:287855597640554238 > > zeros: > zeros: > zeros: > maxLeaseId:737090000 maxTxnTs:101900000 maxRaftId:17 removed: removed: removed: removed: removed: removed: removed: removed: removed: cid:"1582cefb-03ec-4014-b336-a4857ccc1344" license: I0115 05:02:50.934823 16 log.go:34] 11 [logterm: 7, index: 88920945, vote: 0] cast MsgVote for 8 [logterm: 7, index: 88920945] at term 9 I0115 05:02:50.986236 16 log.go:34] raft.node: 11 elected leader 8 at term 9 E0115 05:02:57.851066 16 groups.go:1000] No longer the leader of group 3. Exiting E0115 05:02:57.851212 16 groups.go:937] Error in oracle delta stream. Error: rpc error: code = Canceled desc = context canceled I0115 05:04:53.176769 16 log.go:34] Block cache metrics: hit: 49357512 miss: 22908926 keys-added: 8942542 keys-updated: 114597 keys-evicted: 7660230 cost-added: 38604902650 cost-evicted: 33086877796 sets-dropped: 0 sets-rejected: 3681766 gets-dropped: 63040 gets-kept: 45449152 gets-total: 72266438 hit-ratio: 0.68 W0115 05:05:45.993845 16 groups.go:869] No membership update for 10s. Closing connection to Zero. E0115 05:06:16.362005 16 groups.go:773] While sending membership update: rpc error: code = DeadlineExceeded desc = context deadline exceeded W0115 05:06:46.276630 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 29.982s. Breakdown: [{disk 10.568s} {proposals 6.922s} {advance 0s}] Num entries: 0. MustSync: false I0115 05:07:05.831906 16 groups.go:807] Got address of a Zero leader: 192.168.3.11:5080 W0115 05:07:05.985084 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 3.379s. Breakdown: [{proposals 1.699s} {advance 1.65s} {disk 0s}] Num entries: 0. MustSync: false W0115 05:09:17.622240 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 11.85s. Breakdown: [{proposals 11.841s} {disk 0s} {advance 0s}] Num entries: 0. MustSync: false W0115 05:10:40.531906 16 groups.go:869] No membership update for 10s. Closing connection to Zero. I0115 05:10:42.535046 16 groups.go:821] Starting a new membership stream receive from 192.168.3.11:5080. I0115 05:10:38.874861 16 log.go:34] Block cache metrics: hit: 49357547 miss: 22908929 keys-added: 8942544 keys-updated: 114597 keys-evicted: 7660232 cost-added: 38604911321 cost-evicted: 33086886312 sets-dropped: 0 sets-rejected: 3681767 gets-dropped: 63040 gets-kept: 45449152 gets-total: 72266476 hit-ratio: 0.68 W0115 05:12:49.342913 16 raft_server.go:239] Error while raft.Step from 0x8: context deadline exceeded. Closing RaftMessage stream. I0115 05:12:49.350625 16 groups.go:838] Received first state update from Zero: counter:36743384 groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101868742 checksum:18290414373100373824 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101884062 checksum:2757678515920796499 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101872755 checksum:287855597640554238 > > zeros: > zeros: > zeros: > maxLeaseId:737100000 maxTxnTs:101900000 maxRaftId:17 removed: removed: removed: removed: removed: removed: removed: removed: removed: cid:"1582cefb-03ec-4014-b336-a4857ccc1344" license: W0115 05:14:54.084952 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 2m7.01s. Breakdown: [{disk 2m7.006s} {proposals 4ms} {advance 0s}] Num entries: 0. MustSync: false I0115 05:15:00.285574 16 groups.go:807] Got address of a Zero leader: 192.168.3.11:5080 W0115 05:16:11.441430 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 18.909s. Breakdown: [{disk 0s} {proposals 0s} {advance 0s}] Num entries: 0. MustSync: false I0115 05:16:44.453818 16 groups.go:821] Starting a new membership stream receive from 192.168.3.11:5080. W0115 05:17:03.612728 16 groups.go:869] No membership update for 10s. Closing connection to Zero. W0115 05:21:22.043555 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 4m18.597s. Breakdown: [{disk 3m26.944s} {proposals 26.012s} {advance 10.578s}] Num entries: 0. MustSync: false I0115 05:21:22.048331 16 groups.go:807] Got address of a Zero leader: 192.168.3.11:5080 I0115 05:21:22.048514 16 groups.go:821] Starting a new membership stream receive from 192.168.3.11:5080. I0115 05:21:22.067696 16 groups.go:838] Received first state update from Zero: counter:36743384 groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101868742 checksum:18290414373100373824 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101884062 checksum:2757678515920796499 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101872755 checksum:287855597640554238 > > zeros: > zeros: > zeros: > maxLeaseId:737100000 maxTxnTs:101900000 maxRaftId:17 removed: removed: removed: removed: removed: removed: removed: removed: removed: cid:"1582cefb-03ec-4014-b336-a4857ccc1344" license: I0115 05:20:48.684774 16 groups.go:838] Received first state update from Zero: counter:36743384 groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101868742 checksum:18290414373100373824 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101884062 checksum:2757678515920796499 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101872755 checksum:287855597640554238 > > zeros: > zeros: > zeros: > maxLeaseId:737100000 maxTxnTs:101900000 maxRaftId:17 removed: removed: removed: removed: removed: removed: removed: removed: removed: cid:"1582cefb-03ec-4014-b336-a4857ccc1344" license: I0115 05:21:22.024500 16 log.go:34] Block cache metrics: hit: 49357547 miss: 22908929 keys-added: 8942544 keys-updated: 114597 keys-evicted: 7660232 cost-added: 38604911321 cost-evicted: 33086886312 sets-dropped: 0 sets-rejected: 3681767 gets-dropped: 63040 gets-kept: 45449152 gets-total: 72266476 hit-ratio: 0.68 W0115 05:21:22.814060 16 server_state.go:205] Error while retrieving timestamps: rpc error: code = DeadlineExceeded desc = context deadline exceeded with delay: 10ms. Will retry... W0115 05:21:24.029161 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 922ms. Breakdown: [{advance 6ms} {disk 0s} {proposals 0s}] Num entries: 0. MustSync: false W0115 05:22:35.084131 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 749ms. Breakdown: [{proposals 647ms} {disk 101ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:23:43.698443 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 582ms. Breakdown: [{proposals 567ms} {disk 0s} {advance 0s}] Num entries: 0. MustSync: false W0115 05:23:50.898900 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 3.163s. Breakdown: [{proposals 1.17s} {advance 223ms} {disk 21ms}] Num entries: 0. MustSync: false W0115 05:24:00.537641 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 9.334s. Breakdown: [{disk 5.551s} {advance 1.022s} {proposals 921ms}] Num entries: 0. MustSync: false W0115 05:24:11.999113 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 1.36s. Breakdown: [{disk 1.193s} {proposals 2ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:24:14.840905 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 859ms. Breakdown: [{proposals 847ms} {disk 12ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:24:26.927457 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 755ms. Breakdown: [{proposals 341ms} {disk 168ms} {advance 58ms}] Num entries: 0. MustSync: false W0115 05:24:29.475420 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 1.575s. Breakdown: [{proposals 837ms} {disk 739ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:24:32.405517 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 754ms. Breakdown: [{disk 147ms} {proposals 30ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:24:37.416357 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 621ms. Breakdown: [{proposals 394ms} {disk 226ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:24:40.583693 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 1.01s. Breakdown: [{disk 716ms} {proposals 284ms} {advance 0s}] Num entries: 1. MustSync: true W0115 05:24:44.314053 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 821ms. Breakdown: [{proposals 460ms} {disk 10ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:24:46.207925 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 1.269s. Breakdown: [{advance 680ms} {disk 555ms} {proposals 11ms}] Num entries: 0. MustSync: false W0115 05:24:53.107659 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 1.791s. Breakdown: [{disk 361ms} {proposals 159ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:24:57.098458 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 1.63s. Breakdown: [{disk 1.628s} {proposals 2ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:25:01.648698 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 558ms. Breakdown: [{proposals 518ms} {disk 32ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:26:00.091128 16 groups.go:869] No membership update for 10s. Closing connection to Zero. I0115 05:26:00.387557 16 log.go:34] Block cache metrics: hit: 49360284 miss: 22908990 keys-added: 8942595 keys-updated: 114599 keys-evicted: 7660283 cost-added: 38605129290 cost-evicted: 33087106411 sets-dropped: 0 sets-rejected: 3681775 gets-dropped: 63040 gets-kept: 45449984 gets-total: 72269274 hit-ratio: 0.68 I0115 05:26:48.285912 16 groups.go:807] Got address of a Zero leader: 192.168.3.11:5080 E0115 05:27:33.817794 16 groups.go:773] While sending membership update: rpc error: code = Unimplemented desc = grpc: Decompressor is not installed for grpc-encoding "snappy" W0115 05:27:45.897247 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 5.167s. Breakdown: [{proposals 5.068s} {disk 68ms} {advance 0s}] Num entries: 0. MustSync: false I0115 05:27:47.500759 16 groups.go:821] Starting a new membership stream receive from 192.168.3.11:5080. I0115 05:27:47.861353 16 groups.go:838] Received first state update from Zero: counter:36743390 groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101868742 checksum:18290414373100373824 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101884062 checksum:2757678515920796499 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101872755 checksum:287855597640554238 > > zeros: > zeros: > zeros: > maxLeaseId:737100000 maxTxnTs:101900000 maxRaftId:17 removed: removed: removed: removed: removed: removed: removed: removed: removed: cid:"1582cefb-03ec-4014-b336-a4857ccc1344" license: W0115 05:27:47.942800 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 892ms. Breakdown: [{proposals 885ms} {advance 6ms} {disk 2ms}] Num entries: 6. MustSync: true W0115 05:28:26.533257 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 782ms. Breakdown: [{disk 620ms} {proposals 7ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:29:52.131373 16 groups.go:869] No membership update for 10s. Closing connection to Zero. W0115 05:29:58.765713 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 35.051s. Breakdown: [{disk 8.349s} {advance 5.892s} {proposals 313ms}] Num entries: 0. MustSync: false W0115 05:30:02.562874 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 2.31s. Breakdown: [{proposals 2.068s} {advance 237ms} {disk 0s}] Num entries: 0. MustSync: false I0115 05:30:03.276070 16 groups.go:807] Got address of a Zero leader: 192.168.3.11:5080 I0115 05:30:03.276521 16 groups.go:821] Starting a new membership stream receive from 192.168.3.11:5080. I0115 05:30:04.181040 16 groups.go:838] Received first state update from Zero: counter:36743390 groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101868742 checksum:18290414373100373824 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101884062 checksum:2757678515920796499 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101872755 checksum:287855597640554238 > > zeros: > zeros: > zeros: > maxLeaseId:737100000 maxTxnTs:101900000 maxRaftId:17 removed: removed: removed: removed: removed: removed: removed: removed: removed: cid:"1582cefb-03ec-4014-b336-a4857ccc1344" license: W0115 05:30:49.390830 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 1.086s. Breakdown: [{proposals 312ms} {disk 167ms} {advance 122ms}] Num entries: 0. MustSync: false W0115 05:30:51.292815 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 763ms. Breakdown: [{disk 671ms} {proposals 80ms} {advance 12ms}] Num entries: 0. MustSync: false W0115 05:30:54.362752 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 1.425s. Breakdown: [{disk 772ms} {proposals 652ms} {advance 0s}] Num entries: 0. MustSync: false I0115 05:30:53.912681 16 log.go:34] Block cache metrics: hit: 49360309 miss: 22909031 keys-added: 8942627 keys-updated: 114599 keys-evicted: 7660314 cost-added: 38605265303 cost-evicted: 33087240064 sets-dropped: 0 sets-rejected: 3681784 gets-dropped: 63040 gets-kept: 45449984 gets-total: 72269340 hit-ratio: 0.68 W0115 05:30:55.538526 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 526ms. Breakdown: [{disk 364ms} {proposals 0s} {advance 0s}] Num entries: 0. MustSync: false W0115 05:31:00.852333 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 814ms. Breakdown: [{disk 807ms} {proposals 0s} {advance 0s}] Num entries: 0. MustSync: false W0115 05:32:00.762062 16 groups.go:869] No membership update for 10s. Closing connection to Zero. I0115 05:32:15.984891 16 groups.go:807] Got address of a Zero leader: 192.168.3.11:5080 E0115 05:32:15.987193 16 groups.go:773] While sending membership update: rpc error: code = DeadlineExceeded desc = context deadline exceeded I0115 05:32:15.991723 16 groups.go:821] Starting a new membership stream receive from 192.168.3.11:5080. W0115 05:32:15.440120 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 32.057s. Breakdown: [{proposals 11.232s} {disk 6.233s} {advance 0s}] Num entries: 1. MustSync: true I0115 05:32:16.249036 16 groups.go:838] Received first state update from Zero: counter:36743390 groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101868742 checksum:18290414373100373824 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101884062 checksum:2757678515920796499 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101872755 checksum:287855597640554238 > > zeros: > zeros: > zeros: > maxLeaseId:737100000 maxTxnTs:101900000 maxRaftId:17 removed: removed: removed: removed: removed: removed: removed: removed: removed: cid:"1582cefb-03ec-4014-b336-a4857ccc1344" license: W0115 05:32:40.135135 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 2.765s. Breakdown: [{disk 1.266s} {proposals 0s} {advance 0s}] Num entries: 0. MustSync: false W0115 05:32:55.847713 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 921ms. Breakdown: [{disk 592ms} {proposals 0s} {advance 0s}] Num entries: 0. MustSync: false W0115 05:33:17.071782 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 3.389s. Breakdown: [{proposals 1.852s} {advance 109ms} {disk 14ms}] Num entries: 0. MustSync: false W0115 05:33:32.077349 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 8.499s. Breakdown: [{proposals 5.199s} {disk 560ms} {advance 1ms}] Num entries: 0. MustSync: false E0115 05:33:36.721363 16 groups.go:773] While sending membership update: rpc error: code = DeadlineExceeded desc = context deadline exceeded W0115 05:33:39.412071 16 draft.go:1313] Raft.Ready took too long to process: Timer Total: 7.334s. Breakdown: [{disk 3.825s} {proposals 3.505s} {advance 4ms}] Num entries: 1. MustSync: true W0115 05:33:54.043603 16 pool.go:267] Connection lost with 192.168.3.12:7085. Error: rpc error: code = Unavailable desc = transport is closing I0115 05:35:53.128119 16 log.go:34] Block cache metrics: hit: 49360326 miss: 22909044 keys-added: 8942637 keys-updated: 114600 keys-evicted: 7660324 cost-added: 38605308840 cost-evicted: 33087282928 sets-dropped: 0 sets-rejected: 3681786 gets-dropped: 63040 gets-kept: 45449984 gets-total: 72269370 hit-ratio: 0.68 I0115 05:40:53.128347 16 log.go:34] Block cache metrics: hit: 49360326 miss: 22909044 keys-added: 8942637 keys-updated: 114600 keys-evicted: 7660324 cost-added: 38605308840 cost-evicted: 33087282928 sets-dropped: 0 sets-rejected: 3681786 gets-dropped: 63040 gets-kept: 45449984 gets-total: 72269370 hit-ratio: 0.68 I0115 05:44:43.993481 16 run.go:744] Caught Ctrl-C. Terminating now (this may take a few seconds)... E0115 05:44:43.993548 16 run.go:393] GRPC listener canceled: accept tcp [::]:9087: use of closed network connection E0115 05:44:43.993766 16 server.go:24] error from cmux serve: accept tcp [::]:8087: use of closed network connection E0115 05:44:43.993791 16 server.go:66] Stopped taking more http(s) requests. Err: mux: listener closed I0115 05:44:43.993828 16 run.go:788] GRPC and HTTP stopped. I0115 05:44:43.993845 16 worker.go:120] Stopping group... I0115 05:44:43.993892 16 server.go:70] All http(s) requests finished. I0115 05:44:43.993930 16 groups.go:739] Closing sendMembershipUpdates I0115 05:44:43.993947 16 groups.go:786] Closing receiveMembershipUpdates I0115 05:44:43.993966 16 groups.go:885] Closing processOracleDeltaStream I0115 05:44:43.993982 16 worker.go:124] Updating RAFT state before shutting down... E0115 05:44:43.993987 16 groups.go:829] Unable to sync memberships. Error: rpc error: code = Canceled desc = context canceled. State: E0115 05:44:43.994023 16 groups.go:1143] Error during SubscribeForUpdates for prefix "\x00\x00\vdgraph.cors\x00": while receiving from stream: rpc error: code = Canceled desc = context canceled. closer err: context canceled I0115 05:44:43.994072 16 groups.go:1112] SubscribeForUpdates closing for prefix: "\x00\x00\vdgraph.cors\x00" I0115 05:44:43.994213 16 worker.go:129] Stopping node... I0115 05:44:43.994260 16 draft.go:207] Stopped all ongoing registered tasks. I0115 05:44:43.994260 16 draft.go:1021] Stopping node.Run I0115 05:44:43.994303 16 draft.go:124] Operation completed with id: opRollup I0115 05:44:43.994347 16 draft.go:1095] Raft node done. I0115 05:44:43.994371 16 worker.go:132] Stopping worker server... I0115 05:44:43.995000 16 run.go:795] worker stopped. E0115 05:44:43.995091 16 groups.go:1143] Error during SubscribeForUpdates for prefix "\x00\x00\x15dgraph.graphql.schema\x00": while receiving from stream: rpc error: code = Canceled desc = context canceled. closer err: context canceled I0115 05:44:43.995128 16 groups.go:1112] SubscribeForUpdates closing for prefix: "\x00\x00\x15dgraph.graphql.schema\x00" I0115 05:44:43.995144 16 run.go:798] adminCloser closed. I0115 05:44:43.995159 16 log.go:34] Lifetime L0 stalled for: 0s I0115 05:44:43.995176 16 server_state.go:162] Exiting fillTimestampRequests W0115 05:44:44.292103 16 pool.go:267] Connection lost with 192.168.3.12:7086. Error: rpc error: code = Unavailable desc = transport is closing I0115 05:44:45.111346 16 log.go:34] Level 0 [ ]: NumTables: 04. Size: 87 MiB of 0 B. Score: 0.00->0.00 Target FileSize: 64 MiB Level 1 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 Target FileSize: 2.0 MiB Level 2 [B]: NumTables: 05. Size: 9.7 MiB of 10 MiB. Score: 0.00->0.00 Target FileSize: 2.0 MiB Level 3 [ ]: NumTables: 23. Size: 48 MiB of 49 MiB. Score: 0.00->0.00 Target FileSize: 4.0 MiB Level 4 [ ]: NumTables: 85. Size: 481 MiB of 488 MiB. Score: 0.00->0.00 Target FileSize: 8.0 MiB Level 5 [ ]: NumTables: 186. Size: 1.5 GiB of 4.8 GiB. Score: 0.00->0.00 Target FileSize: 16 MiB Level 6 [ ]: NumTables: 799. Size: 48 GiB of 48 GiB. Score: 0.00->0.00 Target FileSize: 32 MiB Level Done I0115 05:44:47.335089 16 run.go:802] worker.State disposed. I0115 05:44:47.335125 16 run.go:805] updaters closed. I0115 05:44:47.335138 16 run.go:807] Server shutdown. Bye! [Sentry] 2021/01/15 05:44:47 Buffer flushed successfully. [Decoder]: Using assembly version of decoder Page Size: 4096 [Sentry] 2021/01/15 05:47:14 Integration installed: ContextifyFrames [Sentry] 2021/01/15 05:47:14 Integration installed: Environment [Sentry] 2021/01/15 05:47:14 Integration installed: Modules [Sentry] 2021/01/15 05:47:14 Integration installed: IgnoreErrors [Decoder]: Using assembly version of decoder Page Size: 4096 [Sentry] 2021/01/15 05:47:14 Integration installed: ContextifyFrames [Sentry] 2021/01/15 05:47:14 Integration installed: Environment [Sentry] 2021/01/15 05:47:14 Integration installed: Modules [Sentry] 2021/01/15 05:47:14 Integration installed: IgnoreErrors I0115 05:47:14.750479 19 sentry_integration.go:48] This instance of Dgraph will send anonymous reports of panics back to Dgraph Labs via Sentry. No confidential information is sent. These reports help improve Dgraph. To opt-out, restart your instance with the --enable_sentry=false flag. For more info, see https://dgraph.io/docs/howto/#data-handling. W0115 05:47:14.765932 19 run.go:573] --lru_mb is deprecated, use --cache_mb instead I0115 05:47:14.767375 19 util_ee.go:126] KeyReader instantiated of type I0115 05:47:14.999349 19 init.go:107]