Slice bounds out of range in table.(*Builder)

gozssky · September 16, 2020, 5:23am

What version of Go are you using (`go version`)?

$ go version
go version go1.14.2 linux/amd64

What operating system are you using?

CentOS 7

What version of Badger are you using?

v2.0.3

Does this issue reproduce with the latest master?

yes

Steps to Reproduce the issue

This issue can be reproduced in our environment. But the db files are too large to post here.

What Badger options were set?

badger.LSMOnlyOptions(path)

What did you do?

I use badger in our service. I continuously put key/value pairs into badger.
After about 180 million key/value pairs being put, panic occurred in badger.

What did you expect to see?

Everything is ok.

What did you see instead?

badger 2020/09/16 13:17:33 INFO: 170 tables out of 290 opened in 3.009s
badger 2020/09/16 13:17:36 INFO: All 290 tables opened in 5.947s
badger 2020/09/16 13:17:36 INFO: Replaying file id: 0 at offset: 0
badger 2020/09/16 13:17:37 INFO: Got compaction priority: {level:1 score:8.369675766676664 dropPrefix:[]}
badger 2020/09/16 13:17:37 INFO: Running for level: 1
badger 2020/09/16 13:17:37 DEBUG: Flushing memtable, mt.size=67108900 size of flushChan: 0
badger 2020/09/16 13:17:37 DEBUG: Storing value log head: {Fid:0 Len:0 Offset:0}
badger 2020/09/16 13:17:37 DEBUG: LOG Compact. Added 655847 keys. Skipped 0 keys. Iteration took: 310.334714ms
badger 2020/09/16 13:17:38 INFO: Got compaction priority: {level:1 score:8.110338136553764 dropPrefix:[]}
badger 2020/09/16 13:17:38 INFO: Running for level: 1
badger 2020/09/16 13:17:38 DEBUG: Discard stats: map[]
badger 2020/09/16 13:17:38 INFO: LOG Compact 1->2, del 1 tables, add 1 tables, took 489.661115ms
badger 2020/09/16 13:17:38 INFO: Compaction for level: 1 DONE
badger 2020/09/16 13:17:38 DEBUG: LOG Compact. Added 656847 keys. Skipped 0 keys. Iteration took: 331.506869ms
badger 2020/09/16 13:17:38 DEBUG: Discard stats: map[]
badger 2020/09/16 13:17:38 INFO: LOG Compact 1->2, del 1 tables, add 1 tables, took 534.885312ms
badger 2020/09/16 13:17:38 INFO: Compaction for level: 1 DONE
badger 2020/09/16 13:17:38 INFO: Replay took: 2.212160092s
badger 2020/09/16 13:17:38 INFO: Replaying file id: 1 at offset: 0
badger 2020/09/16 13:17:38 INFO: Got compaction priority: {level:1 score:7.850994620472193 dropPrefix:[]}
badger 2020/09/16 13:17:38 INFO: Running for level: 1
badger 2020/09/16 13:17:38 DEBUG: LOG Compact. Added 658045 keys. Skipped 0 keys. Iteration took: 278.10349ms
badger 2020/09/16 13:17:39 INFO: Got compaction priority: {level:1 score:7.591644961386919 dropPrefix:[]}
badger 2020/09/16 13:17:39 INFO: Running for level: 1
badger 2020/09/16 13:17:39 DEBUG: Discard stats: map[]
badger 2020/09/16 13:17:39 INFO: LOG Compact 1->2, del 1 tables, add 1 tables, took 449.334211ms
badger 2020/09/16 13:17:39 INFO: Compaction for level: 1 DONE
badger 2020/09/16 13:17:39 DEBUG: LOG Compact. Added 657738 keys. Skipped 0 keys. Iteration took: 326.138929ms
badger 2020/09/16 13:17:39 DEBUG: Flushing memtable, mt.size=67109020 size of flushChan: 0
badger 2020/09/16 13:17:39 DEBUG: Storing value log head: {Fid:0 Len:0 Offset:0}
badger 2020/09/16 13:17:39 DEBUG: LOG Compact. Added 654329 keys. Skipped 0 keys. Iteration took: 265.576534ms
badger 2020/09/16 13:17:39 INFO: Got compaction priority: {level:1 score:7.332296442240477 dropPrefix:[]}
badger 2020/09/16 13:17:39 INFO: Running for level: 1
badger 2020/09/16 13:17:39 DEBUG: Discard stats: map[]
badger 2020/09/16 13:17:39 INFO: LOG Compact 1->2, del 2 tables, add 2 tables, took 787.045364ms
badger 2020/09/16 13:17:39 INFO: Compaction for level: 1 DONE
badger 2020/09/16 13:17:40 INFO: Got compaction priority: {level:1 score:6.628732442855835 dropPrefix:[]}
badger 2020/09/16 13:17:40 INFO: Running for level: 1
badger 2020/09/16 13:17:40 DEBUG: LOG Compact. Added 658000 keys. Skipped 0 keys. Iteration took: 322.532963ms
badger 2020/09/16 13:17:40 DEBUG: LOG Compact. Added 653966 keys. Skipped 0 keys. Iteration took: 378.973742ms
badger 2020/09/16 13:17:40 INFO: Replay took: 2.267340519s
badger 2020/09/16 13:17:40 INFO: Replaying file id: 2 at offset: 0
badger 2020/09/16 13:17:40 DEBUG: Discard stats: map[]
badger 2020/09/16 13:17:40 INFO: LOG Compact 1->2, del 2 tables, add 2 tables, took 919.092319ms
badger 2020/09/16 13:17:40 INFO: Compaction for level: 1 DONE
badger 2020/09/16 13:17:41 INFO: Got compaction priority: {level:1 score:6.369383972138166 dropPrefix:[]}
badger 2020/09/16 13:17:41 INFO: Running for level: 1
badger 2020/09/16 13:17:41 DEBUG: Flushing memtable, mt.size=67108881 size of flushChan: 0
badger 2020/09/16 13:17:41 DEBUG: Storing value log head: {Fid:0 Len:0 Offset:0}
badger 2020/09/16 13:17:41 DEBUG: LOG Compact. Added 657753 keys. Skipped 0 keys. Iteration took: 532.515694ms
badger 2020/09/16 13:17:41 DEBUG: LOG Compact. Added 657625 keys. Skipped 0 keys. Iteration took: 372.81374ms
badger 2020/09/16 13:17:42 DEBUG: Discard stats: map[]
badger 2020/09/16 13:17:42 INFO: LOG Compact 1->2, del 2 tables, add 2 tables, took 1.122241569s
badger 2020/09/16 13:17:42 INFO: Compaction for level: 1 DONE
badger 2020/09/16 13:17:42 INFO: Got compaction priority: {level:1 score:6.110035698860884 dropPrefix:[]}
badger 2020/09/16 13:17:42 INFO: Running for level: 1
badger 2020/09/16 13:17:42 DEBUG: LOG Compact. Added 658016 keys. Skipped 0 keys. Iteration took: 372.516384ms
badger 2020/09/16 13:17:42 DEBUG: LOG Compact. Added 8010060 keys. Skipped 0 keys. Iteration took: 2.909417403s
badger 2020/09/16 13:17:42 DEBUG: LOG Compact. Added 653584 keys. Skipped 0 keys. Iteration took: 400.80333ms
badger 2020/09/16 13:17:43 DEBUG: Discard stats: map[]
badger 2020/09/16 13:17:43 INFO: LOG Compact 1->2, del 2 tables, add 2 tables, took 1.050056332s
badger 2020/09/16 13:17:43 INFO: Compaction for level: 1 DONE
badger 2020/09/16 13:17:43 INFO: Got compaction priority: {level:1 score:5.8506876938045025 dropPrefix:[]}
badger 2020/09/16 13:17:43 INFO: Running for level: 1
badger 2020/09/16 13:17:43 DEBUG: Flushing memtable, mt.size=67108923 size of flushChan: 0
badger 2020/09/16 13:17:43 DEBUG: Storing value log head: {Fid:0 Len:0 Offset:0}
badger 2020/09/16 13:17:43 DEBUG: Discard stats: map[]
badger 2020/09/16 13:17:43 INFO: LOG Compact 1->2, del 1 tables, add 1 tables, took 3.989466663s
badger 2020/09/16 13:17:43 INFO: Compaction for level: 1 DONE
badger 2020/09/16 13:17:43 INFO: Got compaction priority: {level:1 score:5.591339271515608 dropPrefix:[]}
badger 2020/09/16 13:17:43 INFO: Running for level: 1
badger 2020/09/16 13:17:43 DEBUG: LOG Compact. Added 657999 keys. Skipped 0 keys. Iteration took: 482.223137ms
badger 2020/09/16 13:17:43 INFO: Replay took: 2.943364087s
badger 2020/09/16 13:17:43 INFO: Replaying file id: 3 at offset: 0
badger 2020/09/16 13:17:43 DEBUG: LOG Compact. Added 654763 keys. Skipped 0 keys. Iteration took: 335.593609ms
badger 2020/09/16 13:17:43 DEBUG: LOG Compact. Added 653221 keys. Skipped 0 keys. Iteration took: 326.393897ms
badger 2020/09/16 13:17:44 DEBUG: Discard stats: map[]
badger 2020/09/16 13:17:44 INFO: LOG Compact 1->2, del 2 tables, add 2 tables, took 998.225745ms
badger 2020/09/16 13:17:44 INFO: Compaction for level: 1 DONE
badger 2020/09/16 13:17:44 INFO: Got compaction priority: {level:1 score:5.331962700933218 dropPrefix:[]}
badger 2020/09/16 13:17:44 INFO: Running for level: 1
badger 2020/09/16 13:17:44 DEBUG: LOG Compact. Added 656367 keys. Skipped 0 keys. Iteration took: 337.037357ms
badger 2020/09/16 13:17:44 DEBUG: Discard stats: map[]
badger 2020/09/16 13:17:44 INFO: LOG Compact 1->2, del 2 tables, add 2 tables, took 872.330603ms
badger 2020/09/16 13:17:44 INFO: Compaction for level: 1 DONE
badger 2020/09/16 13:17:44 INFO: Got compaction priority: {level:2 score:1.8276873964816331 dropPrefix:[]}
badger 2020/09/16 13:17:44 INFO: Running for level: 2
badger 2020/09/16 13:17:44 DEBUG: LOG Compact. Added 664249 keys. Skipped 0 keys. Iteration took: 342.202774ms
badger 2020/09/16 13:17:45 DEBUG: Flushing memtable, mt.size=67108991 size of flushChan: 0
badger 2020/09/16 13:17:45 DEBUG: Storing value log head: {Fid:0 Len:0 Offset:0}
badger 2020/09/16 13:17:45 DEBUG: LOG Compact. Added 664266 keys. Skipped 0 keys. Iteration took: 383.852505ms
badger 2020/09/16 13:17:45 DEBUG: LOG Compact. Added 664248 keys. Skipped 0 keys. Iteration took: 370.152538ms
badger 2020/09/16 13:17:45 DEBUG: LOG Compact. Added 664229 keys. Skipped 0 keys. Iteration took: 398.949536ms
badger 2020/09/16 13:17:46 INFO: Replay took: 2.339639031s
badger 2020/09/16 13:17:46 INFO: Replaying file id: 4 at offset: 0
badger 2020/09/16 13:17:46 DEBUG: LOG Compact. Added 664249 keys. Skipped 0 keys. Iteration took: 361.962077ms
badger 2020/09/16 13:17:46 DEBUG: LOG Compact. Added 664242 keys. Skipped 0 keys. Iteration took: 357.28307ms
badger 2020/09/16 13:17:46 DEBUG: Flushing memtable, mt.size=67109018 size of flushChan: 0
badger 2020/09/16 13:17:46 DEBUG: Storing value log head: {Fid:0 Len:0 Offset:0}
badger 2020/09/16 13:17:47 DEBUG: LOG Compact. Added 664244 keys. Skipped 0 keys. Iteration took: 335.988872ms
badger 2020/09/16 13:17:47 DEBUG: LOG Compact. Added 664237 keys. Skipped 0 keys. Iteration took: 339.508552ms
badger 2020/09/16 13:17:47 DEBUG: LOG Compact. Added 664217 keys. Skipped 0 keys. Iteration took: 346.138337ms
badger 2020/09/16 13:17:48 DEBUG: LOG Compact. Added 664226 keys. Skipped 0 keys. Iteration took: 455.195625ms
badger 2020/09/16 13:17:48 DEBUG: LOG Compact. Added 664223 keys. Skipped 0 keys. Iteration took: 339.473412ms
badger 2020/09/16 13:17:48 DEBUG: Flushing memtable, mt.size=67108903 size of flushChan: 0
badger 2020/09/16 13:17:48 DEBUG: Storing value log head: {Fid:0 Len:0 Offset:0}
badger 2020/09/16 13:17:48 INFO: Replay took: 2.47231243s
badger 2020/09/16 13:17:48 INFO: Replaying file id: 5 at offset: 0
badger 2020/09/16 13:17:48 DEBUG: LOG Compact. Added 664248 keys. Skipped 0 keys. Iteration took: 343.678523ms
badger 2020/09/16 13:17:49 DEBUG: LOG Compact. Added 664226 keys. Skipped 0 keys. Iteration took: 358.10809ms
badger 2020/09/16 13:17:49 DEBUG: LOG Compact. Added 664220 keys. Skipped 0 keys. Iteration took: 362.261894ms
badger 2020/09/16 13:17:49 DEBUG: Discard stats: map[]
badger 2020/09/16 13:17:49 INFO: LOG Compact 2->3, del 15 tables, add 14 tables, took 5.460314169s
badger 2020/09/16 13:17:49 INFO: Compaction for level: 2 DONE
badger 2020/09/16 13:17:49 INFO: Got compaction priority: {level:0 score:1.4 dropPrefix:[]}
badger 2020/09/16 13:17:49 INFO: Got compaction priority: {level:2 score:1.801767410710454 dropPrefix:[]}
badger 2020/09/16 13:17:49 INFO: Running for level: 2
badger 2020/09/16 13:17:50 DEBUG: Flushing memtable, mt.size=67108929 size of flushChan: 0
badger 2020/09/16 13:17:50 DEBUG: Storing value log head: {Fid:0 Len:0 Offset:0}
badger 2020/09/16 13:17:50 DEBUG: LOG Compact. Added 664233 keys. Skipped 0 keys. Iteration took: 488.272338ms
badger 2020/09/16 13:17:51 DEBUG: LOG Compact. Added 664236 keys. Skipped 0 keys. Iteration took: 947.951242ms
badger 2020/09/16 13:17:51 INFO: Replay took: 3.095571918s
badger 2020/09/16 13:17:51 INFO: Replaying file id: 6 at offset: 0
badger 2020/09/16 13:17:51 DEBUG: LOG Compact. Added 664242 keys. Skipped 0 keys. Iteration took: 343.542691ms
badger 2020/09/16 13:17:52 DEBUG: LOG Compact. Added 664241 keys. Skipped 0 keys. Iteration took: 322.324699ms
badger 2020/09/16 13:17:52 DEBUG: LOG Compact. Added 664234 keys. Skipped 0 keys. Iteration took: 316.640499ms
badger 2020/09/16 13:17:52 DEBUG: Flushing memtable, mt.size=67108958 size of flushChan: 0
badger 2020/09/16 13:17:52 DEBUG: Storing value log head: {Fid:0 Len:0 Offset:0}
badger 2020/09/16 13:17:52 DEBUG: LOG Compact. Added 664241 keys. Skipped 0 keys. Iteration took: 357.862696ms
badger 2020/09/16 13:17:53 DEBUG: LOG Compact. Added 664253 keys. Skipped 0 keys. Iteration took: 420.599509ms
badger 2020/09/16 13:17:53 DEBUG: LOG Compact. Added 664240 keys. Skipped 0 keys. Iteration took: 387.696377ms
badger 2020/09/16 13:17:53 DEBUG: LOG Compact. Added 664246 keys. Skipped 0 keys. Iteration took: 349.143781ms
badger 2020/09/16 13:17:54 INFO: Replay took: 2.387771872s
badger 2020/09/16 13:17:54 INFO: Replaying file id: 7 at offset: 0
badger 2020/09/16 13:17:54 DEBUG: LOG Compact. Added 664219 keys. Skipped 0 keys. Iteration took: 344.822712ms
badger 2020/09/16 13:17:54 DEBUG: Flushing memtable, mt.size=67109018 size of flushChan: 0
badger 2020/09/16 13:17:54 DEBUG: Storing value log head: {Fid:0 Len:0 Offset:0}
badger 2020/09/16 13:17:54 DEBUG: LOG Compact. Added 664237 keys. Skipped 0 keys. Iteration took: 376.254543ms
badger 2020/09/16 13:17:55 DEBUG: LOG Compact. Added 664245 keys. Skipped 0 keys. Iteration took: 461.258163ms
badger 2020/09/16 13:17:55 DEBUG: LOG Compact. Added 664236 keys. Skipped 0 keys. Iteration took: 310.79548ms
badger 2020/09/16 13:17:55 DEBUG: LOG Compact. Added 664224 keys. Skipped 0 keys. Iteration took: 297.734439ms
badger 2020/09/16 13:17:55 DEBUG: Discard stats: map[]
badger 2020/09/16 13:17:56 INFO: LOG Compact 2->3, del 15 tables, add 14 tables, took 6.088524218s
badger 2020/09/16 13:17:56 INFO: Compaction for level: 2 DONE
badger 2020/09/16 13:17:56 INFO: Got compaction priority: {level:0 score:2 dropPrefix:[]}
badger 2020/09/16 13:17:56 INFO: Got compaction priority: {level:2 score:1.7758323967456817 dropPrefix:[]}
badger 2020/09/16 13:17:56 INFO: Running for level: 2
badger 2020/09/16 13:17:56 DEBUG: Flushing memtable, mt.size=67108945 size of flushChan: 0
badger 2020/09/16 13:17:56 DEBUG: Storing value log head: {Fid:0 Len:0 Offset:0}
badger 2020/09/16 13:17:56 DEBUG: LOG Compact. Added 664224 keys. Skipped 0 keys. Iteration took: 336.263576ms
badger 2020/09/16 13:17:56 INFO: Replay took: 2.321745696s
badger 2020/09/16 13:17:56 INFO: Replaying file id: 8 at offset: 0
badger 2020/09/16 13:17:56 DEBUG: LOG Compact. Added 664220 keys. Skipped 0 keys. Iteration took: 313.620239ms
badger 2020/09/16 13:17:56 DEBUG: LOG Compact. Added 664237 keys. Skipped 0 keys. Iteration took: 316.426117ms
badger 2020/09/16 13:17:57 DEBUG: LOG Compact. Added 664259 keys. Skipped 0 keys. Iteration took: 335.382657ms
badger 2020/09/16 13:17:57 DEBUG: LOG Compact. Added 664222 keys. Skipped 0 keys. Iteration took: 305.460855ms
badger 2020/09/16 13:17:57 DEBUG: Flushing memtable, mt.size=67108967 size of flushChan: 1
badger 2020/09/16 13:17:58 DEBUG: LOG Compact. Added 664232 keys. Skipped 0 keys. Iteration took: 584.09644ms
badger 2020/09/16 13:17:58 DEBUG: LOG Compact. Added 664226 keys. Skipped 0 keys. Iteration took: 426.140368ms
badger 2020/09/16 13:18:00 INFO: Replay took: 4.201446468s
badger 2020/09/16 13:18:00 INFO: Replaying file id: 9 at offset: 0
badger 2020/09/16 13:18:00 DEBUG: LOG Compact. Added 664243 keys. Skipped 0 keys. Iteration took: 1.979342282s
badger 2020/09/16 13:18:00 DEBUG: LOG Compact. Added 664237 keys. Skipped 0 keys. Iteration took: 371.259432ms
badger 2020/09/16 13:18:01 DEBUG: Flushing memtable, mt.size=67108866 size of flushChan: 2
badger 2020/09/16 13:18:01 DEBUG: LOG Compact. Added 664252 keys. Skipped 0 keys. Iteration took: 342.408251ms
badger 2020/09/16 13:18:01 DEBUG: LOG Compact. Added 664231 keys. Skipped 0 keys. Iteration took: 368.207009ms
badger 2020/09/16 13:18:02 DEBUG: LOG Compact. Added 664236 keys. Skipped 0 keys. Iteration took: 362.773366ms
badger 2020/09/16 13:18:02 DEBUG: LOG Compact. Added 664217 keys. Skipped 0 keys. Iteration took: 342.483538ms
badger 2020/09/16 13:18:02 DEBUG: LOG Compact. Added 664244 keys. Skipped 0 keys. Iteration took: 348.755321ms
badger 2020/09/16 13:18:02 DEBUG: Flushing memtable, mt.size=67108934 size of flushChan: 3
badger 2020/09/16 13:18:02 DEBUG: Discard stats: map[]
badger 2020/09/16 13:18:03 INFO: Replay took: 2.420015855s
badger 2020/09/16 13:18:03 INFO: Replaying file id: 10 at offset: 0
badger 2020/09/16 13:18:03 INFO: LOG Compact 2->3, del 15 tables, add 14 tables, took 7.133834116s
badger 2020/09/16 13:18:03 INFO: Compaction for level: 2 DONE
badger 2020/09/16 13:18:03 INFO: Got compaction priority: {level:0 score:2 dropPrefix:[]}
badger 2020/09/16 13:18:03 INFO: Got compaction priority: {level:2 score:1.749897401034832 dropPrefix:[]}
badger 2020/09/16 13:18:03 INFO: Running for level: 2
badger 2020/09/16 13:18:03 DEBUG: LOG Compact. Added 664247 keys. Skipped 0 keys. Iteration took: 329.765199ms
badger 2020/09/16 13:18:03 DEBUG: LOG Compact. Added 664263 keys. Skipped 0 keys. Iteration took: 367.138056ms
badger 2020/09/16 13:18:04 DEBUG: LOG Compact. Added 664231 keys. Skipped 0 keys. Iteration took: 360.942755ms
badger 2020/09/16 13:18:04 DEBUG: LOG Compact. Added 664235 keys. Skipped 0 keys. Iteration took: 366.117115ms
badger 2020/09/16 13:18:04 DEBUG: Flushing memtable, mt.size=67108919 size of flushChan: 4
badger 2020/09/16 13:18:04 DEBUG: LOG Compact. Added 664228 keys. Skipped 0 keys. Iteration took: 395.713017ms
badger 2020/09/16 13:18:05 INFO: Replay took: 2.284835164s
badger 2020/09/16 13:18:05 INFO: Replaying file id: 11 at offset: 0
badger 2020/09/16 13:18:05 DEBUG: LOG Compact. Added 664241 keys. Skipped 0 keys. Iteration took: 543.047843ms
badger 2020/09/16 13:18:06 DEBUG: LOG Compact. Added 664226 keys. Skipped 0 keys. Iteration took: 510.273352ms
badger 2020/09/16 13:18:06 DEBUG: LOG Compact. Added 664249 keys. Skipped 0 keys. Iteration took: 488.443563ms
badger 2020/09/16 13:18:06 DEBUG: Flushing memtable, mt.size=67108942 size of flushChan: 5
badger 2020/09/16 13:18:07 DEBUG: LOG Compact. Added 664231 keys. Skipped 0 keys. Iteration took: 502.97897ms
badger 2020/09/16 13:18:07 DEBUG: LOG Compact. Added 664237 keys. Skipped 0 keys. Iteration took: 526.075361ms
badger 2020/09/16 13:18:08 DEBUG: LOG Compact. Added 664256 keys. Skipped 0 keys. Iteration took: 482.382629ms
badger 2020/09/16 13:18:08 INFO: Replay took: 3.193314697s
badger 2020/09/16 13:18:08 INFO: Replaying file id: 12 at offset: 0
badger 2020/09/16 13:18:08 DEBUG: LOG Compact. Added 664230 keys. Skipped 0 keys. Iteration took: 528.848863ms
badger 2020/09/16 13:18:09 DEBUG: LOG Compact. Added 664228 keys. Skipped 0 keys. Iteration took: 554.891263ms
badger 2020/09/16 13:18:09 DEBUG: LOG Compact. Added 664245 keys. Skipped 0 keys. Iteration took: 485.03771ms
badger 2020/09/16 13:18:09 DEBUG: Discard stats: map[]
badger 2020/09/16 13:18:09 INFO: LOG Compact 2->3, del 15 tables, add 14 tables, took 6.802241235s
badger 2020/09/16 13:18:09 INFO: Compaction for level: 2 DONE
badger 2020/09/16 13:18:09 INFO: Got compaction priority: {level:0 score:2 dropPrefix:[]}
badger 2020/09/16 13:18:09 INFO: Got compaction priority: {level:2 score:1.7239624351263045 dropPrefix:[]}
badger 2020/09/16 13:18:09 INFO: Running for level: 2
badger 2020/09/16 13:18:17 DEBUG: LOG Compact. Added 44929126 keys. Skipped 0 keys. Iteration took: 33.590735522s
panic: runtime error: slice bounds out of range [4052373781:1963281920]

goroutine 108 [running]:
github.com/dgraph-io/badger/v2/table.(*Builder).append(0xc0746486c0, 0xc40fd98000, 0x8ad3796, 0x8ad3796)
	external/com_github_dgraph_io_badger_v2/table/builder.go:247 +0xb9
github.com/dgraph-io/badger/v2/table.(*Builder).Finish(0xc0746486c0, 0xd, 0x1, 0xc00000e028)
	external/com_github_dgraph_io_badger_v2/table/builder.go:421 +0x3eb
github.com/dgraph-io/badger/v2.(*levelsController).compactBuildTables.func2(0x2535, 0x3836306363353735, 0x31363430372f6463, 0x6662323034653264)
	external/com_github_dgraph_io_badger_v2/levels.go:621 +0xc9
github.com/dgraph-io/badger/v2.(*levelsController).compactBuildTables.func3(0xc0002dca80, 0x2535, 0xc02e2d0ae0, 0xc025868000, 0xc0000e10e0, 0xc0746486c0)
	external/com_github_dgraph_io_badger_v2/levels.go:642 +0x14e
created by github.com/dgraph-io/badger/v2.(*levelsController).compactBuildTables
	external/com_github_dgraph_io_badger_v2/levels.go:633 +0x125d

I guess maybe there is some overflow with uint32. Panic happened here:

github.com

dgraph-io/badger/blob/v2.0.3/table/builder.go#L247


	newBuf := make([]byte, l+n)
	copy(newBuf, b.buf)
	b.buf = newBuf
	b.bufLock.Unlock()
}
func (b *Builder) append(data []byte) {
	// Ensure we have enough space to store new data.
	if uint32(len(b.buf)) < b.sz+uint32(len(data)) {
		b.grow(uint32(len(data)))
	}
	copy(b.buf[b.sz:], data)
	b.sz += uint32(len(data))
}

func (b *Builder) addPadding(sz uint32) {
	if uint32(len(b.buf)) < b.sz+sz {
		b.grow(sz)
	}
	b.sz += sz
}

ibrahim · September 16, 2020, 9:11am

Hey @gozssky What’s the size of the biggest SST? Do you insert duplicate keys?

It looks like compactions were trying to build a 4G table and this can happen if you have many duplicate keys.

gozssky · September 16, 2020, 9:35am

The biggest size of SST is 3.6G. There are indeed many duplicate keys. I use a counter to record the numbers of keys. So about 50% put operations are the same key. Is there a work around that I can use?

ibrahim · September 16, 2020, 10:30am

The maximum size that an SST can hold is 4G (uint32 size).

Do you update the same key multiple times? If so, badger should be able to clean up old keys. It might be interesting to look at the SST file. Would you be able to share that file? I just need the 3GB sst file to inspect the content. You can send it to ibrahim [at] dgraph.io

Or if you can do something like this to get the counts of duplicate keys in the file

github.com

dgraph-io/badger/blob/c1fe0ecee5f30c1d88640a83fd40e0fb09ec799d/table/table_test.go#L97-L111


      
          			table, err := OpenTable(f, opts)
          			require.NoError(t, err)
          			defer table.DecrRef()
          			it := table.NewIterator(0)
          			defer it.Close()
          			count := 0
          			for it.Rewind(); it.Valid(); it.Next() {
          				v := it.Value()
          				k := y.KeyWithTs([]byte(key("key", count)), 0)
          				require.EqualValues(t, k, it.Key())
          				require.EqualValues(t, fmt.Sprintf("%d", count), string(v.Value))
          				count++
          			}
          			require.Equal(t, count, n)
          		})

I’m looking for

Number of duplicate keys in that file
Keys that show up multiple times.

gozssky · September 17, 2020, 4:44am

I wrote some code to count keys in a 3.6G sst file.

package main

import (
	"flag"
	"fmt"
	"os"

	"github.com/dgraph-io/badger/v2"
	"github.com/dgraph-io/badger/v2/table"
	"github.com/dgraph-io/badger/v2/y"
)

var (
	flagSSTPath = flag.String("sst-path", "", "Path to sst file")
)

func main() {
	flag.Parse()

	f, err := os.Open(*flagSSTPath)
	if err != nil {
		panic(err)
	}
	defer f.Close()

	opts := badger.DefaultOptions("")
	topts := table.Options{
		TableSize:            uint64(opts.MaxTableSize),
		BlockSize:            opts.BlockSize,
		BloomFalsePositive:   opts.BloomFalsePositive,
		LoadBloomsOnOpen:     opts.LoadBloomsOnOpen,
		LoadingMode:          opts.TableLoadingMode,
		ChkMode:              opts.ChecksumVerificationMode,
		Compression:          opts.Compression,
		ZSTDCompressionLevel: opts.ZSTDCompressionLevel,
	}
	t, err := table.OpenTable(f, topts)
	if err != nil {
		panic(err)
	}
	it := t.NewIterator(false)
	defer it.Close()

	total := 0
	m := make(map[string]int)
	for it.Rewind(); it.Valid(); it.Next() {
		m[string(y.ParseKey(it.Key()))]++
		total++
	}
	for key, count := range m {
		if count > 1 {
			fmt.Printf("key: %v count: %v\n", key, count)
		}
	}
	fmt.Printf("total keys: %v\n", total)
}

The output is:

key: 1/hotv2-alice-0-0-f3d182575cc068cd count: 40372111
total keys: 40739475

Only one key appears multiple times.

ibrahim · September 17, 2020, 7:05am

@gozssky do you run badger with Default options? If it is in default options, the compactions should’ve removed the duplicates. Badger is trying to keep all the duplicate keys in a single table (which cannot go beyond 4G) and that is causing the crash.

gozssky · September 17, 2020, 7:52am

I run with badger.LSMOnlyOptions(path). There are 5 badgers in our environment, each badger was applied with same write operations sequentially. Three of 5 badgers crashed by large SST files. The other two badgers run normally without SST files whose size is larger 100M.

ibrahim · September 18, 2020, 11:44am

Would you be able to upgrade to the latest version of badger Release BadgerDB v2.2007.2 · dgraph-io/badger · GitHub ? We fixed a bunch of issues about cleaning up the LSM Tree. See release notes here Release BadgerDB v2.2007.0 · dgraph-io/badger · GitHub

v2.0.3 and v2.2007 have API breaking changes, no data format changes.

gozssky · September 21, 2020, 4:22am

I will do some test. Thanks for your support.

ibrahim · September 21, 2020, 6:44am

@gozssky Please do let us know if you see this issue on master. Badger supports table upto 4GB. Since you’re using badger with default options, you shouldn’t be seeing such big tables. It could be possible that compaction is storing extra keys (which should’ve been dropped) and this is a bug.

Please do try it out and let know if it fixed the issue.

Topic		Replies	Views
Panic : Slice Range Out Of Bound. InitIndex Crashed Badger badger	0	281	September 21, 2023
Badger panics with Index Out of Range Badger kind:bug	6	1064	December 9, 2020
Unable to read some data Badger kind:bug	3	821	November 4, 2020
Data recovery after a slice bounds out of range error Badger	3	1278	December 5, 2021
[badger v1.6.2] panic: slice bounds out of range Badger kind:bug	3	570	March 24, 2021