@Naman I am currently doing a re-run, so in theory you get telemetry.
Thanks @Dave_Aitel . Yeah, we would get the telemetry that would show us the stack trace, but we won’t be able to get the entire logs. It would be helpful if you can share them separately.
I’ve been running this now for a couple days with the latest version of my code, and it hasn’t crashed yet. Will keep you informed. If it finishes the job, I will revert and try with my earlier, more crashy code.
(I assume you haven’t changed the master docker image?)
@Naman So did ya’ll change something or did my code-cleanups make it so I’m not finding the bug anymore?
@Dave_Aitel we didn’t change anything that could fix this issue. Dgraph master is updated though, so will be your docker image.
Ok, in that case I will try to revert my code and do a new import…hopefully to trigger the crash.
-dave
1 Like
@Naman I apologize, but it’s just not crashing right now. I’ve been running my old code for a couple days…I’m not sure what changed! This was very reproducible before!
@Dave_Aitel that’s strange. We didn’t change something that should fix this directly.
BTW, thanks for trying to reproduce.
Oooh, finally got it. I had to back up to some very old code on my part. (still docker master version on your part).
I1007 11:09:37.817001 20 log.go:34] [0] [E] LOG Compact 0->4 (12, 9 → 13 tables with 3 splits). [550469 550527 550530 550534 550545 550547 550549 550554 550560 550564 550567 550571 . 550508 550520 550514 550507 550503 550509 550511 550512 550504 .] → [550580 550577 550585 550592 550594 550578 550579 550590 550595 550596 550581 550587 550588 .], took 2.31s
panic: runtime error: makeslice: len out of range
goroutine 18694554 [running]:
github.com/dgraph-io/badger/v3.(*levelsController).fillTables(0xc0001ce1c0, 0xc015aa1780, 0xc000044000)
/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:1387 +0x453
github.com/dgraph-io/badger/v3.(*levelsController).doCompact(0xc0001ce1c0, 0x0, 0x4, 0x4002e9161999999a, 0x3ffe0e3a62014f9c, 0x0, 0x0, 0x0, 0x4, 0xc00a2d6180, …)
/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:1529 +0x91e
github.com/dgraph-io/badger/v3.(*levelsController).runCompactor.func2(0x4, 0x4002e9161999999a, 0x3ffe0e3a62014f9c, 0x0, 0x0, 0x0, 0x4, 0xc00a2d6180, 0x7, 0x7, …)
/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:466 +0x7a
github.com/dgraph-io/badger/v3.(*levelsController).runCompactor.func3(0xc015aa1f30)
/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:489 +0x158
github.com/dgraph-io/badger/v3.(*levelsController).runCompactor(0xc0001ce1c0, 0x0, 0xc066005ce0)
/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:518 +0x362
created by github.com/dgraph-io/badger/v3.(*levelsController).startCompact
/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:355 +0x89
[Sentry] 2021/10/07 11:09:38 Sending fatal event [48f03d04713045c2a14f76585c4306ad] to o318308.ingest.sentry.io project: 5208688
[Sentry] 2021/10/07 11:09:38 Buffer flushed successfully.
@naman Here is just a cut and paste from the terminal .
I1007 11:08:45.725185 20 log.go:34] [3] [E] LOG Compact 4->5 (1, 9 → 10 tables with 3 splits). [550032 . 550057 550058 549946 549957 549936 549939 549948 549732 549792 .] → [550063 550070 550072 550064 550067 550071 550066 550074 550075 550076 .], took 2.44s
I1007 11:08:45.881801 20 log.go:34] L0 was stalled for 3.987s
I1007 11:08:46.540285 20 log.go:34] [2] [E] LOG Compact 5->6 (1, 2 → 1 tables with 1 splits). [549779 . 550060 549764 .] → [550095 .], took 2.148s
I1007 11:08:48.323989 20 log.go:34] [3] [E] LOG Compact 5->6 (1, 5 → 2 tables with 2 splits). [549795 . 547199 547189 547215 547221 549836 .] → [550105 550100 .], took 2.541s
I1007 11:08:50.330013 20 log.go:34] [1] [E] LOG Compact 5->6 (1, 1 → 1 tables with 1 splits). [550104 . 549983 .] → [550125 .], took 2.277s
I1007 11:08:51.028618 20 log.go:34] [2] [E] LOG Compact 5->6 (1, 5 → 2 tables with 2 splits). [549929 . 545756 545766 545699 545811 545824 .] → [550124 550115 .], took 4.488s
I1007 11:08:52.064273 20 log.go:34] [3] [E] LOG Compact 4->5 (1, 3 → 4 tables with 1 splits). [550073 . 549858 549912 549922 .] → [550130 550132 550133 550134 .], took 2.035s
I1007 11:08:52.807305 20 log.go:34] [0] [E] LOG Compact 5->6 (1, 4 → 2 tables with 2 splits). [549852 . 549965 549374 547205 547218 .] → [550135 550129 .], took 2.807s
I1007 11:08:55.186895 20 log.go:34] [2] [E] LOG Compact 5->6 (1, 2 → 1 tables with 1 splits). [549903 . 550123 549879 .] → [550148 .], took 2.39s
I1007 11:08:55.193260 20 log.go:34] [3] [E] LOG Compact 5->6 (1, 2 → 1 tables with 1 splits). [550134 . 546992 546997 .] → [550147 .], took 3.11s
I1007 11:08:55.656507 20 log.go:34] [0] [E] LOG Compact 0->3 (15, 15 → 24 tables with 5 splits). [550051 550097 550099 550102 550107 550109 550110 550111 550118 550119 550120 550122 550126 550127 550128 . 550084 550085 550083 550089 550090 550094 550077 550082 550092 550093 550078 550086 550087 550081 550088 .] → [550140 550143 550149 550155 550158 550159 550163 550164 550138 550151 550152 550141 550153 550157 550162 550139 550146 550150 550144 550145 550154 550156 550160 550161 .], took 2.777s
I1007 11:08:55.681596 20 log.go:34] L0 was stalled for 4.663s
I1007 11:08:55.848489 20 log.go:34] [1] [E] LOG Compact 5->6 (1, 6 → 3 tables with 2 splits). [550053 . 550125 549964 545821 545818 545823 550121 .] → [550137 550165 550142 .], took 5.478s
I1007 11:09:00.427821 20 log.go:34] [2] [E] LOG Compact 5->6 (1, 2 → 1 tables with 1 splits). [550167 . 549460 549463 .] → [550211 .], took 2.175s
I1007 11:09:00.443277 20 log.go:34] [3] [E] LOG Compact 5->6 (1, 1 → 1 tables with 1 splits). [549906 . 547521 .] → [550210 .], took 2.22s
I1007 11:09:02.475199 20 log.go:34] [0] [E] LOG Compact 0->3 (15, 17 → 22 tables with 5 splits). [550131 550171 550174 550177 550181 550187 550196 550197 550199 550200 550201 550202 550204 550205 550206 . 550140 550149 550163 550164 550138 550151 550152 550141 550153 550157 550162 550139 550146 550150 550144 550145 550154 .] → [550214 550220 550228 550233 550237 550238 550213 550215 550217 550223 550224 550216 550229 550235 550236 550212 550218 550221 550222 550225 550219 550226 .], took 2.02s
I1007 11:09:02.508535 20 log.go:34] L0 was stalled for 3.036s
I1007 11:09:03.826118 20 log.go:34] [3] [E] LOG Compact 4->5 (1, 10 → 12 tables with 4 splits). [550170 . 550052 550063 550070 550072 550064 550067 550071 550066 550074 550075 .] → [550239 550242 550250 550251 550227 550230 550231 550234 550244 550245 550240 550241 .], took 3.324s
I1007 11:09:03.993887 20 log.go:34] [1] [E] LOG Compact 5->6 (1, 2 → 2 tables with 1 splits). [549875 . 547604 549686 .] → [550208 550247 .], took 6.426s
I1007 11:09:04.233536 20 log.go:34] [2] [E] LOG Compact 5->6 (1, 5 → 3 tables with 2 splits). [549985 . 550211 549457 541139 546529 546534 .] → [550252 550254 550232 .], took 3.732s
I1007 11:09:07.795970 20 log.go:34] [2] [E] LOG Compact 5->6 (1, 2 → 1 tables with 1 splits). [549989 . 550232 546602 .] → [550277 .], took 3.434s
I1007 11:09:09.597680 20 log.go:34] [3] [E] LOG Compact 5->6 (1, 1 → 1 tables with 1 splits). [550241 . 550280 .] → [550284 .], took 2.198s
I1007 11:09:10.742841 20 log.go:34] [2] [E] LOG Compact 5->6 (1, 5 → 2 tables with 2 splits). [550186 . 550169 546984 546993 544891 544897 .] → [550292 550285 .], took 2.842s
I1007 11:09:10.929402 20 log.go:34] [1] [E] LOG Compact 4->5 (1, 2 → 2 tables with 1 splits). [550173 . 550267 550269 .] → [550286 550293 .], took 3.779s
I1007 11:09:11.413999 20 log.go:34] [0] [E] LOG Compact 5->6 (1, 7 → 5 tables with 3 splits). [550103 . 550068 550069 550041 545608 545625 545626 550137 .] → [550283 550294 550289 550288 550290 .], took 6.045s
I1007 11:09:13.305924 20 log.go:34] L0 was stalled for 5.866s
I1007 11:09:13.537357 20 log.go:34] [0] [E] LOG Compact 0->3 (15, 15 → 22 tables with 5 splits). [550207 550246 550255 550259 550260 550261 550264 550266 550268 550271 550273 550275 550276 550278 550279 . 550215 550217 550223 550224 550216 550229 550235 550236 550212 550218 550221 550222 550225 550219 550226 .] → [550307 550308 550315 550320 550321 550324 550326 550327 550306 550309 550310 550305 550311 550312 550302 550313 550316 550323 550303 550304 550318 550319 .], took 2.101s
I1007 11:09:13.585414 20 log.go:34] [1] [E] LOG Compact 5->6 (1, 6 → 2 tables with 2 splits). [550132 . 546868 547370 546686 546712 546714 546689 .] → [550314 550328 .], took 2.641s
I1007 11:09:14.261396 20 log.go:34] [2] [E] LOG Compact 5->6 (1, 6 → 2 tables with 2 splits). [550190 . 549927 549909 545609 545619 545620 550281 .] → [550337 550335 .], took 2.044s
I1007 11:09:16.894458 20 log.go:34] [3] [E] LOG Compact 5->6 (1, 6 → 2 tables with 2 splits). [550191 . 550285 544913 544890 544918 544919 545804 .] → [550358 550363 .], took 2.488s
I1007 11:09:18.429385 20 log.go:34] [2] [E] LOG Compact 5->6 (1, 6 → 2 tables with 2 splits). [550130 . 550129 547224 546935 546979 546953 550314 .] → [550377 550374 .], took 2.193s
I1007 11:09:19.591037 20 log.go:34] [1] [E] LOG Compact 5->6 (1, 5 → 2 tables with 2 splits). [550183 . 550328 546725 546732 546972 550301 .] → [550382 550380 .], took 3.015s
I1007 11:09:20.191058 20 log.go:34] [3] [E] LOG Compact 5->6 (1, 3 → 2 tables with 1 splits). [550182 . 550335 549924 549928 .] → [550383 550387 .], took 2.021s
I1007 11:09:20.386750 20 log.go:34] [0] [E] LOG Compact 5->6 (1, 4 → 4 tables with 2 splits). [550166 . 550124 550136 545800 550252 .] → [550381 550385 550367 550370 .], took 5.481s
I1007 11:09:20.794928 20 log.go:34] [2] [E] LOG Compact 5->6 (1, 5 → 3 tables with 2 splits). [550185 . 550359 545036 545048 549940 550337 .] → [550384 550388 550389 .], took 2.35s
I1007 11:09:21.377409 20 log.go:34] L0 was stalled for 3.263s
I1007 11:09:22.693416 20 log.go:34] [1] [E] LOG Compact 5->6 (1, 2 → 1 tables with 1 splits). [550270 . 547536 550210 .] → [550423 .], took 2.62s
I1007 11:09:27.809298 20 log.go:34] [2] [E] LOG Compact 4->5 (1, 7 → 13 tables with 3 splits). [550437 . 550322 550330 550331 550317 550329 550332 550455 .] → [550473 550479 550482 550486 550488 550471 550472 550474 550477 550480 550481 550470 550475 .], took 2.334s
I1007 11:09:28.738771 20 log.go:34] [0] [E] LOG Compact 4->5 (1, 4 → 6 tables with 2 splits). [550443 . 550456 550461 550366 550357 .] → [550487 550492 550495 550496 550483 550485 .], took 2.413s
I1007 11:09:30.109150 20 log.go:34] L0 was stalled for 4.647s
I1007 11:09:30.211382 20 log.go:34] [3] [E] LOG Compact 5->6 (1, 4 → 7 tables with 2 splits). [550286 . 550203 550209 550208 550247 .] → [550476 550489 550493 550500 550518 550490 550499 .], took 5.045s
I1007 11:09:32.305203 20 log.go:34] [0] [E] LOG Compact 4->5 (1, 8 → 10 tables with 3 splits). [550510 . 550333 550334 550487 550492 550495 550496 550483 550485 .] → [550524 550528 550535 550541 550542 550529 550539 550546 550525 550526 .], took 2.075s
I1007 11:09:33.237853 20 log.go:34] [3] [E] LOG Compact 4->5 (1, 8 → 6 tables with 3 splits). [550515 . 550394 550445 550444 550462 550463 550457 550371 550378 .] → [550544 550548 550555 550536 550537 550532 .], took 2.754s
I1007 11:09:37.817001 20 log.go:34] [0] [E] LOG Compact 0->4 (12, 9 → 13 tables with 3 splits). [550469 550527 550530 550534 550545 550547 550549 550554 550560 550564 550567 550571 . 550508 550520 550514 550507 550503 550509 550511 550512 550504 .] → [550580 550577 550585 550592 550594 550578 550579 550590 550595 550596 550581 550587 550588 .], took 2.31s
panic: runtime error: makeslice: len out of range
goroutine 18694554 [running]:
github.com/dgraph-io/badger/v3.(*levelsController).fillTables(0xc0001ce1c0, 0xc015aa1780, 0xc000044000)
/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:1387 +0x453
github.com/dgraph-io/badger/v3.(*levelsController).doCompact(0xc0001ce1c0, 0x0, 0x4, 0x4002e9161999999a, 0x3ffe0e3a62014f9c, 0x0, 0x0, 0x0, 0x4, 0xc00a2d6180, …)
/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:1529 +0x91e
github.com/dgraph-io/badger/v3.(*levelsController).runCompactor.func2(0x4, 0x4002e9161999999a, 0x3ffe0e3a62014f9c, 0x0, 0x0, 0x0, 0x4, 0xc00a2d6180, 0x7, 0x7, …)
/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:466 +0x7a
github.com/dgraph-io/badger/v3.(*levelsController).runCompactor.func3(0xc015aa1f30)
/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:489 +0x158
github.com/dgraph-io/badger/v3.(*levelsController).runCompactor(0xc0001ce1c0, 0x0, 0xc066005ce0)
/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:518 +0x362
created by github.com/dgraph-io/badger/v3.(*levelsController).startCompact
/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:355 +0x89
[Sentry] 2021/10/07 11:09:38 Sending fatal event [48f03d04713045c2a14f76585c4306ad] to o318308.ingest.sentry.io project: 5208688
[Sentry] 2021/10/07 11:09:38 Buffer flushed successfully.
You can get the container ID using docker ps -a and then do docker logs <container id>
@Naman Except it crashes when I try to start the docker, so I can’t load it and look at the logs . 
root# docker run --rm -it -p “8080:8080” -p “9080:9080” -v ~/dgraph:/dgraph “dgraph/standalone:master”
Warning: This standalone version is meant for quickstart purposes only.
It is NOT RECOMMENDED for production environments.
[Sentry] 2021/10/07 15:32:09 Integration installed: ContextifyFrames
[Sentry] 2021/10/07 15:32:09 Integration installed: Environment
[Sentry] 2021/10/07 15:32:09 Integration installed: Modules
[Sentry] 2021/10/07 15:32:09 Integration installed: IgnoreErrors
[Sentry] 2021/10/07 15:32:09 Integration installed: ContextifyFrames
[Sentry] 2021/10/07 15:32:09 Integration installed: Environment
[Sentry] 2021/10/07 15:32:09 Integration installed: Modules
[Sentry] 2021/10/07 15:32:09 Integration installed: IgnoreErrors
[Sentry] 2021/10/07 15:32:11 Integration installed: ContextifyFrames
[Sentry] 2021/10/07 15:32:11 Integration installed: Environment
[Sentry] 2021/10/07 15:32:11 Integration installed: Modules
[Sentry] 2021/10/07 15:32:11 Integration installed: IgnoreErrors
[Sentry] 2021/10/07 15:32:11 Integration installed: ContextifyFrames
[Sentry] 2021/10/07 15:32:11 Integration installed: Environment
[Sentry] 2021/10/07 15:32:11 Integration installed: Modules
[Sentry] 2021/10/07 15:32:11 Integration installed: IgnoreErrors
I1007 15:32:12.019871 29 sentry_integration.go:48] This instance of Dgraph will send anonymous reports of panics back to Dgraph Labs via Sentry. No confidential information is sent. These reports help improve Dgraph. To opt-out, restart your instance with the --telemetry “sentry=false;” flag. For more info, see https://dgraph.io/docs/howto/#data-handling.
I1007 15:32:12.023201 22 sentry_integration.go:48] This instance of Dgraph will send anonymous reports of panics back to Dgraph Labs via Sentry. No confidential information is sent. These reports help improve Dgraph. To opt-out, restart your instance with the --telemetry “sentry=false;” flag. For more info, see https://dgraph.io/docs/howto/#data-handling.
I1007 15:32:12.713742 29 init.go:110]
Dgraph version : v21.09.0-rc2-13-gf6e8779a3
Dgraph codename : unnamed
Dgraph SHA-256 : 42f94bcd9127fe464482d1dcd04d6aea4746340253d41ddf5ac905a43395c66b
Commit SHA-1 : f6e8779a3
Commit timestamp : 2021-09-16 16:15:27 -0700
Branch : master
Go version : go1.16.2
jemalloc enabled : true
For Dgraph official documentation, visit https://dgraph.io/docs.
For discussions about Dgraph , visit https://discuss.dgraph.io.
For fully-managed Dgraph Cloud , visit https://dgraph.io/cloud.
Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2021 Dgraph Labs, Inc.
I1007 15:32:12.727686 22 init.go:110]
Dgraph version : v21.09.0-rc2-13-gf6e8779a3
Dgraph codename : unnamed
Dgraph SHA-256 : 42f94bcd9127fe464482d1dcd04d6aea4746340253d41ddf5ac905a43395c66b
Commit SHA-1 : f6e8779a3
Commit timestamp : 2021-09-16 16:15:27 -0700
Branch : master
Go version : go1.16.2
jemalloc enabled : true
For Dgraph official documentation, visit https://dgraph.io/docs.
For discussions about Dgraph , visit https://discuss.dgraph.io.
For fully-managed Dgraph Cloud , visit https://dgraph.io/cloud.
Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2021 Dgraph Labs, Inc.
I1007 15:32:12.728623 29 run.go:254] Setting Config to: {raft:0xc000012d88 telemetry:0xc000012d58 limit:0xc000012d90 bindall:true portOffset:0 numReplicas:1 peer: w:zw rebalanceInterval:480000000000 tlsClientConfig: audit: limiterConfig:0xc000045b40}
I1007 15:32:12.728756 29 run.go:144] Setting up grpc listener at: 0.0.0.0:5080
I1007 15:32:12.728933 29 run.go:144] Setting up http listener at: 0.0.0.0:6080
I1007 15:32:12.729087 29 log.go:295] Found file: 145 First Index: 4320001
I1007 15:32:12.729174 29 storage.go:125] Init Raft Storage with snap: 4337263, first: 4337264, last: 4337613
I1007 15:32:12.729348 29 node.go:152] Setting raft.Config to: &{ID:1 peers: learners: ElectionTick:20 HeartbeatTick:1 Storage:0xc00018ef00 Applied:4337263 MaxSizePerMsg:262144 MaxCommittedSizePerReady:67108864 MaxUncommittedEntriesSize:0 MaxInflightMsgs:256 CheckQuorum:false PreVote:true ReadOnlyOption:0 Logger:0x3939e90 DisableProposalForwarding:false}
I1007 15:32:12.729966 29 node.go:310] Found Snapshot.Metadata: {ConfState:{Nodes:[1] Learners: XXX_unrecognized:} Index:4337263 Term:3 XXX_unrecognized:}
I1007 15:32:12.730301 29 node.go:321] Found hardstate: {Term:3 Vote:1 Commit:4337613 XXX_unrecognized:}
I1007 15:32:12.734163 22 run.go:901] x.Config: {PortOffset:0 Limit:max-pending-queries=64; mutations=allow; txn-abort-after=5m; mutations-nquad=1000000; disallow-drop=false; query-timeout=0ms; max-retries=-1; shared-instance=false; query-edge=1000000; normalize-node=10000 LimitMutationsNquad:1000000 LimitQueryEdge:1000000 BlockClusterWideDrop:false LimitNormalizeNode:10000 QueryTimeout:0s MaxRetries:-1 SharedInstance:false GraphQL:{Introspection:true Debug:false Extensions:true PollInterval:1s} Lambda:{Url: Num:1 Port:20000 RestartAfter:10s}}
I1007 15:32:12.734522 22 run.go:902] x.WorkerConfig: {TmpDir:t ExportPath:export Trace:ratio=0.01; jaeger=; datadog= MyAddr: ZeroAddr:[localhost:5080] TLSClientConfig: TLSServerConfig: Raft:learner=false; snapshot-after-entries=10000; snapshot-after-duration=30m; pending-proposals=256; idx=; group= Badger:{Dir: ValueDir: SyncWrites:false NumVersionsToKeep:1 ReadOnly:false Logger:0xc0001c6fd0 Compression:1 InMemory:false MetricsEnabled:true NumGoroutines:8 MemTableSize:67108864 BaseTableSize:2097152 BaseLevelSize:10485760 LevelSizeMultiplier:10 TableSizeMultiplier:2 MaxLevels:7 VLogPercentile:0 ValueThreshold:1048576 NumMemtables:15 BlockSize:4096 BloomFalsePositive:0.01 BlockCacheSize:322122547 IndexCacheSize:214748364 NumLevelZeroTables:5 NumLevelZeroTablesStall:15 ValueLogFileSize:1073741823 ValueLogMaxEntries:1000000 NumCompactors:4 CompactL0OnClose:false LmaxCompaction:false ZSTDCompressionLevel:0 VerifyValueChecksum:false EncryptionKey: EncryptionKeyRotationDuration:240h0m0s BypassLockGuard:false ChecksumVerificationMode:0 AllowStopTheWorld:true DetectConflicts:true NamespaceOffset:-1 ExternalMagicVersion:0 managedTxns:false maxBatchCount:0 maxBatchSize:0 maxValueThreshold:0} WhiteListedIPRanges:[{Lower:0.0.0.0 Upper:255.255.255.255}] StrictMutations:false AclEnabled:false HmacSecret:**** AbortOlderThan:5m0s ProposedGroupId:0 StartTime:2021-10-07 15:32:10.725288244 +0000 UTC m=+0.679980568 Security:whitelist=0.0.0.0/0; token= EncryptionKey:**** LogRequest:0 HardSync:false Audit:false}
I1007 15:32:12.734862 22 run.go:903] worker.Config: {PostingDir:p WALDir:w MutationsMode:1 AuthToken: HmacSecret:**** AccessJwtTtl:0s RefreshJwtTtl:0s CachePercentage:50,30,20 CacheMb:1024 Audit: ChangeDataConf:file=; kafka=; sasl_user=; sasl_password=; ca_cert=; client_cert=; client_key=; sasl-mechanism=PLAIN;}
I1007 15:32:12.735259 22 log.go:295] Found file: 455 First Index: 13620001
I1007 15:32:12.735562 22 storage.go:125] Init Raft Storage with snap: 13620395, first: 13620396, last: 13625451
I1007 15:32:12.735822 22 server_state.go:140] Opening postings BadgerDB with options: {Dir:p ValueDir:p SyncWrites:false NumVersionsToKeep:2147483647 ReadOnly:false Logger:0x3939e90 Compression:1 InMemory:false MetricsEnabled:true NumGoroutines:8 MemTableSize:67108864 BaseTableSize:2097152 BaseLevelSize:10485760 LevelSizeMultiplier:10 TableSizeMultiplier:2 MaxLevels:7 VLogPercentile:0 ValueThreshold:1048576 NumMemtables:15 BlockSize:4096 BloomFalsePositive:0.01 BlockCacheSize:322122547 IndexCacheSize:214748364 NumLevelZeroTables:5 NumLevelZeroTablesStall:15 ValueLogFileSize:1073741823 ValueLogMaxEntries:1000000 NumCompactors:4 CompactL0OnClose:false LmaxCompaction:false ZSTDCompressionLevel:0 VerifyValueChecksum:false EncryptionKey: EncryptionKeyRotationDuration:240h0m0s BypassLockGuard:false ChecksumVerificationMode:0 AllowStopTheWorld:true DetectConflicts:false NamespaceOffset:1 ExternalMagicVersion:1 managedTxns:false maxBatchCount:0 maxBatchSize:0 maxValueThreshold:0}
I1007 15:32:12.766987 29 node.go:326] Group 0 found 17613 entries
I1007 15:32:12.767309 29 raft.go:621] Restarting node for dgraphzero
I1007 15:32:12.767441 29 node.go:189] Setting conf state to nodes:1
I1007 15:32:12.767802 29 pool.go:164] CONN: Connecting to localhost:7080
I1007 15:32:12.768001 29 log.go:34] 1 became follower at term 3
I1007 15:32:12.768156 29 log.go:34] newRaft 1 [peers: [1], term: 3, commit: 4337613, applied: 4337263, lastindex: 4337613, lastterm: 3]
I1007 15:32:12.769588 29 run.go:388] Running Dgraph Zero…
I1007 15:32:12.842589 22 log.go:34] All 167 tables opened in 27ms
[Sentry] 2021/10/07 15:32:13 Sending fatal event [b8da5cc02c044ab0acd0f92a7e16737f] to o318308.ingest.sentry.io project: 5208688
E1007 15:32:13.034831 22 log.go:32] Received err: Level validation err: Levels Controller err: Inter: Biggest(j-1)[550577]
00000000 00 00 00 00 00 00 00 00 00 00 0a 65 6d 61 69 6c |…email|
00000010 5f 74 65 78 74 02 0a 20 71 75 ff ff ff ff ff 30 |_text… qu…0|
00000020 dd 86 |…|
vs Smallest(j)[550585]:
00000000 00 00 00 00 00 00 00 00 00 00 0a 65 6d 61 69 6c |…email|
00000010 5f 74 65 78 74 02 0a 0a 09 6d ff ff ff ff ff 30 |_text…m…0|
00000020 de 56 |.V|
: level=4 j=1 numTables=13
github.com/dgraph-io/badger/v3.(*levelHandler).validate
/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/util.go:55
github.com/dgraph-io/badger/v3.(*levelsController).validate
/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/util.go:33
github.com/dgraph-io/badger/v3.newLevelsController
/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:180
github.com/dgraph-io/badger/v3.Open
/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:344
github.com/dgraph-io/badger/v3.OpenManaged
/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/managed_db.go:26
github.com/dgraph-io/dgraph/worker.(*ServerState).initStorage
/ext-go/1/src/github.com/dgraph-io/dgraph/worker/server_state.go:143
github.com/dgraph-io/dgraph/worker.InitServerState
/ext-go/1/src/github.com/dgraph-io/dgraph/worker/server_state.go:78
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run
/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:905
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.init.1.func1
/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:101
github.com/spf13/cobra.(*Command).execute
/go/pkg/mod/github.com/spf13/[email protected]/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
/go/pkg/mod/github.com/spf13/[email protected]/command.go:914
github.com/spf13/cobra.(*Command).Execute
/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:79
main.main
/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/main.go:112
runtime.main
/usr/local/go/src/runtime/proc.go:225
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1371. Cleaning up…
2021/10/07 15:32:13 Level validation err: Levels Controller err: Inter: Biggest(j-1)[550577]
00000000 00 00 00 00 00 00 00 00 00 00 0a 65 6d 61 69 6c |…email|
00000010 5f 74 65 78 74 02 0a 20 71 75 ff ff ff ff ff 30 |_text… qu…0|
00000020 dd 86 |…|
vs Smallest(j)[550585]:
00000000 00 00 00 00 00 00 00 00 00 00 0a 65 6d 61 69 6c |…email|
00000010 5f 74 65 78 74 02 0a 0a 09 6d ff ff ff ff ff 30 |_text…m…0|
00000020 de 56 |.V|
: level=4 j=1 numTables=13
github.com/dgraph-io/badger/v3.(*levelHandler).validate
/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/util.go:55
github.com/dgraph-io/badger/v3.(*levelsController).validate
/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/util.go:33
github.com/dgraph-io/badger/v3.newLevelsController
/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:180
github.com/dgraph-io/badger/v3.Open
/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:344
github.com/dgraph-io/badger/v3.OpenManaged
/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/managed_db.go:26
github.com/dgraph-io/dgraph/worker.(*ServerState).initStorage
/ext-go/1/src/github.com/dgraph-io/dgraph/worker/server_state.go:143
github.com/dgraph-io/dgraph/worker.InitServerState
/ext-go/1/src/github.com/dgraph-io/dgraph/worker/server_state.go:78
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run
/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:905
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.init.1.func1
/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:101
github.com/spf13/cobra.(*Command).execute
/go/pkg/mod/github.com/spf13/[email protected]/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
/go/pkg/mod/github.com/spf13/[email protected]/command.go:914
github.com/spf13/cobra.(*Command).Execute
/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:79
main.main
/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/main.go:112
runtime.main
/usr/local/go/src/runtime/proc.go:225
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1371
Error while creating badger KV posting store
github.com/dgraph-io/dgraph/x.Checkf
/ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:51
github.com/dgraph-io/dgraph/worker.(*ServerState).initStorage
/ext-go/1/src/github.com/dgraph-io/dgraph/worker/server_state.go:144
github.com/dgraph-io/dgraph/worker.InitServerState
/ext-go/1/src/github.com/dgraph-io/dgraph/worker/server_state.go:78
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run
/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:905
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.init.1.func1
/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:101
github.com/spf13/cobra.(*Command).execute
/go/pkg/mod/github.com/spf13/[email protected]/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
/go/pkg/mod/github.com/spf13/[email protected]/command.go:914
github.com/spf13/cobra.(*Command).Execute
/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:79
main.main
/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/main.go:112
runtime.main
/usr/local/go/src/runtime/proc.go:225
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1371
@Naman I guess the good news is, I can run this over and over again and it will spam you with the crash telemetry
Would it help if I tarred this whole directory up and you uncompressed it yourself to crash it ? @Naman
Just want to re-up this for the new release!
1 Like