XidMap panic: Key cannot be empty

Users Dgraph
#1

Hi, I sometimes get this error when using the bulk loader with a xidmap:

I0112 01:34:53.476194 3138539 xidmap.go:282] Writing xid map to DB
panic: Key cannot be empty

goroutine 227 [running]:
github.com/dgraph-io/dgraph/x.Panic(...)
        /ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:81
github.com/dgraph-io/dgraph/xidmap.(*XidMap).dbWriter(0xc0001ee620)
        /ext-go/1/src/github.com/dgraph-io/dgraph/xidmap/xidmap.go:186 +0x2bf
created by github.com/dgraph-io/dgraph/xidmap.New
        /ext-go/1/src/github.com/dgraph-io/dgraph/xidmap/xidmap.go:104 +0x213
dgraph bulk -f  -s  --map_shards=2 --format=rdf --xidmap xidmap --http     3476.93s user 164.59s system 1657% cpu 3:39.66 total

My first assumption was that there was something wrong with the data files, but it doesn’t happen every time (even with the exact same data). I have been able to reproduce the error several times. It’s just that it’s not certain to happen every time.

If I restart but keep the previous xidmap it seems to never happen.

[Decoder]: Using assembly version of decoder
Page Size: 4096

Dgraph version : v20.11.0
Dgraph codename : tchalla
Dgraph SHA-256 : 8acb886b24556691d7d74929817a4ac7d9db76bb8b77de00f44650931a16b6ac
Commit SHA-1 : c4245ad55
Commit timestamp : 2020-12-16 15:55:40 +0530
Branch : HEAD
Go version : go1.15.5
jemalloc enabled : true

For Dgraph official documentation, visit https://dgraph.io/docs/.
For discussions about Dgraph , visit https://discuss.dgraph.io.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2020 Dgraph Labs, Inc.

#2

Hi, @apete

I’m wondering if you could help me reproduce this? We can keep the data private (I’ll send you a PM). If this keeps happening then this would be a bug.

#3

I definitely think this is a bug, but right now I haven’t seen it in a while. I’ll see if I can get it back with the settings I was using before. For a while I was running with --map_shards=2 and no other settings.

Not sure I can send data to you. I’ll have to ask about.

#4

@ibrahim can you help with debugging this?

#6

Hey @apete, the check here https://github.com/dgraph-io/dgraph/blob/ee0b99586ab740fbdec6da78605624c7feba2e5d/xidmap/xidmap.go#L193 is failing because the xid is empty.

An xid can be the subject or the object in an nquad (<subject> predicate <object> format). I ran a few tests locally but it seems like we have enough checks to validate the subject/object. I believe we have some edge case conditions because of which we’re getting an empty subject/predicate.

@apete if you can share the data that causes this crash, we can try to investigate it. We don’t have a way to reproduce the crash right now.