Valdanito
September 23, 2020, 2:16am
1
When I add an int index to a predicate, I get this error.
error in building indexes, aborting :: Request size offset 4789371283 is bigger than maximum offset 4294967295
The version of dgraph I use is v20.07.
Valdanito
September 23, 2020, 2:28am
2
The location of the error has been found.
// validateWrites will check whether the given requests can fit into 4GB vlog file.
// NOTE: 4GB is the maximum size we can create for vlog because value pointer offset is of type
// uint32. If we create more than 4GB, it will overflow uint32. So, limiting the size to 4GB.
func (vlog *valueLog) validateWrites(reqs []*request) error {
vlogOffset := uint64(vlog.woffset())
for _, req := range reqs {
// calculate size of the request.
size := estimateRequestSize(req)
estimatedVlogOffset := vlogOffset + size
if estimatedVlogOffset > uint64(maxVlogFileSize) {
return errors.Errorf("Request size offset %d is bigger than maximum offset %d",
estimatedVlogOffset, maxVlogFileSize)
}
if estimatedVlogOffset >= uint64(vlog.opt.ValueLogFileSize) {
// We'll create a new vlog file if the estimated offset is greater or equal to
// max vlog size. So, resetting the vlogOffset.
vlogOffset = 0
continue
}
// Estimated vlog offset will become current vlog offset if the vlog is not rotated.
chewxy
September 23, 2020, 3:37am
3
@ibrahim - what happens when the index is too big for the maxVlogFileSize ? How does one get there?
ibrahim
September 23, 2020, 6:23am
4
@Valdanito @chewxy Badger uses pointers to record the offset of the value in the vlog file. Since @Valdanito has a very big predicate, we’re trying to write at an offset bigger than 4 GB but the offset type in badger is uint32 which cannot point to more than 4 GB of data. The error means that we’re writing at an offset that we can never read.
@Valdanito what kind of index does this predicate have? I could try to reproduce this locally to better understand why are we generating so much data.
1 Like
Valdanito
September 23, 2020, 6:35am
5
It is an int type predicate, I tried to add an int index to it.