// validateWrites will check whether the given requests can fit into 4GB vlog file.
// NOTE: 4GB is the maximum size we can create for vlog because value pointer offset is of type
// uint32. If we create more than 4GB, it will overflow uint32. So, limiting the size to 4GB.
https://github.com/dgraph-io/badger/blob/master/value.go#L736
What if change the valuepointer from uint32 to uint64?
Interesting. Is there a reason why vlog files must be 4GiB? Does this have anything to do with the way hard drives are segmented? @naman
We mmap the value log files. Hence, to support 32-bit builds we have kept it to uint32.
Yes, you can change it to uint64 on 64-bit machines if you want these files to be bigger. But you would have to make these changes a bunch of places.
As a curiosity, may I know why you want to change it to uint64? Any significant benefit you see?
we have meet some error when adding index on a predicate which holds much data.
Request size offset %d is bigger than maximum offset %d",
estimatedVlogOffset, maxVlogFileSize
Have you made some changes to the code?
not yet
We validate the request size here so that it fits into a vlog file defined by maxVlogFileSize. Each of the request correspond to a transaction. We reject the big transactions in the validateWrites function. So either your transaction size is too big (say 20 Bytes of key and 4GB of value).
Another possibility is a bug. Say vlog file size is already 3.9 GB and now a request worth 200MB comes. As the sum exceeds 4GB limit, we reject the request(throwing this error). Ideally, the request should move to next vlog file.
Thanks for reporting this issue. Marking it as good first issue.
One way to fix
The validate write function should cover both the cases and inform(return) to write function if the new vlog file has to created upfront or not.
Will there be an issue or PR to fix this problem?