LOCK Issue on windows on exposed API

Hello all and sorry if my question is not at the right place.
I reposted from Github since I realized it was not the appropriate place.

Its related to a miss usage of the library and probably not a bug

What version of Go are you using ( go version )?

$ go version 1.14

What operating system are you using?

Windows 10

What version of Badger are you using?

Last one from master (v2 I guess)

Does this issue reproduce with the latest master?

Yes

Steps to Reproduce the issue

I uses badger to create a DB and store items. I created a function that use a txn.View to Get and it provided its key. This function defer Close() the db after the transaction.

func ReadStatsFromDb(dbname string) (*models.Stats, error) {
	db, err := OpenDB("databases/stats")
	if err != nil {
		return nil, errors.New("databases: could not read stats from db - " + err.Error())
	}
	defer db.Close()
	var data []byte
	err = db.View(func(txn *badger.Txn) error {
		item, err := txn.Get([]byte(dbname))
		if err != nil {
			return err
		}
		data, err = item.ValueCopy(nil)
		if err != nil {
			return err
		}
		return nil
	})
	if err != nil {
		return nil, errors.New("databases: could not read stats from db - " + err.Error())
	}
	stats, err := helpers.DecodeStats(data)
	if err != nil {
		return nil, errors.New("databases: could not read stats from db - " + err.Error())
	}
	return stats, nil
}

I then exposed this function using Beego as an API that responds to GET call on localhost:8080/stats/:key (where key il the key to look for in the badger DB)
Lastly the same Beego runtime serve a HTML page on localhost:8080/ with javascrit that pulls data from the API.
In the javascript code served with the html is some code:

["key1", "key2", "key3"].forEach((key) => {
	fetch("/stats/${key}").then(Do some stuff..)
}

the forEach function iter over the value in the array and execute the request to the API for each of the key value. This process is sequential…
The thing is I can retrieve the first value… but then the other fail:

Cannot create lock file …\LOCK Another process is using this Badger

If I introduce delay in the forEach loop it solves the issue after at least 200ms delay between each call…

My question how would you tackle this issue? Should I not close the DB between each search for key? I suppose it has to do with the lock and file system and since the DB is on my slow hard drive the OS don’t have time to suppress the lock file between each call? Is there a specific setting to change this behavior?

What Badger options were set?

options := badger.DefaultOptions(path)
options.Truncate = true

What did you see instead?

Here I iterate over 4 keys in JS and the first one succeeded, the next two failed LOCK issue and the last succeeded again…
Here is the logger output :

badger 2020/04/14 18:40:24 INFO: All 1 tables opened in 0s
badger 2020/04/14 18:40:24 INFO: Replaying file id: 0 at offset: 421594
badger 2020/04/14 18:40:24 INFO: Replay took: 0s
badger 2020/04/14 18:40:24 DEBUG: Value log discard stats empty
badger 2020/04/14 18:40:24 INFO: Got compaction priority: {level:0 score:1.73 dropPrefix:}
badger 2020/04/14 18:40:24 INFO: All 1 tables opened in 1ms
badger 2020/04/14 18:40:24 INFO: Replaying file id: 0 at offset: 421594
badger 2020/04/14 18:40:24 INFO: Replay took: 0s
badger 2020/04/14 18:40:24 DEBUG: Value log discard stats empty
badger 2020/04/14 18:40:24 INFO: Got compaction priority: {level:0 score:1.73 dropPrefix:}

Thank you for any help provided.

Best regards;

Hey @supergeoff, ideally you should be using a single DB instance across all your queries. Open a DB during server initialization and pass it around the system.

The problem with your current approach is that if two requests are received at the same time, only one of them will be able to open badger and read data from it. This could become a serious problem eventually.

If you wish you open a new DB instance for every request, you might want to set the BypassDirLock option https://github.com/dgraph-io/badger/blob/09dd2e1a4195dbd801ffd09292aa8c9da31e25fc/options.go#L88-L91 . Please be aware that you might corrupt your data if BypassDir option is set and two write requests are being processed at the same time.

As for the 200ms delay, I think it depends on your disk/OS. We use a system call to lock/unlock the directories https://github.com/dgraph-io/badger/blob/09dd2e1a4195dbd801ffd09292aa8c9da31e25fc/dir_windows.go#L70

Hope this answers your queries.

Awesome answer thank you very much.
Its very clear and I better understand what I did wrong and have options on how to fix that.

Thank you.

1 Like