Update:
After running it for ~3 days, mutations and queries seems to stuck. I am seeing following in client logs.
Time elapsed: 328780, Taskname: insert-friend, Speed: 0
Time elapsed: 328780, Taskname: get-friends-one-hop, Speed: 0
Time elapsed: 328780, Taskname: get-friends-two-hop, Speed: 0
Time elapsed: 328781, Taskname: insert-person, Speed: 0
Time elapsed: 328781, Taskname: get-friends-one-hop, Speed: 0
Time elapsed: 328781, Taskname: insert-friend, Speed: 0
Time elapsed: 328781, Taskname: get-friends-two-hop, Speed: 0
Time elapsed: 328782, Taskname: insert-person, Speed: 0
Time elapsed: 328782, Taskname: insert-friend, Speed: 0
Time elapsed: 328782, Taskname: get-friends-one-hop, Speed: 0
Time elapsed: 328782, Taskname: get-friends-two-hop, Speed: 0
Time elapsed: 328783, Taskname: insert-person, Speed: 0
Time elapsed: 328783, Taskname: insert-friend, Speed: 0
Time elapsed: 328783, Taskname: get-friends-one-hop, Speed: 0
Time elapsed: 328783, Taskname: get-friends-two-hop, Speed: 0
Time elapsed: 328784, Taskname: insert-person, Speed: 0
Time elapsed: 328784, Taskname: insert-friend, Speed: 0
Time elapsed: 328784, Taskname: get-friends-one-hop, Speed: 0
Time elapsed: 328784, Taskname: get-friends-two-hop, Speed: 0
Time elapsed: 328785, Taskname: insert-person, Speed: 0
Time elapsed: 328785, Taskname: insert-friend, Speed: 0
Time elapsed: 328785, Taskname: get-friends-one-hop, Speed: 0
Time elapsed: 328785, Taskname: get-friends-two-hop, Speed: 0
These are alpha logs.
E0713 10:58:16.985323 14079 log.go:32] WatchBatch.Cancel error while finishing: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists
E0713 10:58:16.985329 14079 node.go:285] While trying to save Raft update: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists. Retrying...
E0713 10:58:16.987652 14079 log.go:32] writeRequests: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists
E0713 10:58:16.987682 14079 log.go:32] WatchBatch.Cancel error while finishing: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists
E0713 10:58:16.987688 14079 node.go:285] While trying to save Raft update: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists. Retrying...
E0713 10:58:16.989567 14079 log.go:32] writeRequests: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists
E0713 10:58:16.989593 14079 log.go:32] WatchBatch.Cancel error while finishing: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists
E0713 10:58:16.989600 14079 node.go:285] While trying to save Raft update: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists. Retrying...
E0713 10:58:16.991562 14079 log.go:32] writeRequests: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists
E0713 10:58:16.991587 14079 log.go:32] WatchBatch.Cancel error while finishing: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists
E0713 10:58:16.991592 14079 node.go:285] While trying to save Raft update: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists. Retrying...
E0713 10:58:16.993539 14079 log.go:32] writeRequests: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists
E0713 10:58:16.993563 14079 log.go:32] WatchBatch.Cancel error while finishing: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists
E0713 10:58:16.993567 14079 node.go:285] While trying to save Raft update: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists. Retrying...
E0713 10:58:16.995574 14079 log.go:32] writeRequests: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists
E0713 10:58:16.995597 14079 log.go:32] WatchBatch.Cancel error while finishing: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists
E0713 10:58:16.995602 14079 node.go:285] While trying to save Raft update: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists. Retrying...
E0713 10:58:16.997786 14079 log.go:32] writeRequests: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists
E0713 10:58:16.997812 14079 log.go:32] WatchBatch.Cancel error while finishing: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists
E0713 10:58:16.997817 14079 node.go:285] While trying to save Raft update: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists. Retrying...
E0713 10:58:16.999862 14079 log.go:32] writeRequests: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists
E0713 10:58:16.999893 14079 log.go:32] WatchBatch.Cancel error while finishing: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists
Alpha w directory content:
total 1.8M
-rw-rw-r-- 1 ashish ashish 6 Jul 9 13:29 LOCK
-rw------- 1 ashish ashish 28 Jul 9 13:29 KEYREGISTRY
-rw------- 1 ashish ashish 1.3M Jul 11 00:49 006869.vlog
-rw------- 1 ashish ashish 20 Jul 11 00:49 006871.vlog
-rw------- 1 ashish ashish 385K Jul 11 00:49 004148.sst
-rw------- 1 ashish ashish 90K Jul 11 00:49 MANIFEST
-rw------- 1 ashish ashish 21G Jul 13 10:58 006870.vlog
After restarting Alpha logs are as below:
BadgerCompressionLevel:3 WALDir:w MutationsMode:0 AuthToken: AllottedMemory:15072.5 HmacSecret:**** AccessJwtTtl:0s RefreshJwtTtl:0s AclRefreshInterval:0s}
I0713 11:13:35.759000 1642 server_state.go:78] Setting Badger Compression Level: 3
I0713 11:13:35.759014 1642 server_state.go:87] Setting Badger table load option: mmap
I0713 11:13:35.759019 1642 server_state.go:99] Setting Badger value log load option: mmap
I0713 11:13:35.759026 1642 server_state.go:144] Opening write-ahead log BadgerDB with options: {Dir:w ValueDir:w SyncWrites:false TableLoadingMode:1 ValueLogLoadingMode:2 NumVersionsToKeep:1 ReadOnly:false Truncate:true Logger:0x2ba7138 Compression:2 InMemory:false MaxTableSize:67108864 LevelSizeMultiplier:10 MaxLevels:7 ValueThreshold:1048576 NumMemtables:5 BlockSize:4096 BloomFalsePositive:0.01 KeepL0InMemory:false MaxCacheSize:10485760 MaxBfCacheSize:0 LoadBloomsOnOpen:false NumLevelZeroTables:5 NumLevelZeroTablesStall:10 LevelOneSize:268435456 ValueLogFileSize:1073741823 ValueLogMaxEntries:10000 NumCompactors:2 CompactL0OnClose:true LogRotatesToFlush:2 ZSTDCompressionLevel:3 VerifyValueChecksum:false EncryptionKey:[] EncryptionKeyRotationDuration:240h0m0s BypassLockGuard:false ChecksumVerificationMode:0 DetectConflicts:false KeepBlockIndicesInCache:false KeepBlocksInCache:false managedTxns:false maxBatchCount:0 maxBatchSize:0}
I0713 11:13:35.773452 1642 log.go:34] All 1 tables opened in 1ms
I0713 11:13:35.776038 1642 log.go:34] Replaying file id: 6869 at offset: 1341678
I0713 11:13:35.776107 1642 log.go:34] Replay took: 49.037µs
2020/07/13 11:13:35 file size: 265012884 greater than 4294967295
github.com/dgraph-io/badger/v2/y.AssertTruef
/home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/y/error.go:62
github.com/dgraph-io/badger/v2.(*logFile).open
/home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/value.go:914
github.com/dgraph-io/badger/v2.(*valueLog).open
/home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/value.go:1121
github.com/dgraph-io/badger/v2.Open
/home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/db.go:386
github.com/dgraph-io/dgraph/worker.(*ServerState).initStorage
/home/ashish/projects/src/github.com/dgraph-io/dgraph/worker/server_state.go:147
github.com/dgraph-io/dgraph/worker.InitServerState
/home/ashish/projects/src/github.com/dgraph-io/dgraph/worker/server_state.go:54
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run
/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:655
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.init.2.func1
/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:95
github.com/spf13/cobra.(*Command).execute
/home/ashish/projects/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
/home/ashish/projects/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
/home/ashish/projects/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:70
main.main
/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/main.go:78
runtime.main
/home/ashish/go/src/runtime/proc.go:203
runtime.goexit
/home/ashish/go/src/runtime/asm_amd64.s:1373
We are looking into this.