[WIP] Performance bottleneck on 1 TB data

Update:
After running it for ~3 days, mutations and queries seems to stuck. I am seeing following in client logs.

Time elapsed: 328780, Taskname: insert-friend, Speed: 0
Time elapsed: 328780, Taskname: get-friends-one-hop, Speed: 0
Time elapsed: 328780, Taskname: get-friends-two-hop, Speed: 0
Time elapsed: 328781, Taskname: insert-person, Speed: 0
Time elapsed: 328781, Taskname: get-friends-one-hop, Speed: 0
Time elapsed: 328781, Taskname: insert-friend, Speed: 0
Time elapsed: 328781, Taskname: get-friends-two-hop, Speed: 0
Time elapsed: 328782, Taskname: insert-person, Speed: 0
Time elapsed: 328782, Taskname: insert-friend, Speed: 0
Time elapsed: 328782, Taskname: get-friends-one-hop, Speed: 0
Time elapsed: 328782, Taskname: get-friends-two-hop, Speed: 0
Time elapsed: 328783, Taskname: insert-person, Speed: 0
Time elapsed: 328783, Taskname: insert-friend, Speed: 0
Time elapsed: 328783, Taskname: get-friends-one-hop, Speed: 0
Time elapsed: 328783, Taskname: get-friends-two-hop, Speed: 0
Time elapsed: 328784, Taskname: insert-person, Speed: 0
Time elapsed: 328784, Taskname: insert-friend, Speed: 0
Time elapsed: 328784, Taskname: get-friends-one-hop, Speed: 0
Time elapsed: 328784, Taskname: get-friends-two-hop, Speed: 0
Time elapsed: 328785, Taskname: insert-person, Speed: 0
Time elapsed: 328785, Taskname: insert-friend, Speed: 0
Time elapsed: 328785, Taskname: get-friends-one-hop, Speed: 0
Time elapsed: 328785, Taskname: get-friends-two-hop, Speed: 0

These are alpha logs.

E0713 10:58:16.985323   14079 log.go:32] WatchBatch.Cancel error while finishing: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists
E0713 10:58:16.985329   14079 node.go:285] While trying to save Raft update: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists. Retrying...
E0713 10:58:16.987652   14079 log.go:32] writeRequests: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists
E0713 10:58:16.987682   14079 log.go:32] WatchBatch.Cancel error while finishing: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists
E0713 10:58:16.987688   14079 node.go:285] While trying to save Raft update: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists. Retrying...
E0713 10:58:16.989567   14079 log.go:32] writeRequests: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists
E0713 10:58:16.989593   14079 log.go:32] WatchBatch.Cancel error while finishing: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists
E0713 10:58:16.989600   14079 node.go:285] While trying to save Raft update: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists. Retrying...
E0713 10:58:16.991562   14079 log.go:32] writeRequests: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists
E0713 10:58:16.991587   14079 log.go:32] WatchBatch.Cancel error while finishing: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists
E0713 10:58:16.991592   14079 node.go:285] While trying to save Raft update: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists. Retrying...
E0713 10:58:16.993539   14079 log.go:32] writeRequests: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists
E0713 10:58:16.993563   14079 log.go:32] WatchBatch.Cancel error while finishing: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists
E0713 10:58:16.993567   14079 node.go:285] While trying to save Raft update: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists. Retrying...
E0713 10:58:16.995574   14079 log.go:32] writeRequests: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists
E0713 10:58:16.995597   14079 log.go:32] WatchBatch.Cancel error while finishing: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists
E0713 10:58:16.995602   14079 node.go:285] While trying to save Raft update: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists. Retrying...
E0713 10:58:16.997786   14079 log.go:32] writeRequests: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists
E0713 10:58:16.997812   14079 log.go:32] WatchBatch.Cancel error while finishing: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists
E0713 10:58:16.997817   14079 node.go:285] While trying to save Raft update: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists. Retrying...
E0713 10:58:16.999862   14079 log.go:32] writeRequests: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists
E0713 10:58:16.999893   14079 log.go:32] WatchBatch.Cancel error while finishing: Create value log file. Path=w/006871.vlog. Error=open w/006871.vlog: file exists

Alpha w directory content:

total 1.8M
-rw-rw-r-- 1 ashish ashish    6 Jul  9 13:29 LOCK
-rw------- 1 ashish ashish   28 Jul  9 13:29 KEYREGISTRY
-rw------- 1 ashish ashish 1.3M Jul 11 00:49 006869.vlog
-rw------- 1 ashish ashish   20 Jul 11 00:49 006871.vlog
-rw------- 1 ashish ashish 385K Jul 11 00:49 004148.sst
-rw------- 1 ashish ashish  90K Jul 11 00:49 MANIFEST
-rw------- 1 ashish ashish  21G Jul 13 10:58 006870.vlog

After restarting Alpha logs are as below:

 BadgerCompressionLevel:3 WALDir:w MutationsMode:0 AuthToken: AllottedMemory:15072.5 HmacSecret:**** AccessJwtTtl:0s RefreshJwtTtl:0s AclRefreshInterval:0s}
I0713 11:13:35.759000    1642 server_state.go:78] Setting Badger Compression Level: 3
I0713 11:13:35.759014    1642 server_state.go:87] Setting Badger table load option: mmap
I0713 11:13:35.759019    1642 server_state.go:99] Setting Badger value log load option: mmap
I0713 11:13:35.759026    1642 server_state.go:144] Opening write-ahead log BadgerDB with options: {Dir:w ValueDir:w SyncWrites:false TableLoadingMode:1 ValueLogLoadingMode:2 NumVersionsToKeep:1 ReadOnly:false Truncate:true Logger:0x2ba7138 Compression:2 InMemory:false MaxTableSize:67108864 LevelSizeMultiplier:10 MaxLevels:7 ValueThreshold:1048576 NumMemtables:5 BlockSize:4096 BloomFalsePositive:0.01 KeepL0InMemory:false MaxCacheSize:10485760 MaxBfCacheSize:0 LoadBloomsOnOpen:false NumLevelZeroTables:5 NumLevelZeroTablesStall:10 LevelOneSize:268435456 ValueLogFileSize:1073741823 ValueLogMaxEntries:10000 NumCompactors:2 CompactL0OnClose:true LogRotatesToFlush:2 ZSTDCompressionLevel:3 VerifyValueChecksum:false EncryptionKey:[] EncryptionKeyRotationDuration:240h0m0s BypassLockGuard:false ChecksumVerificationMode:0 DetectConflicts:false KeepBlockIndicesInCache:false KeepBlocksInCache:false managedTxns:false maxBatchCount:0 maxBatchSize:0}
I0713 11:13:35.773452    1642 log.go:34] All 1 tables opened in 1ms
I0713 11:13:35.776038    1642 log.go:34] Replaying file id: 6869 at offset: 1341678
I0713 11:13:35.776107    1642 log.go:34] Replay took: 49.037µs
2020/07/13 11:13:35 file size: 265012884 greater than 4294967295
github.com/dgraph-io/badger/v2/y.AssertTruef
        /home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/y/error.go:62
github.com/dgraph-io/badger/v2.(*logFile).open
        /home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/value.go:914
github.com/dgraph-io/badger/v2.(*valueLog).open
        /home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/value.go:1121
github.com/dgraph-io/badger/v2.Open
        /home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/db.go:386
github.com/dgraph-io/dgraph/worker.(*ServerState).initStorage
        /home/ashish/projects/src/github.com/dgraph-io/dgraph/worker/server_state.go:147
github.com/dgraph-io/dgraph/worker.InitServerState
        /home/ashish/projects/src/github.com/dgraph-io/dgraph/worker/server_state.go:54
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run
        /home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:655
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.init.2.func1
        /home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:95
github.com/spf13/cobra.(*Command).execute
        /home/ashish/projects/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
        /home/ashish/projects/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
        /home/ashish/projects/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
        /home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:70
main.main
        /home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/main.go:78
runtime.main
        /home/ashish/go/src/runtime/proc.go:203
runtime.goexit
        /home/ashish/go/src/runtime/asm_amd64.s:1373

We are looking into this.