# \[WIP\] Performance bottleneck on 1 TB data

**URL:** <https://discuss.dgraph.io/t/wip-performance-bottleneck-on-1-tb-data/7631>\
**Category:** Dev\
**Created:** [July 10, 2020, 1:23pm UTC](https://discuss.dgraph.io/t/wip-performance-bottleneck-on-1-tb-data/7631 "2020-07-10T13:23:04Z")\
**Posts on this page:** 5\
**Page:** 2

<div class="post-metadata">

**Author:** ![ashishgoswami](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/ashishgoswami/32/3416_2.png) [@ashishgoswami](https://discuss.dgraph.io/u/ashishgoswami)\
**Post date:** [July 21, 2020, 3:38pm UTC](https://discuss.dgraph.io/t/wip-performance-bottleneck-on-1-tb-data/7631/21 "2020-07-21T15:38:54Z")

</div>

@mrjn I am running Dgraph in normal mode.

@Paras found the reason for its slowness. Thanks to @harshil_goel. Live loader works well it schema if provided to it before starting it. I was passing schema in schema file and was running live loader on fresh cluster. Since there was no schema, I was seeing lot of transaction abortions. This can an enhancement in live loader.

With new run(setting schema before running live loader), I found ran for around ~4 hours, before OOM.  
Just to confirm the reason I ran it again. Currently logs for live loader are as follows:

```auto
[15:32:54Z] Elapsed: 01h47m15s Txns: 448319 N-Quads: 448319000 N-Quads/s [last 5s]: 70200 Aborts: 98
[15:32:59Z] Elapsed: 01h47m20s Txns: 448640 N-Quads: 448640000 N-Quads/s [last 5s]: 64200 Aborts: 98
[15:33:04Z] Elapsed: 01h47m25s Txns: 449003 N-Quads: 449003000 N-Quads/s [last 5s]: 72600 Aborts: 98
[15:33:09Z] Elapsed: 01h47m30s Txns: 449334 N-Quads: 449334000 N-Quads/s [last 5s]: 66200 Aborts: 98
[15:33:14Z] Elapsed: 01h47m35s Txns: 449712 N-Quads: 449712000 N-Quads/s [last 5s]: 75600 Aborts: 98
[15:33:19Z] Elapsed: 01h47m41s Txns: 450064 N-Quads: 450064000 N-Quads/s [last 5s]: 70400 Aborts: 98

```

RES shown by htop are **~23GB** for alpha and **~16GB** for live loader.

Heap profile is follow:

```auto
File: dgraph
Build ID: 7048c9a5b650a3d32ace7cfaec4253a766ddb1ab
Type: inuse_space
Time: Jul 21, 2020 at 3:29pm (UTC)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 7720.75MB, 97.55% of 7914.99MB total
Dropped 250 nodes (cum <= 39.57MB)
Showing top 10 nodes out of 80
      flat flat% sum% cum cum%
 5306.12MB 67.04% 67.04% 5347.73MB 67.56% github.com/dgraph-io/badger/v2/table.OpenTable
  489.11MB 6.18% 73.22% 716.61MB 9.05% github.com/dgraph-io/badger/v2/pb.(*TableIndex).Unmarshal
  448.04MB 5.66% 78.88% 448.04MB 5.66% github.com/dgraph-io/ristretto/z.(*Bloom).Size (inline)
     414MB 5.23% 84.11% 414MB 5.23% github.com/dgraph-io/badger/v2/table.NewTableBuilder
     384MB 4.85% 88.96% 384MB 4.85% github.com/dgraph-io/ristretto.newCmRow (inline)
  279.93MB 3.54% 92.50% 279.93MB 3.54% github.com/dgraph-io/badger/v2/table.glob..func1
  227.51MB 2.87% 95.37% 227.51MB 2.87% github.com/dgraph-io/badger/v2/pb.(*BlockOffset).Unmarshal
  166.41MB 2.10% 97.47% 166.41MB 2.10% github.com/dgraph-io/badger/v2/skl.newArena
       5MB 0.063% 97.54% 693.04MB 8.76% github.com/dgraph-io/badger/v2/table.(*Table).block
    0.64MB 0.008% 97.55% 576.64MB 7.29% github.com/dgraph-io/ristretto.NewCache
(pprof) list OpenTable
Total: 7.73GB
ROUTINE ======================== github.com/dgraph-io/badger/v2/table.OpenTable in /home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/table/table.go
    5.18GB 5.22GB (flat, cum) 67.56% of Total
         . . 273:	id, ok := ParseFileID(filename)
         . . 274:	if !ok {
         . . 275: _ = fd.Close()
         . . 276: return nil, errors.Errorf("Invalid filename: %s", filename)
         . . 277:	}
  512.11kB 512.11kB 278:	t := &Table{
         . . 279: fd: fd,
         . . 280: ref: 1, // Caller is given one reference.
         . . 281: id: id,
         . . 282: opt: &opts,
         . . 283: IsInmemory: false,
         . . 284:	}
         . . 285:
         . . 286:	t.tableSize = int(fileInfo.Size())
         . . 287:
         . . 288:	switch opts.LoadingMode {
         . . 289:	case options.LoadToRAM:
         . . 290: if _, err := t.fd.Seek(0, io.SeekStart); err != nil {
         . . 291: return nil, err
         . . 292: }
    5.18GB 5.18GB 293: t.mmap = make([]byte, t.tableSize)
         . . 294: n, err := t.fd.Read(t.mmap)
         . . 295: if err != nil {
         . . 296: // It's OK to ignore fd.Close() error because we have only read from the file.
         . . 297: _ = t.fd.Close()
         . . 298: return nil, y.Wrapf(err, "Failed to load file into RAM")
         . . 299: }
         . . 300: if n != t.tableSize {
         . . 301: return nil, errors.Errorf("Failed to read all bytes from the file."+
         . . 302: "Bytes in file: %d Bytes actually Read: %d", t.tableSize, n)
         . . 303: }

```

Profile points to `w` directory Badger, that where we keep SSTs in RAM(_options.LoadToRAM_)  
**I am looking into this.**

---

<div class="post-metadata">

**Author:** ![ashishgoswami](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/ashishgoswami/32/3416_2.png) [@ashishgoswami](https://discuss.dgraph.io/u/ashishgoswami)\
**Post date:** [July 22, 2020, 12:25pm UTC](https://discuss.dgraph.io/t/wip-performance-bottleneck-on-1-tb-data/7631/22 "2020-07-22T12:25:39Z")

</div>

Looked into above issue. Most of the memory is taken by SSTs of `w` directory Badger, as tables there are opened with `LoadToRAM` option. Ideally we should not have much data in `w` directory as we take snapshot periodically and we delete all the previous raft indexes(lower than snapshot index).  
But these indexes (and data belonging to them) are only marked as deleted and will only be deleted from disk during compaction. Hence these take significant amount of memory.  
I did some experiments on `w` directory after stopping Dgraph cluster. At start, size of `w` was around  
**~6.5GB** , once I ran `badger flatten` on it and resultant size of directory is **~1.5GB** , which can be considered reasonable.

#### Output of _dgraph debug_

```auto
[Decoder]: Using assembly version of decoder
I0721 15:59:07.034752 7324 util_ee.go:126] KeyReader instantiated of type <nil>
Opening DB: ./wcopy
badger 2020/07/21 15:59:07 INFO: All 335 tables opened in 300ms
badger 2020/07/21 15:59:07 INFO: Replaying file id: 190 at offset: 38347976
badger 2020/07/21 15:59:07 INFO: Replay took: 3.115µs
rids: map[1:true]
gids: map[1:true]
Iterating with Raft Id = 1 Groupd Id = 1

Snapshot Metadata: {ConfState:{Nodes:[1] Learners:[] XXX_unrecognized:[]} Index:737765 Term:2 XXX_unrecognized:[]}
Snapshot Alpha: {Context:id:1 group:1 addr:"localhost:7080" Index:737765 ReadTs:984245 Done:false SinceTs:0 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}

Hardstate: {Term:2 Vote:1 Commit:770724 XXX_unrecognized:[]}
Checkpoint: 770724
Last Index: 770724 . Num Entries: 32958 .

```

As per above output we should have around **32960** indexes.

I made some change in **badger info** command and ran it as below:

```auto
go run main.go info --dir=/home/ashish/temp/wcopy --read-only=false --show-keys --with-prefix="000000000000000100000001" --history

```

Got below result for **original `w`** directory:

```auto
[Summary]
Total Number of keys: 1195879
Total Number of deleted keys: 737765
Total Number of discardEarilierKey: 0
Total Number of deletedAndPresent: 425154
Total Number of rest: 32960
badger 2020/07/22 10:28:38 INFO: Got compaction priority: {level:0 score:1.73 dropPrefixes:[]}

```

While running above command on **flattened `w`** directory gives following result:

```auto
[Summary]
Total Number of keys: 32960
Total Number of deleted keys: 0
Total Number of discardEarilierKey: 0
Total Number of deletedAndPresent: 0
Total Number of rest: 32960
badger 2020/07/22 10:41:20 INFO: Got compaction priority: {level:0 score:1.73 dropPrefixes:[]}

```

#### Meaning of keys

**deleted keys** : Keys with deletion marker  
**deletedAndPresent** : Keys which have deletion marker but previous version with data is also present, these are responsible for increase of RAM.  
**discardEarlierKey** : Keys marked with discardEarlierVersion  
I made the following changes in `info.go`

```auto
func showKeys(db *badger.DB, prefix []byte) error {
        if len(prefix) > 0 {
                fmt.Printf("Only choosing keys with prefix: \n%s", hex.Dump(prefix))
        }
        txn := db.NewTransaction(false)
        defer txn.Discard()

        iopt := badger.DefaultIteratorOptions
        iopt.Prefix = []byte(prefix)
        iopt.PrefetchValues = false
        iopt.AllVersions = opt.keyHistory
        // iopt.InternalAccess = opt.showInternal
        it := txn.NewIterator(iopt)
        defer it.Close()

        rest := 0
        totalKeys := 0
        deletedKeys := 0
        discardEarlier := 0
        deletedAndPresent := 0
        var prevkey []byte
        for it.Rewind(); it.Valid(); it.Next() {
                item := it.Item()
                if item.IsDeletedOrExpired() {
                        deletedKeys++
                        prevkey = item.KeyCopy(prevkey)
                } else if item.DiscardEarlierVersions() {
                        discardEarlier++
                } else if bytes.Equal(prevkey, item.Key()) {
                                deletedAndPresent++
                } else {
                        rest++
                }
                /*
                if err := printKey(item, false); err != nil {
                        return errors.Wrapf(err, "failed to print information about key: %x(%d)",
                                item.Key(), item.Version())
                }
                */
                totalKeys++
        }
        fmt.Print("\n[Summary]\n")
        fmt.Println("Total Number of keys:", totalKeys)
        fmt.Println("Total Number of deleted keys:", deletedKeys)
        fmt.Println("Total Number of discardEarilierKey:", discardEarlier)
        fmt.Println("Total Number of deletedAndPresent:", deletedAndPresent)
        fmt.Println("Total Number of rest:", rest)
        return nil
}

```

Thanks for help @ibrahim in above analysis.

---

<div class="post-metadata">

**Author:** ![ashishgoswami](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/ashishgoswami/32/3416_2.png) [@ashishgoswami](https://discuss.dgraph.io/u/ashishgoswami)\
**Post date:** [July 28, 2020, 2:49pm UTC](https://discuss.dgraph.io/t/wip-performance-bottleneck-on-1-tb-data/7631/23 "2020-07-28T14:49:45Z")

</div>

_ **UPDATE** _: **We have upgraded the machine to have 96 GB RAM.**

After figuring out `w` directory is reason for most of RAM consumption(it is the only thing shown in heap profile which consumes most of RAM), I ran flatten on `w` directory. With existing `w` and `p` directory RAM(RES) consumption was going upto **60 GB**. Before running flatten on _w_ directory sizes of `p` and `w` directories were as below:

```auto
510G	./p
28G ./w
358M	./zw

```

After running **flatten** on `w` directory, size of `w` directory was:

```auto
1.8GB ./w

```

After restart of cluster, I continued running live loader. Currently we have following size of directories:

```auto
701G	./p
27G ./w
545M	./zw

```

We can see that _w_ directory has again gained the size.

I also captured time to open _p_(~700 GB) and _w_(~27 GB):

#### W directory

```auto
I0728 14:38:45.641366 18411 log.go:34] 99 tables out of 1525 opened in 3.012s
I0728 14:38:48.634343 18411 log.go:34] 206 tables out of 1525 opened in 6.005s
I0728 14:38:51.643964 18411 log.go:34] 325 tables out of 1525 opened in 9.015s
I0728 14:38:54.653474 18411 log.go:34] 424 tables out of 1525 opened in 12.025s
I0728 14:38:57.637190 18411 log.go:34] 528 tables out of 1525 opened in 15.008s
I0728 14:39:00.639431 18411 log.go:34] 599 tables out of 1525 opened in 18.01s
I0728 14:39:03.676010 18411 log.go:34] 683 tables out of 1525 opened in 21.047s
I0728 14:39:06.649594 18411 log.go:34] 748 tables out of 1525 opened in 24.021s
I0728 14:39:09.642489 18411 log.go:34] 850 tables out of 1525 opened in 27.014s
I0728 14:39:12.632476 18411 log.go:34] 961 tables out of 1525 opened in 30.004s
I0728 14:39:15.632171 18411 log.go:34] 1065 tables out of 1525 opened in 33.003s
I0728 14:39:18.632756 18411 log.go:34] 1186 tables out of 1525 opened in 36.004s
I0728 14:39:21.668542 18411 log.go:34] 1278 tables out of 1525 opened in 39.04s
I0728 14:39:24.646270 18411 log.go:34] 1401 tables out of 1525 opened in 42.017s
I0728 14:39:27.928709 18411 log.go:34] 1480 tables out of 1525 opened in 45.3s
I0728 14:39:29.747162 18411 log.go:34] All 1525 tables opened in 47.118s

```

#### P directory

```auto
I0728 14:39:33.312447 18411 log.go:34] 539 tables out of 6798 opened in 3.006s
I0728 14:39:36.329768 18411 log.go:34] 907 tables out of 6798 opened in 6.023s
I0728 14:39:39.321231 18411 log.go:34] 1136 tables out of 6798 opened in 9.015s
I0728 14:39:42.307248 18411 log.go:34] 1317 tables out of 6798 opened in 12.001s
I0728 14:39:45.308684 18411 log.go:34] 2051 tables out of 6798 opened in 15.002s
I0728 14:39:48.321764 18411 log.go:34] 2343 tables out of 6798 opened in 18.015s
I0728 14:39:51.312440 18411 log.go:34] 2436 tables out of 6798 opened in 21.006s
I0728 14:39:54.327446 18411 log.go:34] 2550 tables out of 6798 opened in 24.021s
I0728 14:39:57.312190 18411 log.go:34] 2671 tables out of 6798 opened in 27.006s
I0728 14:40:00.314967 18411 log.go:34] 2790 tables out of 6798 opened in 30.008s
I0728 14:40:03.314505 18411 log.go:34] 2896 tables out of 6798 opened in 33.008s
I0728 14:40:06.325143 18411 log.go:34] 3016 tables out of 6798 opened in 36.018s
I0728 14:40:09.492198 18411 log.go:34] 3138 tables out of 6798 opened in 39.186s
I0728 14:40:12.310874 18411 log.go:34] 3271 tables out of 6798 opened in 42.004s
I0728 14:40:15.412023 18411 log.go:34] 3409 tables out of 6798 opened in 45.105s
I0728 14:40:18.323595 18411 log.go:34] 3531 tables out of 6798 opened in 48.017s
I0728 14:40:21.395090 18411 log.go:34] 3647 tables out of 6798 opened in 51.088s
I0728 14:40:24.308413 18411 log.go:34] 3771 tables out of 6798 opened in 54.002s
I0728 14:40:27.383428 18411 log.go:34] 3906 tables out of 6798 opened in 57.077s
I0728 14:40:30.341263 18411 log.go:34] 4011 tables out of 6798 opened in 1m0.035s
I0728 14:40:33.350558 18411 log.go:34] 4147 tables out of 6798 opened in 1m3.044s
I0728 14:40:36.316099 18411 log.go:34] 4267 tables out of 6798 opened in 1m6.009s
I0728 14:40:39.309938 18411 log.go:34] 4395 tables out of 6798 opened in 1m9.003s
I0728 14:40:42.310931 18411 log.go:34] 4540 tables out of 6798 opened in 1m12.004s
I0728 14:40:45.321744 18411 log.go:34] 4654 tables out of 6798 opened in 1m15.015s
I0728 14:40:48.349800 18411 log.go:34] 4802 tables out of 6798 opened in 1m18.043s
I0728 14:40:51.316373 18411 log.go:34] 4924 tables out of 6798 opened in 1m21.01s
I0728 14:40:54.353456 18411 log.go:34] 5050 tables out of 6798 opened in 1m24.047s
I0728 14:40:57.336612 18411 log.go:34] 5190 tables out of 6798 opened in 1m27.03s
I0728 14:41:00.308123 18411 log.go:34] 5291 tables out of 6798 opened in 1m30.001s
I0728 14:41:03.309762 18411 log.go:34] 5380 tables out of 6798 opened in 1m33.003s
I0728 14:41:06.337339 18411 log.go:34] 5522 tables out of 6798 opened in 1m36.031s
I0728 14:41:09.310620 18411 log.go:34] 5639 tables out of 6798 opened in 1m39.004s
I0728 14:41:12.324492 18411 log.go:34] 5752 tables out of 6798 opened in 1m42.018s
I0728 14:41:15.311486 18411 log.go:34] 5875 tables out of 6798 opened in 1m45.005s
I0728 14:41:18.363185 18411 log.go:34] 6006 tables out of 6798 opened in 1m48.057s
I0728 14:41:21.354727 18411 log.go:34] 6108 tables out of 6798 opened in 1m51.048s
I0728 14:41:24.359366 18411 log.go:34] 6203 tables out of 6798 opened in 1m54.053s
I0728 14:41:27.307654 18411 log.go:34] 6294 tables out of 6798 opened in 1m57.001s
I0728 14:41:30.310361 18411 log.go:34] 6374 tables out of 6798 opened in 2m0.004s
I0728 14:41:33.331970 18411 log.go:34] 6454 tables out of 6798 opened in 2m3.025s
I0728 14:41:36.310228 18411 log.go:34] 6531 tables out of 6798 opened in 2m6.004s
I0728 14:41:39.423715 18411 log.go:34] 6625 tables out of 6798 opened in 2m9.117s
I0728 14:41:42.334863 18411 log.go:34] 6713 tables out of 6798 opened in 2m12.028s
I0728 14:41:45.307700 18411 log.go:34] 6781 tables out of 6798 opened in 2m15.001s
I0728 14:41:46.277541 18411 log.go:34] All 6798 tables opened in 2m15.971s

```

### Next thing to figure out

Til now I have mostly focused on loading data into Dgraph. I start doing experiment on retrieving data now.  
One thing I have observed in existing data directory is small sizes of _SSTs_ from usual _64 MB_. I am seeing many _SSTs_ which are of size _~20 MB_, trying to figure this out.

---

<div class="post-metadata">

**Author:** ![ashishgoswami](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/ashishgoswami/32/3416_2.png) [@ashishgoswami](https://discuss.dgraph.io/u/ashishgoswami)\
**Post date:** [August 10, 2020, 12:54pm UTC](https://discuss.dgraph.io/t/wip-performance-bottleneck-on-1-tb-data/7631/24 "2020-08-10T12:54:27Z")

</div>

## Experiments with Indexing

While inserting data into Dgraph, I didn’t have any schema. So everything got inserted with default types. After insertions, my schema was below:

```auto
<AboutMe>: default .
<Author>: [uid] .
<Body>: [uid] .
<Chosen.Answer>: [uid] .
<Comment>: [uid] .
<CreationDate>: default .
<DisplayName>: default .
<Downvote>: [uid] .
<Has.Answer>: [uid] .
<Id>: default .
<LastAccessDate>: default .
<Location>: default .
<Owner>: [uid] .
<Post>: [uid] .
<PostCount>: default .
<Reputation>: default .
<Score>: default .
<Tag.Text>: default .
<Tag>: [uid] .
<TagName>: default .
<Text>: default .
<Timestamp>: default .
<Title>: [uid] .
<Type>: default .
<Upvote>: [uid] .
<ViewCount>: default .
<asfdasdfljals>: string .
<dgraph.graphql.schema>: string .
<dgraph.graphql.xid>: string @index(exact) @upsert .

```

I performed below indexing changes, with their time of completion:

```auto
| Schema operation | Time | Completed |
|------------------------------------|-------|-----------|
| <Tag.Text>: string @index(hash) | 1m | Yes |
| <Author>: [uid] @reverse . | 1h15m | Yes |
| <Tag>: [uid] @reverse | 15m | Yes |
| <Downvote>: [uid] @reverse | 3m | Yes |
| <Upvote>: [uid] @reverse | 20m | Yes |
| <Text>: string @index(fulltext) | 2h30m | No |

```

While performing indexing for **\<Text\>: string @index(fulltext)**, Dgraph got crashed. It got crashed and restart resulted in below error:

```auto
I0806 11:30:12.819371 18401 log.go:34] 6851 tables out of 6954 opened in 6m15.057s
I0806 11:30:15.811030 18401 log.go:34] 6903 tables out of 6954 opened in 6m18.048s
I0806 11:30:18.500677 18401 log.go:34] All 6954 tables opened in 6m20.738s
[Sentry] 2020/08/06 11:30:19 Sending fatal event [b3b9b8dac9c94c92b0f727d9c948fd29] to o318308.ingest.sentry.io project: 5208688
2020/08/06 11:30:19 Inter: Biggest(j-1) 
00000000 00 00 0e 4c 61 73 74 41 63 63 65 73 73 44 61 74 |...LastAccessDat|
00000010 65 00 00 00 00 00 16 a1 72 cc ff ff ff ff ff be |e.......r.......|
00000020 17 84 |..|

 vs Smallest(j): 
00000000 00 00 04 42 6f 64 79 00 00 00 00 00 08 37 dc cc |...Body......7..|
00000010 ff ff ff ff ff e1 e6 d6 |........|

: level=3 j=5 numTables=1351
github.com/dgraph-io/badger/v2.(*levelHandler).validate
	/home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/util.go:55
github.com/dgraph-io/badger/v2.(*levelsController).validate
	/home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/util.go:33
github.com/dgraph-io/badger/v2.newLevelsController
	/home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/levels.go:191
github.com/dgraph-io/badger/v2.Open
	/home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/db.go:354
github.com/dgraph-io/badger/v2.OpenManaged
	/home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/managed_db.go:26
github.com/dgraph-io/dgraph/worker.(*ServerState).initStorage
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/worker/server_state.go:171
github.com/dgraph-io/dgraph/worker.InitServerState
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/worker/server_state.go:54
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:655
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.init.2.func1
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:95
github.com/spf13/cobra.(*Command).execute
	/home/ashish/projects/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
	/home/ashish/projects/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
	/home/ashish/projects/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:70
main.main
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/main.go:78
runtime.main
	/home/ashish/go/src/runtime/proc.go:203
runtime.goexit
	/home/ashish/go/src/runtime/asm_amd64.s:1373
Levels Controller
github.com/dgraph-io/badger/v2.(*levelsController).validate
	/home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/util.go:34
github.com/dgraph-io/badger/v2.newLevelsController
	/home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/levels.go:191
github.com/dgraph-io/badger/v2.Open
	/home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/db.go:354
github.com/dgraph-io/badger/v2.OpenManaged
	/home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/managed_db.go:26
github.com/dgraph-io/dgraph/worker.(*ServerState).initStorage
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/worker/server_state.go:171
github.com/dgraph-io/dgraph/worker.InitServerState
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/worker/server_state.go:54
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:655
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.init.2.func1
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:95
github.com/spf13/cobra.(*Command).execute
	/home/ashish/projects/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
	/home/ashish/projects/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
	/home/ashish/projects/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:70
main.main
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/main.go:78
runtime.main
	/home/ashish/go/src/runtime/proc.go:203
runtime.goexit
	/home/ashish/go/src/runtime/asm_amd64.s:1373
Level validation
github.com/dgraph-io/badger/v2.newLevelsController
	/home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/levels.go:193
github.com/dgraph-io/badger/v2.Open
	/home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/db.go:354
github.com/dgraph-io/badger/v2.OpenManaged
	/home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/managed_db.go:26
github.com/dgraph-io/dgraph/worker.(*ServerState).initStorage
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/worker/server_state.go:171
github.com/dgraph-io/dgraph/worker.InitServerState
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/worker/server_state.go:54
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:655
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.init.2.func1
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:95
github.com/spf13/cobra.(*Command).execute
	/home/ashish/projects/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
	/home/ashish/projects/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
	/home/ashish/projects/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:70
main.main
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/main.go:78
runtime.main
	/home/ashish/go/src/runtime/proc.go:203
runtime.goexit
	/home/ashish/go/src/runtime/asm_amd64.s:1373
Error while creating badger KV posting store
github.com/dgraph-io/dgraph/x.Checkf
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/x/error.go:51
github.com/dgraph-io/dgraph/worker.(*ServerState).initStorage
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/worker/server_state.go:172
github.com/dgraph-io/dgraph/worker.InitServerState
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/worker/server_state.go:54
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:655
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.init.2.func1
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:95
github.com/spf13/cobra.(*Command).execute
	/home/ashish/projects/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
	/home/ashish/projects/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
	/home/ashish/projects/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:70
main.main
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/main.go:78
runtime.main
	/home/ashish/go/src/runtime/proc.go:203
runtime.goexit
	/home/ashish/go/src/runtime/asm_amd64.s:1373

```

---

<div class="post-metadata">

**Author:** ![ibrahim](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/ibrahim/32/6387_2.png) [@ibrahim](https://discuss.dgraph.io/u/ibrahim)\
**Post date:** [August 14, 2020, 6:43am UTC](https://discuss.dgraph.io/t/wip-performance-bottleneck-on-1-tb-data/7631/25 "2020-08-14T06:43:00Z")

</div>

The crash mentioned above was seen by @ashishgoswami on [Dgraph v20.07.0-beta.Jun22](https://github.com/dgraph-io/dgraph/releases/tag/v20.07.0-beta.Jun22) and it was fixed in dgraph [Dgraph v20.07.0-rc1](https://github.com/dgraph-io/dgraph/releases/tag/v20.07.0-rc1).  
The following was the fix for the crash in badger  
[https://github.com/dgraph-io/badger/commit/e013bfd25899ec014d7117fa6e94b17a114d9921](https://github.com/dgraph-io/badger/commit/e013bfd25899ec014d7117fa6e94b17a114d9921)

I had a chat with @Paras and @ashishgoswami and the next step is to do the same test on the v20.07.0 binary.

[Previous page](https://discuss.dgraph.io/t/wip-performance-bottleneck-on-1-tb-data/7631.md?page=1)