# Badger read performance on 1TB data

**URL:** <https://discuss.dgraph.io/t/badger-read-performance-on-1tb-data/9703>\
**Category:** Badger\
**Created:** [August 27, 2020, 4:16pm UTC](https://discuss.dgraph.io/t/badger-read-performance-on-1tb-data/9703 "2020-08-27T16:16:15Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![ahsan](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/ahsan/32/3517_2.png) [@ahsan](https://discuss.dgraph.io/u/ahsan)\
**Post date:** [August 27, 2020, 4:16pm UTC](https://discuss.dgraph.io/t/badger-read-performance-on-1tb-data/9703/1 "2020-08-27T16:16:15Z")

</div>

I’m working on testing badger read performance on 1TB of data. This posts lists the results of the tests.  
[Note: This is a work in progress document]

# Test 1 (Data without compression)

The badger `benchmark read tool` has been used for this test, and the test was carried out on this [commit](https://github.com/dgraph-io/badger/commit/ed3b219a7c096ab6a7cdaec631bab47eb3542b78). We ran the test on 836GB of data written without compression. The level structure of the LSM tree is as follows:

```auto
Level 0 size: 0 B
Level 1 size: 2.2 GB
Level 2 size: 90 GB
Level 3 size: 361 GB
Total index size: 452 GB
Value log size: 445 GB

Total number of SSTs: 6615
Directory size: 836GB

```

The test samples 1million keys in random order. We then randomly choose a key from these 1million keys and get its value using `txn.Get()`, we repeatedly do this for a fixed time span. The test is running **without block and index cache** , also **LoadBloomsOnOpen=false**.

We did this for the following time spans:

### 1 Minute Duration

In one minute it read 2.3GB of data at a speed of 38MB/s.

```auto
badger 2020/08/27 11:53:27 INFO: All 6615 tables opened in 40.167s
...
Time elapsed: 59s, bytes read: 2.3 GB, speed: 38 MB/sec, entries read: 14142937, speed: 239710/sec

```

The heap profile during the test:

```auto
Showing nodes accounting for 9351.92MB, 100% of 9355.08MB total
Dropped 15 nodes (cum <= 46.78MB)
Showing top 5 nodes out of 19
      flat flat% sum% cum cum%
 5584.05MB 59.69% 59.69% 9267.22MB 99.06% github.com/dgraph-io/badger/v2/pb.(*TableIndex).Unmarshal
 3683.17MB 39.37% 99.06% 3683.17MB 39.37% github.com/dgraph-io/badger/v2/pb.(*BlockOffset).Unmarshal
   83.20MB 0.89% 100% 83.20MB 0.89% github.com/dgraph-io/badger/v2/skl.newArena
       1MB 0.011% 100% 9268.87MB 99.08% github.com/dgraph-io/badger/v2/table.OpenTable
    0.50MB 0.0054% 100% 9271.38MB 99.11% github.com/dgraph-io/badger/v2.newLevelsController.func1

```

### 5 Minutes Duration

```auto
Time elapsed: 04m59s, bytes read: 12 GB, speed: 41 MB/sec, entries read: 77040272, speed: 257659/sec

```

Heap profile after 4 minute of run:

```auto
Showing nodes accounting for 12773.51MB, 99.10% of 12889.37MB total
Dropped 68 nodes (cum <= 64.45MB)
Showing top 5 nodes out of 19
      flat flat% sum% cum cum%
 7647.57MB 59.33% 59.33% 12687.30MB 98.43% github.com/dgraph-io/badger/v2/pb.(*TableIndex).Unmarshal
 5039.73MB 39.10% 98.43% 5039.73MB 39.10% github.com/dgraph-io/badger/v2/pb.(*BlockOffset).Unmarshal
   83.20MB 0.65% 99.08% 83.20MB 0.65% github.com/dgraph-io/badger/v2/skl.newArena
       3MB 0.023% 99.10% 12690.80MB 98.46% github.com/dgraph-io/badger/v2/table.OpenTable
         0 0% 99.10% 86.62MB 0.67% github.com/dgraph-io/badger/v2.Open

```

### 10 Minutes Duration

```auto
Time elapsed: 09m59s, bytes read: 25 GB, speed: 42 MB/sec, entries read: 155817374, speed: 260129/sec

```

Heap profile after 9 minute of run:

```auto
Showing nodes accounting for 12645.74MB, 99.06% of 12765.66MB total
Dropped 100 nodes (cum <= 63.83MB)
Showing top 5 nodes out of 19
      flat flat% sum% cum cum%
 7638.31MB 59.83% 59.83% 12559.53MB 98.39% github.com/dgraph-io/badger/v2/pb.(*TableIndex).Unmarshal
 4921.23MB 38.55% 98.39% 4921.23MB 38.55% github.com/dgraph-io/badger/v2/pb.(*BlockOffset).Unmarshal
   83.20MB 0.65% 99.04% 83.20MB 0.65% github.com/dgraph-io/badger/v2/skl.newArena
       3MB 0.024% 99.06% 12563.53MB 98.42% github.com/dgraph-io/badger/v2/table.OpenTable
         0 0% 99.06% 87.41MB 0.68% github.com/dgraph-io/badger/v2.Open

```

### Full DB Scan

In this test, the full DB is scanned using iterators. It took 57 minutes to read 400GB of entries with an average read speed of 117 MB/s.

```auto
Time elapsed: 56m57s, bytes read: 400 GB, speed: 117 MB/sec, entries read: 2500000000, speed: 731635/sec

```

The heap profile after ~50 minutes of run is as follows:

```auto
Showing nodes accounting for 12572.05MB, 99.91% of 12583.90MB total
Dropped 50 nodes (cum <= 62.92MB)
Showing top 5 nodes out of 19
      flat flat% sum% cum cum%
 7568.12MB 60.14% 60.14% 12487.35MB 99.23% github.com/dgraph-io/badger/v2/pb.(*TableIndex).Unmarshal
 4919.23MB 39.09% 99.23% 4919.23MB 39.09% github.com/dgraph-io/badger/v2/pb.(*BlockOffset).Unmarshal
   83.20MB 0.66% 99.89% 83.20MB 0.66% github.com/dgraph-io/badger/v2/skl.newArena
    1.50MB 0.012% 99.91% 12488.85MB 99.24% github.com/dgraph-io/badger/v2/table.OpenTable
         0 0% 99.91% 86.02MB 0.68% github.com/dgraph-io/badger/v2.Open

```

# Test 2 - on compressed Data

This test was carried out on compressed data. The details about the data is as follows:

```auto
Level 0 size: 0 B
Level 1 size: 1.1 GB
Level 2 size: 28 GB
Level 3 size: 256 GB
Level 4 size: 36 GB
Total index size: 322 GB
Value log size: 584 GB

Total number of SSTs: 21040
Directory size: 844 GB

```

This test runs with **cache enabled** and **LoadBloomsOnOpen:true**. The test is run using the following parameters:

```auto
go run main.go benchmark read \
  --index-cache=500 \ # Index Cache of 500MB
  --block-cache=1000 \ # Block Cache of 1GB
  -d=(x)m \ # Duration of x minutes, x ∈ {1m, 5m and 10m}
  --dir=../../data # Data directory

```

The results are as follows:

### 1 Minute Duration

```auto
badger 2020/08/27 13:21:47 INFO: All 21040 tables opened in 1m50.532s
...
Time elapsed: 59s, bytes read: 2.0 GB, speed: 34 MB/sec, entries read: 12606368, speed: 213667/sec

```

Heap profile during the test:

```auto
Showing nodes accounting for 1174.78MB, 89.28% of 1315.86MB total
Dropped 50 nodes (cum <= 6.58MB)
Showing top 5 nodes out of 79
      flat flat% sum% cum cum%
     384MB 29.18% 29.18% 384MB 29.18% github.com/dgraph-io/ristretto.newCmRow
  354.37MB 26.93% 56.11% 506.88MB 38.52% github.com/dgraph-io/badger/v2/pb.(*TableIndex).Unmarshal
  200.70MB 15.25% 71.37% 200.70MB 15.25% github.com/dgraph-io/ristretto/z.(*Bloom).Size
  152.51MB 11.59% 82.96% 152.51MB 11.59% github.com/dgraph-io/badger/v2/pb.(*BlockOffset).Unmarshal
   83.20MB 6.32% 89.28% 83.20MB 6.32% github.com/dgraph-io/badger/v2/skl.newArena

```

### 5 Minutes Duration

```auto
Time elapsed: 04m59s, bytes read: 11 GB, speed: 36 MB/sec, entries read: 67763608, speed: 226634/sec

```

Heap profile at ~4min during the run:

```auto
Showing nodes accounting for 1178.46MB, 90.21% of 1306.37MB total
Dropped 45 nodes (cum <= 6.53MB)
Showing top 5 nodes out of 79
      flat flat% sum% cum cum%
     384MB 29.39% 29.39% 384MB 29.39% github.com/dgraph-io/ristretto.newCmRow
  379.39MB 29.04% 58.44% 507.39MB 38.84% github.com/dgraph-io/badger/v2/pb.(*TableIndex).Unmarshal
  203.86MB 15.61% 74.04% 203.86MB 15.61% github.com/dgraph-io/ristretto/z.(*Bloom).Size
     128MB 9.80% 83.84% 128MB 9.80% github.com/dgraph-io/badger/v2/pb.(*BlockOffset).Unmarshal
   83.20MB 6.37% 90.21% 83.20MB 6.37% github.com/dgraph-io/badger/v2/skl.newArena

```

### 10 Minutes Duration

```auto
Time elapsed: 09m59s, bytes read: 22 GB, speed: 36 MB/sec, entries read: 134503361, speed: 224546/sec

```

Heap profile at ~9m:

```auto
Showing nodes accounting for 1161.13MB, 90.01% of 1290.04MB total
Dropped 63 nodes (cum <= 6.45MB)
Showing top 5 nodes out of 79
      flat flat% sum% cum cum%
     384MB 29.77% 29.77% 384MB 29.77% github.com/dgraph-io/ristretto.newCmRow
  358.72MB 27.81% 57.57% 493.22MB 38.23% github.com/dgraph-io/badger/v2/pb.(*TableIndex).Unmarshal
  200.70MB 15.56% 73.13% 200.70MB 15.56% github.com/dgraph-io/ristretto/z.(*Bloom).Size
  134.50MB 10.43% 83.56% 134.50MB 10.43% github.com/dgraph-io/badger/v2/pb.(*BlockOffset).Unmarshal
   83.20MB 6.45% 90.01% 83.20MB 6.45% github.com/dgraph-io/badger/v2/skl.newArena

```

### Full DB scan

It took 4h24m to read 8.2 billion entries using the iterator.

```auto
Time elapsed: 04h24m02s, bytes read: 1.2 TB, speed: 76 MB/sec, entries read: 8279106798, speed: 522604/sec

```

The heap profile at ~4h is:

```auto
Showing nodes accounting for 1185.06MB, 92.34% of 1283.34MB total
Dropped 60 nodes (cum <= 6.42MB)
Showing top 5 nodes out of 39
      flat flat% sum% cum cum%
     384MB 29.92% 29.92% 384MB 29.92% github.com/dgraph-io/ristretto.newCmRow
  328.35MB 25.59% 55.51% 525.86MB 40.98% github.com/dgraph-io/badger/v2/pb.(*TableIndex).Unmarshal
  197.51MB 15.39% 70.90% 197.51MB 15.39% github.com/dgraph-io/badger/v2/pb.(*BlockOffset).Unmarshal
     192MB 14.96% 85.86% 192MB 14.96% github.com/dgraph-io/ristretto/z.(*Bloom).Size
   83.20MB 6.48% 92.34% 83.20MB 6.48% github.com/dgraph-io/badger/v2/skl.newArena

```

cc @ibrahim @mrjn