Ever Increasing Memory on Dgraph Alpha

Users Dgraph Cloud
1

We are running dgraph cluster which currently has:

Zero Node : 32 GB RAM and 16 cores [Ubuntu]
Alpha Node 1: 32 GB RAM and 16 cores [Ubuntu]

Alpha Node 2: 32 GB RAM and 8 cores [Ubuntu]

It has 87 GB of data on both alpha nodes.

I am running a load test using JMeter and querying dgraph with a depth of 5 nodes.

For some of the cases, the query takes time and both alpha nodes go out of memory.

Observations:

  1. We earlier had 1 alpha node, we resharded and added node 2. After resharding, we saw an increase in used memory. What is the reason for this increase?
  2. While running the load test, I see a consistent increase in memory in both alpha nodes.
    After the load test, memory settles down after 30 mins. even though there are no pending queries in the system.

Screenshot 2021-11-29 at 5.24.25 PM
Screenshot 2021-11-29 at 5.24.25 PM1920×824 75.6 KB

2
  1. during reshard (/moveTablet) it will stream a whole tablet out to the new destination, which will take memory.
  2. I believe you are seeing the caches in action there.
3

Isn’t re-sharding a one-time process? Or will it trigger every time we run a query?

4

I was just referring to this comment and a possible cause of increased used memory after resharding. Yes, that is a one time thing. As far as why the plateau on the right of your screenshot is higher than the left side, there could be any number of reasons, I only have guesses. A memory profile is easy to take and inspect, if you have the ability to.

5

@ujjawaldx The best way to figure out what’s causing high memory usage would be to add some logs and attach a memory profile. The memory profile will have answers to what’s consuming memory.

https://dgraph.io/docs/howto/retrieving-debug-information/#profiling-information