Memory consumption irregularities

Issues
1

For a while now we have experienced weird behaviour of our alpha regarding memory consumption. For some reason the alpha memory consumption spikes to 4x in a matter of seconds. It does so without any additional or irregular load. All requests to our application end with normal latency.

You can see one incedent of this here:

image
image2484×692 65 KB

You can find the logs produced by the alpha/zero here:
extract-2022-11-23T10_15_09.510Z.csv (27.5 KB)

We run a very simple dgraph installation with a single server running one alpha+the zero. Server spec:

Dgraph version

Dgraph version   : v21.03.2
Dgraph codename  : rocket-2
Dgraph SHA-256   : 00a53ef6d874e376d5a53740341be9b822ef1721a4980e6e2fcb60986b3abfbf
Commit SHA-1     : b17395d33
Commit timestamp : 2021-08-26 01:11:38 -0700
Branch           : HEAD
Go version       : go1.16.2
jemalloc enabled : true

12 Core Intel Xeon Processor
48G of ram
/dgraph is mounted to a luks encrypted SSD drive

3 Likes
2

As it seems that we have come to a conclusion and fixed all of the issues:

It was infact rogue queries that we introduced. How we fixed the issue: increase alpha log verbosity which yields all the executed queries. Watch the DB memory consumption rise and retrigger queries occuring during that timespan. Fix the malicious queries.

The DB is now very chill again.

3

@Joschka Glad you got that sorted. In the past I’ve used cache directives on large (but necessary) queries to mitigate memory spikes: https://dgraph.io/docs/graphql/queries/cached-results

Of course it all depends on the mutation characteristics of the data in question…