Storage Growth Even After Predicate Moves 1.0.5

Users Dgraph
1

Hello,

I am experiencing some odd behavior in dgraph I think. It seems that no matter how many predicates are moved off of my first server in a 5 server cluster, the storage continues to grow on that first machine. It has multiple GB vs the other servers having a few hundred MB and it just seems to continue to grow, even after a predicate move. Running this in AWS. Is there an explanation for this, or something I could have done to cause it?

Thanks,
Will

1 Like
2

Need more details to try to answer that. If necessary, call me in box talking about your experience.

There’s no way to predict what it’s happening without details since there’s no other issues (I believe) like that until now.

Michel Conrado,
Dgraph Support Engineer.

3

AWS machines (5x):

r4.xlarge
4 CPU
30.5 RAM
Up to 10 Gigabit

Dgraph Version:

Dgraph version   : v1.0.5
Commit SHA-1     : 82787414
Commit timestamp : 2018-04-20 15:50:53 +1000
Branch           : HEAD

Configs Per Machine:

where the idx and my variables are per machine
{
  "lru_mb": 15000,
  "postings": "/mnt/dgraph/p",
  "wal": "/mnt/dgraph/w",
  "export": "/mnt/dgraph_exports",
  "bindall": true,
  "debug": true,
  "idx": 1,
  "expand_edge": false,
  "my": "dgraph1-xxx:7080",
  "zero": "dgraphzero1-xxx:5080"
}

Systemd service config:

[Unit]
Description=dgraph service

[Service]
ExecStart=/usr/local/bin/dgraph server --config=/etc/dgraph/dgraph.json
ExecStop=/usr/bin/killall dgraph

[Install]
WantedBy=multi-user.target

Current State:

So basically, even though moves are certainly happening predicate-wise, but the storage just keeps going up on 1.

May 21 00:42:24 dgraphzero1-prod dgraph[29842]: 2018/05/21 00:42:24 tablet.go:69: Going to move predicate: [PROFILE_RELATIONSHIP_rate_twelve_months], size: [3.3 MB] from group 1 to 5
May 21 00:42:24 dgraphzero1-prod dgraph[29842]: 2018/05/21 00:42:24 oracle.go:75: purging below ts:2185012, len(o.commits):927, len(o.aborts):23, len(o.rowCommit):10241
May 21 00:42:27 dgraphzero1-prod dgraph[29842]: 2018/05/21 00:42:27 tablet.go:104: Predicate move done for: [PROFILE_RELATIONSHIP_rate_twelve_months] from group 1 to 5
May 21 00:42:34 dgraphzero1-prod dgraph[29842]: 2018/05/21 00:42:34 oracle.go:75: purging below ts:2185396, len(o.commits):607, len(o.aborts):26, len(o.rowCommit):8107
1 Like
4

Hey @willcj33,

I’d like you to try out a new version of Dgraph. I’m planning to make a new release, but wanted to fix a couple more bugs before I do. Can you try building from master? I’ve pulled in the new Badger (1.5.0), which actually does a better job of reclaiming disk space.

1 Like
5

Ah yeah, I will bring in Master and see how it goes. Will I need to run data from scratch?

6

The format of data shouldn’t change. It should read the directories directly back. (But always better to export the data)

1 Like