What mean rolling up in Dgraph?

(Nguyễn Thanh Phi (Backend MWG)) #1

Hi everyone,
I’m new in Dgraph and when i check server status, i see this

Rolling up Created batch of size: 4.4 MB in 4.552918ms.
Rolling up Created batch of size: 4.4 MB in 4.080644ms.

image
image.png1285×336 14.7 KB

I did not understand what it’s mean? is it alway run everyday? Because when i run query , the response is too slow (up to 2s for simple query). I think task “Rolling up…” may be effect to the perfomance.
I use server with 32GB RAM and 300GB HDD

(Michel Conrado) #2

This is part of Dgraph’s balancing system. So probably you are injecting a lot of data. And Dgraph is doing its work.

(Nguyễn Thanh Phi (Backend MWG)) #3

yes. i have injected a lot of data (about 5 millions predicate data) and my program has finished this job, but now, i still see Dgraph Server doing its work. Maybe have delay ? or another reason i don’t know?
Please give me detail

(Martin Martinez Rivera) #4

This is part of Dgraph’s normal operation. This operation happens periodically so it’s normal to see it happen after all the mutations have finished. There’s nothing to worry about here.

Rollup is an internal operation and kind of hard to explain but I’ll try my best. Data in Dgraph is stored in a data structured called a posting list. When doing a mutation, the main posting list is not modified immediately. Instead, a small posting list is written describing what changed (this is called a delta posting list). Rollup consists of reading all these deltas, along with the complete posting list to generate a new complete posting list with all the changes applied. Reads and writes that happen after the rollup will not need to read any of the previous delta posting lists. If this process didn’t happen, reads and writes would slow down as they would need to read an increasing number of delta posting lists before they can perform their work.

(Quanghaisoft) #5

index data but all query alway > 1second, all query slow down , how to fix it, thank you.

(Nguyễn Thanh Phi (Backend MWG)) #6

It’s mean: while rollup processing, query will be slow? If like this , it’s very bad

(Eason) #7

Hi all,

I am using similar hardware and got rather long rolling-up period (> 30min) as shown below. Just wondered if this is normal:

server_1  | I0702 15:47:47.596019       1 node.go:88] Rolling up Time elapsed: 30m04s, bytes sent: 4.2 GB, speed: 2.3 MB/sec
server_1  | I0702 15:47:48.595972       1 node.go:88] Rolling up Time elapsed: 30m05s, bytes sent: 4.2 GB, speed: 2.3 MB/sec
server_1  | I0702 15:47:49.595907       1 node.go:88] Rolling up Time elapsed: 30m06s, bytes sent: 4.2 GB, speed: 2.3 MB/sec
server_1  | I0702 15:47:50.596005       1 node.go:88] Rolling up Time elapsed: 30m07s, bytes sent: 4.2 GB, speed: 2.3 MB/sec
server_1  | I0702 15:47:51.596014       1 node.go:88] Rolling up Time elapsed: 30m08s, bytes sent: 4.2 GB, speed: 2.3 MB/sec
server_1  | I0702 15:47:52.051266       1 node.go:88] Rolling up Created batch of size: 3.4 MB in 240.654755ms.
server_1  | I0702 15:47:52.051303       1 node.go:88] Rolling up Sent 622350 keys
server_1  | I0702 15:47:52.170660       1 draft.go:988] Rolled up 622350 keys. Done
server_1  | I0702 15:47:52.170750       1 draft.go:389] List rollup at Ts 9184267: OK.
  • Number of nodes in dgraph is roughly 40 millions;
  • whereas number of edges can’t be estimated easily but at least 100 millions
  • Data is constantly being injected.
(Martin Martinez Rivera) #8

Yes, this is normal. If you are injecting data and have a lot of keys the rollup will have lots of work to do. This happens in the background so it does not affect normal operations. If data stops being injected, the rollup will eventually complete faster as there is not much work to do anymore.

(Martin Martinez Rivera) #9

Rollup does not block queries or mutations so Dgraph can operate as usual. Slow downs could happen if there is not enough CPU to do both but this could happen also if there are other processes running. As long as your machine can handle your expected load, rollups should not have much of an effect in performance.