Dgraph is not responding while new data is loading

(Nikita Zaletov) #1

while data is loading into dgraph, even simple queries like

{
  lalala(func: eq(entity_key, ["2da123cd-1831-11e9-b616-1d7484867b10"])) {
    expand(_all_)
  }
}

took are taking than few minutes. right after data loading completes, it is responding fine.
are you going to fix it? this issues makes hard to use dgraph in production where new data is coming every second.

(we are using 1.0.10, if this problem was already fixed in newer version, could you please share an issue?)

(Michel Conrado) #2

hmm, Hi Nikita,

Can you be more specific?

new data is coming from where? Live Load? client usage?
Share your specs too.

Cheers.

(Nikita Zaletov) #3

client usage - we are loading data in batches containing few thousands mutation in each other. it’s done in 2 parallel threads.
dgraph is running on Amazon xlarge (16 gb ram)

(Michel Conrado) #4

How many Nodes? if so, are you using balancing?
or it’s a single config?

Sorry, when I said “client usage” I wanted to refer to a real users.

are you testing a load or they are real users?

Try for a more distributed configuration. So then use a load balancer to send users to less overloaded nodes.

(Nikita Zaletov) #5

we have 3 nodes with replicas=3, so data is not sharded, just replicated. we have only few tens of millions edges for now, so i don’t think we need sharding. anyway, new data is ingested sometimes and it’s normal case.

my question is - is there some kind of index locking for read operations during data insertion, or waiting for quorum on read operations? i didn’t dig into dgraph code since i am not familiar with go. but if it’s just cpu/disk load, then it should just answer a little bit later, but seems like something is locking query.

(Manish R Jain) #6

You might want to use best effort read-only transactions – so they won’t block on guaranteeing linearizability while data is being loaded. We can make it part of v1.0.12 release.

(Nikita Zaletov) #7

that’s cool, thank you so much for it
do i need to specify additional flag when using dgraph4j, or just open read-only transaction? if yes, then when will be that new dgraph4j version released?
thanks