# Dgraph server log

**URL:** <https://discuss.dgraph.io/t/dgraph-server-log/2795>\
**Category:** Dgraph\
**Created:** [June 26, 2018, 1:12am UTC](https://discuss.dgraph.io/t/dgraph-server-log/2795 "2018-06-26T01:12:05Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![shanghai-Jerry](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/shanghai-jerry/32/863_2.png) [@shanghai-Jerry](https://discuss.dgraph.io/u/shanghai-Jerry)\
**Post date:** [June 26, 2018, 1:12am UTC](https://discuss.dgraph.io/t/dgraph-server-log/2795/1 "2018-06-26T01:12:05Z")

</div>

![02](https://canada1.discourse-cdn.com/flex007/uploads/dgraph/original/1X/86adad4a2de8cdab154cec64f786be9ddd172483.png)

what happened to server?  
it this normal for server’s log?  
all server in cluster had this kind of log.  
and the most important, servers’ speed are pretty slow.

i guess something happened. i need your help.  
@mrjn

---

<div class="post-metadata">

**Author:** ![mrjn](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/mrjn/32/775_2.png) [@mrjn](https://discuss.dgraph.io/u/mrjn)\
**Post date:** [June 26, 2018, 1:26am UTC](https://discuss.dgraph.io/t/dgraph-server-log/2795/2 "2018-06-26T01:26:33Z")

</div>

It was this issue: [During replication server consumes all RAM and crashes · Issue #2424 · dgraph-io/dgraph · GitHub](https://github.com/dgraph-io/dgraph/issues/2424)

Got fixed in master. Would be present in v1.0.6.

---

<div class="post-metadata">

**Author:** ![shanghai-Jerry](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/shanghai-jerry/32/863_2.png) [@shanghai-Jerry](https://discuss.dgraph.io/u/shanghai-Jerry)\
**Post date:** [June 26, 2018, 1:33am UTC](https://discuss.dgraph.io/t/dgraph-server-log/2795/3 "2018-06-26T01:33:22Z")

</div>

![12](https://canada1.discourse-cdn.com/flex007/uploads/dgraph/original/1X/0079b3094d7a3f149ae9db88434b0d6f54b94ee5.png)

one of the server can not start again while stopped. it seems the commit is out of range. how to fix it to let start .  
by the way, when the v1.0.6 would be presented?

---

<div class="post-metadata">

**Author:** ![mrjn](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/mrjn/32/775_2.png) [@mrjn](https://discuss.dgraph.io/u/mrjn)\
**Post date:** [June 26, 2018, 1:39am UTC](https://discuss.dgraph.io/t/dgraph-server-log/2795/4 "2018-06-26T01:39:34Z")

</div>

Aiming for it this week. Maybe run Dgraph with `--replicas=1` flag until then.

---

<div class="post-metadata">

**Author:** ![shanghai-Jerry](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/shanghai-jerry/32/863_2.png) [@shanghai-Jerry](https://discuss.dgraph.io/u/shanghai-Jerry)\
**Post date:** [June 26, 2018, 2:02am UTC](https://discuss.dgraph.io/t/dgraph-server-log/2795/5 "2018-06-26T02:02:06Z")

</div>

i set three servers in cluster, while --replicas=1, there are too many log like : xx predicate is removed xxxx.

---

<div class="post-metadata">

**Author:** ![mrjn](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/mrjn/32/775_2.png) [@mrjn](https://discuss.dgraph.io/u/mrjn)\
**Post date:** [June 26, 2018, 2:06am UTC](https://discuss.dgraph.io/t/dgraph-server-log/2795/6 "2018-06-26T02:06:40Z")

</div>

You might have to clear out both Zero and Server directories, so the old state is gone.

---

<div class="post-metadata">

**Author:** ![shanghai-Jerry](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/shanghai-jerry/32/863_2.png) [@shanghai-Jerry](https://discuss.dgraph.io/u/shanghai-Jerry)\
**Post date:** [June 26, 2018, 2:10am UTC](https://discuss.dgraph.io/t/dgraph-server-log/2795/7 "2018-06-26T02:10:02Z")

</div>

i need to clear both Zero and Server directories in cluster, and all data are gone.

that’s not acceptable

---

<div class="post-metadata">

**Author:** ![mrjn](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/mrjn/32/775_2.png) [@mrjn](https://discuss.dgraph.io/u/mrjn)\
**Post date:** [June 26, 2018, 2:15am UTC](https://discuss.dgraph.io/t/dgraph-server-log/2795/8 "2018-06-26T02:15:16Z")

</div>

Actually, you could remove nodes from Zero, by using `removeNode` endpoint. You can achieve this without removing dirs.

---

<div class="post-metadata">

**Author:** ![karan28aug](https://avatars.discourse-cdn.com/v4/letter/k/4da419/32.png) [@karan28aug](https://discuss.dgraph.io/u/karan28aug)\
**Post date:** [June 26, 2018, 5:05am UTC](https://discuss.dgraph.io/t/dgraph-server-log/2795/9 "2018-06-26T05:05:46Z")

</div>

`--replicas` : is the option that controls the replication factor.  
If `--replicas=1` then each server node will serve different group.  
If the replica goes down and can not be recovered, you can clear out both the Zero and Server directories for removing the old state and add a new node to the quorum.  
The endpoint can be used to remove a Zero or Dgraph server node.

```auto
For example : Start Zero servers : 
./dgraph zero --replicas 3 --idx 1

```

```auto
# Stop the leader first. Assuming node with idx 1 is leader : 
example: curl "localhost:6081/removeNode?group=0&id=1"

```

```auto
#Restart the node with a peer : 
example : ./dgraph zero --replicas 3 --idx 1 --peer localhost:5081

```

```auto
Start a dgraph server pointing it to the restarted node.

```

For the new one, you should not use the same idx as that of a node that was removed.

---

<div class="post-metadata">

**Author:** ![shanghai-Jerry](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/shanghai-jerry/32/863_2.png) [@shanghai-Jerry](https://discuss.dgraph.io/u/shanghai-Jerry)\
**Post date:** [June 26, 2018, 6:51am UTC](https://discuss.dgraph.io/t/dgraph-server-log/2795/10 "2018-06-26T06:51:03Z")

</div>

> [@karan28aug](#):
>
> If --replicas=1 then each server node will serve different group.

how to understand this sentence? can you give me more information about this?

---

<div class="post-metadata">

**Author:** ![shanghai-Jerry](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/shanghai-jerry/32/863_2.png) [@shanghai-Jerry](https://discuss.dgraph.io/u/shanghai-Jerry)\
**Post date:** [June 26, 2018, 12:46pm UTC](https://discuss.dgraph.io/t/dgraph-server-log/2795/11 "2018-06-26T12:46:53Z")

</div>

if i set replicas=1, it happened:

UNKNOWN: Predicate is being moved, please retry later

server log

 ![33%20PM](https://canada1.discourse-cdn.com/flex007/uploads/dgraph/original/1X/b8695f5fe1c1e6c325a89ce60974540fa0a47ee2.png)

---

<div class="post-metadata">

**Author:** ![karan28aug](https://avatars.discourse-cdn.com/v4/letter/k/4da419/32.png) [@karan28aug](https://discuss.dgraph.io/u/karan28aug)\
**Post date:** [June 26, 2018, 6:02pm UTC](https://discuss.dgraph.io/t/dgraph-server-log/2795/12 "2018-06-26T18:02:57Z")

</div>

Predicate is being moved : this is to make sure that all the servers have equal data stored in them.  
You can

```auto
--rebalance_interval duration Interval for trying a predicate move. (default 8m0s)

```

you can change the duration to some xx hours to avoid predicate move.

---

<div class="post-metadata">

**Author:** ![karan28aug](https://avatars.discourse-cdn.com/v4/letter/k/4da419/32.png) [@karan28aug](https://discuss.dgraph.io/u/karan28aug)\
**Post date:** [June 26, 2018, 6:04pm UTC](https://discuss.dgraph.io/t/dgraph-server-log/2795/13 "2018-06-26T18:04:18Z")

</div>

```auto
https://docs.dgraph.io/deploy#more-about-dgraph-zero

```

You can check this for more information on

```auto
 --replicas=1 then each server node will serve different group.

```

---

<div class="post-metadata">

**Author:** ![shanghai-Jerry](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/shanghai-jerry/32/863_2.png) [@shanghai-Jerry](https://discuss.dgraph.io/u/shanghai-Jerry)\
**Post date:** [June 27, 2018, 1:31am UTC](https://discuss.dgraph.io/t/dgraph-server-log/2795/14 "2018-06-27T01:31:27Z")

</div>

but bulk loader took several hours, i don’t know how long to set rebalance\_interval. is this matters?

---

<div class="post-metadata">

**Author:** ![shanghai-Jerry](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/shanghai-jerry/32/863_2.png) [@shanghai-Jerry](https://discuss.dgraph.io/u/shanghai-Jerry)\
**Post date:** [June 27, 2018, 1:37am UTC](https://discuss.dgraph.io/t/dgraph-server-log/2795/15 "2018-06-27T01:37:21Z")

</div>

> [@karan28aug](#):
>
> Predicate is being moved : this is to make sure that all the servers have equal data stored in them.

and --replicas=1 then each server node will serve different group.

first : i don’t know why, if one server died, the dgraph cluster can not server if all the servers have equal data stored in them.

second: if all the servers have equal data stored in them, why does the server need to move not just copy the Predicate, what’s the differences?

how to understand the operation “Predicate is moved”?  
what really happened to servers?

Much thanks

---

<div class="post-metadata">

**Author:** ![mrjn](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/mrjn/32/775_2.png) [@mrjn](https://discuss.dgraph.io/u/mrjn)\
**Post date:** [June 27, 2018, 5:02pm UTC](https://discuss.dgraph.io/t/dgraph-server-log/2795/16 "2018-06-27T17:02:49Z")

</div>

If replicas=1, then there’s only one copy of each data. And data is sharded across groups.

> [@shanghai-Jerry](#):
>
> first : i don’t know why, if one server died, the dgraph cluster can not server if all the servers have equal data stored in them.

So, if a server dies, then that data is not being serviced at all. So, queries which need those predicates won’t be able to function.

> [@shanghai-Jerry](#):
>
> second: if all the servers have equal data stored in them, why does the server need to move not just copy the Predicate, what’s the differences?

The reason move is happening is that the servers don’t have equal data stored in them. Zero is trying to adjust the data between groups so that each group would contain roughly the same amount of data, by moving some predicates from one group to another.