# Crash replicas and recover **URL:** \ **Category:** Dgraph\ **Created:** [July 29, 2018, 10:22pm UTC](https://discuss.dgraph.io/t/crash-replicas-and-recover/2965 "2018-07-29T22:22:32Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![vladimir](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/vladimir/32/972_2.png) [@vladimir](https://discuss.dgraph.io/u/vladimir)\ **Post date:** [July 29, 2018, 10:22pm UTC](https://discuss.dgraph.io/t/crash-replicas-and-recover/2965/1 "2018-07-29T22:22:32Z")
Hello. I’m new in Dgraph I use Dgraph v1.0.7-rc4 release fo Windows10 for experiments i make 1 instance Dgraph Zero and 2 Dgraph server replicas dgraph zero --my=localhost:5080 --replicas 3 -w zeroCluster dgraph server --my=localhost:7080 --lru\_mb=3512 --zero=localhost:5080 -o 0 -p server0/p -w server0/w dgraph server --my=localhost:7081 --lru\_mb=3512 --zero=localhost:5080 -o 1 -p server1/p -w server1/w dgraph server --my=localhost:7082 --lru\_mb=3512 --zero=localhost:5080 -o 2 -p server2/p -w server2/w If I destroy instance of replicas for 2 dgraph servers(ctrl+c many time or shutdown terminal), for example localhost:7081 and localhost:7080 I can’t get access to data and localhost:7082 show errors. How can I correct this situation, recover data and cluster schema? ---
**Author:** ![mrjn](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/mrjn/32/775_2.png) [@mrjn](https://discuss.dgraph.io/u/mrjn)\ **Post date:** [July 29, 2018, 10:48pm UTC](https://discuss.dgraph.io/t/crash-replicas-and-recover/2965/2 "2018-07-29T22:48:32Z")
@MichelDiz can help more, but you must have 2 out of 3 replicas up and running for data application and retrieval. Dgraph does consistent replication via Raft, which enforces this stipulation. ---
**Author:** ![MichelDiz](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/micheldiz/32/11873_2.png) [@MichelDiz](https://discuss.dgraph.io/u/MichelDiz)\ **Post date:** [July 29, 2018, 11:16pm UTC](https://discuss.dgraph.io/t/crash-replicas-and-recover/2965/3 "2018-07-29T23:16:51Z")
Vladimir, Can you paste here the logs you get? What is the content of your experiment? Did you use LIVE or Bulk? or manually entered data via mutation? How long did it take between you to load the data and run the shutdown? (just to know if the servers have had time to sync) did you see logs like “moving predicate A”? and waited it to finish? When you force a shutdown or there is a sudden shutdown. The Dgraph leaves a Lock in the folders that were being used. Before use it is necessary to remove them. After recovering the server instances did you get errors? paste logs pls. Cheers. ---
**Author:** ![vladimir](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/vladimir/32/972_2.png) [@vladimir](https://discuss.dgraph.io/u/vladimir)\ **Post date:** [July 30, 2018, 5:44pm UTC](https://discuss.dgraph.io/t/crash-replicas-and-recover/2965/4 "2018-07-30T17:44:51Z")
Can you help me, how I can get logs when I start dgraph server on Windows 10? I use mutation from tour: [https://tour.dgraph.io/intro/3/](https://tour.dgraph.io/intro/3/) [https://tour.dgraph.io/intro/4/](https://tour.dgraph.io/intro/4/) [https://tour.dgraph.io/intro/5/](https://tour.dgraph.io/intro/5/) When I destroy the first replica. It’s ok. Then a new leader is chosen. It’s two replicas on. If I destroyed leader. I got mistakes. But if leader on last server, it’s ok too. Then I destroy all. After that, I try this algorithm: I deleting Lock from zeroCluster and start zero server dgraph zero --my=localhost:5080 --replicas 3 -w zeroCluster I deleting Lock from server0 p&w and start server0 dgraph server --my=localhost:7080 --lru\_mb=3512 --zero=localhost:5080 -o 0 -p server0/p -w server0/w I’m deleting Lock from server1 p&w and start server1 dgraph server --my=localhost:7081 --lru\_mb=3512 --zero=localhost:5080 -o 1 -p server1/p -w server1/w I’m deleting Lock from server2 p&w and start server2 dgraph server --my=localhost:7082 --lru\_mb=3512 --zero=localhost:5080 -o 2 -p server2/p -w server2/w And all ok )) Can I manually change a leader from one to another host? ---
**Author:** ![MichelDiz](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/micheldiz/32/11873_2.png) [@MichelDiz](https://discuss.dgraph.io/u/MichelDiz)\ **Post date:** [July 30, 2018, 6:28pm UTC](https://discuss.dgraph.io/t/crash-replicas-and-recover/2965/5 "2018-07-30T18:28:50Z")
About saving the logs on cmd [How to Copy the Command Output to Windows Clipboard - Digital Inspiration](https://www.labnol.org/software/copy-command-output-to-clipboard/2506/) I believe you can do `dgraph zero --my=localhost:5080 --replicas 3 -w zeroCluster >C:\zeroCluster_output.txt` I’m not sure if Microsoft gives support for pipes (but try): `dgraph cmmand cmmand |& tee -a zeroCluster_output.txt` But you can use the “bash” from github install. > [@vladimir](#): > > Can I manually change a leader from one to another host? > /removeNode?id=3&group=2 If a replica goes down and can’t be recovered, you can remove it and add a new node to the quorum. This endpoint can be used to remove a dead Zero or Dgraph server node. To remove dead Zero nodes, just pass group=0 and the id of the Zero node. Take a read [Get started with Dgraph](https://docs.dgraph.io/deploy/#more-about-dgraph-zero) > Generally, the servers are in leader or follower state. When the leader crashes or the communication breaks down, the followers will wait for election timeout before converting to candidates. The election timeout is randomized. This would allow one of them to declare candidacy before others. The candidate would vote for itself and wait for the majority of the cluster to vote for it as well. If a follower hears from a candidate with a higher term than the current (dead in this case) leader, it would vote for it. The candidate who gets majority votes wins the election and becomes the leader. more: [Get started with Dgraph](https://docs.dgraph.io/design-concepts/#server-states) ---
**Author:** ![vladimir](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/vladimir/32/972_2.png) [@vladimir](https://discuss.dgraph.io/u/vladimir)\ **Post date:** [July 30, 2018, 6:38pm UTC](https://discuss.dgraph.io/t/crash-replicas-and-recover/2965/6 "2018-07-30T18:38:13Z")
I believe you can do dgraph zero --my=localhost:5080 --replicas 3 -w zeroCluster \>C:\zeroCluster\_output.txt It’s work. Thanks. ---
**Author:** ![vladimir](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/vladimir/32/972_2.png) [@vladimir](https://discuss.dgraph.io/u/vladimir)\ **Post date:** [July 30, 2018, 7:02pm UTC](https://discuss.dgraph.io/t/crash-replicas-and-recover/2965/7 "2018-07-30T19:02:54Z")
I can safe close servers with /admin/shutdown How I can safe close zero server. Or **kill pid** is safe?