I’m encountering an issue with Dgraph backup and restore. I have two separate servers, Server A and Server B, each configured with Dgraph version 23.0.0. My goal is to take a backup from Server A and restore it on Server B.
Backup Command used on Server A:
curl -X POST -H "X-Dgraph-AccessToken: $TOKEN" -H "Content-Type: application/json" http://localhost:8080/admin -d '{
"query": "mutation { backup(input: {destination: \"/home/ubuntu/dgraph-backup-02-11-2023\"}) { response { code message } } }"
}'
I’m able to create the backup successfully on Server A.
Restore Command used on Server B:
curl -X POST -H "X-Dgraph-AccessToken: $TOKEN" -H "Content-Type: application/json" http://localhost:8080/admin -d '{
"query": "mutation { restore(input: {location: \"/home/ubuntu/dgraph-backup-02-11-2023/dgraph.20231102.063836.848\"}) { code message } }'
I receive a success message indicating that the restore operation has started:
Issue: The restore operation completes successfully, but the schema is not visible in the Dgraph console. The Dgraph Alpha logs contain errors.
{"data":{"restore":{"code":"Success","message":"Restore operation started."}},"extensions":{"tracing":{"version":1,"startTime":"2023-11-03T08:55:32.028555122Z","endTime":"2023-11-03T08:55:32.031573391Z","duration":3018289}}
However, I’m unable to see the restored schema in the Dgraph console. Upon checking the Dgraph Alpha logs during the restore command, I noticed multiple errors.
Dgraph Alpha Logs (Sample):
Nov 03 08:55:32 ip-172-31-86-179 bash[187622]: E1103 08:55:32.088576 187639 groups.go:481] Error while ShouldServe grpc call rpc error: code = Unavailable desc = error reading from server: EOF
Nov 03 08:55:32 ip-172-31-86-179 bash[187622]: E1103 08:55:32.088653 187639 draft.go:764] Applying proposal. Error: cannot create tablet for restored predicate
Nov 03 08:55:32 ip-172-31-86-179 bash[187622]: dgraph.graphql.xid: rpc error: code = Unavailable desc = error reading from server: EOF. Proposal: {"<nil>" [] "<nil>" "" "<nil>" "<nil>" 'ᇚ' '\x00' "group_id:1 restore_ts:2614037 location:\"/home/ubuntu/dgraph-backup-02-11-2023\" " "<nil>" "<nil>" '\x00'}.
Nov 03 08:55:32 ip-172-31-86-179 bash[187622]: E1103 08:55:32.088957 187639 groups.go:890] Unable to sync memberships. Error: rpc error: code = Unavailable desc = error reading from server: EOF. State: <nil>
Nov 03 08:55:32 ip-172-31-86-179 bash[187622]: I1103 08:55:32.088978 187639 groups.go:868] Got address of a Zero leader: localhost:5080
Nov 03 08:55:32 ip-172-31-86-179 bash[187622]: E1103 08:55:32.089565 187639 groups.go:998] Error in oracle delta stream. Error: rpc error: code = Unavailable desc = error reading from server: EOF
Nov 03 08:55:32 ip-172-31-86-179 bash[187622]: I1103 08:55:32.089690 187639 draft.go:134] Operation completed with id: opRestore
Nov 03 08:55:32 ip-172-31-86-179 bash[187622]: E1103 08:55:32.089771 187639 online_restore.go:159] Error while restoring cannot propose restore request: cannot create tablet for restored predicate
Nov 03 08:55:32 ip-172-31-86-179 bash[187622]: dgraph.graphql.xid: rpc error: code = Unavailable desc = error reading from server: EOF
Nov 03 08:55:32 ip-172-31-86-179 bash[187622]: E1103 08:55:32.090474 187639 groups.go:875] Error while calling update rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:5080: connect: connection refused"
Nov 03 08:55:32 ip-172-31-86-179 bash[187622]: W1103 08:55:32.090522 187639 server_state.go:226] Error while retrieving timestamps: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:5080: connect: connection refused" with delay: 10ms. Will retry...
Nov 03 08:55:32 ip-172-31-86-179 bash[187622]: W1103 08:55:32.101465 187639 server_state.go:226] Error while retrieving timestamps: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:5080: connect: connection refused" with delay: 20ms. Will retry...
Nov 03 08:55:32 ip-172-31-86-179 bash[187622]: W1103 08:55:32.121922 187639 server_state.go:226] Error while retrieving timestamps: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:5080: connect: connection refused" with delay: 40ms. Will retry...
Nov 03 08:55:32 ip-172-31-86-179 bash[187622]: W1103 08:55:32.162660 187639 server_state.go:226] Error while retrieving timestamps: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:5080: connect: connection refused" with delay: 80ms. Will retry...
Nov 03 08:55:32 ip-172-31-86-179 bash[187622]: W1103 08:55:32.243816 187639 server_state.go:226] Error while retrieving timestamps: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:5080: connect: connection refused" with delay: 160ms. Will retry...
Nov 03 08:55:32 ip-172-31-86-179 bash[187622]: E1103 08:55:32.374208 187639 run.go:461] GRPC listener canceled: accept tcp [::]:9080: use of closed network connection
Nov 03 08:55:32 ip-172-31-86-179 bash[187622]: E1103 08:55:32.374248 187639 server.go:31] error from cmux serve: accept tcp [::]:8080: use of closed network connection
Nov 03 08:55:32 ip-172-31-86-179 bash[187622]: E1103 08:55:32.374265 187639 server.go:71] Stopped taking more http(s) requests. Err: mux: listener closed
Nov 03 08:55:32 ip-172-31-86-179 bash[187622]: E1103 08:55:32.375404 187639 groups.go:1229] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x16dgraph.rule.permission": while receiving from stream: rpc error: code = Canceled desc = context canceled. closer err: context canceled
Nov 03 08:55:32 ip-172-31-86-179 bash[187622]: W1103 08:55:32.405001 187639 server_state.go:226] Error while retrieving timestamps: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:5080: connect: connection refused" with delay: 320ms. Will retry...
Nov 03 08:55:32 ip-172-31-86-179 bash[187622]: W1103 08:55:32.726064 187639 server_state.go:226] Error while retrieving timestamps: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:5080: connect: connection refused" with delay: 640ms. Will retry...
Nov 03 08:55:33 ip-172-31-86-179 bash[187622]: E1103 08:55:33.091545 187639 groups.go:1229] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": while receiving from stream: rpc error: code = Unavailable desc = error reading from server: EOF. closer err: <nil>
\Nov 03 08:55:33 ip-172-31-86-179 bash[187622]: E1103 08:55:33.092669 187639 pool.go:314] CONN: Unable to connect with localhost:5080 : rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:5080: connect: connection refused"
Nov 03 08:55:33 ip-172-31-86-179 bash[187622]: W1103 08:55:33.367346 187639 server_state.go:226] Error while retrieving timestamps: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:5080: connect: connection refused" with delay: 1s. Will retry...
Nov 03 08:55:34 ip-172-31-86-179 bash[187622]: E1103 08:55:34.092351 187639 groups.go:1229] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": error from client.subscribe: rpc error: code = Canceled desc = context canceled. closer err: context canceled
Nov 03 08:55:34 ip-172-31-86-179 bash[187622]: E1103 08:55:34.093524 187639 pool.go:314] CONN: Unable to connect with localhost:7080 : rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:7080: connect: connection refused"
Nov 03 08:55:34 ip-172-31-86-179 bash[187622]: E1103 08:55:34.093375 187639 pool.go:314] CONN: Unable to connect with localhost:5080 : rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:5080: connect: connection refused"
Nov 03 08:55:36 ip-172-31-86-179 bash[187724]: E1103 08:55:36.730092 187749 groups.go:1229] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": unable to find any servers for group: 1. closer err: <nil>
Nov 03 08:55:37 ip-172-31-86-179 bash[187724]: E1103 08:55:37.730936 187749 groups.go:1229] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": unable to find any servers for group: 1. closer err: <nil>
Nov 03 08:55:38 ip-172-31-86-179 bash[187724]: E1103 08:55:38.731379 187749 groups.go:1229] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": unable to find any servers for group: 1. closer err: <nil>
Nov 03 08:55:39 ip-172-31-86-179 bash[187724]: E1103 08:55:39.731749 187749 groups.go:1229] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": unable to find any servers for group: 1. closer err: <nil>
Nov 03 08:55:40 ip-172-31-86-179 bash[187724]: E1103 08:55:40.732586 187749 groups.go:1229] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": unable to find any servers for group: 1. closer err: <nil>
Nov 03 08:55:41 ip-172-31-86-179 bash[187724]: E1103 08:55:41.730674 187749 admin.go:854] namespace: 0. Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
This is only error log i have shared, since I couldn’t able to add entire log data.
Request for Help: I’m seeking assistance in understanding and resolving the issues I’m facing during the Dgraph restore process. Any insights or guidance on how to successfully restore the schema would be greatly appreciated.
Thank you for your help!