# Dgraph v21.03.0: Online Restore Fails

**URL:** <https://discuss.dgraph.io/t/dgraph-v21-03-0-online-restore-fails/13526>\
**Category:** Dgraph\
**Tags:** status:accepted, kind:bug, ticket:created\
**Created:** [April 3, 2021, 6:50am UTC](https://discuss.dgraph.io/t/dgraph-v21-03-0-online-restore-fails/13526 "2021-04-03T06:50:30Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![joaquin](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/joaquin/32/2619_2.png) [@joaquin](https://discuss.dgraph.io/u/joaquin)\
**Post date:** [April 3, 2021, 6:50am UTC](https://discuss.dgraph.io/t/dgraph-v21-03-0-online-restore-fails/13526/1 "2021-04-03T06:50:30Z")

</div>

## Report a Dgraph Bug

The online restore fails with an error message that restore `cannot write backup` and that `stat` fails on a file that exists on an s3 bucket.

### What version of Dgraph are you using?

- commit `32f1f5893` (`release/v21.03`)

### Have you tried reproducing the issue with the latest release?

n/a

### What is the hardware spec (RAM, OS)?

- Ubuntu 20.04 from docker image

### Steps to reproduce the issue (comm

1. From an existing 6 node cluster (3xA, 3xZ) that has access to an s3 bucket.  
Example `helm install backup -n backup --values backup-cluster.yaml ~/charts/charts/dgraph/`:

```auto
# backup-cluster.yaml
image:
  repository: darknerd/dgraph
  tag: v21.03
ratel:
  enabled: false
alpha:
  acl:
    enabled: true
    file:
      hmac_secret_file: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
  extraEnvs:
    - name: DGRAPH_ALPHA_SECURITY
      value: whitelist=10.0.0.0/8,172.0.0.0/8,192.168.0.0/16
    - name: DGRAPH_ALPHA_ACL
      value: secret-file=/dgraph/acl/hmac_secret_file
backups:
  full:
    enabled: true
    schedule: "0 0 * * *"
  incremental:
    enabled: true
    schedule: "0 1-23 * * *"
  admin:
    user: groot
    password: password
  destination: s3://s3.us-east-2.amazonaws.com/dgraph-dev-backups/bugbuster/backups
  keys:
    s3:
      access: REDACTED
      secret: REDACTED

```

2. Backup the existing cluster (3 tenants were previously created)  
Example backup mutation:

```auto
mutation {
  backup(input: {
    destination: "s3://s3.us-east-2.amazonaws.com/dgraph-dev-backups/jira"
    forceFull: true
  }) {
    response {
      message
      code
    }
  }
}

```

3. Create a similar cluster used for the restore.  
Example: `helm install restore -n restore --values restore-cluster.yaml ~/charts/charts/dgraph/`:

```auto
# restore-cluster.yaml
image:
  repository: darknerd/dgraph
  tag: v21.03
ratel:
  enabled: false
alpha:
  acl:
    enabled: true
    file:
      hmac_secret_file: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
  extraEnvs:
    - name: DGRAPH_ALPHA_SECURITY
      value: whitelist=10.0.0.0/8,172.0.0.0/8,192.168.0.0/16
    - name: DGRAPH_ALPHA_ACL
      value: secret-file=/dgraph/acl/hmac_secret_file

```

4. Perform a restore:

```auto
mutation {
  restore(input:{
    location:"s3://s3.us-east-2.amazonaws.com/dgraph-dev-backups/jira"
    accessKey: "REDACTED"
    secretKey: "REDACTED"
  }) {
    message
    code
  }
}

```

### Expected behaviour and actual result.

#### Expected

There are two expectations from this:

- that a restore would be attempted, not a backup (see error message in actual results below)
- given the file does exist (from actual results below), a restore procedure would be successful.

The file in question does actually exist:

```bash
$ aws s3 ls s3://dgraph-dev-backups/jira/dgraph.20210403.043719.140/
2021-04-02 21:37:20 490166521 r42587-g1.backup

```

#### Actual

From the logs, the restore operation causes this:

```auto
E0403 04:55:58.897630 19 draft.go:720] Applying proposal. Error: cannot write backup: cannot write backup: Stat failed "dgraph.20210403.043719.140/r42587-g1.backup": The specified key does not exist.. Proposal: {"<nil>" [] "<nil>" "" "<nil>" "<nil>" '$' '\x00' "group_id:1 restore_ts:17 location:\"s3://s3.us-east-2.amazonaws.com/dgraph-dev-backups/jira\" access_key:\"REDACTED\" secret_key:\"REDACTED\" " "<nil>" "<nil>"}.
I0403 04:55:58.897745 19 draft.go:124] Operation completed with id: opRestore
E0403 04:55:58.897790 19 online_restore_ee.go:111] Error while restoring cannot propose restore request: cannot write backup: cannot write backup: Stat failed "dgraph.20210403.043719.140/r42587-g1.backup": The specified key does not exist.

```

---

<div class="post-metadata">

**Author:** ![joaquin](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/joaquin/32/2619_2.png) [@joaquin](https://discuss.dgraph.io/u/joaquin)\
**Post date:** [April 5, 2021, 8:30pm UTC](https://discuss.dgraph.io/t/dgraph-v21-03-0-online-restore-fails/13526/3 "2021-04-05T20:30:40Z")

</div>

The listBackups works fine using the same credentials.

```auto
query {
  listBackups(input: {
    location: "s3://s3.us-east-2.amazonaws.com/dgraph-dev-backups/joaquin/backup"
    accessKey: "REDACTED"
    secretKey: "REDACTED"
  }) {
    backupId
    backupNum
    encrypted
    path
    since
    type
  }
}

```

---

<div class="post-metadata">

**Author:** ![Naman](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/naman/32/3498_2.png) [@Naman](https://discuss.dgraph.io/u/Naman)\
**Post date:** [April 7, 2021, 6:47pm UTC](https://discuss.dgraph.io/t/dgraph-v21-03-0-online-restore-fails/13526/4 "2021-04-07T18:47:13Z")

</div>

This has been resolved by [fix(restore): append the object path prefix while reading backup by NamanJain8 · Pull Request #7686 · dgraph-io/dgraph · GitHub](https://github.com/dgraph-io/dgraph/pull/7686)

---

<div class="post-metadata">

**Author:** ![Naman](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/naman/32/3498_2.png) [@Naman](https://discuss.dgraph.io/u/Naman)\
**Post date:** [April 7, 2021, 6:47pm UTC](https://discuss.dgraph.io/t/dgraph-v21-03-0-online-restore-fails/13526/5 "2021-04-07T18:47:17Z")

</div>