How to do backup/restore with export / live-loader

Users Dgraph
1

How can I get a list of all UIDs with a simple DQL query?

@MichelDiz any idea?

2

All UIDs of what?

You can use https://dgraph.io/docs/query-language/functions/#has

3

All UIDs that exist in the whole database.

Some Background:
I want to use the live-loader to restore a backup which I previously made using the export functionality from the admin interface (dgraph graphql)

The issue is, that data that has been added since the snapshot was taken is still present in the DB.

Now I want to try to delete all UIDs that are not present in my backup manually.

Btw, I’d love to use backup/restore but it’s an enterprise feature unfortunately.

Edit:
I tried

{
  all(func: has(uid)) {
    uid
  }
}

but it gave me: while converting to subgraph: Argument cannot be \"uid\"",

4

By query it is not possible. You can do it by debug. But it is not so useful.

That’s a hard task. You should use Upsert Block instead.

In fact, if your dataset was exported with its UIDs and nothing was changed in the DB. You can simply import it that it will be “merged” just fine.

5

Unfortunately, this is not given in my case:

  • User makes backup (export to RDF)
  • Time goes by, things get deleted, other things get added
  • Some error emerges, User restores from backup

With live-loader, all data that has been added after the backup is still present after the live-loader has finished. Meh.

Do you have any other idea how I could do backup/restore (on a self-hosted instance) without having to pay, quote “substantial amounts” ?

“Our on prem enterprise license costs are substantially higher and also include 24x7 support. We do not sell license for only backup functionality.” - Dgraph Sales E-Mail Contact

The last resort would be to shutdown dgraph, delete the database, restart dgraph, use live-loader to import the data. This would however introduce significant downtime.

6

Kind of, no.

You have only the export and live/bulkload option.

In your case I would try to develop my own way of backing-up. It will give you more work? yes. As there no other option, you have to invest on it.

Live and Bulk there are good options that you can take advantage of. From now and beyond you should create a XID for all your objects. And keep the track of all of it. So, in the future you can use the Upsert by XID option in the Liveload tool. With this you will have no worries about UID.

UPDATE:

I would also create a timestamp for each object and labels setting things as “deleted” or other rules. And control it in your end application.

7

Whats xid?

All of my objects have a custom id field. Could that be useful?

8

You can try to copy paste that ID to the XID predicate(fill it up). And also do the same in your dataset(by hand in your case). And that should work.

Do in the terminal

dgraph live -h | grep XID
dgraph live -h | grep upsert

And read the descriptions.

9

Thanks so far, let’s see how this goes!

10

For anyone stumbling across this thread, here’s how I solved it:

Backup
Use export

Restore

  1. send { "drop_all": true } to /alter endpoint
  2. post schema from backup to /admin/schema
  3. use live loader to load rdf data from backup