Mapping all UIDs before Upgrade HELP!

Issues Dgraph
1

We currently have 20.11.X on Dgraph Cloud and have a pending upgrade window for 21.03.

For context, we have setup a sync between our data and our old rdbms and have not completed migrating users yet, so we still need to keep this sync going for a little longer after our upgrade. The sync uses… uids that are stored alongside the data in the rdbms.

That was the logical thing to do in the architecture because they were gaurenteed to be globally unique and we did not have to generate them using uuid.

But the problem is that all uids change in the upgrade process. :scream: :sob: :angry:

So what can I do right now before the upgrade to map all of these uids over to a predicate old_uid so that we can map the rdbms uids over after the upgrade?

2

The exported data has the UIDs? Dgraph exports it by default.
Are you using liveloader? How did you imported it? To have new uids you have to explicitly set --new_uids - Or maybe you are using a shared instance? In shared instances I think it isn’t possible to control the UIDs.

3

We are in a shared instance. Going to 21.03 it becomes a truly shared instance and gets migrated in a namespaced instance of Dgraph. Which is all great minus this uid issue.

How can I somehow preserve these uids before the migration?

Even if I can just simply map them to a predicate I can reference after the upgrade would be satisfactory. But there is no forEach or anything like that in Dgraph still.

4

Try to use XIDs instead of UIDs. I think liveload doesn’t have --store_xids need to check (I’m using macOS, not sure if the core devs have added xids to the liveloader).

Using XIDs you can use

-U, --upsertPredicate string       
run in upsertPredicate mode. the value would be used to store 
blank nodes  as an xid

With this you can send the data and it will create an upsert block based on the XIDs stored in the XID edge.

5

Ok, yes XID, but how to add the current UID as an XID for all my nodes, no idea how many.

6

I would start a new cluster*, and bulkload it using --store_xids. What it gonna do is, it will take the old UIDs and store it in the XID edge.

1 Like
7

@anand is this possible to do in my upgrade path?

@MichelDiz if that is not possible in my upgrade path by Dgraph Cloud, then how would I go about getting these uids mapped back to my upgraded deployment if I was to to this offline right now?

We only have 2,111,426 nodes counting only the ones with types.

8

I am trying an upsert block:

upsert {
  query {
    v as var(func: has(dgraph.type), first: 100) @filter(NOT has(xid)) {
      getValues as uid
    }
  }
  mutation {
    set {
      uid(v) <xid> val(getValues) .
    }
  }
}

But this is not changing any data. I think it might be because I am using val on a uid?

Edit: the val function does not work in this way. :frowning:

https://dgraph.io/docs/mutations/upsert-block/#val-function

9

The type of the variable will be always the same as the origin, so it won’t convert UID to String in this case. You might use lambda to make this work.

10

I added a predicate to my DQL schema:

  old_xid: string .

Added a field to all of my GraphQL interfaces and non implementing types:

type Atype {
  xid: String @dgraph(pred:"old_xid") @search(by: [exact])
  ...
}

Wrote a quick and ugly node script:

const fetch = require('node-fetch')

const commit = true
const endpoint = 'https://<my-endpoint>.cloud.dgraph.io'
// TODO: You must enter a valid token from Dgraph Cloud GUI to use this script.
const xAuthToken = '<my-dgraph-cloud-token>'

async function main() {
  return fetch(`${endpoint}/query`, {
    method: 'post',
    headers: {
      'x-auth-token': xAuthToken,
      'Content-type': 'application/dql',
    },
    body: `
        query {
          n(func: has(dgraph.type), first: 100000) @filter(NOT has(old_xid)) {
            uid
          }
        }
      `,
  })
    .then((res) => res.json())
    .then((res) => {
      if (res.data.n.length === 0) {
        console.log('Completed UID Map!')
        process.exit()
      }
      console.log(res.data.n[0])
      let rdf = ''
      res.data.n.forEach((node) => {
        rdf += `<${node.uid}> <old_xid> "${node.uid}" .\n`
      })
      fetch(`${endpoint}/mutate${commit ? '?commitNow=true' : ''}`, {
        method: 'post',
        headers: {
          'x-auth-token': xAuthToken,
          'Content-type': 'application/rdf',
        },
        body: `{
            set {
              ${rdf}
            }
          }`,
      })
        .then((res) => res.json())
        .then((res) => {
          console.log(res)
          return main()
        })
    })
}

return main()

And Done!

Now after the upgrade we will be able to query a list of xids and get their mapped new uid:

query ($xids:[String]) {
  queryAtype(filter: {xid: {in: $xids}}) {
    xid
    id
  }
}

@MichelDiz thank you for your help as always!!

2 Likes