Live loader on Dgraph Cloud XID mapping

Users GraphQL
1

When trying to use XID mapping with cloud graphql it doesn’t seem to work. Here’s an example JSON:

[
   {
		"namespace":"0x0",
		"uid": "_:africa",
		"dgraph.type": "Region",
		"Region.changed": "2021-06-17T12:21:44-04:00",
		"Region.created": "2021-06-17T12:21:44-04:00",
		"Region.name": "Africa",
		"Region.published": true,
		"Region.code": "AF"
	},
    ...
]

Here’s the live loader script i’m using:

docker run -it --rm -v $(pwd)/migration/xids:/tmp/xids -v $(pwd)/migration/$1.json:/tmp/file.json dgraph/dgraph:v21.03-slash dgraph live --slash_grpc_endpoint=MY_ENDPOINT:443 -f /tmp/file.json -x /tmp/xids -t MY_TOKEN --creds user=groot

It mutates the results but if I run it again it just duplicates the data. I can confirm there are files being written to the xid directory. It seems like “uid” doesn’t work for GraphQL JSON loading with live loader to cloud?

2

Your example data and command look right to me. In fact, I tried out your example in a Dgraph Cloud backend and the queries only return back the single node for the same blank node. e.g., I ran your [dgraph live -x /tmp/xids ...] multiple times and I only see one node for the data point. Here’s the query and response I see:

[... ran live command multiple times]

Query:

{
  q(func: has(Region.code)) {
    uid
    dgraph.type
    Region.changed
    Region.created
    Region.name
    Region.published
    Region.code
  }
}

Response:

{
  "data": {
    "q": [
      {
        "uid": "0x1",
        "dgraph.type": [
          "Region"
        ],
        "Region.changed": "2021-06-17T12:21:44-04:00",
        "Region.created": "2021-06-17T12:21:44-04:00",
        "Region.name": "Africa",
        "Region.published": true,
        "Region.code": "AF"
      }
    ]
  }
}
3

I’m still getting duplicates each time it’s run

4

Can you share full steps to reproduce the issue? I haven’t been able to reproduce the issue with the steps you shared so far.

5

I’ve attached the regions JSON i’m importing. That is located in a ./migrations folder which i’m mounting. I’m also mounting an xids folder. The xids folder DOES get populated but doesn’t seem to do anything on subsequent loads. Here is my xids folder after multiple runs:

xids.zip (3.6 KB)

regions.json (1.8 KB)

6

And here is the results after querying:

Screen Shot 2021-08-27 at 1.05.28 AM
Screen Shot 2021-08-27 at 1.05.28 AM1658×1115 94.7 KB

7

Thanks for sharing. I still can’t reproduce the issue. I don’t see any duplicates in the Dgraph Cloud backend I’m live loading to.

explorer
explorer1467×869 77.4 KB

Are you seeing new duplicates whenever you load? i.e., there are two nodes with the name region name and then when you run live loader there are now three?

Here’s the live loader logs I see. Note the line that says Loaded up 7 xid to uid mappings, which shows that 7 UIDs were stored in the xidmap and then brought back in this live load.

I0826 23:02:29.929983  724212 init.go:110] 

Dgraph version   : v21.03.0-78-ge4ad0b113
Dgraph codename  : rocket
Dgraph SHA-256   : 674eb1cb564ab928c8da418e025dd75d4547a262510b9bf74296f020a6039c72
Commit SHA-1     : e4ad0b113
Commit timestamp : 2021-08-24 14:34:35 -0700
Branch           : release/v21.03-slash
Go version       : go1.16.3
jemalloc enabled : true

For Dgraph official documentation, visit https://dgraph.io/docs.
For discussions about Dgraph     , visit http://discuss.dgraph.io.
For fully-managed Dgraph Cloud   , visit https://dgraph.io/cloud.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2021 Dgraph Labs, Inc.



Running transaction with dgraph endpoint: frozen-mango.grpc.us-west-2.cloud.cloud.dgraph.io:443
badger 2021/08/26 23:02:30 INFO: All 1 tables opened in 3ms
badger 2021/08/26 23:02:30 INFO: Discard stats nextEmptySlot: 0
badger 2021/08/26 23:02:30 INFO: Set nextTxnTs to 1
badger 2021/08/26 23:02:30 INFO: Deleting empty file: ./xids/000002.vlog
I0826 23:02:30.122854  724212 xidmap.go:145] Loaded up 7 xid to uid mappings
Found 1 data file(s) to process
Processing data file "regions.json"
Number of TXs run            : 1                                                                    
Number of N-Quads processed  : 42
Time spent                   : 29.780632ms
N-Quads processed per second : 42
I0826 23:02:30.153026  724212 xidmap.go:365] Writing xid map to DB
I0826 23:02:30.153117  724212 xidmap.go:367] Finished writing xid map to DB
badger 2021/08/26 23:02:30 INFO: Lifetime L0 stalled for: 0s
badger 2021/08/26 23:02:30 INFO: 
Level 0 [ ]: NumTables: 01. Size: 313 B of 0 B. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 64 MiB
Level 1 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 2 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 3 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 4 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 5 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 6 [B]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level Done
8

Here’s what i’m seeing:

Screen Shot 2021-08-27 at 8.52.46 AM
Screen Shot 2021-08-27 at 8.52.46 AM776×616 88.6 KB

They are duplicate items. Each time I run it it creates new nodes.

More than happy to jump on a zoom if that’s easier for you. If not no worries I can continue to give you info.