# Dgraph live - way to prevent duplication of data? **URL:** \ **Category:** Users\ **Created:** [January 12, 2018, 2:30am UTC](https://discuss.dgraph.io/t/dgraph-live-way-to-prevent-duplication-of-data/2122 "2018-01-12T02:30:27Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![artooro](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/artooro/32/1129_2.png) [@artooro](https://discuss.dgraph.io/u/artooro)\ **Post date:** [January 12, 2018, 2:30am UTC](https://discuss.dgraph.io/t/dgraph-live-way-to-prevent-duplication-of-data/2122/1 "2018-01-12T02:30:27Z")
My desire is to be able to write an RDF file that when inserted into Dgraph using `dgraph live -r myfile.rdf.gz` that no matter how many times I run it, it will not insert duplicates of any of the triples. So far from what I’ve been reading and testing, it doesn’t seem like there is a way to do this. So my last ditch is to post here and see what you guys think or whether there are alternative suggestions. The use case is our data is being collected asynchronously and sent to the graph via events. I’d like for each event to generate an RDF file that ends up being fed to `dgraph live` but from what I’m seeing I’ll actually have to write code to do a transaction and manual query for each predicate. On a side note, also not sure what to query to find out which node is the “lead” node. Querying [http://127.0.0.1:8080/state](http://127.0.0.1:8080/state) doesn’t return anything. It would be nice to have a list of HTTP endpoints in the docs, or maybe I missed it. ---
**Author:** ![pawan](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/pawan/32/1946_2.png) [@pawan](https://discuss.dgraph.io/u/pawan)\ **Post date:** [January 12, 2018, 2:41am UTC](https://discuss.dgraph.io/t/dgraph-live-way-to-prevent-duplication-of-data/2122/2 "2018-01-12T02:41:55Z")
Hey @artooro > [@artooro](#): > > So far from what I’ve been reading and testing, it doesn’t seem like there is a way to do this. That’s right. There is no way to currently for Dgraph live to do this right now. You’d have to add an index to store your `xid`, query it and use it. The `/state` endpoint is present on Zero. So considering that Zero is serving HTTP on port `6080` (assuming you started with offset `-2000`), you could goto `http://127.0.0.1:6080/state`. ---
**Author:** ![llonchj](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/llonchj/32/1134_2.png) [@llonchj](https://discuss.dgraph.io/u/llonchj)\ **Post date:** [January 12, 2018, 2:42am UTC](https://discuss.dgraph.io/t/dgraph-live-way-to-prevent-duplication-of-data/2122/3 "2018-01-12T02:42:31Z")
Arthur, Consider using a xidmap directory when invoking live `dgraph live --xidmap .....` ---
**Author:** ![artooro](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/artooro/32/1129_2.png) [@artooro](https://discuss.dgraph.io/u/artooro)\ **Post date:** [January 12, 2018, 2:45am UTC](https://discuss.dgraph.io/t/dgraph-live-way-to-prevent-duplication-of-data/2122/4 "2018-01-12T02:45:25Z")
Gotcha, I’ll build my own query+insert tool then. So I have zero running on port 5080 (kubernetes HA setup) and if I run `curl http://127.0.0.1:5080/state` it returns `?` not sure what the question mark means. ---
**Author:** ![artooro](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/artooro/32/1129_2.png) [@artooro](https://discuss.dgraph.io/u/artooro)\ **Post date:** [January 12, 2018, 2:48am UTC](https://discuss.dgraph.io/t/dgraph-live-way-to-prevent-duplication-of-data/2122/5 "2018-01-12T02:48:27Z")
@llonchj thats an interesting idea. So I could store a map on the local file system and lookup uids and do a search/replace maybe… Or because of concurrency might be better to just write my own script. ---
**Author:** ![pawan](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/pawan/32/1946_2.png) [@pawan](https://discuss.dgraph.io/u/pawan)\ **Post date:** [January 12, 2018, 2:49am UTC](https://discuss.dgraph.io/t/dgraph-live-way-to-prevent-duplication-of-data/2122/6 "2018-01-12T02:49:12Z")
@artooro If you always plan to use the same client to upload the data, then this should work. We store the xid =\> uid mapping in the `--xidmap` directory locally on the client. * * * Update - There is a small issue ([Xidmap in live loader is not completely pushed to badger · Issue #2006 · dgraph-io/dgraph · GitHub](https://github.com/dgraph-io/dgraph/issues/2006)) because of which the above solution won’t work for now. If you are using the Kubernetes HA setup, try `http://127.0.0.1:6080/state`. ---
**Author:** ![llonchj](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/llonchj/32/1134_2.png) [@llonchj](https://discuss.dgraph.io/u/llonchj)\ **Post date:** [January 17, 2018, 2:12am UTC](https://discuss.dgraph.io/t/dgraph-live-way-to-prevent-duplication-of-data/2122/7 "2018-01-17T02:12:01Z")
@artooro I think the xidmap folder keeps mapping of the xid and the uid so you can use IRI’s dgraph live -xidmap my-xidmap folder … “john” . dgraph live -xidmap my-xidmap folder … \<_:1\> “mark” . \<_:1\> . ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex007/uploads/dgraph/original/2X/2/2b38fdece2abe5814f2e51bee69d1e67fc304b0a.png) [@system](https://discuss.dgraph.io/u/system)\ **Post date:** [February 16, 2018, 2:12am UTC](https://discuss.dgraph.io/t/dgraph-live-way-to-prevent-duplication-of-data/2122/8 "2018-02-16T02:12:06Z")
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.