# Loading data using pydgraph **URL:** \ **Category:** Dgraph\ **Tags:** dgraph\ **Created:** [November 17, 2022, 7:03am UTC](https://discuss.dgraph.io/t/loading-data-using-pydgraph/17985 "2022-11-17T07:03:49Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Kashifa\_Khursheed](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/kashifa_khursheed/32/10123_2.png) [@Kashifa\_Khursheed](https://discuss.dgraph.io/u/Kashifa_Khursheed)\ **Post date:** [November 17, 2022, 7:03am UTC](https://discuss.dgraph.io/t/loading-data-using-pydgraph/17985/1 "2022-11-17T07:03:49Z")
I’m new to dgraph and I wanted to know if I can use something like (shown in below images) to load my dataset using python library pydgraph. The below images display doing the same in neo4j. Or is it only to be done using live loader and bulk loader in dgraph? (PS: my data is in CSV format) ![1](https://canada1.discourse-cdn.com/flex007/uploads/dgraph/original/2X/2/2a464d5c1f3d299722b4e37d4e60f363b90d9f6a.png) ![2](https://canada1.discourse-cdn.com/flex007/uploads/dgraph/original/2X/8/807e74e4e666dbbbba5e19d0246c3b5cac9b8785.png) ---
**Author:** ![Anurag](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/anurag/32/2599_2.png) [@Anurag](https://discuss.dgraph.io/u/Anurag)\ **Post date:** [November 17, 2022, 11:22am UTC](https://discuss.dgraph.io/t/loading-data-using-pydgraph/17985/2 "2022-11-17T11:22:05Z")
Hi @Kashifa_Khursheed ! Welcome to Dgraph. You can use pydgraph to load data in dgraph. The only catch is that you need a way to transform the csv data into a RDF format. If you can share more details about your schema and data, I can probably suggest a way to do that. Once you have a schema and RDFs in place, its easy to do a live-load or a bulk-load based on the data size. A simple script is [here](https://github.com/dgraph-io/pydgraph/blob/master/examples/simple/simple.py). If you need help let me know. ---
**Author:** ![Kashifa\_Khursheed](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/kashifa_khursheed/32/10123_2.png) [@Kashifa\_Khursheed](https://discuss.dgraph.io/u/Kashifa_Khursheed)\ **Post date:** [November 17, 2022, 11:37am UTC](https://discuss.dgraph.io/t/loading-data-using-pydgraph/17985/3 "2022-11-17T11:37:42Z")
The CSVs I have are of the form: movie\_id, title, genres, budget, revenue, director\_name, director\_id, cast\_ids, cast\_names, cast\_characters 1, “Sunshine”, [‘Drama’,‘Romance’], 100000, 8000, “David”, 154, [2,5,8], ['Hannah",“Laura”,“Carlos”], [“A”,“B”,“C”] (genres, cast\_ids, cast\_names are in the form of lists) So, I would like to have two Nodes mainly Person and Movie Person (person\_id, person\_name)–[:ACTED\_IN or :DIRECTED\_BY]–\>Movie (movie\_id, title, budget, revenue, genres (in list format)) - only ACTED\_IN relationship has an edge attribute ‘character’ and ‘count’ - DIRECTED\_BY relationship has ‘count’ edge attribute For example, for the above row, the nodes and relationships would be like shown below: PERSON (person\_id:154, person\_name:‘David’) – [e:DIRECTED\_BY (e.count)] → MOVIE (movie\_id:1, title:‘Sunshine’, budget:100000, revenue:8000, genres:[‘Drama’,‘Romance’]) PERSON (person\_id:2, person\_name:‘Hannah’) – [r:ACTED\_IN (r.character:“A”, r.count)] → MOVIE (movie\_id:1, title:‘Sunshine’, budget:100000, revenue:8000, genres:[‘Drama’,‘Romance’]) PERSON (person\_id:5, person\_name:‘Laura’) – [r:ACTED\_IN (r.character:“B”, r.count)] → MOVIE (movie\_id:1, title:‘Sunshine’, budget:100000, revenue:8000, genres:[‘Drama’,‘Romance’]) PERSON (person\_id:8, person\_name:‘Carlos’) – [r:ACTED\_IN (r.character:“C”, r.count)] → MOVIE (movie\_id:1, title:‘Sunshine’, budget:100000, revenue:8000, genres:[‘Drama’,‘Romance’]) The count variable would increment everytime the same data is loaded (I’m having this extra count variable just to see if the count attribute is incremented when the same data is loaded again) ---
**Author:** ![MichelDiz](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/micheldiz/32/11873_2.png) [@MichelDiz](https://discuss.dgraph.io/u/MichelDiz)\ **Post date:** [November 17, 2022, 2:26pm UTC](https://discuss.dgraph.io/t/loading-data-using-pydgraph/17985/4 "2022-11-17T14:26:32Z")
Try this converter > > > This PR includes a sample Neo4j export csv and a converter program written in go…lang. > > > \--- > This change is \[\Reviewable\](https://reviewable.io/reviews/dgraph-io/dgraph/7545) Related [Neo4JConvertor(test):Fixed some issues regarding Conversion of Neo4J csv to rdf by AvinashRatnam608 · Pull Request #7892 · dgraph-io/dgraph · GitHub](https://github.com/dgraph-io/dgraph/pull/7892) You can also try > **[GitHub - rderbier/dgraphtools: Some tools in Go for graph database Dgraph](https://github.com/rderbier/dgraphtools)** > > Some tools in Go for graph database Dgraph. Contribute to rderbier/dgraphtools development by creating an account on GitHub. I also have done this > > > //github.com/OpenDgraph/ItisTimetoReproduce/tree/master/reproduce006 But it is for JSON exported files from Neo4J. ---
**Author:** ![Kashifa\_Khursheed](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/kashifa_khursheed/32/10123_2.png) [@Kashifa\_Khursheed](https://discuss.dgraph.io/u/Kashifa_Khursheed)\ **Post date:** [November 17, 2022, 3:31pm UTC](https://discuss.dgraph.io/t/loading-data-using-pydgraph/17985/5 "2022-11-17T15:31:57Z")
Can you please elaborate on how to use Neo4j CSV TO RDF Converter? I would like to convert a lot of CSV files to RDF situated in a folder. It would be of much help. Thank you. ---
**Author:** ![MichelDiz](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/micheldiz/32/11873_2.png) [@MichelDiz](https://discuss.dgraph.io/u/MichelDiz)\ **Post date:** [November 17, 2022, 9:16pm UTC](https://discuss.dgraph.io/t/loading-data-using-pydgraph/17985/6 "2022-11-17T21:16:59Z")
I never used them. Just my own code. But basically it would be just check the flags in the code and follow your guts. Note that none of these codes are officially supported. And they run differently. You can have different results in terms of entities. Cuz Neo4J is weird… ---
**Author:** ![Kashifa\_Khursheed](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/kashifa_khursheed/32/10123_2.png) [@Kashifa\_Khursheed](https://discuss.dgraph.io/u/Kashifa_Khursheed)\ **Post date:** [November 18, 2022, 1:40am UTC](https://discuss.dgraph.io/t/loading-data-using-pydgraph/17985/7 "2022-11-18T01:40:28Z")
Update: I have exploded the columns with lists so as to make it simple to import the data into dgraph (below image shows the same). So, while going for ‘dgraph live load’ command, am I supposed to also provide a schema file (written in DQL)? If yes, then are the schema file predicate names same as my column names? Can you just draw a out a sample schema file for the following data? I’ve converted this into a .json file. I’m only attaching CSV’s snapshot so as to make it easy to understand. ![sc](https://canada1.discourse-cdn.com/flex007/uploads/dgraph/original/2X/c/cc51558f1bcff54d454f228ee405b3230d7959c0.png) ---
**Author:** ![MichelDiz](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/micheldiz/32/11873_2.png) [@MichelDiz](https://discuss.dgraph.io/u/MichelDiz)\ **Post date:** [November 18, 2022, 5:34am UTC](https://discuss.dgraph.io/t/loading-data-using-pydgraph/17985/8 "2022-11-18T05:34:04Z")
> [@Kashifa\_Khursheed](#): > > I supposed to also provide a schema file (written in DQL)? Don’t need to. It can be empty. If it asks, provide an empty file that is fine. Schema is Schema, it is not exactly DQL. > [@Kashifa\_Khursheed](#): > > then are the schema file predicate names same as my column names? It will be, Dgraph will infer the predicate name in the RDF file from the generated ones based in the columns. > [@Kashifa\_Khursheed](#): > > Can you just draw a out a sample schema file for the following data? Schema is needed when you need lang or indexes. ---
**Author:** ![Kashifa\_Khursheed](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/kashifa_khursheed/32/10123_2.png) [@Kashifa\_Khursheed](https://discuss.dgraph.io/u/Kashifa_Khursheed)\ **Post date:** [November 18, 2022, 5:41am UTC](https://discuss.dgraph.io/t/loading-data-using-pydgraph/17985/9 "2022-11-18T05:41:40Z")
My idea right now is to have separate files for Movies, Directors, Actors, and a separate file for Relationships. I want to compare the results with both Neo4j and Dgraph. So, I need to have a common format of files to go with before hand. (Movies file will contain only rows of movies data like movie\_id, movie\_name, genres, revenue, budget) (Directors file will contain data like id, name) (Actors file will contain id, name, character) (Relationship file will contain relationships between Movie-Director and Movie-Actor) I would like to have data in such a format so that I load it into neo4j and dgraph and test things out. Can you please suggest such a common data format? ---
**Author:** ![MichelDiz](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/micheldiz/32/11873_2.png) [@MichelDiz](https://discuss.dgraph.io/u/MichelDiz)\ **Post date:** [November 18, 2022, 5:59am UTC](https://discuss.dgraph.io/t/loading-data-using-pydgraph/17985/10 "2022-11-18T05:59:22Z")
> [@Kashifa\_Khursheed](#): > > Can you please suggest such a common data format? Neo4J’s modeling is different from Dgraph. Both are directed graphs, but Neo4J works differently. Each case is a case. A query on Neo4J can be 70% different on Dgraph. And the response too. But conceptually, they can follow a similar structure. The simpler the modeling, the better. But for cases where you are months modeling your Graph in Neo4J. This gets a lot more complicated. Some things need to be remodeled. When I was playing with this thing of exporting data from Neo4J to Dgraph. I had to treat the data a lot. Either via some loop on my code or using Upsert Block in the Dgraph query. CSV is a bad choice for a Graph database. Does not make sense, relationships are detached. It would be better to continue using SQL. > [@Kashifa\_Khursheed](#): > > now is to have separate files for Movies, Directors, Actors, and a separate file for Relationships. Send them separately and do some logic with the relations using e.g. [https://dgraph.io/docs/mutations/upsert-block/#example-of-val-function](https://dgraph.io/docs/mutations/upsert-block/#example-of-val-function) if you don’t wanna code. ---
**Author:** ![Kashifa\_Khursheed](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/kashifa_khursheed/32/10123_2.png) [@Kashifa\_Khursheed](https://discuss.dgraph.io/u/Kashifa_Khursheed)\ **Post date:** [November 18, 2022, 6:36am UTC](https://discuss.dgraph.io/t/loading-data-using-pydgraph/17985/11 "2022-11-18T06:36:44Z")
So, I’ll just stick to CSV for Neo4j and JSON for Dgraph. The problem now is, I’ll have data in the form of separate JSONs (Movies, Actors, Directors, relationships). I want also want to have an edge attribute called “strength” which would change if there are more than one overlapping edges (let’s say there’s a formula to increment or update strength values). Noting that when initially loading the data, there’s unique edges and hence strength is set to a default value (such as 0.5 or 1 initially). While loading more data later, Should I loop this relationship file and perform an upsert query for each of the relation in relationships file to update the strength? Because while using dgraph live loader or bulk loader, I do not think I can incorporate a formula anywhere to update ‘strength’ edge attribute. ---
**Author:** ![Kashifa\_Khursheed](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/kashifa_khursheed/32/10123_2.png) [@Kashifa\_Khursheed](https://discuss.dgraph.io/u/Kashifa_Khursheed)\ **Post date:** [November 19, 2022, 5:12am UTC](https://discuss.dgraph.io/t/loading-data-using-pydgraph/17985/12 "2022-11-19T05:12:16Z")
Data is not getting loaded when using live loader. Encountering this error. I’ve separate files for Movies, Persons and Relationships (JSON format): **Movies – movie\_id, title, revenue, budget, genres** **Persons – person\_id, person\_name** **Relationships – movie\_id, RELATIONSHIP, person\_id** I’m getting this error while trying to load Movies.json file. Zero and alpha are already running … ![error](https://canada1.discourse-cdn.com/flex007/uploads/dgraph/original/2X/8/8a4882a7d43d1e6d01236b929eef26a9bf294afd.png) ![error1](https://canada1.discourse-cdn.com/flex007/uploads/dgraph/original/2X/f/fde0d52e0ebddaec79c1b42cebc740f51e540ddd.png) NOTE: I’ve not created any schema anywhere, since live loader automatically detects predicates, I wanted to do this ---
**Author:** ![MichelDiz](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/micheldiz/32/11873_2.png) [@MichelDiz](https://discuss.dgraph.io/u/MichelDiz)\ **Post date:** [November 19, 2022, 5:58am UTC](https://discuss.dgraph.io/t/loading-data-using-pydgraph/17985/13 "2022-11-19T05:58:16Z")
You are using 20.03, only v21.03 would be acceptable to v22.0.x. Upgrade liveloader. Can you share a example of your JSON? ---
**Author:** ![Kashifa\_Khursheed](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/kashifa_khursheed/32/10123_2.png) [@Kashifa\_Khursheed](https://discuss.dgraph.io/u/Kashifa_Khursheed)\ **Post date:** [November 19, 2022, 6:03am UTC](https://discuss.dgraph.io/t/loading-data-using-pydgraph/17985/14 "2022-11-19T06:03:32Z")
```auto [ { "movie_id": 86838, "title": "Seven Psychopaths", "revenue": 19422261, "budget": 15000000, "genres": "['Comedy', 'Crime']" }, { "movie_id": 44154, "title": "A Touch of Zen", "revenue": 0, "budget": 0, "genres": "['Action', 'Adventure']" }, { "movie_id": 211798, "title": "Wallace & Gromit's Cracking Contraptions", "revenue": 0, "budget": 0, "genres": "['Animation', 'Comedy']" }, { "movie_id": 84066, "title": "Bonnie and Clyde Italian Style", "revenue": 0, "budget": 0, "genres": "['Crime', 'Comedy']" }, { "movie_id": 150473, "title": "Bad Hair Friday", "revenue": 0, "budget": 0, "genres": "['Thriller']" }, { "movie_id": 40478, "title": "Baby Doll", "revenue": 0, "budget": 0, "genres": "['Drama']" } ] ``` ---
**Author:** ![Kashifa\_Khursheed](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/kashifa_khursheed/32/10123_2.png) [@Kashifa\_Khursheed](https://discuss.dgraph.io/u/Kashifa_Khursheed)\ **Post date:** [November 19, 2022, 6:37am UTC](https://discuss.dgraph.io/t/loading-data-using-pydgraph/17985/15 "2022-11-19T06:37:35Z")
Thanks, I could load all 3 files and it was pretty fast. But now, I would like to form relationships between nodes. Should I define a schema and then import all files again? I’d like to have movie\_id, title, genres, revenue, budget and node properties for node MOVIE person\_id, person\_name node properties for node PERSON and relationship is the edge between PERSON and MOVIE nodes … how can I achieve this? ![pred](https://canada1.discourse-cdn.com/flex007/uploads/dgraph/original/2X/9/909d515159ff21f16ddf780ab75198f14c39aa45.png) ---
**Author:** ![MichelDiz](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/micheldiz/32/11873_2.png) [@MichelDiz](https://discuss.dgraph.io/u/MichelDiz)\ **Post date:** [November 19, 2022, 4:49pm UTC](https://discuss.dgraph.io/t/loading-data-using-pydgraph/17985/16 "2022-11-19T16:49:58Z")
> [@Kashifa\_Khursheed](#): > > Should I define a schema and then import all files again? relationships don’t come from Schema. Schema just define the types. > [@Kashifa\_Khursheed](#): > > and relationship is the edge between PERSON and MOVIE nodes Which edges? What are the refecences? If you have something like “relationship table” you can use [https://dgraph.io/docs/mutations/upsert-block/](https://dgraph.io/docs/mutations/upsert-block/) to link them. But you’d better try using the tools I mentioned before. Because they try their best to create a valid RDF already with the relations. ---
**Author:** ![Kashifa\_Khursheed](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/kashifa_khursheed/32/10123_2.png) [@Kashifa\_Khursheed](https://discuss.dgraph.io/u/Kashifa_Khursheed)\ **Post date:** [November 20, 2022, 2:24pm UTC](https://discuss.dgraph.io/t/loading-data-using-pydgraph/17985/18 "2022-11-20T14:24:52Z")
I’ve used dgraph live loader to load movies.rdf, persons.rdf and relationships.rdf files. But edges aren’t created. Why is that? (Attaching three files snapshots below) ![1](https://canada1.discourse-cdn.com/flex007/uploads/dgraph/original/2X/9/9683f8f8f7559c194d8999bee6fb67b1cd068c3a.png) ![2](https://canada1.discourse-cdn.com/flex007/uploads/dgraph/original/2X/3/32d7ea3e96a60ac9f810f4101bd541c105a5ac59.png) ![3](https://canada1.discourse-cdn.com/flex007/uploads/dgraph/original/2X/3/3c34c7a1bce8a9ba2f0204b5dbec3b47ba048471.png) ---
**Author:** ![MichelDiz](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/micheldiz/32/11873_2.png) [@MichelDiz](https://discuss.dgraph.io/u/MichelDiz)\ **Post date:** [November 20, 2022, 4:18pm UTC](https://discuss.dgraph.io/t/loading-data-using-pydgraph/17985/19 "2022-11-20T16:18:53Z")
> [@Kashifa\_Khursheed](#): > > I’ve used dgraph live loader to load movies.rdf, persons.rdf and relationships.rdf files. But edges aren’t created. Why is that? Really long story. In short, you have to send all together. Or use the upsert flag in live load ` -U, --upsertPredicate string run in upsertPredicate mode. the value would be used to store blank nodes as an xid`. The Blank node context isn’t saved “globaly”. So, each new mutation the blank nodes are considered new uids. You can try also(I think it is easier) xidmapping ```auto "xidmap", "x", "", "Directory to store xid to uid mapping" ``` It will save the UID context to a path called `x` and you have to use this flag in every load. ---
**Author:** ![Kashifa\_Khursheed](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/kashifa_khursheed/32/10123_2.png) [@Kashifa\_Khursheed](https://discuss.dgraph.io/u/Kashifa_Khursheed)\ **Post date:** [November 21, 2022, 4:02am UTC](https://discuss.dgraph.io/t/loading-data-using-pydgraph/17985/21 "2022-11-21T04:02:36Z")
It’s not working out for me. Can we connect over zoom and sort this out? It’s alright if you cannot, I’ll post more information regarding the problem. ---
**Author:** ![MichelDiz](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/micheldiz/32/11873_2.png) [@MichelDiz](https://discuss.dgraph.io/u/MichelDiz)\ **Post date:** [November 21, 2022, 5:34am UTC](https://discuss.dgraph.io/t/loading-data-using-pydgraph/17985/22 "2022-11-21T05:34:09Z")
Not sure due to Timezone. Monday I’m having some meetings. PS. If this data is public, give me the Link that I will run on my side. ### The logic is simple. You need to make sure that all entities are using the same BlankNode. BlankNode is a unique temporary identifier that we use in RDF and mutations in general. If you run two different transactions for Blank Node \<\_:New01\> the Dgraph will not identify the UID of the first transaction. Because when committing the transaction, the context of the leasing UID is lost. So the solution is as follows. 1. Run all RDFs in one transaction. Or upload them all in one batch via Liveloader. Liveloader can read all RDFs that are within a given Path. 2. Or Start the first transaction with xidmap. Bulkloader also generates a map of xids. XID stands for “External IDs”. When running the command: `dgraph live -x "~/pathTo/dgraphData/XIDs"` You will save all UIDs mapped to each Blank Node identifier from your dump data. You need to certify that they are unique according to their entity uniqueness. That is, Bob and all his attributes must have the same Blank Node. However, Alice and all her attributes do not. [Next page](https://discuss.dgraph.io/t/loading-data-using-pydgraph/17985.md?page=2)