# \#area:bulk-loader **URL:** https://discuss.dgraph.io/tag/area-bulk-loader/151.md [Latest](https://discuss.dgraph.io/latest.md) · [Categories](https://discuss.dgraph.io/categories.md) · [Tags](https://discuss.dgraph.io/tags.md) --- ## [Init loading data and upsert](https://discuss.dgraph.io/t/init-loading-data-and-upsert/19368)
**Author:** [@CyberSputnik](https://discuss.dgraph.io/u/CyberSputnik)\ **Replies:** 0\ **Last updated:** [June 21, 2024, 7:22am UTC](https://discuss.dgraph.io/t/init-loading-data-and-upsert/19368 "2024-06-21T07:22:34Z")
Hello everyone! Deployed dgraph in docker, version 23.1.0 (3 zeros, 6 alphas). I upload the initial data via graph bulk command: dgraph bulk -f data.rdf -s schema.txt --map\_shards=4 --reduce\_shards=2 --http localhost… --- ## [Load Data using Dgraph w/ Kubernetes](https://discuss.dgraph.io/t/load-data-using-dgraph-w-kubernetes/10732)
**Author:** [@saugat](https://discuss.dgraph.io/u/saugat)\ **Replies:** 8\ **Last updated:** [March 24, 2024, 2:23am UTC](https://discuss.dgraph.io/t/load-data-using-dgraph-w-kubernetes/10732 "2024-03-24T02:23:00Z")
Hi Everyone, I am new to DGraph and was trying to do the bulk loader in kubernetes. I found the yml file from the github https://raw.githubusercontent.com/dgraph-io/dgraph/master/contrib/config/kubernetes/dgraph-ha/dgra… --- ## [Bulk Load doesn't import / save the schema file](https://discuss.dgraph.io/t/bulk-load-doesnt-import-save-the-schema-file/17657)
**Author:** [@amc](https://discuss.dgraph.io/u/amc)\ **Replies:** 8\ **Last updated:** [August 17, 2022, 1:54pm UTC](https://discuss.dgraph.io/t/bulk-load-doesnt-import-save-the-schema-file/17657 "2022-08-17T13:54:04Z")
Trying to stand up a replica copy of my production data: Using the docker standalone image download the rdf and schema files from S3 bucket and gunzip run bulk load command specifying the --files and --schema arguments … --- ## [Cannot import with bulk loader](https://discuss.dgraph.io/t/cannot-import-with-bulk-loader/12341)
**Author:** [@anargu](https://discuss.dgraph.io/u/anargu)\ **Replies:** 12\ **Last updated:** [May 12, 2022, 10:17am UTC](https://discuss.dgraph.io/t/cannot-import-with-bulk-loader/12341 "2022-05-12T10:17:19Z")
I Want to Do Restore my backup data by using bulk loader to a new instance of Dgraph (v20.11.0) What I Did I started dgraph zero then alpha via docker containers I get into dgraph zero container and executed the comma… --- ## [Why feature team is silent on Bulk loader performance issues?](https://discuss.dgraph.io/t/why-feature-team-is-silent-on-bulk-loader-performance-issues/16856)
**Author:** [@porsche](https://discuss.dgraph.io/u/porsche)\ **Replies:** 4\ **Last updated:** [February 18, 2022, 5:56pm UTC](https://discuss.dgraph.io/t/why-feature-team-is-silent-on-bulk-loader-performance-issues/16856 "2022-02-18T17:56:36Z")
Not sure if Bulk loader team is watching this forum, we are experiencing serious issues with bulk loader. Appreciate if anyone chime-in to help us out. Thanks in advance. --- ## [Can we use live uploader followed by bulk uploader?](https://discuss.dgraph.io/t/can-we-use-live-uploader-followed-by-bulk-uploader/16867)
**Author:** [@ddude](https://discuss.dgraph.io/u/ddude)\ **Replies:** 0\ **Last updated:** [February 18, 2022, 6:17am UTC](https://discuss.dgraph.io/t/can-we-use-live-uploader-followed-by-bulk-uploader/16867 "2022-02-18T06:17:15Z")
Can can we use live uploader followed by bulk uploader? In out PoC (Proof of Concept) implementation… We wanted upload initial data using bulk uploader. Bulk loader command is something like below dgraph bulk -f /bulk… --- ## [How to trouble shoot bulk loader hung](https://discuss.dgraph.io/t/how-to-trouble-shoot-bulk-loader-hung/16853)
**Author:** [@porsche](https://discuss.dgraph.io/u/porsche)\ **Replies:** 0\ **Last updated:** [February 16, 2022, 4:51pm UTC](https://discuss.dgraph.io/t/how-to-trouble-shoot-bulk-loader-hung/16853 "2022-02-16T16:51:10Z")
Version – v21.12.0 Cluster Zeros: 3 – Leader config: 32 core, 256 GB, 8 TB – Follower config: 8 Core, 64 GB, 2 TB SSD Dataset -1 – Count of Data Files: 1044 – Size of each file: 30 MB (compressed) – Result:… --- ## [Stuck at REDUCE 100% for more than 30 hours](https://discuss.dgraph.io/t/stuck-at-reduce-100-for-more-than-30-hours/13327)
**Author:** [@parly](https://discuss.dgraph.io/u/parly)\ **Replies:** 14\ **Last updated:** [February 16, 2022, 4:33am UTC](https://discuss.dgraph.io/t/stuck-at-reduce-100-for-more-than-30-hours/13327 "2022-02-16T04:33:57Z")
I’m trying to convert for about 7.2 billion lines of rdf file to dgraph, it takes for about 5 hours reaching to REDUCE 100.00%, and then it stuck at REDUCE 100.00% for more than 35 hours. I didn’t see any error logs. Is … --- ## [Does xid attribute needs to be defined in the type for bulk loader to xid --store\_xids flag](https://discuss.dgraph.io/t/does-xid-attribute-needs-to-be-defined-in-the-type-for-bulk-loader-to-xid-store-xids-flag/16787)
**Author:** [@porsche](https://discuss.dgraph.io/u/porsche)\ **Replies:** 6\ **Last updated:** [February 7, 2022, 8:55pm UTC](https://discuss.dgraph.io/t/does-xid-attribute-needs-to-be-defined-in-the-type-for-bulk-loader-to-xid-store-xids-flag/16787 "2022-02-07T20:55:17Z")
Officecial docs says \*--store\_xids: Generate a xid edge for each node. It will store the XIDs (The identifier / Blank-nodes) in an attribute named xid in the entity itself. It is useful if you gonna use External IDs.\* D… --- ## [Bulk loader: What am I doing wrong, my data doesn't show up](https://discuss.dgraph.io/t/bulk-loader-what-am-i-doing-wrong-my-data-doesnt-show-up/16786)
**Author:** [@porsche](https://discuss.dgraph.io/u/porsche)\ **Replies:** 0\ **Last updated:** [February 6, 2022, 10:27pm UTC](https://discuss.dgraph.io/t/bulk-loader-what-am-i-doing-wrong-my-data-doesnt-show-up/16786 "2022-02-06T22:27:09Z")
Our cluster config: zero: 1 (say z1) Alpha: 3 (say a1, a2 and a3) replica: 3 Deployed the cluster using Helm Charts Question: My data & schema does show up in Ratte, what am I doing wrong? How I copied /out folder… --- ## [Where is bulk uploader /out folder](https://discuss.dgraph.io/t/where-is-bulk-uploader-out-folder/16777)
**Author:** [@porsche](https://discuss.dgraph.io/u/porsche)\ **Replies:** 3\ **Last updated:** [February 6, 2022, 8:20pm UTC](https://discuss.dgraph.io/t/where-is-bulk-uploader-out-folder/16777 "2022-02-06T20:20:13Z")
Question: Where is /out folder created, bulk upload seems to have run successfully? – We run bulk uploader from /dgraph directory on Zero – We launch bulk uploader multiple time as our data pipeline keeps writing .rd… --- ## [Can we run bulk uploader multiple times?](https://discuss.dgraph.io/t/can-we-run-bulk-uploader-multiple-times/16779)
**Author:** [@porsche](https://discuss.dgraph.io/u/porsche)\ **Replies:** 2\ **Last updated:** [February 5, 2022, 6:12pm UTC](https://discuss.dgraph.io/t/can-we-run-bulk-uploader-multiple-times/16779 "2022-02-05T18:12:43Z")
Question: Can we run bulk uploader multiple times? We were getting below error when bulk uploader was launched more than onceOutput directory exists and is not empty. Use --replace\_out to overwrite it. We launch bulk… --- ## [How to copy out/0/p from zero to alpha (blocked with init)](https://discuss.dgraph.io/t/how-to-copy-out-0-p-from-zero-to-alpha-blocked-with-init/16773)
**Author:** [@porsche](https://discuss.dgraph.io/u/porsche)\ **Replies:** 1\ **Last updated:** [February 5, 2022, 2:09am UTC](https://discuss.dgraph.io/t/how-to-copy-out-0-p-from-zero-to-alpha-blocked-with-init/16773 "2022-02-05T02:09:22Z")
dgraph:v21.03.1 Question: Where is my /out folder located? – We run ‘dgraph bulk’ command from /dgraph directory – /out directory wasn’t found in /dgraph, out .rdf.gz or schema directory – Bulk uploader seems to ha… --- ## [Bulk Loader with cluster - 'Attribute xyz is not indexed' on followers](https://discuss.dgraph.io/t/bulk-loader-with-cluster-attribute-xyz-is-not-indexed-on-followers/15273)
**Author:** [@omerenesbayram](https://discuss.dgraph.io/u/omerenesbayram)\ **Replies:** 3\ **Last updated:** [September 19, 2021, 7:59pm UTC](https://discuss.dgraph.io/t/bulk-loader-with-cluster-attribute-xyz-is-not-indexed-on-followers/15273 "2021-09-19T19:59:40Z")
Report a Dgraph Bug What version of Dgraph are you using? ▶ Dgraph Version v21.03.0 Have you tried reproducing the issue with the latest release? Yes, the issue is with the latest release. What is the hardware spec (RAM, … --- ## [Bulk loader still OOM during reduce phase](https://discuss.dgraph.io/t/bulk-loader-still-oom-during-reduce-phase/15030)
**Author:** [@sbv1976](https://discuss.dgraph.io/u/sbv1976)\ **Replies:** 18\ **Last updated:** [August 1, 2021, 4:37pm UTC](https://discuss.dgraph.io/t/bulk-loader-still-oom-during-reduce-phase/15030 "2021-08-01T16:37:34Z")
What I want to do Import an existing data set that consists of around 350M RDF quads. There are a lot of uids but I don’t have precise statistics on those. There are also a large number of unique predicates… I don’t ha… --- ## [Does triple order matter for bulk loading](https://discuss.dgraph.io/t/does-triple-order-matter-for-bulk-loading/13156)
**Author:** [@danieljuhl](https://discuss.dgraph.io/u/danieljuhl)\ **Replies:** 8\ **Last updated:** [April 9, 2021, 4:52am UTC](https://discuss.dgraph.io/t/does-triple-order-matter-for-bulk-loading/13156 "2021-04-09T04:52:36Z")
I haven’t been able to find anything on the question, but using bulk loader if the same edge is defined multiple times, which edge/triple “wins”? If I perform the following mutation, the last triple will be the winner. … --- ## [Not all of JSON file consumed](https://discuss.dgraph.io/t/not-all-of-json-file-consumed/12584)
**Author:** [@Prateek](https://discuss.dgraph.io/u/Prateek)\ **Replies:** 2\ **Last updated:** [March 29, 2021, 7:42pm UTC](https://discuss.dgraph.io/t/not-all-of-json-file-consumed/12584 "2021-03-29T19:42:06Z")
Hey Guys, I’m trying to import the JSON file of the Dgraph database on my system using the data loading tools live loader and bulk loader but getting the following error while importing the data - Not all of JSON file … --- ## [Questions about importing data](https://discuss.dgraph.io/t/questions-about-importing-data/13278)
**Author:** [@03B037](https://discuss.dgraph.io/u/03B037)\ **Replies:** 5\ **Last updated:** [March 19, 2021, 2:21am UTC](https://discuss.dgraph.io/t/questions-about-importing-data/13278 "2021-03-19T02:21:53Z")
In the process of importing the data, the waiting time was too long, so I stopped the importing event manually, but part of the data had already been introduced, which affected the integrity of my original data. Could yo… --- ## [How can you speed up live import and bull import data, and where are their performance bottlenecks](https://discuss.dgraph.io/t/how-can-you-speed-up-live-import-and-bull-import-data-and-where-are-their-performance-bottlenecks/13247)
**Author:** [@03B037](https://discuss.dgraph.io/u/03B037)\ **Replies:** 1\ **Last updated:** [March 17, 2021, 7:14pm UTC](https://discuss.dgraph.io/t/how-can-you-speed-up-live-import-and-bull-import-data-and-where-are-their-performance-bottlenecks/13247 "2021-03-17T19:14:44Z")
When I import 100 million points and 200 million edges, it seems to be particularly slow, and even the market, how should I optimize？ --- ## [Inconsistent bulk loader failures](https://discuss.dgraph.io/t/inconsistent-bulk-loader-failures/8849)
**Author:** [@diggy](https://discuss.dgraph.io/u/diggy)\ **Replies:** 13\ **Last updated:** [January 27, 2021, 10:24am UTC](https://discuss.dgraph.io/t/inconsistent-bulk-loader-failures/8849 "2021-01-27T10:24:07Z")
Moved from GitHub dgraph/5361 Posted by dzygmundfelt: What version of Dgraph are you using? v1.1.1, v1.2.2, v20.03.0 Have you tried reproducing the issue with the latest release? No. Latest is currently v20.03.1. Wh… --- ## [Error Log should print the exact line of the RDF/JSON](https://discuss.dgraph.io/t/error-log-should-print-the-exact-line-of-the-rdf-json/12414)
**Author:** [@MichelDiz](https://discuss.dgraph.io/u/MichelDiz)\ **Replies:** 4\ **Last updated:** [January 23, 2021, 7:02pm UTC](https://discuss.dgraph.io/t/error-log-should-print-the-exact-line-of-the-rdf-json/12414 "2021-01-23T19:02:00Z")
Experience Report for Feature Request What you wanted to do Analise the error log and try to find out where is the issue in my dataset to do sanitation. PS. This also should be port to Liveloader. What you actually did … --- ## [Make bulk loader code available as a package](https://discuss.dgraph.io/t/make-bulk-loader-code-available-as-a-package/12179)
**Author:** [@brr-fraugster](https://discuss.dgraph.io/u/brr-fraugster)\ **Replies:** 3\ **Last updated:** [January 6, 2021, 6:31am UTC](https://discuss.dgraph.io/t/make-bulk-loader-code-available-as-a-package/12179 "2021-01-06T06:31:28Z")
Hello Dgraphers, I’m busy building a pipeline to move data from Athena to new Dgraph clusters using the bulk loader, but I’d like to be able to control the bulk loader programmatically from inside Go. Are there plans t… --- ## [Error importing data from v1.0.8 to Slash GraphQL backend](https://discuss.dgraph.io/t/error-importing-data-from-v1-0-8-to-slash-graphql-backend/11391)
**Author:** [@peterwhitesell](https://discuss.dgraph.io/u/peterwhitesell)\ **Replies:** 6\ **Last updated:** [November 27, 2020, 9:45pm UTC](https://discuss.dgraph.io/t/error-importing-data-from-v1-0-8-to-slash-graphql-backend/11391 "2020-11-27T21:45:08Z")
In 2018 I prototyped a mindmapping app using self hosted dgraph v1.0.8 in a kubernetes cluster to store user content. I’m decommissioning the app for a period of time, intending to relaunch in the near future using Slash… --- ## [Data from neo4j to dgraph](https://discuss.dgraph.io/t/data-from-neo4j-to-dgraph/9555)
**Author:** [@tss](https://discuss.dgraph.io/u/tss)\ **Replies:** 2\ **Last updated:** [August 19, 2020, 11:57am UTC](https://discuss.dgraph.io/t/data-from-neo4j-to-dgraph/9555 "2020-08-19T11:57:40Z")
I want to directly import the data of neo4j into dgraph. Is there a feasible scheme？ --- ## [Better logs for Live and bulkloader](https://discuss.dgraph.io/t/better-logs-for-live-and-bulkloader/9140)
**Author:** [@MichelDiz](https://discuss.dgraph.io/u/MichelDiz)\ **Replies:** 0\ **Last updated:** [July 31, 2020, 5:16pm UTC](https://discuss.dgraph.io/t/better-logs-for-live-and-bulkloader/9140 "2020-07-31T17:16:21Z")
Experience Report for Feature Request What you wanted to do Have a quick and easy reading of the Live and Bulk loaders logs. Why that wasn’t great, with examples For those who are already used to it, reading live and bu… --- ## [Bulkload OOM when loading big dataset](https://discuss.dgraph.io/t/bulkload-oom-when-loading-big-dataset/8548)
**Author:** [@diggy](https://discuss.dgraph.io/u/diggy)\ **Replies:** 6\ **Last updated:** [July 11, 2020, 12:05am UTC](https://discuss.dgraph.io/t/bulkload-oom-when-loading-big-dataset/8548 "2020-07-11T00:05:40Z")
Moved from GitHub dgraph/5574 Posted by JimWen: What version of Dgraph are you using? Dgraph version : v20.03.3 Dgraph SHA-256 : 08424035910be6b6720570427948bab8352a0b5a6d59a0d20c3ec5ed29533121 Commit SHA-1 … --- ## [Bulk loader aborts on single corrupted input file instead of continuing with other valid inputs](https://discuss.dgraph.io/t/bulk-loader-aborts-on-single-corrupted-input-file-instead-of-continuing-with-other-valid-inputs/8769)
**Author:** [@diggy](https://discuss.dgraph.io/u/diggy)\ **Replies:** 7\ **Last updated:** [June 23, 2020, 1:43pm UTC](https://discuss.dgraph.io/t/bulk-loader-aborts-on-single-corrupted-input-file-instead-of-continuing-with-other-valid-inputs/8769 "2020-06-23T13:43:55Z")
Moved from GitHub dgraph/3375 Posted by vipulmathur: Filing this using the bug template, but this can be considered an enhancement as well. Bulk load crashes on encountering an error (unexpected EOF) in an input file.… --- ## [Bulk loader uses too much memory](https://discuss.dgraph.io/t/bulk-loader-uses-too-much-memory/8760)
**Author:** [@diggy](https://discuss.dgraph.io/u/diggy)\ **Replies:** 2\ **Last updated:** [May 20, 2020, 4:44am UTC](https://discuss.dgraph.io/t/bulk-loader-uses-too-much-memory/8760 "2020-05-20T04:44:09Z")
Moved from GitHub dgraph/3017 Posted by codexnull: The bulk loader’s memory usage can grow beyond the size of physical RAM and cause swap space to be used. As the image below shows, memory usage grows monotonically dur… --- ## [Add a validator for bulk and live loader](https://discuss.dgraph.io/t/add-a-validator-for-bulk-and-live-loader/8790)
**Author:** [@diggy](https://discuss.dgraph.io/u/diggy)\ **Replies:** 1\ **Last updated:** [April 24, 2020, 1:46am UTC](https://discuss.dgraph.io/t/add-a-validator-for-bulk-and-live-loader/8790 "2020-04-24T01:46:00Z")
Moved from GitHub dgraph/3984 Posted by campoy: Experience Report What you wanted to do I wanted to load a big dataset use bulk loader. Bulk loading and live loading can take quite a while for large datasets, it’s ver… --- ## [Deal better with Unrecognized RDF types](https://discuss.dgraph.io/t/deal-better-with-unrecognized-rdf-types/8490)
**Author:** [@diggy](https://discuss.dgraph.io/u/diggy)\ **Replies:** 1\ **Last updated:** [April 7, 2020, 6:55pm UTC](https://discuss.dgraph.io/t/deal-better-with-unrecognized-rdf-types/8490 "2020-04-07T18:55:03Z")
Moved from GitHub dgraph/4915 Posted by MichelDiz: Experience Report reference: http://dgraph.discourse.group/t/support-json-ld-on-dgraph/7162 What you wanted to do Import an RDF from a public dump. What you actually did … [Next page](https://discuss.dgraph.io/tag/area-bulk-loader/151.md?match_all_tags=true&page=1&tags%5B%5D=area%3Abulk-loader)