# Getting error while doing live loading **URL:** \ **Category:** Dgraph\ **Tags:** kind:question, dgraph\ **Created:** [August 8, 2021, 1:16pm UTC](https://discuss.dgraph.io/t/getting-error-while-doing-live-loading/15235 "2021-08-08T13:16:08Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![Sagar\_Choudhary](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/sagar_choudhary/32/7537_2.png) [@Sagar\_Choudhary](https://discuss.dgraph.io/u/Sagar_Choudhary)\ **Post date:** [August 8, 2021, 1:16pm UTC](https://discuss.dgraph.io/t/getting-error-while-doing-live-loading/15235/1 "2021-08-08T13:16:08Z")
## What I want to do I want to do live loading in a single node cluster. although I did the same thing many times this time I am getting an error. below are the steps I am following. Live Load:- Step 1:- start Zero node. - ./dgraph zero --my=localhost:5080 Step 2:- start Alpha node. ./dgraph alpha --my=localhost:7080 --zero=localhost:5080 Step 3:- Live load the data in the Dgraph. ./dgraph live -f rdf-file-path -s schema-file-path -zero=localhost:5080 -a=localhost:9080 after live loading, it starts processing files but after 1-2 mins of processing. it gets killed automatically. even didn’t get any error on the terminal. please refer attached screenshot ![image](https://canada1.discourse-cdn.com/flex007/uploads/dgraph/original/2X/3/38bba5c2326e6986481916248aad63570918e941.png) I tried the same steps on the 2-3 different servers but getting the same issue. ---
**Author:** ![MichelDiz](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/micheldiz/32/11873_2.png) [@MichelDiz](https://discuss.dgraph.io/u/MichelDiz)\ **Post date:** [August 8, 2021, 6:16pm UTC](https://discuss.dgraph.io/t/getting-error-while-doing-live-loading/15235/2 "2021-08-08T18:16:56Z")
Can you try with half of the dataset? Have you checked the limits of opened files in the OS? ---
**Author:** ![Sagar\_Choudhary](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/sagar_choudhary/32/7537_2.png) [@Sagar\_Choudhary](https://discuss.dgraph.io/u/Sagar_Choudhary)\ **Post date:** [August 9, 2021, 2:47am UTC](https://discuss.dgraph.io/t/getting-error-while-doing-live-loading/15235/3 "2021-08-09T02:47:03Z")
yes, I have checked the limit also. earlier I got the error: too many files open then I increase the limit from 1024 to 10 lakh as well. I have already completed bulk loading with the same dataset and it is working fine. but I am getting this issue while doing the live load. ---
**Author:** ![Sagar\_Choudhary](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/sagar_choudhary/32/7537_2.png) [@Sagar\_Choudhary](https://discuss.dgraph.io/u/Sagar_Choudhary)\ **Post date:** [August 12, 2021, 4:20am UTC](https://discuss.dgraph.io/t/getting-error-while-doing-live-loading/15235/5 "2021-08-12T04:20:19Z")
Can anyone please help. ---
**Author:** ![MichelDiz](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/micheldiz/32/11873_2.png) [@MichelDiz](https://discuss.dgraph.io/u/MichelDiz)\ **Post date:** [August 12, 2021, 2:23pm UTC](https://discuss.dgraph.io/t/getting-error-while-doing-live-loading/15235/6 "2021-08-12T14:23:55Z")
Please share the dataset so I can try to reproduce. ---
**Author:** ![Sagar\_Choudhary](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/sagar_choudhary/32/7537_2.png) [@Sagar\_Choudhary](https://discuss.dgraph.io/u/Sagar_Choudhary)\ **Post date:** [August 12, 2021, 2:46pm UTC](https://discuss.dgraph.io/t/getting-error-while-doing-live-loading/15235/7 "2021-08-12T14:46:06Z")
The size of dataset is 256 gb and its confidential. I can’t share. please give any suggestion. ---
**Author:** ![MichelDiz](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/micheldiz/32/11873_2.png) [@MichelDiz](https://discuss.dgraph.io/u/MichelDiz)\ **Post date:** [August 12, 2021, 2:56pm UTC](https://discuss.dgraph.io/t/getting-error-while-doing-live-loading/15235/8 "2021-08-12T14:56:13Z")
> [@Sagar\_Choudhary](#): > > 256 gb What are the stats of your machine? looks like there is your problem. ---
**Author:** ![Sagar\_Choudhary](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/sagar_choudhary/32/7537_2.png) [@Sagar\_Choudhary](https://discuss.dgraph.io/u/Sagar_Choudhary)\ **Post date:** [August 13, 2021, 4:15am UTC](https://discuss.dgraph.io/t/getting-error-while-doing-live-loading/15235/9 "2021-08-13T04:15:52Z")
1024 GB disc size and 128gb ram. I have completed bulk load with the same data on the same configuration machine. ---
**Author:** ![MichelDiz](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/micheldiz/32/11873_2.png) [@MichelDiz](https://discuss.dgraph.io/u/MichelDiz)\ **Post date:** [August 13, 2021, 3:06pm UTC](https://discuss.dgraph.io/t/getting-error-while-doing-live-loading/15235/10 "2021-08-13T15:06:51Z")
No Alpha and Zero logs? Without a way to reproduce and collect data, we can’t find a solution. Give us some debug logs [https://dgraph.io/docs/howto/retrieving-debug-information/#sidebar](https://dgraph.io/docs/howto/retrieving-debug-information/#sidebar) - Even tho, I think we need a way to reproduce it. ---
**Author:** ![Sagar\_Choudhary](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/sagar_choudhary/32/7537_2.png) [@Sagar\_Choudhary](https://discuss.dgraph.io/u/Sagar_Choudhary)\ **Post date:** [August 14, 2021, 3:47pm UTC](https://discuss.dgraph.io/t/getting-error-while-doing-live-loading/15235/11 "2021-08-14T15:47:28Z")
Actually live loading is consuming 124 gb memory and it killed. let me know one thing if we live load data from a folder of rdf file i.e 256gb and if we live load data from a rdf.gz file i.e 24 gb. will there be any difference in live loading or both the ways are same. because if a live load data from zip file i.e rdf.gz format then it start loading data i.e elapsed. please refer below screenshot. it will not consume much memory only 4 to 5 gb memory has been taken. ![image](https://canada1.discourse-cdn.com/flex007/uploads/dgraph/original/2X/2/277cdaf35470c922d59e56aff93c6d734d5d2403.png) and if I live load data from a directory of rdf files then it starts processing in different way. first it showing processing data. please refer below screenshot. it takes almost all the memory of my system and killed the system after consuming all the memory. in my case I have 124 GiB server. and it took all 124 gb memory and kill the process. ![image](https://canada1.discourse-cdn.com/flex007/uploads/dgraph/original/2X/3/36c7c3d46b4ed6c96f7846b2876e688582367a90.png) please let me know the difference in both the approaches. why it is taking 124GiB memory while loading data from a folder of RDF file and if read data from rdf.gz format then it takes much less memory. only 4-5 GiB memory. ---
**Author:** ![MichelDiz](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/micheldiz/32/11873_2.png) [@MichelDiz](https://discuss.dgraph.io/u/MichelDiz)\ **Post date:** [August 18, 2021, 11:42pm UTC](https://discuss.dgraph.io/t/getting-error-while-doing-live-loading/15235/12 "2021-08-18T23:42:50Z")
It might be related to the concurrency of loading multiple files at once. In the last case. Can you give us a memory profile? [https://dgraph.io/docs/howto/retrieving-debug-information/#memory-profile](https://dgraph.io/docs/howto/retrieving-debug-information/#memory-profile) ---
**Author:** ![Sagar\_Choudhary](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/sagar_choudhary/32/7537_2.png) [@Sagar\_Choudhary](https://discuss.dgraph.io/u/Sagar_Choudhary)\ **Post date:** [August 20, 2021, 8:33am UTC](https://discuss.dgraph.io/t/getting-error-while-doing-live-loading/15235/13 "2021-08-20T08:33:47Z")
Hi, yes, this is the concurrency issue. can you help me in compressing the data like if I have 256 gb of rdf file in a directory my\_data. How can I compress this data in rdf.gz format. eg:- my\_data.rdf.gz it will solve my problem because this compressed data can be easily live-loaded. ---
**Author:** ![MichelDiz](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/micheldiz/32/11873_2.png) [@MichelDiz](https://discuss.dgraph.io/u/MichelDiz)\ **Post date:** [August 20, 2021, 5:15pm UTC](https://discuss.dgraph.io/t/getting-error-while-doing-live-loading/15235/14 "2021-08-20T17:15:09Z")
Just merge all files in a single RDF and then gzip it normally. ---
**Author:** ![Sagar\_Choudhary](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/sagar_choudhary/32/7537_2.png) [@Sagar\_Choudhary](https://discuss.dgraph.io/u/Sagar_Choudhary)\ **Post date:** [August 25, 2021, 1:18pm UTC](https://discuss.dgraph.io/t/getting-error-while-doing-live-loading/15235/15 "2021-08-25T13:18:48Z")
> [@MichelDiz](#): > > ust merge all files in a single RDF can you please suggest how to merge all files in a single RDF? ---
**Author:** ![MichelDiz](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/micheldiz/32/11873_2.png) [@MichelDiz](https://discuss.dgraph.io/u/MichelDiz)\ **Post date:** [August 25, 2021, 1:35pm UTC](https://discuss.dgraph.io/t/getting-error-while-doing-live-loading/15235/16 "2021-08-25T13:35:44Z")
In general, we do a custom code for this, but you can use tools like grep, sed, or ` cat * >` ---
**Author:** ![Sagar\_Choudhary](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/sagar_choudhary/32/7537_2.png) [@Sagar\_Choudhary](https://discuss.dgraph.io/u/Sagar_Choudhary)\ **Post date:** [August 25, 2021, 1:55pm UTC](https://discuss.dgraph.io/t/getting-error-while-doing-live-loading/15235/17 "2021-08-25T13:55:56Z")
> [@MichelDiz](#): > > n general, we do a custom code for this, but you can use tools like that’s great! Thankyou 🙂 it would be really helpful if you can provide any link for reference. or any example for merging 2-3 rdf files. ---
**Author:** ![Sagar\_Choudhary](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/sagar_choudhary/32/7537_2.png) [@Sagar\_Choudhary](https://discuss.dgraph.io/u/Sagar_Choudhary)\ **Post date:** [August 25, 2021, 2:08pm UTC](https://discuss.dgraph.io/t/getting-error-while-doing-live-loading/15235/18 "2021-08-25T14:08:40Z")
> [@MichelDiz](#): > > `cat * >` cat \* \> this command is working for less no of files but I have 1 million files. need to merge them in a single rdf. need to know 1 more thing If I load files direclty from directory them ot shows processing and I have a zip file when I upload that file then it shlows elapsed. what is it. please refer below screenshot. ![image](https://canada1.discourse-cdn.com/flex007/uploads/dgraph/original/2X/6/6d5b9f973a2922f99601e649be49a4f6088ca854.png) ---
**Author:** ![MichelDiz](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/micheldiz/32/11873_2.png) [@MichelDiz](https://discuss.dgraph.io/u/MichelDiz)\ **Post date:** [August 25, 2021, 2:51pm UTC](https://discuss.dgraph.io/t/getting-error-while-doing-live-loading/15235/19 "2021-08-25T14:51:03Z")
> [@Sagar\_Choudhary](#): > > what is it. Not sure about your question. But the work “elapsed” means “elapsed time”, it is just a time count. > [@Sagar\_Choudhary](#): > > it would be really helpful if you can provide any link for reference. or any example for merging 2-3 rdf files. I know that we have several code spread across repositories. This one is an example of code [benchmarks/convert/main.go at master · dgraph-io/benchmarks · GitHub](https://github.com/dgraph-io/benchmarks/blob/master/convert/main.go) But I guess this converts from CVS to RDF. Or from a custom dataset(From Google) to RDF. You can create your own code for this case based on that principles in that code.