How to load a large mutation with blank nodes?

I have a large (9 GB / 2.1 GB gzipped) JSON file containing a mutation with 15M+ objects like the three in the example bellow.
Tried to load it over HTTP with curl but it runs out of memory and dgraph live process gets killed by the system.

Is there a way to commit this mutation at once?

{
    "set": [
        {
            "uid": "_:Title_1",
            "xid": "6f850c84a0006d9aab70ba5261a0de168ad992c3",
            "title": "Title 1",
            "links": [{ "uid": "_:Link_1"},{"uid": "_:Link_2"},{ "uid": "_:Link_3"},{"uid": "_:Link_4"}]
        },
        {
            "uid": "_:Title_2",
            "xid": "7380e7dbff3c35705194836c7e9d602ccda7f207",
            "title": "Title 2",
            "links": [{"uid": "_:Link_5" },{"uid": "_:Link_6"},{"uid": "_:Link_7"}]
        },
        {
            "uid": "_:Link_1",
            "xid": "6d06831d5ed613040f148520d38b8ab099309b6e",
            "title": "Link 1",
            "links": [{"uid": "_:Title_1"},{"uid": "_:Title_2"},{"uid": "_:Link_2"}]
        }
    ]
}

This is the output of dgraph live -f file.json -a localhost:9080:

Running transaction with dgraph endpoint: localhost:9080
Found 1 data file(s) to process
Processing data file "file.json"
[06:28:41-0600] Elapsed: 05s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[06:28:46-0600] Elapsed: 10s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[06:28:51-0600] Elapsed: 15s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[06:28:56-0600] Elapsed: 20s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[06:29:01-0600] Elapsed: 25s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[06:29:06-0600] Elapsed: 30s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[06:29:11-0600] Elapsed: 35s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[06:29:16-0600] Elapsed: 40s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[06:29:21-0600] Elapsed: 45s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[06:29:26-0600] Elapsed: 50s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[06:29:31-0600] Elapsed: 55s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[06:29:36-0600] Elapsed: 01m00s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[06:29:41-0600] Elapsed: 01m05s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[06:29:46-0600] Elapsed: 01m10s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[06:29:51-0600] Elapsed: 01m15s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[1]    7489 killed     dgraph live -f file.json -a localhost:9080
dgraph version

Dgraph version : v21.12.0

You can’t load a big chunk into Dgraph like this(via cURL). You have to use Bulk or liveloader.
If your liveloader instance is being killed, you may haven’t enough resources to do so.

Thanks for your response, is 8 GB of memory not enough for liveloader?

The minimum is 20GB+, but your logs feels odd. it’s not even starting. Add the flag --format json in your command.

Tried with --format json, even lowered the batch size to 100, checked memory consumption of the process and it doesn’t even get higher than 3.5 GB.
this is the output of dgraph live -f file.json -a localhost:9080 --format json -b 100:

Running transaction with dgraph endpoint: localhost:9080
Found 1 data file(s) to process
Processing data file "file.json.gz"
[11:12:31-0600] Elapsed: 05s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[11:12:36-0600] Elapsed: 10s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[11:12:41-0600] Elapsed: 15s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[11:12:46-0600] Elapsed: 20s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[11:12:51-0600] Elapsed: 25s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[11:12:56-0600] Elapsed: 30s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[11:13:01-0600] Elapsed: 35s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[11:13:06-0600] Elapsed: 40s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[11:13:11-0600] Elapsed: 45s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[11:13:16-0600] Elapsed: 50s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[11:13:21-0600] Elapsed: 55s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[11:13:26-0600] Elapsed: 01m00s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[11:13:31-0600] Elapsed: 01m05s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[11:13:36-0600] Elapsed: 01m10s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[11:13:41-0600] Elapsed: 01m15s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[11:13:46-0600] Elapsed: 01m20s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[11:13:51-0600] Elapsed: 01m25s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[11:13:56-0600] Elapsed: 01m30s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[11:14:01-0600] Elapsed: 01m35s Txns: 0 N-Quads: 0 N-Quads/s: 0 Inflight:  0/10 Aborts: 0
[1]    17278 killed     dgraph live -f file.json.gz -a localhost:9080 --format json -b 100

I’ll try it with 16 GB of memory but i feel like memory isn’t the problem at all.

Share a piece of the json file.

it goes like this:

{
    "set": [
        {
            "uid": "_:Ally_McBeal",
            "xid": "dd1cfb010e26a865349c5be4f698c492e5e472bb",
            "title": "Ally McBeal",
            "dgraph.type": "article",
            "links": [
                {
                    "uid": "_:Sting_(musician)"
                },
                {
                    "uid": "_:Gloria_Steinem"
                },
                {
                    "uid": "_:Both_Sides,_Now"
                },
                {
                    "uid": "_:Legal_drama"
                },
                {
                    "uid": "_:Lucy_Liu"
                },
                {
                    "uid": "_:Danny_Lux"
                },
                {
                    "uid": "_:Bill_D'Elia"
                },
                {
                    "uid": "_:Barry_White"
                },
                {
                    "uid": "_:Tell_Him_(Bert_Berns_song)"
                },
                {
                    "uid": "_:Law_&_Order"
                },
                {
                    "uid": "_:Portia_de_Rossi"
                },
                {
                    "uid": "_:Continuity_(fiction)"
                },
                {
                    "uid": "_:Gloria_Gaynor"
                },
                {
                    "uid": "_:Single-camera_setup"
                },
                {
                    "uid": "_:Boston"
                },
                {
                    "uid": "_:Greg_Germann"
                },
                {
                    "uid": "_:Billy_Dickson"
                },
                {
                    "uid": "_:Mark_Albert"
                },
                {
                    "uid": "_:Ted_Danson"
                },
                {
                    "uid": "_:Hooked_on_a_Feeling"
                },
                {
                    "uid": "_:Surrealism"
                },
                {
                    "uid": "_:Kay_Starr"
                },
                {
                    "uid": "_:Hayden_Panettiere"
                },
                {
                    "uid": "_:Mariah_Carey"
                },
                {
                    "uid": "_:20th_Television"
                },
                {
                    "uid": "_:Ally_McBeal_(character)"
                },
                {
                    "uid": "_:The_Wizard_of_Oz_(1939_film)"
                },
                {
                    "uid": "_:Network_10"
                },
                {
                    "uid": "_:Golden_Globe_Award_for_Best_Television_Series –_Musical_or_Comedy"
                },
                {
                    "uid": "_:List_of_awards_and_nominations_received_by_Ally_McBeal"
                },
                {
                    "uid": "_:Susan_B._Anthony"
                },
                {
                    "uid": "_:Lisa_Nicole_Carson"
                },
                {
                    "uid": "_:Seven_Network"
                },
                {
                    "uid": "_:dancing_baby"
                },
                {
                    "uid": "_:Calista_Flockhart"
                },
                {
                    "uid": "_:James_LeGros"
                },
                {
                    "uid": "_:James_Marsden"
                },
                {
                    "uid": "_:comedy-drama"
                },
                {
                    "uid": "_:Josh_Hopkins"
                },
                {
                    "uid": "_:Anastacia"
                },
                {
                    "uid": "_:Comedy-drama"
                },
                {
                    "uid": "_:Nelle_Porter"
                },
                {
                    "uid": "_:Time_(magazine)"
                },
                {
                    "uid": "_:The_Good_Place"
                },
                {
                    "uid": "_:Entertainment_Weekly"
                },
                {
                    "uid": "_:Primetime_Emmy_Award_for_Outstanding_Comedy_Series"
                },
                {
                    "uid": "_:Lara_Flynn_Boyle"
                },
                {
                    "uid": "_:DVD-Video"
                },
                {
                    "uid": "_:Chayanne"
                },
                {
                    "uid": "_:Gil_Bellows"
                },
                {
                    "uid": "_:Dolby_Pro_Logic"
                },
                {
                    "uid": "_:Ally_McBeal_(season_1)"
                },
                {
                    "uid": "_:Jane_Krakowski"
                },
                {
                    "uid": "_:leitmotif"
                },
                {
                    "uid": "_:Steve_Robin"
                },
                {
                    "uid": "_:legal_drama"
                },
                {
                    "uid": "_:Ally_McBeal_(season_3)"
                },
                {
                    "uid": "_:music_licensing"
                },
                {
                    "uid": "_:EzyDVD"
                },
                {
                    "uid": "_:The_Practice"
                },
                {
                    "uid": "_:plot_device"
                },
                {
                    "uid": "_:Wattle_(anatomy)"
                },
                {
                    "uid": "_:Songs_from_Ally_McBeal"
                },
                {
                    "uid": "_:Tina_Turner"
                },
                {
                    "uid": "_:Dylan_McDermott"
                },
                {
                    "uid": "_:Deadline_Hollywood"
                },
                {
                    "uid": "_:American_Broadcasting_Company"
                },
                {
                    "uid": "_:Ally_McBeal_(season_2)"
                },
                {
                    "uid": "_:Tim_Suhrstedt"
                },
                {
                    "uid": "_:fantasy"
                },
                {
                    "uid": "_:Ally_McBeal_(season_4)"
                },
                {
                    "uid": "_:Josh_Groban"
                },
                {
                    "uid": "_:Michael_Badalucco"
                },
                {
                    "uid": "_:Betty_Friedan"
                },
                {
                    "uid": "_:Gladys_Knight"
                },
                {
                    "uid": "_:Heather_Locklear"
                },
                {
                    "uid": "_:Julianne_Nicholson"
                },
                {
                    "uid": "_:Vonda_Shepard"
                },
                {
                    "uid": "_:Robert_Downey_Jr."
                },
                {
                    "uid": "_:John_Cage_(character)"
                },
                {
                    "uid": "_:Fox_Broadcasting_Company"
                },
                {
                    "uid": "_:Elton_John"
                },
                {
                    "uid": "_:Ally_McBeal_(season_5)"
                },
                {
                    "uid": "_:Al_Green"
                },
                {
                    "uid": "_:20th_Century_Studios_Home_Entertainment"
                },
                {
                    "uid": "_:Wicked_Witch_of_the_West"
                },
                {
                    "uid": "_:Futurama"
                },
                {
                    "uid": "_:Mohamedou_Ould_Salahi"
                },
                {
                    "uid": "_:Ling_Woo"
                },
                {
                    "uid": "_:piano_bar"
                },
                {
                    "uid": "_:Boston_Legal"
                },
                {
                    "uid": "_:David_E._Kelley"
                },
                {
                    "uid": "_:Guantanamo_Bay_detention_camp"
                },
                {
                    "uid": "_:Regina_Hall"
                },
                {
                    "uid": "_:(Everybody's_Waitin'_for)_The_Man_with_the_Bag"
                },
                {
                    "uid": "_:List_of_The_Practice_episodes#Season_2"
                },
                {
                    "uid": "_:The_Mauritanian"
                },
                {
                    "uid": "_:When_Aliens_Attack"
                },
                {
                    "uid": "_:Mike_Listo"
                },
                {
                    "uid": "_:Law_firms_in_fiction"
                },
                {
                    "uid": "_:Courtney_Thorne-Smith"
                },
                {
                    "uid": "_:Peter_MacNicol"
                },
                {
                    "uid": "_:Macy_Gray"
                },
                {
                    "uid": "_:Barry_Manilow"
                },
                {
                    "uid": "_:The_Adam_and_Joe_Show"
                },
                {
                    "uid": "_:You're_the_First,_the_Last,_My_Everything"
                }
            ]
        },
        {
            "uid": "_:Ally_McBeal",
            "xid": "dd1cfb010e26a865349c5be4f698c492e5e472bb",
            "title": "Ally McBeal",
            "dgraph.type": "article",
            "links": [
                {
                    "uid": "_:Sting_(musician)"
                },
                {
                    "uid": "_:Gloria_Steinem"
                },
                {
                    "uid": "_:Both_Sides,_Now"
                },
                {
                    "uid": "_:Legal_drama"
                },
                {
                    "uid": "_:Lucy_Liu"
                },
                {
                    "uid": "_:Danny_Lux"
                },
                {
                    "uid": "_:Bill_D'Elia"
                },
                {
                    "uid": "_:Barry_White"
                },
                {
                    "uid": "_:Tell_Him_(Bert_Berns_song)"
                },
                {
                    "uid": "_:Law_&_Order"
                },
                {
                    "uid": "_:Portia_de_Rossi"
                },
                {
                    "uid": "_:Continuity_(fiction)"
                },
                {
                    "uid": "_:Gloria_Gaynor"
                },
                {
                    "uid": "_:Single-camera_setup"
                },
                {
                    "uid": "_:Boston"
                },
                {
                    "uid": "_:Greg_Germann"
                },
                {
                    "uid": "_:Billy_Dickson"
                },
                {
                    "uid": "_:Mark_Albert"
                },
                {
                    "uid": "_:Ted_Danson"
                },
                {
                    "uid": "_:Hooked_on_a_Feeling"
                },
                {
                    "uid": "_:Surrealism"
                },
                {
                    "uid": "_:Kay_Starr"
                },
                {
                    "uid": "_:Hayden_Panettiere"
                },
                {
                    "uid": "_:Mariah_Carey"
                },
                {
                    "uid": "_:20th_Television"
                },
                {
                    "uid": "_:Ally_McBeal_(character)"
                },
                {
                    "uid": "_:The_Wizard_of_Oz_(1939_film)"
                },
                {
                    "uid": "_:Network_10"
                },
                {
                    "uid": "_:Golden_Globe_Award_for_Best_Television_Series –_Musical_or_Comedy"
                },
                {
                    "uid": "_:List_of_awards_and_nominations_received_by_Ally_McBeal"
                },
                {
                    "uid": "_:Susan_B._Anthony"
                },
                {
                    "uid": "_:Lisa_Nicole_Carson"
                },
                {
                    "uid": "_:Seven_Network"
                },
                {
                    "uid": "_:dancing_baby"
                },
                {
                    "uid": "_:Calista_Flockhart"
                },
                {
                    "uid": "_:James_LeGros"
                },
                {
                    "uid": "_:James_Marsden"
                },
                {
                    "uid": "_:comedy-drama"
                },
                {
                    "uid": "_:Josh_Hopkins"
                },
                {
                    "uid": "_:Anastacia"
                },
                {
                    "uid": "_:Comedy-drama"
                },
                {
                    "uid": "_:Nelle_Porter"
                },
                {
                    "uid": "_:Time_(magazine)"
                },
                {
                    "uid": "_:The_Good_Place"
                },
                {
                    "uid": "_:Entertainment_Weekly"
                },
                {
                    "uid": "_:Primetime_Emmy_Award_for_Outstanding_Comedy_Series"
                },
                {
                    "uid": "_:Lara_Flynn_Boyle"
                },
                {
                    "uid": "_:DVD-Video"
                },
                {
                    "uid": "_:Chayanne"
                },
                {
                    "uid": "_:Gil_Bellows"
                },
                {
                    "uid": "_:Dolby_Pro_Logic"
                },
                {
                    "uid": "_:Ally_McBeal_(season_1)"
                },
                {
                    "uid": "_:Jane_Krakowski"
                },
                {
                    "uid": "_:leitmotif"
                },
                {
                    "uid": "_:Steve_Robin"
                },
                {
                    "uid": "_:legal_drama"
                },
                {
                    "uid": "_:Ally_McBeal_(season_3)"
                },
                {
                    "uid": "_:music_licensing"
                },
                {
                    "uid": "_:EzyDVD"
                },
                {
                    "uid": "_:The_Practice"
                },
                {
                    "uid": "_:plot_device"
                },
                {
                    "uid": "_:Wattle_(anatomy)"
                },
                {
                    "uid": "_:Songs_from_Ally_McBeal"
                },
                {
                    "uid": "_:Tina_Turner"
                },
                {
                    "uid": "_:Dylan_McDermott"
                },
                {
                    "uid": "_:Deadline_Hollywood"
                },
                {
                    "uid": "_:American_Broadcasting_Company"
                },
                {
                    "uid": "_:Ally_McBeal_(season_2)"
                },
                {
                    "uid": "_:Tim_Suhrstedt"
                },
                {
                    "uid": "_:fantasy"
                },
                {
                    "uid": "_:Ally_McBeal_(season_4)"
                },
                {
                    "uid": "_:Josh_Groban"
                },
                {
                    "uid": "_:Michael_Badalucco"
                },
                {
                    "uid": "_:Betty_Friedan"
                },
                {
                    "uid": "_:Gladys_Knight"
                },
                {
                    "uid": "_:Heather_Locklear"
                },
                {
                    "uid": "_:Julianne_Nicholson"
                },
                {
                    "uid": "_:Vonda_Shepard"
                },
                {
                    "uid": "_:Robert_Downey_Jr."
                },
                {
                    "uid": "_:John_Cage_(character)"
                },
                {
                    "uid": "_:Fox_Broadcasting_Company"
                },
                {
                    "uid": "_:Elton_John"
                },
                {
                    "uid": "_:Ally_McBeal_(season_5)"
                },
                {
                    "uid": "_:Al_Green"
                },
                {
                    "uid": "_:20th_Century_Studios_Home_Entertainment"
                },
                {
                    "uid": "_:Wicked_Witch_of_the_West"
                },
                {
                    "uid": "_:Futurama"
                },
                {
                    "uid": "_:Mohamedou_Ould_Salahi"
                },
                {
                    "uid": "_:Ling_Woo"
                },
                {
                    "uid": "_:piano_bar"
                },
                {
                    "uid": "_:Boston_Legal"
                },
                {
                    "uid": "_:David_E._Kelley"
                },
                {
                    "uid": "_:Guantanamo_Bay_detention_camp"
                },
                {
                    "uid": "_:Regina_Hall"
                },
                {
                    "uid": "_:(Everybody's_Waitin'_for)_The_Man_with_the_Bag"
                },
                {
                    "uid": "_:List_of_The_Practice_episodes#Season_2"
                },
                {
                    "uid": "_:The_Mauritanian"
                },
                {
                    "uid": "_:When_Aliens_Attack"
                },
                {
                    "uid": "_:Mike_Listo"
                },
                {
                    "uid": "_:Law_firms_in_fiction"
                },
                {
                    "uid": "_:Courtney_Thorne-Smith"
                },
                {
                    "uid": "_:Peter_MacNicol"
                },
                {
                    "uid": "_:Macy_Gray"
                },
                {
                    "uid": "_:Barry_Manilow"
                },
                {
                    "uid": "_:The_Adam_and_Joe_Show"
                },
                {
                    "uid": "_:You're_the_First,_the_Last,_My_Everything"
                }
            ]
        },
        {
            "uid": "_:Andreas_Capellanus",
            "xid": "18c2119b64b51ca6eead4bfa4167897df4e44f84",
            "title": "Andreas Capellanus",
            "dgraph.type": "article",
            "links": [
                {
                    "uid": "_:Bela_III_of_Hungary"
                },
                {
                    "uid": "_:Margaret_of_France,_Queen_of_England_and_Hungary"
                },
                {
                    "uid": "_:William_IX_of_Aquitaine"
                },
                {
                    "uid": "_:Quadrivium"
                },
                {
                    "uid": "_:Courtly_Love"
                },
                {
                    "uid": "_:De_amore_(Andreas_Capellanus)"
                },
                {
                    "uid": "_:Louis_VII_of_France"
                },
                {
                    "uid": "_:poetical_debate_(Occitan_literature)"
                },
                {
                    "uid": "_:Provençal_literature"
                },
                {
                    "uid": "_:John_Jay_Parry"
                },
                {
                    "uid": "_:Martianus_Capella"
                },
                {
                    "uid": "_:troubadour"
                },
                {
                    "uid": "_:Eleanor_of_Aquitaine"
                },
                {
                    "uid": "_:courtly_love"
                },
                {
                    "uid": "_:Marie_de_Champagne"
                },
                {
                    "uid": "_:trobairitz"
                }
            ]
        }
    ]
}

remove this, this key is used in Ratel or clients. Via Live or Bulk you have to send pure JSON.

Works just fine now, thanks for your time!

1 Like