Bulk-Import does not really work

AndiK · May 14, 2019, 10:30am

In terms of bulk import, documentation and reality don’t quite match, which means that bulk import only seems to work at the first moment.

Assumption: cluster 3-fold redundant with 3 zeros and 3 alphas

The documentation says that only the “p” directory has to be transferred to all 3 alphas and bulk import only works with a new installation, not with a running system:
“Warning Don’t use bulk loader once the Dgraph cluster is up and running. Use it to import your existing data to a new cluster.”

OK. But afterwards the DB does not work correctly, it is extremely slow and inserts lead to errors, Ratel Explorer does not know any counts, and so on…
I found out that you can “repair” the DB by setting up UIDs and timestamps by URL call:
/assign?what=uids&num=10000000
/assign?what=timestamps&num=10000000
I think ZERO usually knows the current max values, but they are probably not transferred to the production system with the “p” directory.

Is there a way to read out the max values after bulk import and to set them correctly in the production system?

Or does the “zw” directory have to be transferred to the production system in addition to the “p” directory?

MichelDiz · May 14, 2019, 1:51pm

Hum, can you share the commands used?

which version are you using?

dgraph bulk --version

AndiK · May 14, 2019, 2:23pm

# dgraph bulk --version
I0514 14:20:47.762953     106 init.go:88]

Dgraph version   : v1.0.14
Commit SHA-1     : 26cb2f94
Commit timestamp : 2019-04-12 13:21:56 -0700
Branch           : HEAD
Go version       : go1.11.5

Running in docker, container from docker hub:
dgraph/dgraph:v1.0.14

I created following script:

docker run -d --name dgraph-zero-bulk --rm \
        -p 55080:5080 \
        -v /storage/local-1/data/dgraph/bulk:/dgraph \
        -v /storage/global-1/data/dgraphexport:/export \
        dgraph/dgraph:v1.0.14 dgraph zero --my=10.20.2.12:55080 --idx 1


docker run --rm --name dgraph-bulk  -v /storage/local-1/data/dgraph/bulk:/dgraph -v /storage/global-1/data/dgraphexport:/export  dgraph/dgraph:v1.0.14 dgraph bulk -r /export/$1 -s /export/$2 --map_shards=1 --reduce_shards=1 --http localhost:8000 --zero=10.20.2.12:55080

MichelDiz · May 14, 2019, 2:47pm

Are you sure? this port is wrong. The default port is 5080 not 55080. If you need change it use
-o, --port_offset int "Value added to all listening port numbers". [Grpc=5080, HTTP=6080]

selmeci · May 14, 2019, 2:49pm

Hi there!

I can confirm that I have same issue with 1.0.14 running directly on server.

I have setup of three zeros and three alphas with replication factor 3. When I use replication factor 1 and split data among servers in /p/0, p/1, … everything is working.

AndiK · May 14, 2019, 2:52pm

Yes sure: Its just a docker port mapping: dgraph needs 5080, but the access through docker is 10.20.2.12:55080. Bulk import is running while another instance is serving…

AndiK · May 14, 2019, 2:53pm

With replication factor 1 i only get a single directory /p/0 ?

selmeci · May 14, 2019, 2:57pm

Yes, of course. I use this bulk load command:

dgraph bulk -r /migration/data.rdf.gz -s /migration/g01.schema --custom_tokenizers="/dgraph/plugins/nfd.so" --zero=1.zero.weave.local:5080 --out=/migration/out --reduce_shards=3 --map_shards=3

MichelDiz · May 14, 2019, 2:57pm

Not sure about it. Docker port mapping isn’t reverse proxy or something. Also If you set the wrong “my: addr” this can lead to issues in your Cluster.

AndiK · May 14, 2019, 3:01pm

The serving instance always uses the original ports, only one instance is running per server.
I just configured the bulk import to run on a different port, to be able to run in parallel. No problems or errors while bulk import is running.

MichelDiz · May 14, 2019, 3:01pm

Try with my scripts GitHub - MichelDiz/Dgraph-Bulk-Script: Just a simple Sh to use Dgraph's Bulk Loader.

AndiK · May 14, 2019, 3:02pm

Ah, i understand. Good idea, i didn´t think of that. I´ll try it.
Hmm… but thinking about it: I think there is a risk that your data is no longer redundant? Did you check it?

selmeci · May 14, 2019, 3:10pm

You can try it. But this not solve problem with incorrect bulk import when somebody want to set up cluster with replication.

MichelDiz · May 14, 2019, 4:47pm

I’ve just tested locally what you mention using v1.0.14.

1 - ./dgraph zero --my=localhost:5080
2 - ./dgraph bulk -r 21million.rdf.gz -s 21million.schema --map_shards=1 --reduce_shards=1 --http localhost:8000 --zero=localhost:5080

{
	"RDFDir": "21million.rdf.gz",
	"JSONDir": "",
	"SchemaFile": "millionOLD.schema",
	"DgraphsDir": "out",
	"TmpDir": "tmp",
	"NumGoroutines": 16,
	"MapBufSize": 67108864,
	"ExpandEdges": true,
	"SkipMapPhase": false,
	"CleanupTmp": true,
	"NumShufflers": 1,
	"Version": false,
	"StoreXids": false,
	"ZeroAddr": "localhost:5080",
	"HttpAddr": "localhost:8000",
	"IgnoreErrors": false,
	"CustomTokenizers": "",
	"MapShards": 1,
	"ReduceShards": 1
}

3 - rm zeroDir
4 - ./dgraph zero --my=localhost:5080 (new zero, from scratch)
5 - cp out/0/p dgraphDir
6 - ./dgraph alpha --lru_mb 8000 --zero=localhost:5080

And is running just fine. Close and opens fast, queries are fast. No issues on logs.

Mac-Pro-de-Michel:dgraph-v1.0.14 micheldiz$ ./dgraph alpha --lru_mb 8000
I0514 13:38:48.949788   72454 init.go:88] 

Dgraph version   : v1.0.14
Commit SHA-1     : 26cb2f94
Commit timestamp : 2019-04-12 13:21:56 -0700
Branch           : HEAD
Go version       : go1.11.5

For Dgraph official documentation, visit https://docs.dgraph.io.
For discussions about Dgraph     , visit http://discuss.dgraph.io.
To say hi to the community       , visit https://dgraph.slack.com.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2018 Dgraph Labs, Inc.


I0514 13:38:48.950420   72454 server.go:117] Setting Badger table load option: mmap
I0514 13:38:48.950433   72454 server.go:129] Setting Badger value log load option: mmap
I0514 13:38:48.950441   72454 server.go:157] Opening write-ahead log BadgerDB with options: {Dir:w ValueDir:w SyncWrites:true TableLoadingMode:1 ValueLogLoadingMode:2 NumVersionsToKeep:1 MaxTableSize:67108864 LevelSizeMultiplier:10 MaxLevels:7 ValueThreshold:65500 NumMemtables:5 NumLevelZeroTables:5 NumLevelZeroTablesStall:10 LevelOneSize:268435456 ValueLogFileSize:1073741823 ValueLogMaxEntries:10000 NumCompactors:2 CompactL0OnClose:true managedTxns:false maxBatchCount:0 maxBatchSize:0 ReadOnly:false Truncate:true Logger:0x5556178}
I0514 13:38:48.954967   72454 node.go:85] All 0 tables opened in 0s
I0514 13:38:48.955251   72454 server.go:117] Setting Badger table load option: mmap
I0514 13:38:48.955262   72454 server.go:129] Setting Badger value log load option: mmap
I0514 13:38:48.955267   72454 server.go:171] Opening postings BadgerDB with options: {Dir:p ValueDir:p SyncWrites:true TableLoadingMode:2 ValueLogLoadingMode:2 NumVersionsToKeep:2147483647 MaxTableSize:67108864 LevelSizeMultiplier:10 MaxLevels:7 ValueThreshold:1024 NumMemtables:5 NumLevelZeroTables:5 NumLevelZeroTablesStall:10 LevelOneSize:268435456 ValueLogFileSize:1073741823 ValueLogMaxEntries:1000000 NumCompactors:2 CompactL0OnClose:true managedTxns:false maxBatchCount:0 maxBatchSize:0 ReadOnly:false Truncate:true Logger:0x5556178}
I0514 13:38:49.434758   72454 node.go:85] All 13 tables opened in 477ms
I0514 13:38:49.435645   72454 node.go:85] Replaying file id: 22 at offset: 29452205
I0514 13:38:49.435829   72454 node.go:85] Replay took: 172.759µs
I0514 13:38:49.437184   72454 groups.go:95] Current Raft Id: 0x0
I0514 13:38:49.437301   72454 worker.go:81] Worker listening at address: [::]:7080
I0514 13:38:49.437452   72454 run.go:387] gRPC server started.  Listening on port 9080
I0514 13:38:49.437467   72454 run.go:388] HTTP server started.  Listening on port 8080
I0514 13:38:49.539999   72454 pool.go:138] CONNECTED to localhost:5080
I0514 13:38:49.544440   72454 groups.go:118] Connected to group zero. Assigned group: 1
I0514 13:38:49.544454   72454 groups.go:120] Raft Id after connection to Zero: 0x1
I0514 13:38:49.544863   72454 draft.go:74] Node ID: 0x1 with GroupID: 1
I0514 13:38:49.544915   72454 node.go:153] Setting raft.Config to: &{ID:1 peers:[] learners:[] ElectionTick:100 HeartbeatTick:1 Storage:0xc00fc42270 Applied:0 MaxSizePerMsg:1048576 MaxCommittedSizePerReady:0 MaxUncommittedEntriesSize:0 MaxInflightMsgs:256 CheckQuorum:false PreVote:true ReadOnlyOption:0 Logger:0x5556178 DisableProposalForwarding:false}
I0514 13:38:49.546175   72454 node.go:297] Group 1 found 1 entries
I0514 13:38:49.546191   72454 draft.go:1143] New Node for group: 1
I0514 13:38:49.546241   72454 node.go:85] 1 became follower at term 0
I0514 13:38:49.546280   72454 node.go:85] newRaft 1 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
I0514 13:38:49.546287   72454 node.go:85] 1 became follower at term 1
I0514 13:38:49.546312   72454 groups.go:138] Server is ready
I0514 13:38:49.546351   72454 groups.go:672] Got address of a Zero leader: localhost:5080
I0514 13:38:49.546552   72454 groups.go:685] Starting a new membership stream receive from localhost:5080.
I0514 13:38:49.546497   72454 node.go:175] Setting conf state to nodes:1 
I0514 13:38:49.547280   72454 node.go:85] 1 is starting a new election at term 1
I0514 13:38:49.547295   72454 node.go:85] 1 became pre-candidate at term 1
I0514 13:38:49.547306   72454 node.go:85] 1 received MsgPreVoteResp from 1 at term 1
I0514 13:38:49.547318   72454 node.go:85] 1 became candidate at term 2
I0514 13:38:49.546720   72454 groups.go:702] Received first state update from Zero: counter:4 groups:<key:1 value:<members:<key:1 value:<id:1 group_id:1 addr:"localhost:7080" > > > > zeros:<key:1 value:<id:1 addr:"localhost:5080" leader:true > > maxRaftId:1 cid:"e2eb834b-75e3-4e03-8b09-c661eef4fe18" 
I0514 13:38:49.547323   72454 node.go:85] 1 received MsgVoteResp from 1 at term 2
I0514 13:38:49.547340   72454 node.go:85] 1 became leader at term 2
I0514 13:38:49.547346   72454 node.go:85] raft.node: 1 elected leader 1 at term 2
I0514 13:38:49.547563   72454 groups.go:394] Serving tablet for: _predicate_
I0514 13:38:49.548659   72454 mutation.go:145] Done schema update predicate:"_predicate_" value_type:STRING list:true 
I0514 13:38:49.549329   72454 groups.go:394] Serving tablet for: dgraph.xid
I0514 13:38:49.549780   72454 index.go:686] Deleting index for attr dgraph.xid and tokenizers []
I0514 13:38:49.549791   72454 index.go:704] Rebuilding index for attr dgraph.xid and tokenizers [exact]
I0514 13:38:49.550385   72454 mutation.go:145] Done schema update predicate:"dgraph.xid" value_type:STRING directive:INDEX tokenizer:"exact" 
I0514 13:38:49.550976   72454 groups.go:394] Serving tablet for: dgraph.password
I0514 13:38:49.551139   72454 mutation.go:145] Done schema update predicate:"dgraph.password" value_type:PASSWORD 
I0514 13:38:49.551649   72454 groups.go:394] Serving tablet for: dgraph.user.group
I0514 13:38:49.551816   72454 index.go:861] Deleting reverse index for dgraph.user.group
I0514 13:38:49.551872   72454 index.go:871] Rebuilding reverse index for dgraph.user.group
I0514 13:38:49.552060   72454 mutation.go:145] Done schema update predicate:"dgraph.user.group" value_type:UID directive:REVERSE 
I0514 13:38:49.552601   72454 groups.go:394] Serving tablet for: dgraph.group.acl
I0514 13:38:49.552759   72454 mutation.go:145] Done schema update predicate:"dgraph.group.acl" value_type:STRING 
I0514 13:38:50.550270   72454 groups.go:747] Leader idx=0x1 of group=1 is connecting to Zero for txn updates
I0514 13:38:50.550293   72454 groups.go:756] Got Zero leader: localhost:5080
I0514 13:38:50.550837   72454 groups.go:394] Serving tablet for: personal_appearance.person
I0514 13:38:50.551250   72454 groups.go:394] Serving tablet for: festival.sponsoring_organization
I0514 13:38:50.551781   72454 groups.go:394] Serving tablet for: topic_server.webref_cluster_members_type
I0514 13:38:50.552202   72454 groups.go:394] Serving tablet for: costumer_designer.costume_design_for_film
I0514 13:38:50.552569   72454 groups.go:394] Serving tablet for: cut.film
I0514 13:38:50.553053   72454 groups.go:394] Serving tablet for: set_decoration_by
I0514 13:38:50.553501   72454 groups.go:394] Serving tablet for: featured_locations
I0514 13:38:50.553965   72454 groups.go:394] Serving tablet for: content_rating.rating_system
I0514 13:38:50.554470   72454 groups.go:394] Serving tablet for: festival.focus
I0514 13:38:50.554960   72454 groups.go:394] Serving tablet for: festival.location
I0514 13:38:50.555362   72454 groups.go:394] Serving tablet for: netflix_id
I0514 13:38:50.555805   72454 groups.go:394] Serving tablet for: cut.type_of_cut
I0514 13:38:50.556164   72454 groups.go:394] Serving tablet for: email
I0514 13:38:50.556584   72454 groups.go:394] Serving tablet for: rated
I0514 13:38:50.556956   72454 groups.go:394] Serving tablet for: personal_appearances
I0514 13:38:50.557392   72454 groups.go:394] Serving tablet for: nytimes_id
I0514 13:38:50.557832   72454 groups.go:394] Serving tablet for: crew_gig.film
I0514 13:38:50.558295   72454 groups.go:394] Serving tablet for: cut.runtime
I0514 13:38:50.558658   72454 groups.go:394] Serving tablet for: songs
I0514 13:38:50.559053   72454 groups.go:394] Serving tablet for: featured_song.featured_in_film
I0514 13:38:50.559597   72454 groups.go:394] Serving tablet for: character.portrayed_in_films
I0514 13:38:50.560047   72454 groups.go:394] Serving tablet for: content_rating_system.ratings
I0514 13:38:50.560549   72454 groups.go:394] Serving tablet for: producer.films_executive_produced
I0514 13:38:50.561115   72454 groups.go:394] Serving tablet for: person_or_entity_appearing_in_films
I0514 13:38:50.561625   72454 groups.go:394] Serving tablet for: performance.special_performance_type
I0514 13:38:50.562114   72454 groups.go:394] Serving tablet for: production_companies
I0514 13:38:50.562528   72454 groups.go:394] Serving tablet for: content_rating.country
I0514 13:38:50.563029   72454 groups.go:394] Serving tablet for: collections
I0514 13:38:50.563467   72454 groups.go:394] Serving tablet for: release_date_s
I0514 13:38:50.563898   72454 groups.go:394] Serving tablet for: apple_movietrailer_id
I0514 13:38:50.564329   72454 groups.go:394] Serving tablet for: story_contributor.story_credits
I0514 13:38:50.564764   72454 groups.go:394] Serving tablet for: festival_focus.festivals_with_this_focus
I0514 13:38:50.565241   72454 groups.go:394] Serving tablet for: distribution_medium.films_distributed_in_this_medium
I0514 13:38:50.565655   72454 groups.go:394] Serving tablet for: rating
I0514 13:38:50.566083   72454 groups.go:394] Serving tablet for: song.films
I0514 13:38:50.566486   72454 groups.go:394] Serving tablet for: dubbing_performances
I0514 13:38:50.566891   72454 groups.go:394] Serving tablet for: set_designer.sets_designed
I0514 13:38:50.567385   72454 groups.go:394] Serving tablet for: http://www.w3.org/2000/01/rdf-schema#domain
I0514 13:38:50.568023   72454 groups.go:394] Serving tablet for: writer.film
I0514 13:38:50.568444   72454 groups.go:394] Serving tablet for: cut.release_region
I0514 13:38:50.568838   72454 groups.go:394] Serving tablet for: costume_design_by
I0514 13:38:50.569336   72454 groups.go:394] Serving tablet for: production_company.films
I0514 13:38:50.569724   72454 groups.go:394] Serving tablet for: content_rating_system.jurisdiction
I0514 13:38:50.570213   72454 groups.go:394] Serving tablet for: special_performance_type.performance_type
I0514 13:38:50.570692   72454 groups.go:394] Serving tablet for: country
I0514 13:38:50.571124   72454 groups.go:394] Serving tablet for: metacritic_id
I0514 13:38:50.571560   72454 groups.go:394] Serving tablet for: song_performer.songs
I0514 13:38:50.572028   72454 groups.go:394] Serving tablet for: festivals
I0514 13:38:50.572469   72454 groups.go:394] Serving tablet for: initial_release_date
I0514 13:38:50.572959   72454 groups.go:394] Serving tablet for: topic_server.schemastaging_corresponding_entities_type
I0514 13:38:50.573404   72454 groups.go:394] Serving tablet for: producer.film
I0514 13:38:50.573847   72454 groups.go:394] Serving tablet for: crew_gig.crew_role
I0514 13:38:50.574289   72454 groups.go:394] Serving tablet for: art_direction_by
I0514 13:38:50.574695   72454 groups.go:394] Serving tablet for: cinematography
I0514 13:38:50.575150   72454 groups.go:394] Serving tablet for: post_production
I0514 13:38:50.575657   72454 groups.go:394] Serving tablet for: http://www.w3.org/2000/01/rdf-schema#range
I0514 13:38:50.576219   72454 groups.go:394] Serving tablet for: locations
I0514 13:38:50.576675   72454 groups.go:394] Serving tablet for: _share_hash_
I0514 13:38:50.577242   72454 groups.go:394] Serving tablet for: distributors
I0514 13:38:50.577737   72454 groups.go:394] Serving tablet for: director.film
I0514 13:38:50.578236   72454 groups.go:394] Serving tablet for: primary_language
I0514 13:38:50.578725   72454 groups.go:394] Serving tablet for: job.films_with_this_crew_job
I0514 13:38:50.579095   72454 groups.go:394] Serving tablet for: personal_appearance.type_of_appearance
I0514 13:38:50.579544   72454 groups.go:394] Serving tablet for: sequel
I0514 13:38:50.580039   72454 groups.go:394] Serving tablet for: story_by
I0514 13:38:50.580515   72454 groups.go:394] Serving tablet for: casting_director
I0514 13:38:50.580948   72454 groups.go:394] Serving tablet for: performance.film
I0514 13:38:50.581386   72454 groups.go:394] Serving tablet for: character.portrayed_in_films_dubbed
I0514 13:38:50.581799   72454 groups.go:394] Serving tablet for: typ
I0514 13:38:50.582186   72454 groups.go:394] Serving tablet for: starring
I0514 13:38:50.582661   72454 groups.go:394] Serving tablet for: production_designer.films_production_designed
I0514 13:38:50.583041   72454 groups.go:394] Serving tablet for: personal_appearance.film
I0514 13:38:50.583538   72454 groups.go:394] Serving tablet for: casting_director.films_casting_directed
I0514 13:38:50.584004   72454 groups.go:394] Serving tablet for: actor.dubbing_performances
I0514 13:38:50.584398   72454 groups.go:394] Serving tablet for: regional_release_date.release_date
I0514 13:38:50.584875   72454 groups.go:394] Serving tablet for: http://www.w3.org/2002/07/owl#inverseOf
I0514 13:38:50.585504   72454 groups.go:394] Serving tablet for: music
I0514 13:38:50.586006   72454 groups.go:394] Serving tablet for: language
I0514 13:38:50.586472   72454 groups.go:394] Serving tablet for: music_contributor.film
I0514 13:38:50.586910   72454 groups.go:394] Serving tablet for: series.films_in_series
I0514 13:38:50.587373   72454 groups.go:394] Serving tablet for: regional_release_date.release_region
I0514 13:38:50.587786   72454 groups.go:394] Serving tablet for: trailers
I0514 13:38:50.588235   72454 groups.go:394] Serving tablet for: rottentomatoes_id
I0514 13:38:50.588747   72454 groups.go:394] Serving tablet for: cinematographer.film
I0514 13:38:50.589161   72454 groups.go:394] Serving tablet for: festival_event.films
I0514 13:38:50.589629   72454 groups.go:394] Serving tablet for: performance.character
I0514 13:38:50.590015   72454 groups.go:394] Serving tablet for: type.property.expected_type
I0514 13:38:50.590433   72454 groups.go:394] Serving tablet for: edited_by
I0514 13:38:50.590862   72454 groups.go:394] Serving tablet for: editor.film
I0514 13:38:50.591296   72454 groups.go:394] Serving tablet for: crewmember.films_crewed
I0514 13:38:50.591800   72454 groups.go:394] Serving tablet for: performance.character_note
I0514 13:38:50.592272   72454 groups.go:394] Serving tablet for: type.property.reverse_property
I0514 13:38:50.592796   72454 groups.go:394] Serving tablet for: name
I0514 13:38:50.593391   72454 groups.go:394] Serving tablet for: subject.films
I0514 13:38:50.593862   72454 groups.go:394] Serving tablet for: written_by
I0514 13:38:50.594457   72454 groups.go:394] Serving tablet for: gross_revenue
I0514 13:38:50.595046   72454 groups.go:394] Serving tablet for: festival_event.festival
I0514 13:38:50.595471   72454 groups.go:394] Serving tablet for: festival.date_founded
I0514 13:38:50.596027   72454 groups.go:394] Serving tablet for: art_director.films_art_directed
I0514 13:38:50.596482   72454 groups.go:394] Serving tablet for: estimated_budget
I0514 13:38:50.596943   72454 groups.go:394] Serving tablet for: crew_gig.crewmember
I0514 13:38:50.597407   72454 groups.go:394] Serving tablet for: format.format
I0514 13:38:50.597834   72454 groups.go:394] Serving tablet for: content_rating.minimum_accompanied_age
I0514 13:38:50.598374   72454 groups.go:394] Serving tablet for: content_rating.minimum_unaccompanied_age
I0514 13:38:50.598789   72454 groups.go:394] Serving tablet for: loc
I0514 13:38:50.599320   72454 groups.go:394] Serving tablet for: genre
I0514 13:38:50.599835   72454 groups.go:394] Serving tablet for: featured_song
I0514 13:38:50.600325   72454 groups.go:394] Serving tablet for: traileraddict_id
I0514 13:38:50.600819   72454 groups.go:394] Serving tablet for: featured_song.performed_by
I0514 13:38:50.601217   72454 groups.go:394] Serving tablet for: festival.individual_festivals
I0514 13:38:50.601649   72454 groups.go:394] Serving tablet for: company_role_or_service.companies_performing_this_role_or_service
I0514 13:38:50.602046   72454 groups.go:394] Serving tablet for: format
I0514 13:38:50.602557   72454 groups.go:394] Serving tablet for: series
I0514 13:38:50.603006   72454 groups.go:394] Serving tablet for: type.property.schema
I0514 13:38:50.603450   72454 groups.go:394] Serving tablet for: personal_appearance_type.appearances
I0514 13:38:50.603867   72454 groups.go:394] Serving tablet for: tagline
I0514 13:38:50.604275   72454 groups.go:394] Serving tablet for: actor.film
I0514 13:38:50.604712   72454 groups.go:394] Serving tablet for: executive_produced_by
I0514 13:38:50.605103   72454 groups.go:394] Serving tablet for: festival_sponsor.festivals_sponsored
I0514 13:38:50.605545   72454 groups.go:394] Serving tablet for: prequel
I0514 13:38:50.605960   72454 groups.go:394] Serving tablet for: produced_by
I0514 13:38:50.606418   72454 groups.go:394] Serving tablet for: soundtrack
I0514 13:38:50.606837   72454 groups.go:394] Serving tablet for: other_companies
I0514 13:38:50.607338   72454 groups.go:394] Serving tablet for: fandango_id
I0514 13:38:50.607727   72454 groups.go:394] Serving tablet for: company.films
I0514 13:38:50.608149   72454 groups.go:394] Serving tablet for: production_design_by
I0514 13:38:50.608620   72454 groups.go:394] Serving tablet for: location.featured_in_films
I0514 13:38:50.609149   72454 groups.go:394] Serving tablet for: distributor.films_distributed
I0514 13:38:50.609651   72454 groups.go:394] Serving tablet for: cut.note
I0514 13:38:50.610314   72454 groups.go:394] Serving tablet for: other_crew
I0514 13:38:50.610744   72454 groups.go:394] Serving tablet for: subjects
I0514 13:38:50.611397   72454 groups.go:394] Serving tablet for: pre_production
I0514 13:38:50.611917   72454 groups.go:394] Serving tablet for: performance.actor
I0514 13:38:50.612466   72454 groups.go:394] Serving tablet for: collection.films_in_collection
I0514 13:38:50.612954   72454 groups.go:394] Serving tablet for: filming
I0514 13:38:50.613383   72454 groups.go:394] Serving tablet for: runtime

selmeci · May 14, 2019, 5:06pm

One server is not problem, try to run 3 alphas and 3 zeros with replication factor 3. Copy /p to every server and you will see errors like UID is out of range (or something like that) when you will query alphas.

MichelDiz · May 14, 2019, 5:55pm

Okay, I’ve just tested it too (same result)

selmeci · May 15, 2019, 10:17am

You are right. I tried it one more time with my dataset on local PC and it works with last version of dGraph.

But I swear, that I’ve some issue with it few weeks ago. Maybe I used release candidate or it’s some glitch.

MichelDiz · May 15, 2019, 3:57pm

There was an issue long time ago with Bulk and multiple sharding. But it was solved. Never happened issues with a single shard as far I know. Since I always follow Bulk/Live issues.

I believe that any small error is the result of (before anything) misconfiguration that a user without noticing does. I do this a lot, as I’m doing a lot things at the same time, sometimes I do silly mistakes. And it’s easy to happen.

So, the case of @AndiK should be the address (--my) thing. As we can see that is possible to use only ‘p’ directory and get the results predicted in Docs. The main point is clear. Now just see where the error is in the whole process. It could also be in the process of moving the data between directories. Who knows.

I’d start from scratch tho.

Topic		Replies	Views
Bulk-Import does not really work but live successful Dgraph kind:question , dgraph	5	465	October 6, 2020
How to import bulk data into cluster？ Users	8	1525	September 27, 2019
About bulk loader Users	7	1855	September 12, 2018
Dgraph is using actually assigned uids after bulk import Dgraph	2	529	January 12, 2020
Documentation issue with bulk loader Dgraph dgraph	2	521	January 24, 2022

Bulk-Import does not really work

Related topics