Bulk loader: What am I doing wrong, my data doesn't show up

Our cluster config:
zero: 1 (say z1)
Alpha: 3 (say a1, a2 and a3)
replica: 3
Deployed the cluster using Helm Charts

Question:
My data & schema does show up in Ratte, what am I doing wrong?

How I copied /out folder contnents to Alphas

Since our above mentioned cluster configuration created 3 groups, after bulk uploader finished on z1, per my understand of official docs, below is how copied data files /out folder…

  • On a1
    – it was in container init state (a2 & a3 were not created/scheduled yet)
    – copy /out/0/p to p folder
    – touch doneinit
    – this brought a1 to ready state and a2 in container init state
  • On a2
    – it was in container init state (a1 in ready state & a3 was not created/scheduled yet)
    – copy /out/1/p to p folder
    – touch doneinit
    – this brought a2 to ready state and a3 in container init state
  • On a3
    – it was in container init state (a1/2 in ready state)
    – copy /out/2/p to p folder
    – touch doneinit
    – a3 in ready state

Below is how we ran bulk uploader

  1. Launched bulk uploader on zero (z1) using below command
 dgraph bulk -f ${files_in_ready_state} -s ${schemaFile} --format=rdf --out /coldstart/out --replace_out --xidmap xid --store_xids --map_shards=6 --reduce_shards=3 --zero=dgraph-dgraph-zero:5080
  1. Out out directory contents (cleaned a bit for brevity)
    # pwd
    /coldstart/out
    # ls
    0  1  2
    # cd 0/p
    # ls
    000001.sst   DISCARD  KEYREGISTRY  MANIFEST group_id
    # cd ../../1/p
    # ls
    000001.sst   DISCARD  KEYREGISTRY  MANIFEST group_id
    # cd ../../2/p
    # ls
    000001.sst   DISCARD  KEYREGISTRY  MANIFEST group_id
    #
    
  2. logs of a1
I0206 21:34:58.399256      22 run.go:752] x.Config: {PortOffset:0 Limit:mutations=allow; normalize-node=10000; mutations-nquad=1000000; query-timeout=0ms; txn-abort-after=5m; max-retries=-1; query-edge=1000000; disallow-drop=false; max-pending-queries=10000 LimitMutationsNquad:1000000 LimitQueryEdge:1000000 BlockClusterWideDrop:false LimitNormalizeNode:10000 QueryTimeout:0s MaxRetries:-1 GraphQL:introspection=true; debug=false; extensions=true; poll-interval=1s; lambda-url= GraphQLDebug:false}
I0206 21:34:58.399328      22 run.go:753] x.WorkerConfig: {TmpDir:t ExportPath:export Trace:ratio=0.01; jaeger=; datadog= MyAddr:dgraph-dgraph-alpha-0.dgraph-dgraph-alpha-headless.graph.svc.cluster.local:7080 ZeroAddr:[dgraph-dgraph-zero-0.dgraph-dgraph-zero-headless.graph.svc.cluster.local:5080] TLSClientConfig:<nil> TLSServerConfig:<nil> Raft:pending-proposals=256; idx=; group=; learner=false; snapshot-after-entries=10000; snapshot-after-duration=30m Badger:{Dir: ValueDir: SyncWrites:false NumVersionsToKeep:1 ReadOnly:false Logger:0xc0004fa440 Compression:1 InMemory:false MetricsEnabled:true NumGoroutines:8 MemTableSize:67108864 BaseTableSize:2097152 BaseLevelSize:10485760 LevelSizeMultiplier:10 TableSizeMultiplier:2 MaxLevels:7 VLogPercentile:0 ValueThreshold:1048576 NumMemtables:5 BlockSize:4096 BloomFalsePositive:0.01 BlockCacheSize:697932185 IndexCacheSize:375809638 NumLevelZeroTables:5 NumLevelZeroTablesStall:15 ValueLogFileSize:1073741823 ValueLogMaxEntries:1000000 NumCompactors:4 CompactL0OnClose:false LmaxCompaction:false ZSTDCompressionLevel:0 VerifyValueChecksum:false EncryptionKey:[] EncryptionKeyRotationDuration:240h0m0s BypassLockGuard:false ChecksumVerificationMode:0 DetectConflicts:true NamespaceOffset:-1 managedTxns:false maxBatchCount:0 maxBatchSize:0 maxValueThreshold:0} WhiteListedIPRanges:[{Lower:10.155.0.0 Upper:10.155.63.255}] StrictMutations:false AclEnabled:false HmacSecret:**** AbortOlderThan:5m0s ProposedGroupId:0 StartTime:2022-02-06 21:34:57.81824603 +0000 UTC m=+0.213185717 Ludicrous:enabled=false; concurrency=2000 LudicrousEnabled:false Security:whitelist=10.155.0.0/18; token= EncryptionKey:**** LogRequest:0 HardSync:false Audit:false}
I0206 21:34:58.399459      22 run.go:754] worker.Config: {PostingDir:p WALDir:w MutationsMode:0 AuthToken: HmacSecret:**** AccessJwtTtl:0s RefreshJwtTtl:0s CachePercentage:0,65,35 CacheMb:1024 Audit:<nil> ChangeDataConf:file=; kafka=; sasl_user=; sasl_password=; ca_cert=; client_cert=; client_key=; sasl-mechanism=PLAIN;}
I0206 21:34:58.400943      22 storage.go:125] Init Raft Storage with snap: 0, first: 1, last: 0
I0206 21:34:58.400988      22 server_state.go:140] Opening postings BadgerDB with options: {Dir:p ValueDir:p SyncWrites:false NumVersionsToKeep:2147483647 ReadOnly:false Logger:0x33e3080 Compression:1 InMemory:false MetricsEnabled:true NumGoroutines:8 MemTableSize:67108864 BaseTableSize:2097152 BaseLevelSize:10485760 LevelSizeMultiplier:10 TableSizeMultiplier:2 MaxLevels:7 VLogPercentile:0 ValueThreshold:1048576 NumMemtables:5 BlockSize:4096 BloomFalsePositive:0.01 BlockCacheSize:697932185 IndexCacheSize:375809638 NumLevelZeroTables:5 NumLevelZeroTablesStall:15 ValueLogFileSize:1073741823 ValueLogMaxEntries:1000000 NumCompactors:4 CompactL0OnClose:false LmaxCompaction:false ZSTDCompressionLevel:0 VerifyValueChecksum:false EncryptionKey:[] EncryptionKeyRotationDuration:240h0m0s BypassLockGuard:false ChecksumVerificationMode:0 DetectConflicts:false NamespaceOffset:1 managedTxns:false maxBatchCount:0 maxBatchSize:0 maxValueThreshold:0}
I0206 21:34:58.841367      22 log.go:34] All 0 tables opened in 0s
I0206 21:34:58.842239      22 log.go:34] Discard stats nextEmptySlot: 0
I0206 21:34:58.842293      22 log.go:34] Set nextTxnTs to 0
I0206 21:34:58.843313      22 groups.go:99] Current Raft Id: 0x0
I0206 21:34:58.843338      22 groups.go:115] Sending member request to Zero: addr:"dgraph-dgraph-alpha-0.dgraph-dgraph-alpha-headless.graph.svc.cluster.local:7080"
I0206 21:34:58.843590      22 worker.go:114] Worker listening at address: [::]:7080
I0206 21:34:58.845073      22 run.go:565] Bringing up GraphQL HTTP API at 0.0.0.0:8080/graphql
I0206 21:34:58.845096      22 run.go:566] Bringing up GraphQL HTTP admin API at 0.0.0.0:8080/admin
I0206 21:34:58.845116      22 run.go:593] gRPC server started.  Listening on port 9080
E0206 21:34:58.845102      22 groups.go:1181] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
I0206 21:34:58.845132      22 run.go:594] HTTP server started.  Listening on port 8080
I0206 21:34:58.944585      22 pool.go:162] CONNECTING to dgraph-dgraph-zero-0.dgraph-dgraph-zero-headless.graph.svc.cluster.local:5080
I0206 21:34:58.951255      22 groups.go:134] Connected to group zero. Assigned group: 1
I0206 21:34:58.951280      22 groups.go:136] Raft Id after connection to Zero: 0x1
I0206 21:34:58.951290      22 draft.go:270] Node ID: 0x1 with GroupID: 1
I0206 21:34:58.951298      22 draft.go:279] RaftContext: id:1 group:1 addr:"dgraph-dgraph-alpha-0.dgraph-dgraph-alpha-headless.graph.svc.cluster.local:7080"
I0206 21:34:58.951377      22 node.go:152] Setting raft.Config to: &{ID:1 peers:[] learners:[] ElectionTick:20 HeartbeatTick:1 Storage:0xc00015cb80 Applied:0 MaxSizePerMsg:262144 MaxCommittedSizePerReady:67108864 MaxUncommittedEntriesSize:0 MaxInflightMsgs:256 CheckQuorum:false PreVote:true ReadOnlyOption:0 Logger:0x33e3080 DisableProposalForwarding:false}
I0206 21:34:58.951466      22 node.go:326] Group 1 found 0 entries
I0206 21:34:58.951492      22 draft.go:1857] New Node for group: 1
I0206 21:34:58.951512      22 log.go:34] 1 became follower at term 0
I0206 21:34:58.951522      22 log.go:34] newRaft 1 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
I0206 21:34:58.951533      22 log.go:34] 1 became follower at term 1
I0206 21:34:58.951572      22 draft.go:211] Operation started with id: opRollup
I0206 21:34:58.951656      22 draft.go:1208] Found Raft progress: 0
I0206 21:34:58.951662      22 groups.go:820] Got address of a Zero leader: dgraph-dgraph-zero-0.dgraph-dgraph-zero-headless.graph.svc.cluster.local:5080
I0206 21:34:58.951747      22 groups.go:834] Starting a new membership stream receive from dgraph-dgraph-zero-0.dgraph-dgraph-zero-headless.graph.svc.cluster.local:5080.
I0206 21:34:58.951788      22 node.go:189] Setting conf state to nodes:1
I0206 21:34:58.952032      22 log.go:34] 1 is starting a new election at term 1
I0206 21:34:58.952046      22 log.go:34] 1 became pre-candidate at term 1
I0206 21:34:58.952053      22 log.go:34] 1 received MsgPreVoteResp from 1 at term 1
I0206 21:34:58.952064      22 log.go:34] 1 became candidate at term 2
I0206 21:34:58.952070      22 log.go:34] 1 received MsgVoteResp from 1 at term 2
I0206 21:34:58.952081      22 log.go:34] 1 became leader at term 2
I0206 21:34:58.952087      22 log.go:34] raft.node: 1 elected leader 1 at term 2
I0206 21:34:58.952317      22 groups.go:851] Received first state update from Zero: counter:5686 groups:<key:1 value:<members:<key:1 value:<id:1 group_id:1 addr:"dgraph-dgraph-alpha-0.dgraph-dgraph-alpha-headless.graph.svc.cluster.local:7080" > > > > zeros:<key:1 value:<id:1 addr:"dgraph-dgraph-zero-0.dgraph-dgraph-zero-headless.graph.svc.cluster.local:5080" leader:true > > maxUID:624800000 maxTxnTs:10000 maxRaftId:1 cid:"cdfe11d2-2104-45fd-ab90-9e07e0ec0d37" license:<maxNodes:18446744073709551615 expiryTs:1646711439 enabled:true >
I0206 21:34:59.846668      22 pool.go:162] CONNECTING to dgraph-dgraph-alpha-0.dgraph-dgraph-alpha-headless.graph.svc.cluster.local:7080
I0206 21:34:59.952218      22 groups.go:906] Leader idx=0x1 of group=1 is connecting to Zero for txn updates
I0206 21:34:59.952252      22 groups.go:918] Got Zero leader: dgraph-dgraph-zero-0.dgraph-dgraph-zero-headless.graph.svc.cluster.local:5080
I0206 21:34:59.953704      22 groups.go:495] Serving tablet for: 0-dgraph.type
I0206 21:34:59.954608      22 groups.go:495] Serving tablet for: 0-dgraph.drop.op
I0206 21:34:59.955951      22 groups.go:495] Serving tablet for: 0-dgraph.graphql.schema
I0206 21:34:59.957113      22 groups.go:495] Serving tablet for: 0-dgraph.graphql.xid
I0206 21:34:59.958214      22 groups.go:495] Serving tablet for: 0-dgraph.graphql.p_query
I0206 21:34:59.958379      22 groups.go:166] Server is ready
I0206 21:34:59.958403      22 access_ee.go:408] ResetAcl closed
I0206 21:34:59.958409      22 access_ee.go:318] RefreshAcls closed
I0206 21:35:03.847506      22 admin.go:868] namespace: 0. No GraphQL schema in Dgraph; serving empty GraphQL API
I0206 21:38:28.960742      22 pool.go:162] CONNECTING to dgraph-dgraph-alpha-1.dgraph-dgraph-alpha-headless.graph.svc.cluster.local:7080
I0206 21:38:28.960775      22 node.go:585] Trying to add 0x2 to cluster. Addr: dgraph-dgraph-alpha-1.dgraph-dgraph-alpha-headless.graph.svc.cluster.local:7080
I0206 21:38:28.960785      22 node.go:586] Current confstate at 0x1: nodes:1
I0206 21:38:28.960930      22 node.go:189] Setting conf state to nodes:1 nodes:2
I0206 21:38:28.961009      22 node.go:755] [0x2] Done joining cluster with err: <nil>
I0206 21:41:07.834568      22 pool.go:162] CONNECTING to dgraph-dgraph-alpha-2.dgraph-dgraph-alpha-headless.graph.svc.cluster.local:7080
I0206 21:41:07.834591      22 node.go:585] Trying to add 0x3 to cluster. Addr: dgraph-dgraph-alpha-2.dgraph-dgraph-alpha-headless.graph.svc.cluster.local:7080
I0206 21:41:07.834599      22 node.go:586] Current confstate at 0x1: nodes:1 nodes:2
I0206 21:41:07.835263      22 node.go:189] Setting conf state to nodes:1 nodes:2 nodes:3
I0206 21:41:07.835312      22 node.go:755] [0x3] Done joining cluster with err: <nil>