Our cluster config:
zero: 1 (say z1)
Alpha: 3 (say a1, a2 and a3)
replica: 3
Deployed the cluster using Helm Charts
Question:
My data & schema does show up in Ratte, what am I doing wrong?
How I copied /out folder contnents to Alphas
Since our above mentioned cluster configuration created 3 groups, after bulk uploader finished on z1, per my understand of official docs, below is how copied data files /out folder…
- On a1
– it was in container init state (a2 & a3 were not created/scheduled yet)
– copy /out/0/p to p folder
– touch doneinit
– this brought a1 to ready state and a2 in container init state - On a2
– it was in container init state (a1 in ready state & a3 was not created/scheduled yet)
– copy /out/1/p to p folder
– touch doneinit
– this brought a2 to ready state and a3 in container init state - On a3
– it was in container init state (a1/2 in ready state)
– copy /out/2/p to p folder
– touch doneinit
– a3 in ready state
Below is how we ran bulk uploader
- Launched bulk uploader on zero (z1) using below command
dgraph bulk -f ${files_in_ready_state} -s ${schemaFile} --format=rdf --out /coldstart/out --replace_out --xidmap xid --store_xids --map_shards=6 --reduce_shards=3 --zero=dgraph-dgraph-zero:5080
- Out out directory contents (cleaned a bit for brevity)
# pwd /coldstart/out # ls 0 1 2 # cd 0/p # ls 000001.sst DISCARD KEYREGISTRY MANIFEST group_id # cd ../../1/p # ls 000001.sst DISCARD KEYREGISTRY MANIFEST group_id # cd ../../2/p # ls 000001.sst DISCARD KEYREGISTRY MANIFEST group_id # - logs of a1
I0206 21:34:58.399256 22 run.go:752] x.Config: {PortOffset:0 Limit:mutations=allow; normalize-node=10000; mutations-nquad=1000000; query-timeout=0ms; txn-abort-after=5m; max-retries=-1; query-edge=1000000; disallow-drop=false; max-pending-queries=10000 LimitMutationsNquad:1000000 LimitQueryEdge:1000000 BlockClusterWideDrop:false LimitNormalizeNode:10000 QueryTimeout:0s MaxRetries:-1 GraphQL:introspection=true; debug=false; extensions=true; poll-interval=1s; lambda-url= GraphQLDebug:false}
I0206 21:34:58.399328 22 run.go:753] x.WorkerConfig: {TmpDir:t ExportPath:export Trace:ratio=0.01; jaeger=; datadog= MyAddr:dgraph-dgraph-alpha-0.dgraph-dgraph-alpha-headless.graph.svc.cluster.local:7080 ZeroAddr:[dgraph-dgraph-zero-0.dgraph-dgraph-zero-headless.graph.svc.cluster.local:5080] TLSClientConfig:<nil> TLSServerConfig:<nil> Raft:pending-proposals=256; idx=; group=; learner=false; snapshot-after-entries=10000; snapshot-after-duration=30m Badger:{Dir: ValueDir: SyncWrites:false NumVersionsToKeep:1 ReadOnly:false Logger:0xc0004fa440 Compression:1 InMemory:false MetricsEnabled:true NumGoroutines:8 MemTableSize:67108864 BaseTableSize:2097152 BaseLevelSize:10485760 LevelSizeMultiplier:10 TableSizeMultiplier:2 MaxLevels:7 VLogPercentile:0 ValueThreshold:1048576 NumMemtables:5 BlockSize:4096 BloomFalsePositive:0.01 BlockCacheSize:697932185 IndexCacheSize:375809638 NumLevelZeroTables:5 NumLevelZeroTablesStall:15 ValueLogFileSize:1073741823 ValueLogMaxEntries:1000000 NumCompactors:4 CompactL0OnClose:false LmaxCompaction:false ZSTDCompressionLevel:0 VerifyValueChecksum:false EncryptionKey:[] EncryptionKeyRotationDuration:240h0m0s BypassLockGuard:false ChecksumVerificationMode:0 DetectConflicts:true NamespaceOffset:-1 managedTxns:false maxBatchCount:0 maxBatchSize:0 maxValueThreshold:0} WhiteListedIPRanges:[{Lower:10.155.0.0 Upper:10.155.63.255}] StrictMutations:false AclEnabled:false HmacSecret:**** AbortOlderThan:5m0s ProposedGroupId:0 StartTime:2022-02-06 21:34:57.81824603 +0000 UTC m=+0.213185717 Ludicrous:enabled=false; concurrency=2000 LudicrousEnabled:false Security:whitelist=10.155.0.0/18; token= EncryptionKey:**** LogRequest:0 HardSync:false Audit:false}
I0206 21:34:58.399459 22 run.go:754] worker.Config: {PostingDir:p WALDir:w MutationsMode:0 AuthToken: HmacSecret:**** AccessJwtTtl:0s RefreshJwtTtl:0s CachePercentage:0,65,35 CacheMb:1024 Audit:<nil> ChangeDataConf:file=; kafka=; sasl_user=; sasl_password=; ca_cert=; client_cert=; client_key=; sasl-mechanism=PLAIN;}
I0206 21:34:58.400943 22 storage.go:125] Init Raft Storage with snap: 0, first: 1, last: 0
I0206 21:34:58.400988 22 server_state.go:140] Opening postings BadgerDB with options: {Dir:p ValueDir:p SyncWrites:false NumVersionsToKeep:2147483647 ReadOnly:false Logger:0x33e3080 Compression:1 InMemory:false MetricsEnabled:true NumGoroutines:8 MemTableSize:67108864 BaseTableSize:2097152 BaseLevelSize:10485760 LevelSizeMultiplier:10 TableSizeMultiplier:2 MaxLevels:7 VLogPercentile:0 ValueThreshold:1048576 NumMemtables:5 BlockSize:4096 BloomFalsePositive:0.01 BlockCacheSize:697932185 IndexCacheSize:375809638 NumLevelZeroTables:5 NumLevelZeroTablesStall:15 ValueLogFileSize:1073741823 ValueLogMaxEntries:1000000 NumCompactors:4 CompactL0OnClose:false LmaxCompaction:false ZSTDCompressionLevel:0 VerifyValueChecksum:false EncryptionKey:[] EncryptionKeyRotationDuration:240h0m0s BypassLockGuard:false ChecksumVerificationMode:0 DetectConflicts:false NamespaceOffset:1 managedTxns:false maxBatchCount:0 maxBatchSize:0 maxValueThreshold:0}
I0206 21:34:58.841367 22 log.go:34] All 0 tables opened in 0s
I0206 21:34:58.842239 22 log.go:34] Discard stats nextEmptySlot: 0
I0206 21:34:58.842293 22 log.go:34] Set nextTxnTs to 0
I0206 21:34:58.843313 22 groups.go:99] Current Raft Id: 0x0
I0206 21:34:58.843338 22 groups.go:115] Sending member request to Zero: addr:"dgraph-dgraph-alpha-0.dgraph-dgraph-alpha-headless.graph.svc.cluster.local:7080"
I0206 21:34:58.843590 22 worker.go:114] Worker listening at address: [::]:7080
I0206 21:34:58.845073 22 run.go:565] Bringing up GraphQL HTTP API at 0.0.0.0:8080/graphql
I0206 21:34:58.845096 22 run.go:566] Bringing up GraphQL HTTP admin API at 0.0.0.0:8080/admin
I0206 21:34:58.845116 22 run.go:593] gRPC server started. Listening on port 9080
E0206 21:34:58.845102 22 groups.go:1181] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
I0206 21:34:58.845132 22 run.go:594] HTTP server started. Listening on port 8080
I0206 21:34:58.944585 22 pool.go:162] CONNECTING to dgraph-dgraph-zero-0.dgraph-dgraph-zero-headless.graph.svc.cluster.local:5080
I0206 21:34:58.951255 22 groups.go:134] Connected to group zero. Assigned group: 1
I0206 21:34:58.951280 22 groups.go:136] Raft Id after connection to Zero: 0x1
I0206 21:34:58.951290 22 draft.go:270] Node ID: 0x1 with GroupID: 1
I0206 21:34:58.951298 22 draft.go:279] RaftContext: id:1 group:1 addr:"dgraph-dgraph-alpha-0.dgraph-dgraph-alpha-headless.graph.svc.cluster.local:7080"
I0206 21:34:58.951377 22 node.go:152] Setting raft.Config to: &{ID:1 peers:[] learners:[] ElectionTick:20 HeartbeatTick:1 Storage:0xc00015cb80 Applied:0 MaxSizePerMsg:262144 MaxCommittedSizePerReady:67108864 MaxUncommittedEntriesSize:0 MaxInflightMsgs:256 CheckQuorum:false PreVote:true ReadOnlyOption:0 Logger:0x33e3080 DisableProposalForwarding:false}
I0206 21:34:58.951466 22 node.go:326] Group 1 found 0 entries
I0206 21:34:58.951492 22 draft.go:1857] New Node for group: 1
I0206 21:34:58.951512 22 log.go:34] 1 became follower at term 0
I0206 21:34:58.951522 22 log.go:34] newRaft 1 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
I0206 21:34:58.951533 22 log.go:34] 1 became follower at term 1
I0206 21:34:58.951572 22 draft.go:211] Operation started with id: opRollup
I0206 21:34:58.951656 22 draft.go:1208] Found Raft progress: 0
I0206 21:34:58.951662 22 groups.go:820] Got address of a Zero leader: dgraph-dgraph-zero-0.dgraph-dgraph-zero-headless.graph.svc.cluster.local:5080
I0206 21:34:58.951747 22 groups.go:834] Starting a new membership stream receive from dgraph-dgraph-zero-0.dgraph-dgraph-zero-headless.graph.svc.cluster.local:5080.
I0206 21:34:58.951788 22 node.go:189] Setting conf state to nodes:1
I0206 21:34:58.952032 22 log.go:34] 1 is starting a new election at term 1
I0206 21:34:58.952046 22 log.go:34] 1 became pre-candidate at term 1
I0206 21:34:58.952053 22 log.go:34] 1 received MsgPreVoteResp from 1 at term 1
I0206 21:34:58.952064 22 log.go:34] 1 became candidate at term 2
I0206 21:34:58.952070 22 log.go:34] 1 received MsgVoteResp from 1 at term 2
I0206 21:34:58.952081 22 log.go:34] 1 became leader at term 2
I0206 21:34:58.952087 22 log.go:34] raft.node: 1 elected leader 1 at term 2
I0206 21:34:58.952317 22 groups.go:851] Received first state update from Zero: counter:5686 groups:<key:1 value:<members:<key:1 value:<id:1 group_id:1 addr:"dgraph-dgraph-alpha-0.dgraph-dgraph-alpha-headless.graph.svc.cluster.local:7080" > > > > zeros:<key:1 value:<id:1 addr:"dgraph-dgraph-zero-0.dgraph-dgraph-zero-headless.graph.svc.cluster.local:5080" leader:true > > maxUID:624800000 maxTxnTs:10000 maxRaftId:1 cid:"cdfe11d2-2104-45fd-ab90-9e07e0ec0d37" license:<maxNodes:18446744073709551615 expiryTs:1646711439 enabled:true >
I0206 21:34:59.846668 22 pool.go:162] CONNECTING to dgraph-dgraph-alpha-0.dgraph-dgraph-alpha-headless.graph.svc.cluster.local:7080
I0206 21:34:59.952218 22 groups.go:906] Leader idx=0x1 of group=1 is connecting to Zero for txn updates
I0206 21:34:59.952252 22 groups.go:918] Got Zero leader: dgraph-dgraph-zero-0.dgraph-dgraph-zero-headless.graph.svc.cluster.local:5080
I0206 21:34:59.953704 22 groups.go:495] Serving tablet for: 0-dgraph.type
I0206 21:34:59.954608 22 groups.go:495] Serving tablet for: 0-dgraph.drop.op
I0206 21:34:59.955951 22 groups.go:495] Serving tablet for: 0-dgraph.graphql.schema
I0206 21:34:59.957113 22 groups.go:495] Serving tablet for: 0-dgraph.graphql.xid
I0206 21:34:59.958214 22 groups.go:495] Serving tablet for: 0-dgraph.graphql.p_query
I0206 21:34:59.958379 22 groups.go:166] Server is ready
I0206 21:34:59.958403 22 access_ee.go:408] ResetAcl closed
I0206 21:34:59.958409 22 access_ee.go:318] RefreshAcls closed
I0206 21:35:03.847506 22 admin.go:868] namespace: 0. No GraphQL schema in Dgraph; serving empty GraphQL API
I0206 21:38:28.960742 22 pool.go:162] CONNECTING to dgraph-dgraph-alpha-1.dgraph-dgraph-alpha-headless.graph.svc.cluster.local:7080
I0206 21:38:28.960775 22 node.go:585] Trying to add 0x2 to cluster. Addr: dgraph-dgraph-alpha-1.dgraph-dgraph-alpha-headless.graph.svc.cluster.local:7080
I0206 21:38:28.960785 22 node.go:586] Current confstate at 0x1: nodes:1
I0206 21:38:28.960930 22 node.go:189] Setting conf state to nodes:1 nodes:2
I0206 21:38:28.961009 22 node.go:755] [0x2] Done joining cluster with err: <nil>
I0206 21:41:07.834568 22 pool.go:162] CONNECTING to dgraph-dgraph-alpha-2.dgraph-dgraph-alpha-headless.graph.svc.cluster.local:7080
I0206 21:41:07.834591 22 node.go:585] Trying to add 0x3 to cluster. Addr: dgraph-dgraph-alpha-2.dgraph-dgraph-alpha-headless.graph.svc.cluster.local:7080
I0206 21:41:07.834599 22 node.go:586] Current confstate at 0x1: nodes:1 nodes:2
I0206 21:41:07.835263 22 node.go:189] Setting conf state to nodes:1 nodes:2 nodes:3
I0206 21:41:07.835312 22 node.go:755] [0x3] Done joining cluster with err: <nil>