Hi.
I have on my own machine a zero and one alpha node running. I have create more than 1.000.000 nodes.
When I run a query the alpha node restart The query will hit almost all the nodes but is limited to the first 100 nodes.
I do not understand from the log what the root cause are. Can anyone please help me in the right direction?
Dgraph version : v22.0.0
Dgraph codename : dgraph
Dgraph SHA-256 : bc4cc6d649fa2328df1d9d5702b30e2204934ba0b58cbbfa6816d52e98a9d191
Commit SHA-1 : c36206a
Commit timestamp : 2022-10-21 11:32:45 +0000
Branch : release/v22.0.0
Go version : go1.18.5
jemalloc enabled : true
For Dgraph official documentation, visit Get started with Dgraph.
For discussions about Dgraph , visit http://discuss.dgraph.io.
For fully-managed Dgraph Cloud , visit Dgraph | GraphQL Cloud Platform, Distributed Graph Engine.
Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2021 Dgraph Labs, Inc.
I0508 07:55:00.650552 1 run.go:754] x.Config: {PortOffset:0 Limit:mutations-nquad=1000000; disallow-drop=false; max-pending-queries=10000; query-edge=1000000; normalize-node=10000; query-timeout=0ms; txn-abort-after=5m; max-retries=-1; mutations=allow LimitMutationsNquad:1000000 LimitQueryEdge:1000000 BlockClusterWideDrop:false LimitNormalizeNode:10000 QueryTimeout:0s MaxRetries:-1 GraphQL:lambda-url=; introspection=true; debug=false; extensions=true; poll-interval=1s GraphQLDebug:false}
I0508 07:55:00.650599 1 run.go:755] x.WorkerConfig: {TmpDir:t ExportPath:/exports/soi Trace:jaeger=; datadog=; ratio=0.01 MyAddr:soidgraphalpha:7080 ZeroAddr:[soidgraphzero:5080] TLSClientConfig: TLSServerConfig: Raft:learner=false; snapshot-after-entries=10000; snapshot-after-duration=30m; pending-proposals=256; idx=; group= Badger:{Dir: ValueDir: SyncWrites:false NumVersionsToKeep:1 ReadOnly:false Logger:0xc000416840 Compression:1 InMemory:false MetricsEnabled:true NumGoroutines:8 MemTableSize:67108864 BaseTableSize:2097152 BaseLevelSize:10485760 LevelSizeMultiplier:10 TableSizeMultiplier:2 MaxLevels:7 VLogPercentile:0 ValueThreshold:1048576 NumMemtables:5 BlockSize:4096 BloomFalsePositive:0.01 BlockCacheSize:697932185 IndexCacheSize:375809638 NumLevelZeroTables:5 NumLevelZeroTablesStall:15 ValueLogFileSize:1073741823 ValueLogMaxEntries:1000000 NumCompactors:4 CompactL0OnClose:false LmaxCompaction:false ZSTDCompressionLevel:0 VerifyValueChecksum:false EncryptionKey: EncryptionKeyRotationDuration:240h0m0s BypassLockGuard:false ChecksumVerificationMode:0 DetectConflicts:true NamespaceOffset:-1 managedTxns:false maxBatchCount:0 maxBatchSize:0 maxValueThreshold:0} WhiteListedIPRanges:[{Lower:0.0.0.0 Upper:255.255.255.255}] StrictMutations:false AclEnabled:false HmacSecret:**** AbortOlderThan:5m0s ProposedGroupId:0 StartTime:2023-05-08 07:55:00.429748888 +0000 UTC m=+0.169656269 Ludicrous:concurrency=2000; enabled=false LudicrousEnabled:false Security:token=; whitelist=0.0.0.0/0 EncryptionKey:**** LogRequest:0 HardSync:false Audit:false}
I0508 07:55:00.650669 1 run.go:756] worker.Config: {PostingDir:p WALDir:w MutationsMode:0 AuthToken: HmacSecret:**** AccessJwtTtl:0s RefreshJwtTtl:0s CachePercentage:0,65,35 CacheMb:1024 Audit: ChangeDataConf:file=; kafka=; sasl_user=; sasl_password=; ca_cert=; client_cert=; client_key=; sasl-mechanism=PLAIN;}
I0508 07:55:00.652161 1 log.go:295] Found file: 177 First Index: 5280001
I0508 07:55:00.653274 1 storage.go:125] Init Raft Storage with snap: 5303072, first: 5303073, last: 5303103
I0508 07:55:00.653312 1 server_state.go:140] Opening postings BadgerDB with options: {Dir:p ValueDir:p SyncWrites:false NumVersionsToKeep:2147483647 ReadOnly:false Logger:0x3111300 Compression:1 InMemory:false MetricsEnabled:true NumGoroutines:8 MemTableSize:67108864 BaseTableSize:2097152 BaseLevelSize:10485760 LevelSizeMultiplier:10 TableSizeMultiplier:2 MaxLevels:7 VLogPercentile:0 ValueThreshold:1048576 NumMemtables:5 BlockSize:4096 BloomFalsePositive:0.01 BlockCacheSize:697932185 IndexCacheSize:375809638 NumLevelZeroTables:5 NumLevelZeroTablesStall:15 ValueLogFileSize:1073741823 ValueLogMaxEntries:1000000 NumCompactors:4 CompactL0OnClose:false LmaxCompaction:false ZSTDCompressionLevel:0 VerifyValueChecksum:false EncryptionKey: EncryptionKeyRotationDuration:240h0m0s BypassLockGuard:false ChecksumVerificationMode:0 DetectConflicts:false NamespaceOffset:1 managedTxns:false maxBatchCount:0 maxBatchSize:0 maxValueThreshold:0}
I0508 07:55:00.820868 1 log.go:34] All 239 tables opened in 130ms
I0508 07:55:00.825620 1 log.go:34] Discard stats nextEmptySlot: 0
I0508 07:55:00.825844 1 log.go:34] Set nextTxnTs to 4899992
I0508 07:55:00.826433 1 log.go:34] Deleting empty file: p/000009.vlog
I0508 07:55:00.853665 1 groups.go:100] Current Raft Id: 0x1
I0508 07:55:00.853776 1 worker.go:114] Worker listening at address: [::]:7080
I0508 07:55:00.853720 1 groups.go:116] Sending member request to Zero: id:1 addr:“soidgraphalpha:7080”
I0508 07:55:00.854389 1 run.go:567] Bringing up GraphQL HTTP API at 0.0.0.0:8080/graphql
I0508 07:55:00.854405 1 run.go:568] Bringing up GraphQL HTTP admin API at 0.0.0.0:8080/admin
I0508 07:55:00.854410 1 run.go:595] gRPC server started. Listening on port 9080
I0508 07:55:00.854413 1 run.go:596] HTTP server started. Listening on port 8080
E0508 07:55:00.854429 1 groups.go:1224] Error during SubscribeForUpdates for prefix “\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00”: Unable to find any servers for group: 1. closer err:
I0508 07:55:00.954282 1 pool.go:164] CONN: Connecting to soidgraphzero:5080
I0508 07:55:00.956977 1 groups.go:135] Connected to group zero. Assigned group: 0
I0508 07:55:00.956993 1 groups.go:137] Raft Id after connection to Zero: 0x1
I0508 07:55:00.957024 1 draft.go:270] Node ID: 0x1 with GroupID: 1
I0508 07:55:00.957026 1 draft.go:279] RaftContext: id:1 group:1 addr:“soidgraphalpha:7080”
I0508 07:55:00.957086 1 node.go:153] Setting raft.Config to: &{ID:1 peers: learners: ElectionTick:20 HeartbeatTick:1 Storage:0xc0003d99c0 Applied:5303072 MaxSizePerMsg:262144 MaxCommittedSizePerReady:67108864 MaxUncommittedEntriesSize:0 MaxInflightMsgs:256 CheckQuorum:false PreVote:true ReadOnlyOption:0 Logger:0x3111300 DisableProposalForwarding:false}
I0508 07:55:01.854760 1 pool.go:164] CONN: Connecting to soidgraphalpha:7080
I0508 07:55:01.958678 1 node.go:312] Found Snapshot.Metadata: {ConfState:{Nodes:[1] Learners: XXX_unrecognized:} Index:5303072 Term:3 XXX_unrecognized:}
I0508 07:55:01.958703 1 node.go:323] Found hardstate: {Term:11 Vote:1 Commit:5303103 XXX_unrecognized:}
I0508 07:55:01.982997 1 node.go:328] Group 1 found 23103 entries
I0508 07:55:01.983031 1 draft.go:1826] Restarting node for group: 1
I0508 07:55:01.983036 1 node.go:190] Setting conf state to nodes:1
I0508 07:55:01.983169 1 log.go:34] 1 became follower at term 11
I0508 07:55:01.983208 1 log.go:34] newRaft 1 [peers: [1], term: 11, commit: 5303103, applied: 5303072, lastindex: 5303103, lastterm: 11]
I0508 07:55:01.983246 1 draft.go:211] Operation started with id: opRollup
I0508 07:55:01.983311 1 groups.go:863] Got address of a Zero leader: soidgraphzero:5080
I0508 07:55:01.983310 1 draft.go:1208] Found Raft progress: 5303101
I0508 07:55:01.983417 1 groups.go:877] Starting a new membership stream receive from soidgraphzero:5080.
I0508 07:55:01.985964 1 groups.go:894] Received first state update from Zero: counter:1813801 groups:<key:1 value:<members:<key:1 value:<id:1 group_id:1 addr:“soidgraphalpha:7080” last_update:1683532501 > > tablets:<key:"\000\000\000\000\000\000\000\000AccountAttributes.displayname" value:<group_id:1 predicate:"\000\000\000\000\000\000\000\000AccountAttributes.displayname" > >
…
<key:"\000\000\000\000\000\000\000\000set" value:<group_id:1 predicate:"\000\000\000\000\000\000\000\000set" on_disk_bytes:48932903 uncompressed_bytes:93727713 > > snapshot_ts:4899992 checksum:12817227327644144052 > > zeros:<key:1 value:<id:1 addr:“soidgraphzero:5080” leader:true > > maxUID:4630000 maxTxnTs:4910000 maxRaftId:1 cid:“240d16e7-b855-4865-a386-bda0a27f091f” license:<maxNodes:18446744073709551615 expiryTs:1684406336 enabled:true >
W0508 07:55:01.986301 1 sentry_integration.go:130] unable to write CID to file open : no such file or directory
I0508 07:55:02.856912 1 pool.go:327] CONN: Re-established connection with soidgraphalpha:7080.
E0508 07:55:02.856938 1 groups.go:1224] Error during SubscribeForUpdates for prefix “\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00”: while receiving from stream: rpc error: code = Canceled desc = grpc: the client connection is closing. closer err:
I0508 07:55:02.984121 1 groups.go:166] Server is ready
I0508 07:55:02.984146 1 access_ee.go:418] InitializeAcl closed
I0508 07:55:02.986470 1 access_ee.go:353] RefreshAcls closed
I0508 07:55:04.983539 1 log.go:34] 1 is starting a new election at term 11
I0508 07:55:04.983560 1 log.go:34] 1 became pre-candidate at term 11
I0508 07:55:04.983562 1 log.go:34] 1 received MsgPreVoteResp from 1 at term 11
I0508 07:55:04.983569 1 log.go:34] 1 became candidate at term 12
I0508 07:55:04.983570 1 log.go:34] 1 received MsgVoteResp from 1 at term 12
I0508 07:55:04.983574 1 log.go:34] 1 became leader at term 12
I0508 07:55:04.983577 1 log.go:34] raft.node: 1 elected leader 1 at term 12
I0508 07:55:05.983510 1 groups.go:949] Leader idx=0x1 of group=1 is connecting to Zero for txn updates
I0508 07:55:05.983531 1 groups.go:961] Got Zero leader: soidgraphzero:5080
I0508 07:55:05.987339 1 admin.go:873] namespace: 0. No GraphQL schema in Dgraph; serving empty GraphQL API