Lambda server at port: 20000 stopped with error

Issues Dgraph
,
1

I am getting this error for fresh dgraph standalone start:
dgraph zero: dgraph zero --cwd /home/.../dgraph
dgraph alpha: dgraph alpha --limit mutations=strict --cwd /home/.../dgraph

Dgraph version   : v21.12.0
Dgraph codename  : zion
Dgraph SHA-256   : 078c75df9fa1057447c8c8afc10ea57cb0a29dfb22f9e61d8c334882b4b4eb37
Commit SHA-1     : d62ed5f15
Commit timestamp : 2021-12-02 21:20:09 +0530
Branch           : HEAD
Go version       : go1.17.3
jemalloc enabled : true

For Dgraph official documentation, visit https://dgraph.io/docs.
For discussions about Dgraph     , visit http://discuss.dgraph.io.
For fully-managed Dgraph Cloud   , visit https://dgraph.io/cloud.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2021 Dgraph Labs, Inc.


I0210 11:08:19.747941  112887 run.go:910] x.Config: {PortOffset:0 Limit:disallow-drop=false; normalize-node=10000; mutations-nquad=1000000; txn-abort-after=5m; max-pending-queries=64; max-splits=1000; query-edge=1000000; query-timeout=0ms; mutations=strict; shared-instance=false; max-retries=-1 LimitMutationsNquad:1000000 LimitQueryEdge:1000000 BlockClusterWideDrop:false LimitNormalizeNode:10000 QueryTimeout:0s MaxRetries:-1 SharedInstance:false GraphQL:{Introspection:true Debug:false Extensions:true PollInterval:1s} Lambda:{Url: Num:1 Port:20000 RestartAfter:30s}}
I0210 11:08:19.748005  112887 run.go:911] x.WorkerConfig: {TmpDir:t ExportPath:export Trace:datadog=; ratio=0.01; jaeger= MyAddr: ZeroAddr:[localhost:5080] TLSClientConfig:<nil> TLSServerConfig:<nil> Raft:group=; learner=false; snapshot-after-entries=10000; snapshot-after-duration=30m; pending-proposals=256; idx= Badger:{Dir: ValueDir: SyncWrites:false NumVersionsToKeep:1 ReadOnly:false Logger:0xc0003e6670 Compression:1 InMemory:false MetricsEnabled:true NumGoroutines:8 MemTableSize:67108864 BaseTableSize:2097152 BaseLevelSize:10485760 LevelSizeMultiplier:10 TableSizeMultiplier:2 MaxLevels:7 VLogPercentile:0 ValueThreshold:1048576 NumMemtables:15 BlockSize:4096 BloomFalsePositive:0.01 BlockCacheSize:322122547 IndexCacheSize:214748364 NumLevelZeroTables:5 NumLevelZeroTablesStall:15 ValueLogFileSize:1073741823 ValueLogMaxEntries:1000000 NumCompactors:4 CompactL0OnClose:false LmaxCompaction:false ZSTDCompressionLevel:0 VerifyValueChecksum:false EncryptionKey:[] EncryptionKeyRotationDuration:240h0m0s BypassLockGuard:false ChecksumVerificationMode:0 AllowStopTheWorld:true DetectConflicts:true NamespaceOffset:-1 ExternalMagicVersion:0 managedTxns:false maxBatchCount:0 maxBatchSize:0 maxValueThreshold:0} WhiteListedIPRanges:[] StrictMutations:true AclEnabled:false HmacSecret:**** AbortOlderThan:5m0s ProposedGroupId:0 StartTime:2022-02-10 11:08:18.750374207 +0330 +0330 m=+0.362270586 Security:token=; whitelist= EncryptionKey:**** LogRequest:0 HardSync:false Audit:false}
I0210 11:08:19.748118  112887 run.go:912] worker.Config: {PostingDir:p WALDir:w MutationsMode:3 AuthToken: HmacSecret:**** AccessJwtTtl:0s RefreshJwtTtl:0s CachePercentage:50,30,20 CacheMb:1024 Audit:<nil> ChangeDataConf:file=; kafka=; sasl_user=; sasl_password=; ca_cert=; client_cert=; client_key=; sasl-mechanism=PLAIN; tls=false;}
I0210 11:08:19.749397  112887 storage.go:125] Init Raft Storage with snap: 0, first: 1, last: 0
I0210 11:08:19.749494  112887 server_state.go:140] Opening postings BadgerDB with options: {Dir:p ValueDir:p SyncWrites:false NumVersionsToKeep:2147483647 ReadOnly:false Logger:0x3771ec8 Compression:1 InMemory:false MetricsEnabled:true NumGoroutines:8 MemTableSize:67108864 BaseTableSize:2097152 BaseLevelSize:10485760 LevelSizeMultiplier:10 TableSizeMultiplier:2 MaxLevels:7 VLogPercentile:0 ValueThreshold:1048576 NumMemtables:15 BlockSize:4096 BloomFalsePositive:0.01 BlockCacheSize:322122547 IndexCacheSize:214748364 NumLevelZeroTables:5 NumLevelZeroTablesStall:15 ValueLogFileSize:1073741823 ValueLogMaxEntries:1000000 NumCompactors:4 CompactL0OnClose:false LmaxCompaction:false ZSTDCompressionLevel:0 VerifyValueChecksum:false EncryptionKey:[] EncryptionKeyRotationDuration:240h0m0s BypassLockGuard:false ChecksumVerificationMode:0 AllowStopTheWorld:true DetectConflicts:false NamespaceOffset:1 ExternalMagicVersion:1 managedTxns:false maxBatchCount:0 maxBatchSize:0 maxValueThreshold:0}
I0210 11:08:19.781183  112887 log.go:34] All 0 tables opened in 0s
I0210 11:08:19.786895  112887 log.go:34] Discard stats nextEmptySlot: 0
I0210 11:08:19.787053  112887 log.go:34] Set nextTxnTs to 0
I0210 11:08:19.796237  112887 worker.go:114] Worker listening at address: [::]:7080
I0210 11:08:19.796461  112887 groups.go:100] Current Raft Id: 0x0
I0210 11:08:19.796481  112887 groups.go:116] Sending member request to Zero: addr:"localhost:7080" 
I0210 11:08:19.797912  112887 run.go:713] Bringing up GraphQL HTTP API at 0.0.0.0:8080/graphql
I0210 11:08:19.798095  112887 run.go:714] Bringing up GraphQL HTTP admin API at 0.0.0.0:8080/admin
E0210 11:08:19.797958  112887 groups.go:1223] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
I0210 11:08:19.800956  112887 run.go:514] Setting up lambda servers
I0210 11:08:19.800994  112887 run.go:742] gRPC server started.  Listening on port 9080
I0210 11:08:19.801007  112887 run.go:743] HTTP server started.  Listening on port 8080
I0210 11:08:19.897565  112887 pool.go:164] CONN: Connecting to localhost:5080
I0210 11:08:19.904707  112887 groups.go:135] Connected to group zero. Assigned group: 1
I0210 11:08:19.904725  112887 groups.go:137] Raft Id after connection to Zero: 0x1
I0210 11:08:19.905081  112887 draft.go:348] Node ID: 0x1 with GroupID: 1
I0210 11:08:19.905100  112887 draft.go:357] RaftContext: id:1 group:1 addr:"localhost:7080" 
I0210 11:08:19.905205  112887 node.go:152] Setting raft.Config to: &{ID:1 peers:[] learners:[] ElectionTick:20 HeartbeatTick:1 Storage:0xc0002d2bc0 Applied:0 MaxSizePerMsg:262144 MaxCommittedSizePerReady:67108864 MaxUncommittedEntriesSize:0 MaxInflightMsgs:256 CheckQuorum:false PreVote:true ReadOnlyOption:0 Logger:0x3771ec8 DisableProposalForwarding:false}
I0210 11:08:19.905954  112887 log.go:34] Number of ranges found: 1
I0210 11:08:19.905974  112887 log.go:34] Sent range 0 for iteration: [, ) of size: 0 B
I0210 11:08:19.906404  112887 log.go:34] LoadFromDb Schema Streaming about 0 B of uncompressed data (0 B on disk)
I0210 11:08:19.906519  112887 log.go:34] LoadFromDb Schema Sent data of size 0 B
I0210 11:08:19.906680  112887 log.go:34] LoadFromDb Type Streaming about 0 B of uncompressed data (0 B on disk)
I0210 11:08:19.906725  112887 log.go:34] Number of ranges found: 1
I0210 11:08:19.906892  112887 log.go:34] Sent range 0 for iteration: [, ) of size: 0 B
I0210 11:08:19.906991  112887 log.go:34] LoadFromDb Type Sent data of size 0 B
I0210 11:08:19.907003  112887 groups.go:154] Load schema from DB: OK
I0210 11:08:19.907018  112887 node.go:326] Group 1 found 0 entries
I0210 11:08:19.907025  112887 draft.go:2198] New Node for group: 1
I0210 11:08:19.907054  112887 log.go:34] 1 became follower at term 0
I0210 11:08:19.907062  112887 log.go:34] newRaft 1 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
I0210 11:08:19.907071  112887 log.go:34] 1 became follower at term 1
I0210 11:08:19.907097  112887 draft.go:279] Operation started with id: opRollup
I0210 11:08:19.907137  112887 groups.go:157] Init and start Raft node: OK
I0210 11:08:19.907263  112887 groups.go:867] Got address of a Zero leader: localhost:5080
I0210 11:08:19.907444  112887 groups.go:881] Starting a new membership stream receive from localhost:5080.
I0210 11:08:19.907515  112887 draft.go:1507] Found Raft checkpoint: 0
I0210 11:08:19.907640  112887 node.go:189] Setting conf state to nodes:1 
I0210 11:08:19.908349  112887 log.go:34] 1 is starting a new election at term 1
I0210 11:08:19.908390  112887 log.go:34] 1 became pre-candidate at term 1
I0210 11:08:19.908399  112887 log.go:34] 1 received MsgPreVoteResp from 1 at term 1
I0210 11:08:19.908423  112887 log.go:34] 1 became candidate at term 2
I0210 11:08:19.908430  112887 log.go:34] 1 received MsgVoteResp from 1 at term 2
I0210 11:08:19.908444  112887 log.go:34] 1 became leader at term 2
I0210 11:08:19.908452  112887 log.go:34] raft.node: 1 elected leader 1 at term 2
I0210 11:08:19.908611  112887 groups.go:898] Received first state update from Zero: counter:5 groups:<key:1 value:<members:<key:1 value:<id:1 group_id:1 addr:"localhost:7080" > > > > zeros:<key:1 value:<id:1 addr:"localhost:5080" leader:true > > maxRaftId:1 cid:"30314078-8bbf-4e38-9f89-16d3abc68019" license:<maxNodes:18446744073709551615 expiryTs:1647070698 enabled:true > 
I0210 11:08:20.799404  112887 pool.go:164] CONN: Connecting to localhost:7080
I0210 11:08:20.907381  112887 groups.go:165] Informed Zero about tablets I have: OK
I0210 11:08:20.907463  112887 groups.go:953] Leader idx=0x1 of group=1 is connecting to Zero for txn updates
I0210 11:08:20.907486  112887 groups.go:965] Got Zero leader: localhost:5080
I0210 11:08:20.910512  112887 groups.go:493] Serving tablet for: 0-dgraph.type
I0210 11:08:20.912131  112887 groups.go:493] Serving tablet for: 0-dgraph.drop.op
I0210 11:08:20.913958  112887 groups.go:493] Serving tablet for: 0-dgraph.graphql.schema
I0210 11:08:20.915526  112887 groups.go:493] Serving tablet for: 0-dgraph.graphql.xid
I0210 11:08:20.916615  112887 groups.go:493] Serving tablet for: 0-dgraph.graphql.p_query
I0210 11:08:20.916890  112887 groups.go:168] Upserted Schema and Types: OK
I0210 11:08:20.916926  112887 groups.go:171] Server is ready: OK
I0210 11:08:20.916935  112887 access_ee.go:412] ResetAcl closed
I0210 11:08:20.916943  112887 access_ee.go:322] RefreshAcls closed
I0210 11:08:21.801639  112887 run.go:533] Running node command: /usr/bin/node t/index.js
/home/.../dgraph/t/index.js:2
(()=>{var __webpack_modules__={9078:(e,t,n)=>{"use strict";var i=n(159),a=n(983);function r(e){if(!(this instanceof r))return new r(e);this.headers=e.headers,this.negotiator=new i(e)}function o(e){return-1===e.indexOf("/")?a.lookup(e):e}function s(e){return"string"==typeof e}e.exports=r,r.prototype.type=r.prototype.types=function(e){var t=e;if(t&&!Array.isArray(t)){t=new Array(arguments.length);for(var n=0;n<t.length;n++)t[n]=arguments[n]}if(!t||0===t.length)return this.negotiator.mediaTypes();if(!this.headers.accept)return t[0];var i=t.map(o),a=this.negotiator.mediaTypes(i.filter(s)),r=a[0];return!!r&&t[i.indexOf(r)]},r.prototype.encoding=r.prototype.encodings=function(e){var t=e;if(t&&!Array.isArray(t)){t=new Array(arguments.length);for(var n=0;n<t.length;n++)t[n]=arguments[n]}return t&&0!==t.length?this.negotiator.encodings(t)[....function u(e,t){var a=t||{};if(this.defaulE0210 11:08:21.955538  112887 run.go:543] Lambda server at port: 20000 stopped with error: exit status 1

Note: I removed some of the log text.

1 Like
2

Hi @samadadi. Any luck with this issue?

This is not the first problem I see in the logs, though. Before this one, I was getting this repeatedly (every two seconds):

May 30 12:00:41 singularity bash[16609]: I0530 12:00:41.799551   16631 run.go:533] Running node command: node t/index.js
May 30 12:00:41 singularity bash[16609]: E0530 12:00:41.799608   16631 run.go:535] Failed to start lambda server at port: 20000. Got err: exec: "node": executable file not found in $PATH

I reckoned this was due to node not being installed in my system. So I installed it and then I started receiving the error that you describe.

My system:
ArchLinux updated today.
Kernel: 5.18.0-zen1-1-zen
Running dgraph-alpha as a systemd service, installed from the AUR as per the instructions: AUR (en) - dgraph-bin
Nodejs: 18.2.0-1

Update (May 29, 2022):
I kept watching the logs, and found this:

May 30 12:38:32 singularity bash[27077]: Error: node-loader:
May 30 12:38:32 singularity bash[27077]: Error: The module '/var/lib/dgraph/t/238e74e906bbea8f81bc0ffc69ff2e86.node'
May 30 12:38:32 singularity bash[27077]: was compiled against a different Node.js version using
May 30 12:38:32 singularity bash[27077]: NODE_MODULE_VERSION 83. This version of Node.js requires
May 30 12:38:32 singularity bash[27077]: NODE_MODULE_VERSION 108. Please try re-compiling or re-installing
May 30 12:38:32 singularity bash[27077]: the module (for instance, using `npm rebuild` or `npm install`).

It goes without saying that I’m not an expert on node.js (to put it very mildly).

3

Installing node 12.x etc worked for me.

Sadly I get a different error now.

Aug 12 12:24:20 xx-dgraph-XX.xx.xxxx.com bash[30063]: Error: node-loader:
Aug 12 12:24:20 xx-dgraph-XX.xx.xxxx.com bash[30063]: Error: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /data/dgraph/t/238e74e906bbea8f81bc0ffc69ff2e86.node)
Aug 12 12:24:20 xx-dgraph-XX.xx.xxxx.com bash[30063]: at Object.2656 (/data/dgraph/t/index.js:2:142509)
Aug 12 12:24:20 xx-dgraph-XX.xx.xxxx.com bash[30063]: at __webpack_require__ (/data/dgraph/t/index.js:2:596027)
Aug 12 12:24:20 xx-dgraph-XX.xx.xxxx.com bash[30063]: at Module.2276 (/data/dgraph/t/index.js:2:236567)
Aug 12 12:24:20 xx-dgraph-XX.xx.xxxx.com bash[30063]: at __webpack_require__ (/data/dgraph/t/index.js:2:596027)
Aug 12 12:24:20 xx-dgraph-XX.xx.xxxx.com bash[30063]: at Object.6572 (/data/dgraph/t/index.js:2:304761)
Aug 12 12:24:20 xx-dgraph-XX.xx.xxxx.com bash[30063]: at __webpack_require__ (/data/dgraph/t/index.js:2:596027)
Aug 12 12:24:20 xx-dgraph-XX.xx.xxxx.com bash[30063]: at Object.4918 (/data/dgraph/t/index.js:2:301884)
Aug 12 12:24:20 xx-dgraph-XX.xx.xxxx.com bash[30063]: at __webpack_require__ (/data/dgraph/t/index.js:2:596027)
Aug 12 12:24:20 xx-dgraph-XX.xx.xxxx.com bash[30063]: at Object.3607 (/data/dgraph/t/index.js:2:307770)
Aug 12 12:24:20 xx-dgraph-XX.xx.xxxx.com bash[30063]: at __webpack_require__ (/data/dgraph/t/index.js:2:596027)
Aug 12 12:24:20 xx-dgraph-XX.xx.xxxx.com bash[30063]: E0812 12:24:20.304508   30063 run.go:543] Lambda server at port: 20000 stopped with error: exit status 1

Possible to support GLIBCXX_3.4.19… Centos 7 etc???

Do seem to be able to turn it off with

 --lambda="url=""; num=0; port=20000; restart-after=30s;"
4

Is running the lambda in a docker container an option for you? dgraph/dgraph-lambda:1.4.0 is completely self-contained so wouldn’t need a local node installed.

5

It is an option, but extra that I’d preferably not manage. Also prefer bare-metal since we are running bare etc. Centos 7.X is very common (Still gets updates till 2024 i think) and would love to have it supported out of the box like dgraph is.

Is the lambda required at all? Or will the settings I posted above be enough to disable it and run without?

Thanks for your reply btw.

6

The lambda server should only be required if you plan on using lambda resolvers. In other words if there are @lambda directives in your schema, you should be set.

Note also that I’d recommend dgraph v21.03, not 21.12 (12.21 has some scalability issues).