Lambda server at port: 20000 stopped with error

I am getting this error for fresh dgraph standalone start:
dgraph zero: dgraph zero --cwd /home/.../dgraph
dgraph alpha: dgraph alpha --limit mutations=strict --cwd /home/.../dgraph

Dgraph version   : v21.12.0
Dgraph codename  : zion
Dgraph SHA-256   : 078c75df9fa1057447c8c8afc10ea57cb0a29dfb22f9e61d8c334882b4b4eb37
Commit SHA-1     : d62ed5f15
Commit timestamp : 2021-12-02 21:20:09 +0530
Branch           : HEAD
Go version       : go1.17.3
jemalloc enabled : true

For Dgraph official documentation, visit https://dgraph.io/docs.
For discussions about Dgraph     , visit https://discuss.dgraph.io.
For fully-managed Dgraph Cloud   , visit https://dgraph.io/cloud.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2021 Dgraph Labs, Inc.


I0210 11:08:19.747941  112887 run.go:910] x.Config: {PortOffset:0 Limit:disallow-drop=false; normalize-node=10000; mutations-nquad=1000000; txn-abort-after=5m; max-pending-queries=64; max-splits=1000; query-edge=1000000; query-timeout=0ms; mutations=strict; shared-instance=false; max-retries=-1 LimitMutationsNquad:1000000 LimitQueryEdge:1000000 BlockClusterWideDrop:false LimitNormalizeNode:10000 QueryTimeout:0s MaxRetries:-1 SharedInstance:false GraphQL:{Introspection:true Debug:false Extensions:true PollInterval:1s} Lambda:{Url: Num:1 Port:20000 RestartAfter:30s}}
I0210 11:08:19.748005  112887 run.go:911] x.WorkerConfig: {TmpDir:t ExportPath:export Trace:datadog=; ratio=0.01; jaeger= MyAddr: ZeroAddr:[localhost:5080] TLSClientConfig:<nil> TLSServerConfig:<nil> Raft:group=; learner=false; snapshot-after-entries=10000; snapshot-after-duration=30m; pending-proposals=256; idx= Badger:{Dir: ValueDir: SyncWrites:false NumVersionsToKeep:1 ReadOnly:false Logger:0xc0003e6670 Compression:1 InMemory:false MetricsEnabled:true NumGoroutines:8 MemTableSize:67108864 BaseTableSize:2097152 BaseLevelSize:10485760 LevelSizeMultiplier:10 TableSizeMultiplier:2 MaxLevels:7 VLogPercentile:0 ValueThreshold:1048576 NumMemtables:15 BlockSize:4096 BloomFalsePositive:0.01 BlockCacheSize:322122547 IndexCacheSize:214748364 NumLevelZeroTables:5 NumLevelZeroTablesStall:15 ValueLogFileSize:1073741823 ValueLogMaxEntries:1000000 NumCompactors:4 CompactL0OnClose:false LmaxCompaction:false ZSTDCompressionLevel:0 VerifyValueChecksum:false EncryptionKey:[] EncryptionKeyRotationDuration:240h0m0s BypassLockGuard:false ChecksumVerificationMode:0 AllowStopTheWorld:true DetectConflicts:true NamespaceOffset:-1 ExternalMagicVersion:0 managedTxns:false maxBatchCount:0 maxBatchSize:0 maxValueThreshold:0} WhiteListedIPRanges:[] StrictMutations:true AclEnabled:false HmacSecret:**** AbortOlderThan:5m0s ProposedGroupId:0 StartTime:2022-02-10 11:08:18.750374207 +0330 +0330 m=+0.362270586 Security:token=; whitelist= EncryptionKey:**** LogRequest:0 HardSync:false Audit:false}
I0210 11:08:19.748118  112887 run.go:912] worker.Config: {PostingDir:p WALDir:w MutationsMode:3 AuthToken: HmacSecret:**** AccessJwtTtl:0s RefreshJwtTtl:0s CachePercentage:50,30,20 CacheMb:1024 Audit:<nil> ChangeDataConf:file=; kafka=; sasl_user=; sasl_password=; ca_cert=; client_cert=; client_key=; sasl-mechanism=PLAIN; tls=false;}
I0210 11:08:19.749397  112887 storage.go:125] Init Raft Storage with snap: 0, first: 1, last: 0
I0210 11:08:19.749494  112887 server_state.go:140] Opening postings BadgerDB with options: {Dir:p ValueDir:p SyncWrites:false NumVersionsToKeep:2147483647 ReadOnly:false Logger:0x3771ec8 Compression:1 InMemory:false MetricsEnabled:true NumGoroutines:8 MemTableSize:67108864 BaseTableSize:2097152 BaseLevelSize:10485760 LevelSizeMultiplier:10 TableSizeMultiplier:2 MaxLevels:7 VLogPercentile:0 ValueThreshold:1048576 NumMemtables:15 BlockSize:4096 BloomFalsePositive:0.01 BlockCacheSize:322122547 IndexCacheSize:214748364 NumLevelZeroTables:5 NumLevelZeroTablesStall:15 ValueLogFileSize:1073741823 ValueLogMaxEntries:1000000 NumCompactors:4 CompactL0OnClose:false LmaxCompaction:false ZSTDCompressionLevel:0 VerifyValueChecksum:false EncryptionKey:[] EncryptionKeyRotationDuration:240h0m0s BypassLockGuard:false ChecksumVerificationMode:0 AllowStopTheWorld:true DetectConflicts:false NamespaceOffset:1 ExternalMagicVersion:1 managedTxns:false maxBatchCount:0 maxBatchSize:0 maxValueThreshold:0}
I0210 11:08:19.781183  112887 log.go:34] All 0 tables opened in 0s
I0210 11:08:19.786895  112887 log.go:34] Discard stats nextEmptySlot: 0
I0210 11:08:19.787053  112887 log.go:34] Set nextTxnTs to 0
I0210 11:08:19.796237  112887 worker.go:114] Worker listening at address: [::]:7080
I0210 11:08:19.796461  112887 groups.go:100] Current Raft Id: 0x0
I0210 11:08:19.796481  112887 groups.go:116] Sending member request to Zero: addr:"localhost:7080" 
I0210 11:08:19.797912  112887 run.go:713] Bringing up GraphQL HTTP API at 0.0.0.0:8080/graphql
I0210 11:08:19.798095  112887 run.go:714] Bringing up GraphQL HTTP admin API at 0.0.0.0:8080/admin
E0210 11:08:19.797958  112887 groups.go:1223] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
I0210 11:08:19.800956  112887 run.go:514] Setting up lambda servers
I0210 11:08:19.800994  112887 run.go:742] gRPC server started.  Listening on port 9080
I0210 11:08:19.801007  112887 run.go:743] HTTP server started.  Listening on port 8080
I0210 11:08:19.897565  112887 pool.go:164] CONN: Connecting to localhost:5080
I0210 11:08:19.904707  112887 groups.go:135] Connected to group zero. Assigned group: 1
I0210 11:08:19.904725  112887 groups.go:137] Raft Id after connection to Zero: 0x1
I0210 11:08:19.905081  112887 draft.go:348] Node ID: 0x1 with GroupID: 1
I0210 11:08:19.905100  112887 draft.go:357] RaftContext: id:1 group:1 addr:"localhost:7080" 
I0210 11:08:19.905205  112887 node.go:152] Setting raft.Config to: &{ID:1 peers:[] learners:[] ElectionTick:20 HeartbeatTick:1 Storage:0xc0002d2bc0 Applied:0 MaxSizePerMsg:262144 MaxCommittedSizePerReady:67108864 MaxUncommittedEntriesSize:0 MaxInflightMsgs:256 CheckQuorum:false PreVote:true ReadOnlyOption:0 Logger:0x3771ec8 DisableProposalForwarding:false}
I0210 11:08:19.905954  112887 log.go:34] Number of ranges found: 1
I0210 11:08:19.905974  112887 log.go:34] Sent range 0 for iteration: [, ) of size: 0 B
I0210 11:08:19.906404  112887 log.go:34] LoadFromDb Schema Streaming about 0 B of uncompressed data (0 B on disk)
I0210 11:08:19.906519  112887 log.go:34] LoadFromDb Schema Sent data of size 0 B
I0210 11:08:19.906680  112887 log.go:34] LoadFromDb Type Streaming about 0 B of uncompressed data (0 B on disk)
I0210 11:08:19.906725  112887 log.go:34] Number of ranges found: 1
I0210 11:08:19.906892  112887 log.go:34] Sent range 0 for iteration: [, ) of size: 0 B
I0210 11:08:19.906991  112887 log.go:34] LoadFromDb Type Sent data of size 0 B
I0210 11:08:19.907003  112887 groups.go:154] Load schema from DB: OK
I0210 11:08:19.907018  112887 node.go:326] Group 1 found 0 entries
I0210 11:08:19.907025  112887 draft.go:2198] New Node for group: 1
I0210 11:08:19.907054  112887 log.go:34] 1 became follower at term 0
I0210 11:08:19.907062  112887 log.go:34] newRaft 1 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
I0210 11:08:19.907071  112887 log.go:34] 1 became follower at term 1
I0210 11:08:19.907097  112887 draft.go:279] Operation started with id: opRollup
I0210 11:08:19.907137  112887 groups.go:157] Init and start Raft node: OK
I0210 11:08:19.907263  112887 groups.go:867] Got address of a Zero leader: localhost:5080
I0210 11:08:19.907444  112887 groups.go:881] Starting a new membership stream receive from localhost:5080.
I0210 11:08:19.907515  112887 draft.go:1507] Found Raft checkpoint: 0
I0210 11:08:19.907640  112887 node.go:189] Setting conf state to nodes:1 
I0210 11:08:19.908349  112887 log.go:34] 1 is starting a new election at term 1
I0210 11:08:19.908390  112887 log.go:34] 1 became pre-candidate at term 1
I0210 11:08:19.908399  112887 log.go:34] 1 received MsgPreVoteResp from 1 at term 1
I0210 11:08:19.908423  112887 log.go:34] 1 became candidate at term 2
I0210 11:08:19.908430  112887 log.go:34] 1 received MsgVoteResp from 1 at term 2
I0210 11:08:19.908444  112887 log.go:34] 1 became leader at term 2
I0210 11:08:19.908452  112887 log.go:34] raft.node: 1 elected leader 1 at term 2
I0210 11:08:19.908611  112887 groups.go:898] Received first state update from Zero: counter:5 groups:<key:1 value:<members:<key:1 value:<id:1 group_id:1 addr:"localhost:7080" > > > > zeros:<key:1 value:<id:1 addr:"localhost:5080" leader:true > > maxRaftId:1 cid:"30314078-8bbf-4e38-9f89-16d3abc68019" license:<maxNodes:18446744073709551615 expiryTs:1647070698 enabled:true > 
I0210 11:08:20.799404  112887 pool.go:164] CONN: Connecting to localhost:7080
I0210 11:08:20.907381  112887 groups.go:165] Informed Zero about tablets I have: OK
I0210 11:08:20.907463  112887 groups.go:953] Leader idx=0x1 of group=1 is connecting to Zero for txn updates
I0210 11:08:20.907486  112887 groups.go:965] Got Zero leader: localhost:5080
I0210 11:08:20.910512  112887 groups.go:493] Serving tablet for: 0-dgraph.type
I0210 11:08:20.912131  112887 groups.go:493] Serving tablet for: 0-dgraph.drop.op
I0210 11:08:20.913958  112887 groups.go:493] Serving tablet for: 0-dgraph.graphql.schema
I0210 11:08:20.915526  112887 groups.go:493] Serving tablet for: 0-dgraph.graphql.xid
I0210 11:08:20.916615  112887 groups.go:493] Serving tablet for: 0-dgraph.graphql.p_query
I0210 11:08:20.916890  112887 groups.go:168] Upserted Schema and Types: OK
I0210 11:08:20.916926  112887 groups.go:171] Server is ready: OK
I0210 11:08:20.916935  112887 access_ee.go:412] ResetAcl closed
I0210 11:08:20.916943  112887 access_ee.go:322] RefreshAcls closed
I0210 11:08:21.801639  112887 run.go:533] Running node command: /usr/bin/node t/index.js
/home/.../dgraph/t/index.js:2
(()=>{var __webpack_modules__={9078:(e,t,n)=>{"use strict";var i=n(159),a=n(983);function r(e){if(!(this instanceof r))return new r(e);this.headers=e.headers,this.negotiator=new i(e)}function o(e){return-1===e.indexOf("/")?a.lookup(e):e}function s(e){return"string"==typeof e}e.exports=r,r.prototype.type=r.prototype.types=function(e){var t=e;if(t&&!Array.isArray(t)){t=new Array(arguments.length);for(var n=0;n<t.length;n++)t[n]=arguments[n]}if(!t||0===t.length)return this.negotiator.mediaTypes();if(!this.headers.accept)return t[0];var i=t.map(o),a=this.negotiator.mediaTypes(i.filter(s)),r=a[0];return!!r&&t[i.indexOf(r)]},r.prototype.encoding=r.prototype.encodings=function(e){var t=e;if(t&&!Array.isArray(t)){t=new Array(arguments.length);for(var n=0;n<t.length;n++)t[n]=arguments[n]}return t&&0!==t.length?this.negotiator.encodings(t)[....function u(e,t){var a=t||{};if(this.defaulE0210 11:08:21.955538  112887 run.go:543] Lambda server at port: 20000 stopped with error: exit status 1

Note: I removed some of the log text.

1 Like

Hi @samadadi. Any luck with this issue?

This is not the first problem I see in the logs, though. Before this one, I was getting this repeatedly (every two seconds):

May 30 12:00:41 singularity bash[16609]: I0530 12:00:41.799551   16631 run.go:533] Running node command: node t/index.js
May 30 12:00:41 singularity bash[16609]: E0530 12:00:41.799608   16631 run.go:535] Failed to start lambda server at port: 20000. Got err: exec: "node": executable file not found in $PATH

I reckoned this was due to node not being installed in my system. So I installed it and then I started receiving the error that you describe.

My system:
ArchLinux updated today.
Kernel: 5.18.0-zen1-1-zen
Running dgraph-alpha as a systemd service, installed from the AUR as per the instructions: AUR (en) - dgraph-bin
Nodejs: 18.2.0-1


Update (May 29, 2022):
I kept watching the logs, and found this:

May 30 12:38:32 singularity bash[27077]: Error: node-loader:
May 30 12:38:32 singularity bash[27077]: Error: The module '/var/lib/dgraph/t/238e74e906bbea8f81bc0ffc69ff2e86.node'
May 30 12:38:32 singularity bash[27077]: was compiled against a different Node.js version using
May 30 12:38:32 singularity bash[27077]: NODE_MODULE_VERSION 83. This version of Node.js requires
May 30 12:38:32 singularity bash[27077]: NODE_MODULE_VERSION 108. Please try re-compiling or re-installing
May 30 12:38:32 singularity bash[27077]: the module (for instance, using `npm rebuild` or `npm install`).

It goes without saying that I’m not an expert on node.js (to put it very mildly).

Installing node 12.x etc worked for me.

Sadly I get a different error now.

Aug 12 12:24:20 xx-dgraph-XX.xx.xxxx.com bash[30063]: Error: node-loader:
Aug 12 12:24:20 xx-dgraph-XX.xx.xxxx.com bash[30063]: Error: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /data/dgraph/t/238e74e906bbea8f81bc0ffc69ff2e86.node)
Aug 12 12:24:20 xx-dgraph-XX.xx.xxxx.com bash[30063]: at Object.2656 (/data/dgraph/t/index.js:2:142509)
Aug 12 12:24:20 xx-dgraph-XX.xx.xxxx.com bash[30063]: at __webpack_require__ (/data/dgraph/t/index.js:2:596027)
Aug 12 12:24:20 xx-dgraph-XX.xx.xxxx.com bash[30063]: at Module.2276 (/data/dgraph/t/index.js:2:236567)
Aug 12 12:24:20 xx-dgraph-XX.xx.xxxx.com bash[30063]: at __webpack_require__ (/data/dgraph/t/index.js:2:596027)
Aug 12 12:24:20 xx-dgraph-XX.xx.xxxx.com bash[30063]: at Object.6572 (/data/dgraph/t/index.js:2:304761)
Aug 12 12:24:20 xx-dgraph-XX.xx.xxxx.com bash[30063]: at __webpack_require__ (/data/dgraph/t/index.js:2:596027)
Aug 12 12:24:20 xx-dgraph-XX.xx.xxxx.com bash[30063]: at Object.4918 (/data/dgraph/t/index.js:2:301884)
Aug 12 12:24:20 xx-dgraph-XX.xx.xxxx.com bash[30063]: at __webpack_require__ (/data/dgraph/t/index.js:2:596027)
Aug 12 12:24:20 xx-dgraph-XX.xx.xxxx.com bash[30063]: at Object.3607 (/data/dgraph/t/index.js:2:307770)
Aug 12 12:24:20 xx-dgraph-XX.xx.xxxx.com bash[30063]: at __webpack_require__ (/data/dgraph/t/index.js:2:596027)
Aug 12 12:24:20 xx-dgraph-XX.xx.xxxx.com bash[30063]: E0812 12:24:20.304508   30063 run.go:543] Lambda server at port: 20000 stopped with error: exit status 1

Possible to support GLIBCXX_3.4.19… Centos 7 etc???

Do seem to be able to turn it off with

 --lambda="url=""; num=0; port=20000; restart-after=30s;"

Is running the lambda in a docker container an option for you? dgraph/dgraph-lambda:1.4.0 is completely self-contained so wouldn’t need a local node installed.

It is an option, but extra that I’d preferably not manage. Also prefer bare-metal since we are running bare etc. Centos 7.X is very common (Still gets updates till 2024 i think) and would love to have it supported out of the box like dgraph is.

Is the lambda required at all? Or will the settings I posted above be enough to disable it and run without?

Thanks for your reply btw.

The lambda server should only be required if you plan on using lambda resolvers. In other words if there are @lambda directives in your schema, you should be set.

Note also that I’d recommend dgraph v21.03, not 21.12 (12.21 has some scalability issues).