Problem detected when run dgraph alpha with custom tokeniser


Report a Dgraph Bug

What version of Dgraph are you using?

Dgraph Version
$ dgraph version
 
Dgraph version   : v20.11.0-gd79d3797b
Dgraph codename  : unnamed-mod
Dgraph SHA-256   : 936d4f9e15171392e4017378abf2cd853896f624ac22a6d18c5e02b91c4a7c8f
Commit SHA-1     : d79d3797b
Commit timestamp : 2021-02-05 07:09:12 -0800
Branch           : master
Go version       : go1.15.6
jemalloc enabled : true

Have you tried reproducing the issue with the latest release?

yes

What is the hardware spec (RAM, OS)?

Ubuntu 20.10

Steps to reproduce the issue (command/config used to run Dgraph).

./dgraph alpha -p /data/dgraph/p -w /data/dgraph/w --whitelist=0.0.0.0/0 --custom_tokenizers=/usr/lib/dgraph/cidr.so

this is main file of custom tokenizer

package main

import "net"

func Tokenizer() interface{} { return CIDRTokenizer{} }

type CIDRTokenizer struct{}

func (CIDRTokenizer) Name() string     { return "cidr" }
func (CIDRTokenizer) Type() string     { return "string" }
func (CIDRTokenizer) Identifier() byte { return 0xff }

func (t CIDRTokenizer) Tokens(value interface{}) ([]string, error) {
        _, ipnet, err := net.ParseCIDR(value.(string))
        if err != nil {
                return nil, err
        }
        ones, bits := ipnet.Mask.Size()
        var toks []string
        for i := ones; i >= 1; i-- {
                m := net.CIDRMask(i, bits)
                tok := net.IPNet{
                        IP:   ipnet.IP.Mask(m),
                        Mask: m,
                }
                toks = append(toks, tok.String())
        }
        return toks, nil
}

and here’s the error when run dgraph alpha with tokeniser

# dgraph alpha  -p /data/dgraph/p -w /data/dgraph/w --whitelist=0.0.0.0/0 --custom_tokenizers=/usr/lib/dgraph/cidr.so
[Sentry] 2021/02/06 14:33:14 Integration installed: ContextifyFrames
[Sentry] 2021/02/06 14:33:14 Integration installed: Environment
[Sentry] 2021/02/06 14:33:14 Integration installed: Modules
[Sentry] 2021/02/06 14:33:14 Integration installed: IgnoreErrors
[Sentry] 2021/02/06 14:33:14 Integration installed: ContextifyFrames
[Sentry] 2021/02/06 14:33:14 Integration installed: Environment
[Sentry] 2021/02/06 14:33:14 Integration installed: Modules
[Sentry] 2021/02/06 14:33:14 Integration installed: IgnoreErrors
I0206 14:33:15.102695   22930 sentry_integration.go:48] This instance of Dgraph will send anonymous reports of panics back to Dgraph Labs via Sentry. No confidential information is sent. These reports help improve Dgraph. To opt-out, restart your instance with the --enable_sentry=false flag. For more info, see https://dgraph.io/docs/howto/#data-handling.
I0206 14:33:15.103031   22930 tok.go:121] Loading custom tokenizer from "/usr/lib/dgraph/cidr.so"
I0206 14:33:15.240291   22930 init.go:107]

Dgraph version   : v20.11.0-gd79d3797b
Dgraph codename  : unnamed-mod
Dgraph SHA-256   : 936d4f9e15171392e4017378abf2cd853896f624ac22a6d18c5e02b91c4a7c8f
Commit SHA-1     : d79d3797b
Commit timestamp : 2021-02-05 07:09:12 -0800
Branch           : master
Go version       : go1.15.6
jemalloc enabled : true

For Dgraph official documentation, visit https://dgraph.io/docs/.
For discussions about Dgraph     , visit https://discuss.dgraph.io.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2020 Dgraph Labs, Inc.


I0206 14:33:15.240305   22930 run.go:722] x.Config: {PortOffset:0 QueryEdgeLimit:1000000 NormalizeNodeLimit:10000 MutationsNQuadLimit:1000000 PollInterval:1s GraphqlExtension:true GraphqlDebug:false GraphqlLambdaUrl:}
I0206 14:33:15.240322   22930 run.go:723] x.WorkerConfig: {TmpDir:t ExportPath:export NumPendingProposals:256 Tracing:0.01 MyAddr: ZeroAddr:[localhost:5080] TLSClientConfig:<nil> TLSServerConfig:<nil> Raft:snapshot-after=10000; idx=0; group=0; learner=false WhiteListedIPRanges:[{Lower:0.0.0.0 Upper:255.255.255.255}] MaxRetries:-1 StrictMutations:false AclEnabled:false HmacSecret:**** AbortOlderThan:5m0s ProposedGroupId:0 StartTime:2021-02-06 14:33:14.835241972 +0800 CST m=+0.138739977 LudicrousMode:false LudicrousConcurrency:2000 EncryptionKey:**** LogRequest:0 HardSync:false Audit:false}
I0206 14:33:15.240393   22930 run.go:724] worker.Config: {PostingDir:/data/dgraph/p PostingDirCompression:1 PostingDirCompressionLevel:0 WALDir:/data/dgraph/w MutationsMode:0 AuthToken: PBlockCacheSize:697932185 PIndexCacheSize:375809638 WalCache:0 HmacSecret:**** AccessJwtTtl:0s RefreshJwtTtl:0s CachePercentage:0,65,35,0 CacheMb:0 Audit:<nil>}
I0206 14:33:15.240560   22930 log.go:295] Found file: 9 First Index: 0
I0206 14:33:15.241381   22930 storage.go:125] Init Raft Storage with snap: 0, first: 1, last: 0
I0206 14:33:15.241393   22930 server_state.go:76] Setting Posting Dir Compression Level: 0
I0206 14:33:15.241397   22930 server_state.go:120] Opening postings BadgerDB with options: {Dir:/data/dgraph/p ValueDir:/data/dgraph/p SyncWrites:false NumVersionsToKeep:2147483647 ReadOnly:false Logger:0x2c93178 Compression:1 InMemory:false MemTableSize:67108864 BaseTableSize:2097152 BaseLevelSize:10485760 LevelSizeMultiplier:10 TableSizeMultiplier:2 MaxLevels:7 ValueThreshold:1024 NumMemtables:5 BlockSize:4096 BloomFalsePositive:0.01 BlockCacheSize:697932185 IndexCacheSize:375809638 NumLevelZeroTables:5 NumLevelZeroTablesStall:15 ValueLogFileSize:1073741823 ValueLogMaxEntries:1000000 NumCompactors:4 CompactL0OnClose:false LmaxCompaction:false ZSTDCompressionLevel:0 VerifyValueChecksum:false EncryptionKey:[] EncryptionKeyRotationDuration:240h0m0s BypassLockGuard:false ChecksumVerificationMode:0 DetectConflicts:false NamespaceOffset:-1 managedTxns:false maxBatchCount:0 maxBatchSize:0}
[Sentry] 2021/02/06 14:33:15 Sending fatal event [b6cdb9d1363343dca2b5db3bc1ad915e] to o318308.ingest.sentry.io project: 5208688
2021/02/06 14:33:15 manifest has unsupported version: 7 (we support 8).
Please see https://github.com/dgraph-io/badger/blob/master/README.md#i-see-manifest-has-unsupported-version-x-we-support-y-error on how to fix this.
Error while creating badger KV posting store

output gives error below, which is error from badger in dgraph process ,so how do fix this?
2021/02/06 14:33:15 manifest has unsupported version: 7 (we support 8).