Crash When Using Dgraph Alpha Config in HCL

Issues Dgraph
, ,
1

Report a Dgraph Bug

What version of Dgraph are you using?

  • v20.03.3

Have you tried reproducing the issue with the latest release?

  • false

What is the hardware spec (RAM, OS)?

  • Amazon Linux (default AMI on Amazon EKS)

Steps to reproduce the issue (command/config used to run Dgraph).

  • ran alpha with this configuration:
    "mutations" = "disallow"

"badger" = {
  "tables" = "mmap"
  "vlog" = "mmap"
 }

 "export" = "myexport"

Expected behaviour and actual result.

Expected Behavior

The same configuration works fine when the config is in json or yaml.

Actual Behavior

When the config is in HCL, the result is this:

I0721 05:23:34.622289      19 server_state.go:75] Setting Badger Compression Level: 0
I0721 05:23:34.622348      19 server_state.go:84] Setting Badger table load option: 
2020/07/21 05:23:34 Invalid Badger Tables options
github.com/dgraph-io/dgraph/x.Fatalf
	/ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:114
github.com/dgraph-io/dgraph/worker.setBadgerOptions
	/ext-go/1/src/github.com/dgraph-io/dgraph/worker/server_state.go:93
github.com/dgraph-io/dgraph/worker.(*ServerState).initStorage
	/ext-go/1/src/github.com/dgraph-io/dgraph/worker/server_state.go:126
github.com/dgraph-io/dgraph/worker.InitServerState
	/ext-go/1/src/github.com/dgraph-io/dgraph/worker/server_state.go:57
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:612
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.init.2.func1
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:90
github.com/spf13/cobra.(*Command).execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:70
main.main
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/main.go:78
runtime.main
	/usr/local/go/src/runtime/proc.go:203
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1373
2

I tried similar configurations in TOML, YAML, JSON, and Properties work as expected.

3

Have you tried badger.tables and badger.vlog, instead of the nested map in HCL?

4

That seems as a workaround to accept the configuration:

"mutations" = "disallow"

"badger.tables" = "ram"
"badger.vlog" =  "disk"

"export" = "myexport"

Which resulted in a successful configuration:

I0721 21:27:54.239740      21 server_state.go:75] Setting Badger Compression Level: 3
I0721 21:27:54.239794      21 server_state.go:84] Setting Badger table load option: ram
I0721 21:27:54.239832      21 server_state.go:96] Setting Badger value log load option: disk
I0721 21:27:54.239856      21 server_state.go:141] Opening write-ahead log BadgerDB with options: {Dir:w ValueDir:w SyncWrites:false TableLoadingMode:1 ValueLogLoadingMode:0 NumVersionsToKeep:1 ReadOnly:false Truncate:true Logger:0x28325d0 Compression:2 InMemory:false MaxT
ableSize:67108864 LevelSizeMultiplier:10 MaxLevels:7 ValueThreshold:1048576 NumMemtables:5 BlockSize:4096 BloomFalsePositive:0.01 KeepL0InMemory:true MaxCacheSize:10485760 MaxBfCacheSize:0 LoadBloomsOnOpen:false NumLevelZeroTables:5 NumLevelZeroTablesStall:10 LevelOneSize:
268435456 ValueLogFileSize:1073741823 ValueLogMaxEntries:10000 NumCompactors:2 CompactL0OnClose:true LogRotatesToFlush:2 ZSTDCompressionLevel:3 VerifyValueChecksum:false EncryptionKey:[] EncryptionKeyRotationDuration:240h0m0s BypassLockGuard:false ChecksumVerificationMode:
0 KeepBlockIndicesInCache:false KeepBlocksInCache:false managedTxns:false maxBatchCount:0 maxBatchSize:0}

(NOTE: edited as had invalid value pass onto badger that caused stack trace)

Despite this, we should support valid HCL as this will confuse customers. Hash/Map/Object are supported with other formats:

TOML

mutations = 'disallow'
export = 'myexport'

[badger]
tables = 'mmap'
vlog = 'disk'

JSON

{
  "mutations": "disallow",
  "badger": {
    "tables": "mmap",
    "vlog": "disk"
  },
  "export": "myexport"
}

YAML

mutations: disallow
badger:
  tables: mmap
  vlog: disk
export: myexport
5

Are these actually setting the config values? The config flag is badger.tables and badger.vlog. The dgraph command doesn’t try to do anything smart to namespace the flags via the “.” separator.

6

That is correct, those are the actual settings in config values. This is supported in TOML, JSON, YAML, but not HCL.

1 Like
7

The solutions in this are a workaround, but the original issue is that HCLs do not support hierarchical configurations and this has never been resolved.

With v20.11.1, the configuration would be:

{
  "mutations": "disallow",
  "badger": {
    "compression_level": "snappy"
  },
  "export": "myexport"
}

This would correspond to

dgraph alpha --export myexport --mutations disallow \ 
  --badger.compresion_level snappy