Bulk load failed

xxm404 · April 27, 2020, 4:40am

When I use bulk load to import data，show me this info and import failed

2020/04/27 04:03:08 loader.go:77: Connecting to zero at localhost:5086
2020/04/27 04:03:08 Missing colon

github.com/dgraph-io/dgraph/x.Wrap
	/ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:76
github.com/dgraph-io/dgraph/x.Check
	/ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:32
github.com/dgraph-io/dgraph/dgraph/cmd/bulk.readSchema
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/loader.go:132
github.com/dgraph-io/dgraph/dgraph/cmd/bulk.newLoader
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/loader.go:91
github.com/dgraph-io/dgraph/dgraph/cmd/bulk.run
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/run.go:148
github.com/dgraph-io/dgraph/dgraph/cmd/bulk.init.0.func1
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/run.go:34
github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra.(*Command).execute
	/ext-go/1/src/github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra/command.go:702
github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra.(*Command).ExecuteC
	/ext-go/1/src/github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra/command.go:783
github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra.(*Command).Execute
	/ext-go/1/src/github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra/command.go:736
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:43
main.main
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/main.go:24
runtime.main
	/usr/local/go/src/runtime/proc.go:198
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:2361

my dgraph version is 1.0.16

ashishgoswami · April 27, 2020, 7:13am

Hey @xxm404, from trace looks like something is wrong with schema file. Can you verify if it is in correct format. Also please try to reproduce it on any of latest releases.

xxm404 · April 27, 2020, 7:44am

Hey @ashishgoswami thanks for your reply. I used this schema file schema, how to verify the schema file.

ashishgoswami · April 27, 2020, 8:01am

Hey @xxm404, this is correct schema file. Can you try reproducing it on any of latest release(v20.03.1)? If it still persists, let the know the steps to reproduce it.

xxm404 · April 27, 2020, 8:23am

It still failed when I imported the same data in v20.03.1

[Decoder]: Using assembly version of decoder
I0427 08:04:38.217033    2863 init.go:99] 

Dgraph version   : v20.03.1
Dgraph SHA-256   : 6a40b1e084205ae9e29336780b3458a3869db45c0b96b916190881c16d705ba8
Commit SHA-1     : c201611d6
Commit timestamp : 2020-04-24 13:53:41 -0700
Branch           : HEAD
Go version       : go1.14.1

For Dgraph official documentation, visit https://docs.dgraph.io.
For discussions about Dgraph     , visit https://discuss.dgraph.io.
To say hi to the community       , visit https://dgraph.slack.com.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2020 Dgraph Labs, Inc.


{
	"DataFiles": "../example/21million.rdf.gz",
	"DataFormat": "",
	"SchemaFile": "../example/21million.schema",
	"OutDir": "./out",
	"ReplaceOutDir": false,
	"TmpDir": "tmp",
	"NumGoroutines": 1,
	"MapBufSize": 67108864,
	"SkipMapPhase": false,
	"CleanupTmp": true,
	"NumReducers": 1,
	"Version": false,
	"StoreXids": false,
	"ZeroAddr": "localhost:5086",
	"HttpAddr": "localhost:8080",
	"IgnoreErrors": false,
	"CustomTokenizers": "",
	"NewUids": false,
	"Encrypted": false,
	"MapShards": 1,
	"ReduceShards": 1,
	"BadgerKeyFile": "",
	"BadgerCompressionLevel": 1
}

The bulk loader needs to open many files at once. This number depends on the size of the data set loaded, the map file output size, and the level of indexing. 100,000 is adequate for most data set sizes. See `man ulimit` for details of how to change the limit.
Current max open files limit: 100000

Connecting to zero at localhost:5086
2020/04/27 08:04:38 while lexing
 at line 7 column 9: Invalid schema: Unexpected character ' ' while parsing IRI
github.com/dgraph-io/dgraph/lex.(*Lexer).ValidateResult
	/ext-go/1/src/github.com/dgraph-io/dgraph/lex/lexer.go:199
github.com/dgraph-io/dgraph/schema.Parse
	/ext-go/1/src/github.com/dgraph-io/dgraph/schema/parse.go:443
github.com/dgraph-io/dgraph/dgraph/cmd/bulk.readSchema
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/loader.go:165
github.com/dgraph-io/dgraph/dgraph/cmd/bulk.newLoader
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/loader.go:119
github.com/dgraph-io/dgraph/dgraph/cmd/bulk.run
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/run.go:234
github.com/dgraph-io/dgraph/dgraph/cmd/bulk.init.0.func1
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/run.go:49
github.com/spf13/cobra.(*Command).execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:70
main.main
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/main.go:78
runtime.main
	/usr/local/go/src/runtime/proc.go:203
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1373

github.com/dgraph-io/dgraph/x.Check
	/ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:42
github.com/dgraph-io/dgraph/dgraph/cmd/bulk.readSchema
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/loader.go:166
github.com/dgraph-io/dgraph/dgraph/cmd/bulk.newLoader
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/loader.go:119
github.com/dgraph-io/dgraph/dgraph/cmd/bulk.run
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/run.go:234
github.com/dgraph-io/dgraph/dgraph/cmd/bulk.init.0.func1
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/run.go:49
github.com/spf13/cobra.(*Command).execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:70
main.main
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/main.go:78
runtime.main
	/usr/local/go/src/runtime/proc.go:203
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1373

reproduce steps

Start dgraph zero ./bin/dgraph zero --config=./conf/zero.yml
Run bulk load ./bin/dgraph bulk -f ../example/21million.rdf.gz -s ../example/21million.schema --reduce_shards=1 --zero=localhost:5086

This my zero.yml content

idx: 1
my: 172.30.19.63:5086
replicas: 1
wal: /home/work/dgraph_zero/data/zero/zw1
port_offset: 6
telemetry: false

xxm404 · April 27, 2020, 9:03am

Hey @ashishgoswami Import success when I use this schema，but when used my scheme it still failed, this is my scheme content

xid:string @index(hash) @upsert .
ww/rel/invite:uid @reverse .
ww/rel/manage:uid @reverse .
ww/attr/tryFlag:int @index(int) .
ww/attr/enterFee:int @index(int) .
ww/attr/closeFlag:int @index(int) .
ww/attr/storeTime:int @index(int) .
ww/attr/createTime:int @index(int) .
ww/attr/manageLevel:int @index(int) .
ww/attr/socialLevel:int @index(int) .

Import error info

Connecting to zero at localhost:5086
2020/04/27 08:41:43 while lexing xid:string @index(hash) @upsert . 
ww/rel/invite:uid @reverse . 
ww/rel/manage:uid @reverse . 
ww/attr/tryFlag:int @index(int) . 
ww/attr/enterFee:int @index(int) . 
ww/attr/closeFlag:int @index(int) . 
ww/attr/storeTime:int @index(int) . 
ww/attr/createTime:int @index(int) . 
ww/attr/manageLevel:int @index(int) . 
ww/attr/socialLevel:int @index(int) . 
 at line 2 column 2: Invalid schema. Unexpected /
github.com/dgraph-io/dgraph/lex.(*Lexer).ValidateResult
	/ext-go/1/src/github.com/dgraph-io/dgraph/lex/lexer.go:199
github.com/dgraph-io/dgraph/schema.Parse
	/ext-go/1/src/github.com/dgraph-io/dgraph/schema/parse.go:443
github.com/dgraph-io/dgraph/dgraph/cmd/bulk.readSchema
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/loader.go:165
github.com/dgraph-io/dgraph/dgraph/cmd/bulk.newLoader
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/loader.go:119
github.com/dgraph-io/dgraph/dgraph/cmd/bulk.run
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/run.go:234
github.com/dgraph-io/dgraph/dgraph/cmd/bulk.init.0.func1
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/run.go:49
github.com/spf13/cobra.(*Command).execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:70
main.main
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/main.go:78
runtime.main
	/usr/local/go/src/runtime/proc.go:203
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1373

github.com/dgraph-io/dgraph/x.Check
	/ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:42
github.com/dgraph-io/dgraph/dgraph/cmd/bulk.readSchema
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/loader.go:166
github.com/dgraph-io/dgraph/dgraph/cmd/bulk.newLoader
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/loader.go:119
github.com/dgraph-io/dgraph/dgraph/cmd/bulk.run
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/run.go:234
github.com/dgraph-io/dgraph/dgraph/cmd/bulk.init.0.func1
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/run.go:49
github.com/spf13/cobra.(*Command).execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:70
main.main
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/main.go:78
runtime.main
	/usr/local/go/src/runtime/proc.go:203
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1373

ashishgoswami · April 28, 2020, 4:16am

Please try enclosing your predicates with <>.

xxm404 · April 28, 2020, 4:19am

Thanks for your reply,

enclosing your predicates with <>.

this method solved my problem