Zero is restarting again and again

Users
1

Hello, I deploy Dgraph(zion) in my docker, work well in pass 15 days. But I start it today, the zero is restarting again and again, I delete the container and run docker compose up -d again, the same result.

image

the log of ZERO as following:

Dgraph version   : v21.12.0
Dgraph codename  : zion
Dgraph SHA-256   : 078c75df9fa1057447c8c8afc10ea57cb0a29dfb22f9e61d8c334882b4b4eb37
Commit SHA-1     : d62ed5f15
Commit timestamp : 2021-12-02 21:20:09 +0530
Branch           : HEAD
Go version       : go1.17.3
jemalloc enabled : true

For Dgraph official documentation, visit https://dgraph.io/docs.
For discussions about Dgraph     , visit http://discuss.dgraph.io.
For fully-managed Dgraph Cloud   , visit https://dgraph.io/cloud.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2021 Dgraph Labs, Inc.


I0118 01:08:17.805547      16 run.go:254] Setting Config to: {raft:0xc000012410 telemetry:0xc0000123e0 limit:0xc000012418 bindall:true portOffset:0 numReplicas:1 peer: w:zw rebalanceInterval:480000000000 tlsClientConfig:<nil> audit:<nil> limiterConfig:0xc00041edf0}
I0118 01:08:17.805581      16 run.go:144] Setting up grpc listener at: 0.0.0.0:5080
I0118 01:08:17.831747      16 run.go:144] Setting up http listener at: 0.0.0.0:6080
I0118 01:08:17.873362      16 log.go:295] Found file: 1 First Index: 1
2022/01/18 01:08:17 snap index: 1171 + 1 should be equal to first: 1147

github.com/dgraph-io/dgraph/x.AssertTruef
	/ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:107
github.com/dgraph-io/dgraph/raftwal.InitEncrypted
	/ext-go/1/src/github.com/dgraph-io/dgraph/raftwal/storage.go:116
github.com/dgraph-io/dgraph/raftwal.Init
	/ext-go/1/src/github.com/dgraph-io/dgraph/raftwal/storage.go:83
github.com/dgraph-io/dgraph/dgraph/cmd/zero.run
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/zero/run.go:310
github.com/dgraph-io/dgraph/dgraph/cmd/zero.init.0.func1
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/zero/run.go:83
github.com/spf13/cobra.(*Command).execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:81
main.main
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/main.go:112
runtime.main
	/usr/local/go/src/runtime/proc.go:255
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1581
I0118 01:09:11.378995      16 sentry_integration.go:48] This instance of Dgraph will send anonymous reports of panics back to Dgraph Labs via Sentry. No confidential information is sent. These reports help improve Dgraph. To opt-out, restart your instance with the --telemetry "sentry=false;" flag. For more info, see https://dgraph.io/docs/howto/#data-handling.
I0118 01:09:11.517546      16 init.go:110] 

Dgraph version   : v21.12.0
Dgraph codename  : zion
Dgraph SHA-256   : 078c75df9fa1057447c8c8afc10ea57cb0a29dfb22f9e61d8c334882b4b4eb37
Commit SHA-1     : d62ed5f15
Commit timestamp : 2021-12-02 21:20:09 +0530
Branch           : HEAD
Go version       : go1.17.3
jemalloc enabled : true

For Dgraph official documentation, visit https://dgraph.io/docs.
For discussions about Dgraph     , visit http://discuss.dgraph.io.
For fully-managed Dgraph Cloud   , visit https://dgraph.io/cloud.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2021 Dgraph Labs, Inc.


I0118 01:09:11.517623      16 run.go:254] Setting Config to: {raft:0xc00013a8d8 telemetry:0xc00013a8a8 limit:0xc00013a8e0 bindall:true portOffset:0 numReplicas:1 peer: w:zw rebalanceInterval:480000000000 tlsClientConfig:<nil> audit:<nil> limiterConfig:0xc000370b50}
I0118 01:09:11.517697      16 run.go:144] Setting up grpc listener at: 0.0.0.0:5080
I0118 01:09:11.517867      16 run.go:144] Setting up http listener at: 0.0.0.0:6080
I0118 01:09:11.541608      16 log.go:295] Found file: 1 First Index: 1
2022/01/18 01:09:11 snap index: 1171 + 1 should be equal to first: 1147

github.com/dgraph-io/dgraph/x.AssertTruef
	/ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:107
github.com/dgraph-io/dgraph/raftwal.InitEncrypted
	/ext-go/1/src/github.com/dgraph-io/dgraph/raftwal/storage.go:116
github.com/dgraph-io/dgraph/raftwal.Init
	/ext-go/1/src/github.com/dgraph-io/dgraph/raftwal/storage.go:83
github.com/dgraph-io/dgraph/dgraph/cmd/zero.run
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/zero/run.go:310
github.com/dgraph-io/dgraph/dgraph/cmd/zero.init.0.func1
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/zero/run.go:83
github.com/spf13/cobra.(*Command).execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:81
main.main
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/main.go:112
runtime.main
	/usr/local/go/src/runtime/proc.go:255
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1581
[Sentry] 2022/01/18 01:07:04 Integration installed: ContextifyFrames
[Sentry] 2022/01/18 01:07:04 Integration installed: Environment
[Sentry] 2022/01/18 01:07:04 Integration installed: Modules
[Sentry] 2022/01/18 01:07:04 Integration installed: IgnoreErrors
[Sentry] 2022/01/18 01:07:04 Integration installed: ContextifyFrames
[Sentry] 2022/01/18 01:07:04 Integration installed: Environment
[Sentry] 2022/01/18 01:07:04 Integration installed: Modules
[Sentry] 2022/01/18 01:07:04 Integration installed: IgnoreErrors
[Sentry] 2022/01/18 01:07:07 Integration installed: ContextifyFrames
[Sentry] 2022/01/18 01:07:07 Integration installed: Environment
[Sentry] 2022/01/18 01:07:07 Integration installed: Modules
[Sentry] 2022/01/18 01:07:07 Integration installed: IgnoreErrors
[Sentry] 2022/01/18 01:07:07 Integration installed: ContextifyFrames
[Sentry] 2022/01/18 01:07:07 Integration installed: Environment
[Sentry] 2022/01/18 01:07:07 Integration installed: Modules
[Sentry] 2022/01/18 01:07:07 Integration installed: IgnoreErrors
[Sentry] 2022/01/18 01:07:10 Integration installed: ContextifyFrames
[Sentry] 2022/01/18 01:07:10 Integration installed: Environment
[Sentry] 2022/01/18 01:07:10 Integration installed: Modules
[Sentry] 2022/01/18 01:07:10 Integration installed: IgnoreErrors
[Sentry] 2022/01/18 01:07:11 Integration installed: ContextifyFrames
[Sentry] 2022/01/18 01:07:11 Integration installed: Environment
[Sentry] 2022/01/18 01:07:11 Integration installed: Modules
[Sentry] 2022/01/18 01:07:11 Integration installed: IgnoreErrors
[Sentry] 2022/01/18 01:07:13 Integration installed: ContextifyFrames
[Sentry] 2022/01/18 01:07:13 Integration installed: Environment
[Sentry] 2022/01/18 01:07:13 Integration installed: Modules
[Sentry] 2022/01/18 01:07:13 Integration installed: IgnoreErrors
[Sentry] 2022/01/18 01:07:14 Integration installed: ContextifyFrames
[Sentry] 2022/01/18 01:07:14 Integration installed: Environment
[Sentry] 2022/01/18 01:07:14 Integration installed: Modules
[Sentry] 2022/01/18 01:07:14 Integration installed: IgnoreErrors
[Sentry] 2022/01/18 01:07:17 Integration installed: ContextifyFrames
[Sentry] 2022/01/18 01:07:17 Integration installed: Environment
[Sentry] 2022/01/18 01:07:17 Integration installed: Modules
[Sentry] 2022/01/18 01:07:17 Integration installed: IgnoreErrors
[Sentry] 2022/01/18 01:07:18 Integration installed: ContextifyFrames
[Sentry] 2022/01/18 01:07:18 Integration installed: Environment
[Sentry] 2022/01/18 01:07:18 Integration installed: Modules
[Sentry] 2022/01/18 01:07:18 Integration installed: IgnoreErrors
[Sentry] 2022/01/18 01:07:23 Integration installed: ContextifyFrames
[Sentry] 2022/01/18 01:07:23 Integration installed: Environment
[Sentry] 2022/01/18 01:07:23 Integration installed: Modules
[Sentry] 2022/01/18 01:07:23 Integration installed: IgnoreErrors
[Sentry] 2022/01/18 01:07:24 Integration installed: ContextifyFrames
[Sentry] 2022/01/18 01:07:24 Integration installed: Environment
[Sentry] 2022/01/18 01:07:24 Integration installed: Modules
[Sentry] 2022/01/18 01:07:24 Integration installed: IgnoreErrors
[Sentry] 2022/01/18 01:07:32 Integration installed: ContextifyFrames
[Sentry] 2022/01/18 01:07:32 Integration installed: Environment
[Sentry] 2022/01/18 01:07:32 Integration installed: Modules
[Sentry] 2022/01/18 01:07:32 Integration installed: IgnoreErrors
[Sentry] 2022/01/18 01:07:33 Integration installed: ContextifyFrames
[Sentry] 2022/01/18 01:07:33 Integration installed: Environment
[Sentry] 2022/01/18 01:07:33 Integration installed: Modules
[Sentry] 2022/01/18 01:07:33 Integration installed: IgnoreErrors
[Sentry] 2022/01/18 01:07:47 Integration installed: ContextifyFrames
[Sentry] 2022/01/18 01:07:47 Integration installed: Environment
[Sentry] 2022/01/18 01:07:47 Integration installed: Modules
[Sentry] 2022/01/18 01:07:47 Integration installed: IgnoreErrors
[Sentry] 2022/01/18 01:07:48 Integration installed: ContextifyFrames
[Sentry] 2022/01/18 01:07:48 Integration installed: Environment
[Sentry] 2022/01/18 01:07:48 Integration installed: Modules
[Sentry] 2022/01/18 01:07:48 Integration installed: IgnoreErrors
[Sentry] 2022/01/18 01:08:15 Integration installed: ContextifyFrames
[Sentry] 2022/01/18 01:08:15 Integration installed: Environment
[Sentry] 2022/01/18 01:08:15 Integration installed: Modules
[Sentry] 2022/01/18 01:08:15 Integration installed: IgnoreErrors
[Sentry] 2022/01/18 01:08:16 Integration installed: ContextifyFrames
[Sentry] 2022/01/18 01:08:16 Integration installed: Environment
[Sentry] 2022/01/18 01:08:16 Integration installed: Modules
[Sentry] 2022/01/18 01:08:16 Integration installed: IgnoreErrors
[Sentry] 2022/01/18 01:09:10 Integration installed: ContextifyFrames
[Sentry] 2022/01/18 01:09:10 Integration installed: Environment
[Sentry] 2022/01/18 01:09:10 Integration installed: Modules
[Sentry] 2022/01/18 01:09:10 Integration installed: IgnoreErrors
[Sentry] 2022/01/18 01:09:11 Integration installed: ContextifyFrames
[Sentry] 2022/01/18 01:09:11 Integration installed: Environment
[Sentry] 2022/01/18 01:09:11 Integration installed: Modules
[Sentry] 2022/01/18 01:09:11 Integration installed: IgnoreErrors

Blockquote

here is my "docker-compose.yaml"
version: "3.7"

services:

  app:

    build:

      context: .

      dockerfile: app-dockerfile

    command: sh -c "npm install --production && node src/index.js"

    ports:

      - 3000:3000

    restart: unless-stopped

    working_dir: /app

    volumes:

      - ./:/app

    # environment:

    #   MYSQL_HOST: mysql

    #   MYSQL_USER: root

    #   MYSQL_PASSWORD: secret

    #   MYSQL_DB: todos

  zero:

    image: dgraph/dgraph:latest

    volumes:

      - ~/dgraph-auth-new-schema/dgraph:/dgraph

    ports:

      - 5080:5080

      - 6080:6080

    restart: unless-stopped

    command: dgraph zero --my=zero:5080

   

  alpha:

    image: dgraph/dgraph:latest

    volumes:

      - ~/dgraph-auth-new-schema/tmp/data:/dgraph

    ports:

      - 8080:8080

      - 9080:9080

    restart: unless-stopped

    command: dgraph alpha --my=alpha:7080 --cache size-mb=8096 --zero=zero:5080 --security whitelist=0.0.0.0/0

  ratel:

    image: dgraph/ratel:latest

    # volumes:

    #   - type: volume

    #     source: dgraph

    #     target: /dgraph

    #     volume:

    #       nocopy: true

    ports:

      - 8000:8000

    restart: unless-stopped

    command: dgraph-ratel

# volumes:

#   dgraph-auth:

and app-dockfile

# syntax=docker/dockerfile:1

FROM node:12-alpine

RUN apk add --no-cache python3 g++ make

RUN apk update && apk add curl

# RUN curl -X POST alpha:8080/admin/schema --data-binary '@schema.graphql'

# WORKDIR /app

# COPY . .

# RUN npm install --production

#CMD ["node", "src/app.js"]

any help?

2

Looks like to me the disk was probably not synced or the program was killed unexpectedly and the files written do not match the expected state.

It is worth noting there has not been a change to this part of the code between v21.03 and v21.12 (other than a variable rename).

I would suggest running multiple zeros in prod but I assume this is not prod. Periodic backups/exports can protect you as well if you are not worried about a bit of downtime.

Not to minimize the issue or say there is nothing that could be done to protect against this happening in code - but just to say, disk operations can sometimes mess up in real life so protect yourself accordingly

3

Thanks, I delete dgraph files, and it work again, testing data lost.