Panic on update schema

Report a GraphQL Bug

DGraph is panicking when trying to POST to /admin/schema

What edition and version of Dgraph are you using?

Edition:

  • SlashGraphQL
  • Dgraph (community edition)

If you are using the community edition or enterprise edition of Dgraph, please list the version:

Tested on community editions from dockerhub:

  • v20.11.0
  • v20.11.2
  • master (the dgraph version command below has been run on master)

Using my schema on v20.07.3 works fine, but upgrading to either of the above versions results with a panic and the following logs:

Dgraph Version
$ dgraph version
 

Dgraph version   : v20.11.0-rc1-443-ga39b57a9
Dgraph codename  : unnamed
Dgraph SHA-256   : 4798245358c5d048fb7909cfbef612da3ed800ab1956809c4f718643b36865f4
Commit SHA-1     : a39b57a9
Commit timestamp : 2021-03-11 11:09:05 +0000
Branch           : master
Go version       : go1.15.5
jemalloc enabled : true

For Dgraph official documentation, visit https://dgraph.io/docs.
For discussions about Dgraph     , visit https://discuss.dgraph.io.
For fully-managed Dgraph Cloud   , visit https://dgraph.io/cloud.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2021 Dgraph Labs, Inc.

Have you tried reproducing the issue with the latest release?

Yes

Steps to reproduce the issue (paste the query/schema if possible)

I can’t really paste my schema, however I’ve tested a schema which was in use 3 months ago (we’ve done a lot of development since) and it also breaks.

Using the schema from the getting started guide seems to work.

Expected behaviour and actual result.

I expect the schema to successfully update.

Here’s the panic log:

Panic log
I0312 04:14:35.571618      33 middlewares.go:178] GraphQL admin mutation. Name =  updateGQLSchema
I0312 04:14:35.571693      33 schema.go:51] Got updateGQLSchema request
2021/03/12 04:14:35 http: panic serving 172.17.0.1:62206: runtime error: invalid memory address or nil pointer dereference
goroutine 706 [running]:
net/http.(*conn).serve.func1(0xc0005619a0)
        /usr/local/go/src/net/http/server.go:1801 +0x147
panic(0x1bc7700, 0x2994600)
        /usr/local/go/src/runtime/panic.go:975 +0x47a
github.com/dgraph-io/dgraph/graphql/schema.Stringify(0xc001552380, 0xc001086c00, 0x4e, 0x5b, 0x0, 0x0)
        /ext-go/1/src/github.com/dgraph-io/dgraph/graphql/schema/gqlschema.go:2254 +0x1f2d
github.com/dgraph-io/dgraph/graphql/schema.(*handler).GQLSchema(0xc00155f6c0, 0x6c43, 0xc0013b6801)
        /ext-go/1/src/github.com/dgraph-io/dgraph/graphql/schema/schemagen.go:68 +0x4a
github.com/dgraph-io/dgraph/graphql/admin.(*updateSchemaResolver).Resolve(0xc000118268, 0x1f96a40, 0xc001494270, 0x1fb1aa0, 0xc001494bd0, 0x1, 0x1)
        /ext-go/1/src/github.com/dgraph-io/dgraph/graphql/admin/schema.go:65 +0x123
github.com/dgraph-io/dgraph/graphql/resolve.LoggingMWMutation.func1(0x1f96a40, 0xc001494270, 0x1fb1aa0, 0xc001494bd0, 0x0, 0x7fcc4734aa78)
        /ext-go/1/src/github.com/dgraph-io/dgraph/graphql/resolve/middlewares.go:179 +0x10b
github.com/dgraph-io/dgraph/graphql/resolve.MutationResolverFunc.Resolve(0xc000493fa0, 0x1f96a40, 0xc001494270, 0x1fb1aa0, 0xc001494bd0, 0x1d56900, 0xc0004935a0)
        /ext-go/1/src/github.com/dgraph-io/dgraph/graphql/resolve/mutation.go:153 +0x4e
github.com/dgraph-io/dgraph/graphql/resolve.GuardianAuthMW4Mutation.func1(0x1f96a40, 0xc001494270, 0x1fb1aa0, 0xc001494bd0, 0x0, 0x0)
        /ext-go/1/src/github.com/dgraph-io/dgraph/graphql/resolve/middlewares.go:161 +0xe2
github.com/dgraph-io/dgraph/graphql/resolve.MutationResolverFunc.Resolve(0xc000493fc0, 0x1f96a40, 0xc001494270, 0x1fb1aa0, 0xc001494bd0, 0xc0004737b8, 0x182c4d1)
        /ext-go/1/src/github.com/dgraph-io/dgraph/graphql/resolve/mutation.go:153 +0x4e
github.com/dgraph-io/dgraph/graphql/resolve.IpWhitelistingMW4Mutation.func1(0x1f96a40, 0xc001494270, 0x1fb1aa0, 0xc001494bd0, 0x3, 0x2997170)
        /ext-go/1/src/github.com/dgraph-io/dgraph/graphql/resolve/middlewares.go:171 +0xe2
github.com/dgraph-io/dgraph/graphql/resolve.MutationResolverFunc.Resolve(0xc001342ae0, 0x1f96a40, 0xc001494270, 0x1fb1aa0, 0xc001494bd0, 0x0, 0x1f96a40)
        /ext-go/1/src/github.com/dgraph-io/dgraph/graphql/resolve/mutation.go:153 +0x4e
github.com/dgraph-io/dgraph/graphql/resolve.(*RequestResolver).Resolve(0xc000350ea0, 0x1f96a40, 0xc001494270, 0xc0013b6200, 0x0)
        /ext-go/1/src/github.com/dgraph-io/dgraph/graphql/resolve/resolver.go:529 +0x5da
github.com/dgraph-io/dgraph/graphql/web.(*graphqlHandler).Resolve(0xc000350ee0, 0x1f96a40, 0xc001494240, 0xc0013b6200, 0xc001494240)
        /ext-go/1/src/github.com/dgraph-io/dgraph/graphql/web/http.go:91 +0x4c
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.resolveWithAdminServer(0xc0013b6200, 0xc00649a600, 0x1f8b600, 0xc000350ee0, 0xc000ecfb88)
        /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/http.go:647 +0x117
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.adminSchemaHandler(0x1f90200, 0xc0004f42a0, 0xc00649a600, 0x1f8b600, 0xc000350ee0)
        /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/http.go:630 +0x1a5
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.setupServer.func2(0x1f90200, 0xc0004f42a0, 0xc00649a600)
        /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:474 +0x57
net/http.HandlerFunc.ServeHTTP(0x1e08478, 0x1f90200, 0xc0004f42a0, 0xc00649a600)
        /usr/local/go/src/net/http/server.go:2042 +0x44
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.adminAuthHandler.func1(0x1f90200, 0xc0004f42a0, 0xc00649a600)
        /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/admin.go:71 +0x7e
net/http.HandlerFunc.ServeHTTP(0xc000350fa0, 0x1f90200, 0xc0004f42a0, 0xc00649a600)
        /usr/local/go/src/net/http/server.go:2042 +0x44
net/http.(*ServeMux).ServeHTTP(0x2bb8dc0, 0x1f90200, 0xc0004f42a0, 0xc00649a600)
        /usr/local/go/src/net/http/server.go:2417 +0x1ad
net/http.serverHandler.ServeHTTP(0xc00019c0e0, 0x1f90200, 0xc0004f42a0, 0xc00649a600)
        /usr/local/go/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc0005619a0, 0x1f96980, 0xc0013b6080)
        /usr/local/go/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2969 +0x36c

Being a memory problem. What is your hardware setup?

This might help, try to split up your GraphQL schema and add it in pieces. If you build it up piece at a time then there is less change for Dgraph to process all at once.

I first noticed this in a GKE Kubernetes cluster running on COS distro, and then reproduced locally using docker on MacOS, any ideas why this might be happening across multiple distros?

The schema is 27KB, so it’s big for a schema, but not in terms of memory requirements.

I can’t really paste my schema, however I’ve tested a schema which was in use 3 months ago (we’ve done a lot of development since) and it also breaks.

I did the above to check a smaller schema but it turns out that file was actually 28K too, so I’ve not done anything to test a smaller schema just yet. Will try and find time to test it over the weekend but I may not have time.

Hi @dan-j ,

I do understand that you cannot paste the schema over here. Can you share the schema over email ?
I can try reproducing this to find out the root cause.

My Email ID: rajas [at] dgraph [dot] io

Hi @rajas,

I shared a copy with @JatinDevDG here: https://discuss.dgraph.io/t/re-panic-on-update-schema/13175, I don’t have the modified copy to hand so hopefully you can see that thread?

2 Likes

Hi @dan-j , i am able to reproduce this and accepting this as a bug. We will fix it soon. Thanks for reporting it.

Thanks for sharing the schema @dan-j ,

I was able to recreate this on master .

In October 2020, we had merged the PR, Feat(GraphQL): This PR add enable schema cleaning in GraphQL and reduce schema update time. by JatinDevDG · Pull Request #6666 · dgraph-io/dgraph · GitHub which removes some input types which are not used.

The current algorithm removes any input type of the form UpdateTInput (where T could be any non-empty string) containing exactly one field.

The schema provided by you contains a user defined input type satisfying these constraints which gets removed by the algorithm. I was able to update schema successfully by renaming the type of the form, UpdateTInput to something else.

We are working on modifying the algorithm and adding extra checks so that this does not happen. This will be fixed soon.

As a temporary work around, can you rename the type of the form, UpdateTInput to something else ?

1 Like

This has been fixed and merged to release/v20.11 and release/v21.03 . The fix will be part of next 20.11 release.
Related PR: Fix(GraphQL): Add extra checks for deleting UpdateTypeInput (20.11 cherry-pick) by vmrajas · Pull Request #7600 · dgraph-io/dgraph · GitHub

1 Like