Panic on update schema

dan-j · March 12, 2021, 4:31am

Report a GraphQL Bug

DGraph is panicking when trying to POST to /admin/schema

What edition and version of Dgraph are you using?

Edition:

SlashGraphQL
Dgraph (community edition)

If you are using the community edition or enterprise edition of Dgraph, please list the version:

Tested on community editions from dockerhub:

v20.11.0
v20.11.2
master (the dgraph version command below has been run on master)

Using my schema on v20.07.3 works fine, but upgrading to either of the above versions results with a panic and the following logs:

Dgraph Version

$ dgraph version
 

Dgraph version   : v20.11.0-rc1-443-ga39b57a9
Dgraph codename  : unnamed
Dgraph SHA-256   : 4798245358c5d048fb7909cfbef612da3ed800ab1956809c4f718643b36865f4
Commit SHA-1     : a39b57a9
Commit timestamp : 2021-03-11 11:09:05 +0000
Branch           : master
Go version       : go1.15.5
jemalloc enabled : true

For Dgraph official documentation, visit https://dgraph.io/docs.
For discussions about Dgraph     , visit http://discuss.dgraph.io.
For fully-managed Dgraph Cloud   , visit https://dgraph.io/cloud.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2021 Dgraph Labs, Inc.

Have you tried reproducing the issue with the latest release?

Yes

Steps to reproduce the issue (paste the query/schema if possible)

I can’t really paste my schema, however I’ve tested a schema which was in use 3 months ago (we’ve done a lot of development since) and it also breaks.

Using the schema from the getting started guide seems to work.

Expected behaviour and actual result.

I expect the schema to successfully update.

Here’s the panic log:

Panic log

I0312 04:14:35.571618      33 middlewares.go:178] GraphQL admin mutation. Name =  updateGQLSchema
I0312 04:14:35.571693      33 schema.go:51] Got updateGQLSchema request
2021/03/12 04:14:35 http: panic serving 172.17.0.1:62206: runtime error: invalid memory address or nil pointer dereference
goroutine 706 [running]:
net/http.(*conn).serve.func1(0xc0005619a0)
        /usr/local/go/src/net/http/server.go:1801 +0x147
panic(0x1bc7700, 0x2994600)
        /usr/local/go/src/runtime/panic.go:975 +0x47a
github.com/dgraph-io/dgraph/graphql/schema.Stringify(0xc001552380, 0xc001086c00, 0x4e, 0x5b, 0x0, 0x0)
        /ext-go/1/src/github.com/dgraph-io/dgraph/graphql/schema/gqlschema.go:2254 +0x1f2d
github.com/dgraph-io/dgraph/graphql/schema.(*handler).GQLSchema(0xc00155f6c0, 0x6c43, 0xc0013b6801)
        /ext-go/1/src/github.com/dgraph-io/dgraph/graphql/schema/schemagen.go:68 +0x4a
github.com/dgraph-io/dgraph/graphql/admin.(*updateSchemaResolver).Resolve(0xc000118268, 0x1f96a40, 0xc001494270, 0x1fb1aa0, 0xc001494bd0, 0x1, 0x1)
        /ext-go/1/src/github.com/dgraph-io/dgraph/graphql/admin/schema.go:65 +0x123
github.com/dgraph-io/dgraph/graphql/resolve.LoggingMWMutation.func1(0x1f96a40, 0xc001494270, 0x1fb1aa0, 0xc001494bd0, 0x0, 0x7fcc4734aa78)
        /ext-go/1/src/github.com/dgraph-io/dgraph/graphql/resolve/middlewares.go:179 +0x10b
github.com/dgraph-io/dgraph/graphql/resolve.MutationResolverFunc.Resolve(0xc000493fa0, 0x1f96a40, 0xc001494270, 0x1fb1aa0, 0xc001494bd0, 0x1d56900, 0xc0004935a0)
        /ext-go/1/src/github.com/dgraph-io/dgraph/graphql/resolve/mutation.go:153 +0x4e
github.com/dgraph-io/dgraph/graphql/resolve.GuardianAuthMW4Mutation.func1(0x1f96a40, 0xc001494270, 0x1fb1aa0, 0xc001494bd0, 0x0, 0x0)
        /ext-go/1/src/github.com/dgraph-io/dgraph/graphql/resolve/middlewares.go:161 +0xe2
github.com/dgraph-io/dgraph/graphql/resolve.MutationResolverFunc.Resolve(0xc000493fc0, 0x1f96a40, 0xc001494270, 0x1fb1aa0, 0xc001494bd0, 0xc0004737b8, 0x182c4d1)
        /ext-go/1/src/github.com/dgraph-io/dgraph/graphql/resolve/mutation.go:153 +0x4e
github.com/dgraph-io/dgraph/graphql/resolve.IpWhitelistingMW4Mutation.func1(0x1f96a40, 0xc001494270, 0x1fb1aa0, 0xc001494bd0, 0x3, 0x2997170)
        /ext-go/1/src/github.com/dgraph-io/dgraph/graphql/resolve/middlewares.go:171 +0xe2
github.com/dgraph-io/dgraph/graphql/resolve.MutationResolverFunc.Resolve(0xc001342ae0, 0x1f96a40, 0xc001494270, 0x1fb1aa0, 0xc001494bd0, 0x0, 0x1f96a40)
        /ext-go/1/src/github.com/dgraph-io/dgraph/graphql/resolve/mutation.go:153 +0x4e
github.com/dgraph-io/dgraph/graphql/resolve.(*RequestResolver).Resolve(0xc000350ea0, 0x1f96a40, 0xc001494270, 0xc0013b6200, 0x0)
        /ext-go/1/src/github.com/dgraph-io/dgraph/graphql/resolve/resolver.go:529 +0x5da
github.com/dgraph-io/dgraph/graphql/web.(*graphqlHandler).Resolve(0xc000350ee0, 0x1f96a40, 0xc001494240, 0xc0013b6200, 0xc001494240)
        /ext-go/1/src/github.com/dgraph-io/dgraph/graphql/web/http.go:91 +0x4c
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.resolveWithAdminServer(0xc0013b6200, 0xc00649a600, 0x1f8b600, 0xc000350ee0, 0xc000ecfb88)
        /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/http.go:647 +0x117
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.adminSchemaHandler(0x1f90200, 0xc0004f42a0, 0xc00649a600, 0x1f8b600, 0xc000350ee0)
        /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/http.go:630 +0x1a5
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.setupServer.func2(0x1f90200, 0xc0004f42a0, 0xc00649a600)
        /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:474 +0x57
net/http.HandlerFunc.ServeHTTP(0x1e08478, 0x1f90200, 0xc0004f42a0, 0xc00649a600)
        /usr/local/go/src/net/http/server.go:2042 +0x44
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.adminAuthHandler.func1(0x1f90200, 0xc0004f42a0, 0xc00649a600)
        /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/admin.go:71 +0x7e
net/http.HandlerFunc.ServeHTTP(0xc000350fa0, 0x1f90200, 0xc0004f42a0, 0xc00649a600)
        /usr/local/go/src/net/http/server.go:2042 +0x44
net/http.(*ServeMux).ServeHTTP(0x2bb8dc0, 0x1f90200, 0xc0004f42a0, 0xc00649a600)
        /usr/local/go/src/net/http/server.go:2417 +0x1ad
net/http.serverHandler.ServeHTTP(0xc00019c0e0, 0x1f90200, 0xc0004f42a0, 0xc00649a600)
        /usr/local/go/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc0005619a0, 0x1f96980, 0xc0013b6080)
        /usr/local/go/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2969 +0x36c

verneleem · March 12, 2021, 4:51am

Being a memory problem. What is your hardware setup?

This might help, try to split up your GraphQL schema and add it in pieces. If you build it up piece at a time then there is less change for Dgraph to process all at once.

dan-j · March 12, 2021, 2:36pm

I first noticed this in a GKE Kubernetes cluster running on COS distro, and then reproduced locally using docker on MacOS, any ideas why this might be happening across multiple distros?

dan-j · March 12, 2021, 2:55pm

The schema is 27KB, so it’s big for a schema, but not in terms of memory requirements.

I can’t really paste my schema, however I’ve tested a schema which was in use 3 months ago (we’ve done a lot of development since) and it also breaks.

I did the above to check a smaller schema but it turns out that file was actually 28K too, so I’ve not done anything to test a smaller schema just yet. Will try and find time to test it over the weekend but I may not have time.

rajas · March 12, 2021, 3:23pm

Hi @dan-j ,

I do understand that you cannot paste the schema over here. Can you share the schema over email ?
I can try reproducing this to find out the root cause.

My Email ID: rajas [at] dgraph [dot] io

dan-j · March 14, 2021, 3:09pm

Hi @rajas,

I shared a copy with @JatinDevDG here: http://discuss.dgraph.io/t/re-panic-on-update-schema/13175, I don’t have the modified copy to hand so hopefully you can see that thread?

JatinDevDG · March 15, 2021, 7:09am

Hi @dan-j , i am able to reproduce this and accepting this as a bug. We will fix it soon. Thanks for reporting it.

rajas · March 15, 2021, 4:47pm

Thanks for sharing the schema @dan-j ,

I was able to recreate this on master .

In October 2020, we had merged the PR, Feat(GraphQL): This PR add enable schema cleaning in GraphQL and reduce schema update time. by JatinDevDG · Pull Request #6666 · dgraph-io/dgraph · GitHub which removes some input types which are not used.

The current algorithm removes any input type of the form UpdateTInput (where T could be any non-empty string) containing exactly one field.

The schema provided by you contains a user defined input type satisfying these constraints which gets removed by the algorithm. I was able to update schema successfully by renaming the type of the form, UpdateTInput to something else.

We are working on modifying the algorithm and adding extra checks so that this does not happen. This will be fixed soon.

As a temporary work around, can you rename the type of the form, UpdateTInput to something else ?

rajas · March 18, 2021, 11:01am

This has been fixed and merged to release/v20.11 and release/v21.03 . The fix will be part of next 20.11 release.
Related PR: Fix(GraphQL): Add extra checks for deleting UpdateTypeInput (20.11 cherry-pick) by vmrajas · Pull Request #7600 · dgraph-io/dgraph · GitHub

Topic		Replies	Views
Dgraph start to panic out of nowhere on Darwin Dgraph graphql , dgraph , kind:bug	7	907	December 31, 2020
Can't deploy schema Dgraph kind:bug	0	309	May 27, 2021
Applying GraphQL schema times out GraphQL schema , status:accepted , kind:bug , ticket:created	6	663	February 15, 2021
Update mutation removing connection between nodes in one field and setting the connection on a different field fails GraphQL kind:bug	9	835	November 30, 2022
Tour of go panics on 1.2.1 Issues untagged , tutorial	3	550	July 11, 2020