Trying to do some “large” bulk batch delete operations and getting:
failed because Dgraph execution failed because NQuad count in the request: 1002000, is more that threshold: 1000000
How can I fix this?
Edit
For clarity that this is about GraphQL, here the mutation I used:
mutation {
deleteFoo(filter: {has: id}) {numUids}
}
Found it on dgraph alpha:
--mutations_nquad_limit uint
What are the implications when I increase this limit to, let’s say: 2000000? Any downsides apart from a suspected higher CPU/RAM/DISK usage?
Edit
Okay, this is weird. I increased the limit to 2000000 and now the error says:
failed because Dgraph execution failed because NQuad count in the request: 2002000, is more that threshold: 2000000
Why is the threshold added to the actual count?! Something is wrong here…
In general, I guess RAM and CPU usage. But is not that a big deal if you don’t perform those mutations all the time.
Maybe your query is really big and you should keep increasing the threshold to see where it goes.
Also, if you are using Upsert Block. You can limit the query with the pagination params. And perform multiple Upsert mutations.
My point was that the actual number of objects I try to delete is 2000. And when I changed the threshold from 1000000 to 2000000 NQuad count is supposed to be 2002000 instead of 1002000? It seems like that the threshold is somehow added to the actual NQuad number…
Anyways, I’m a little bit puzzled that deleting 2000 objects at once seems to be too much… Could this be because of additional overhead due to @auth rules etc.?
Each object can have Nˆ0(N-quads/Triples) - do your nodes have more than 10 predicates? are they Highly connected? A single node can have Nˆfriends, if a node has 50 friends, without counting the real properties of the node, it has more than 50 “N-quads”.
To give a concrete example for the object which I tried to delete 1000 from:
type Foo {
id: String! @id
createdAt: DateTime! @search
updatedAt: DateTime! @search
owner: User!
bla: Bla!
blub: Blub!
name: String!
bars: [Bar!] @hasInverse(field: foo)
foobars: [FooBar!] @hasInverse(field: foo)
}
type Bar {
id: String! @id
createdAt: DateTime! @search
updatedAt: DateTime! @search
owner: User!
label: String! @search(by: [hash])
a: Int!
b: Int!
c: String!
d: String!
lols: [Lol!] @hasInverse(field: bar)
}
Note that bars had a lenght of 1 while foobars and lols were empty (so 1000*2 objects in total).
How can I calculate the N-Quads here?
Impossible to say from the schema. Only who have access to the cluster could say. You can go and check Ratel statistics for each pred and try to have a number.
Also, as it is a GraphQL Schema, maybe the GraphQL mutation isn’t able to do such bulk delete. You should use DQL in this case.
Okay, thanks.
What do you think about this one? Does this seem normal to you? In both cases it was the exact same database-state with the same amount of total objects and the exact same deleteMutation.
qq, Is it a GraphQL Mutation? if so, I can’t imagine what is happening just the @graphql guys could.
Yes. Did I use the wrong tags for this thread?
It is alright. My hope was that you were using Bulk delete from Upsert Block. Cuz this name “Bulk delete” isn’t used in GraphQL as far as I know. I could be wrong.
Ah okay, sorry for the confusion then. I used “bulk” because I saw it on other threads (looking for answers to my question) and found it to be a fitting name 
FYI, as of 20.07 (haven’t tested with 20.11 yet) you can do a delete mutation with an empty filter object. The filter is required, but the object inside of the filter can be empty. That may better describe what you want here as all nodes should have an id. So instead of “bulk” delete, it can be understood as delete all of type.
mutation {
deleteFoo(filter: {}) {
numUids
}
}
I think this would vaguely make the mutation faster because it does not have to do a has(Foo.id) filter, but can do it with just the type(Foo) root function.
Thank you, good to know!
In reality I do id: {in: [*long list of ids*]} though. Just came up with this example because I don’t want to copy-pasta 2000 cuids into this forum
Probably what’s happening is a malformed data structure in the mutation or misspeled fields. Like, for example, when using JS/Apollo:
const DELETE_DOCUMENT_MUTATION = gql`
mutation deleteDocument($id: [ID!]) {
deleteDocument(filter: { id: $id }) {
document {
id
title}}}
client.mutate({mutation: DELETE_DOCUMENT_MUTATION,
variables: { id: [data.id] }})
, it works. When you provide variables: { Whatever: [data.id] }}on one side or, instead, mutation deleteDocument($whatever: [ID!]) {deleteDocument(filter: { id: $whatever })} on the other side, it’ll throw the mentioned error. Which has nothing to do with actual reason. If you try to do, for example, something like variables: { id: data.id }}, that is, a string instead of the array, it’ll throw “Internal Server Error - a panic was trapped”.
Can you share the panic from the logs or are you using Slash GraphQL in which case we can try to see what causes the panic?
2020-11-12 11:32:11.672
"panic: interface conversion: interface {} is []string, not []interface {}."
2020-11-12 11:32:11.673
trace: goroutine 367010 [running]:
2020-11-12 11:32:11.673
runtime/debug.Stack(0xc0014cccd0, 0x1b45e20, 0xc0021c9f50)
2020-11-12 11:32:11.673
/usr/local/go/src/runtime/debug/stack.go:24 +0x9d
2020-11-12 11:32:11.673
github.com/dgraph-io/dgraph/graphql/api.PanicHandler(0xc0014cdb20)
2020-11-12 11:32:11.673
/ext-go/1/src/github.com/dgraph-io/dgraph/graphql/api/panics.go:33 +0x57
2020-11-12 11:32:11.673
panic(0x1b45e20, 0xc0021c9f50)
2020-11-12 11:32:11.673
/usr/local/go/src/runtime/panic.go:969 +0x166
2020-11-12 11:32:11.673
github.com/dgraph-io/dgraph/graphql/resolve.idFilter(0xc0021c9ef0, 0x1f9d3e0, 0xc0021c9f20, 0x0, 0x1, 0xc0011ac3c8)
2020-11-12 11:32:11.673
/ext-go/1/src/github.com/dgraph-io/dgraph/graphql/resolve/query_rewriter.go:1250 +0x125
2020-11-12 11:32:11.673
github.com/dgraph-io/dgraph/graphql/resolve.RewriteUpsertQueryFromMutation(0x1fb0f20, 0xc0021c9b30, 0xc0014cd130, 0x1cf3000)
2020-11-12 11:32:11.673
/ext-go/1/src/github.com/dgraph-io/dgraph/graphql/resolve/mutation_rewriter.go:707 +0x2de
2020-11-12 11:32:11.673
github.com/dgraph-io/dgraph/graphql/resolve.(*deleteRewriter).Rewrite(0x2fdf160, 0x1f8cfa0, 0xc001c9d740, 0x1fb0f20, 0xc0021c9b30, 0x9, 0x203000, 0x203000, 0x85, 0x18)
2020-11-12 11:32:11.673
/ext-go/1/src/github.com/dgraph-io/dgraph/graphql/resolve/mutation_rewriter.go:782 +0x35d
2020-11-12 11:32:11.673
github.com/dgraph-io/dgraph/graphql/resolve.(*dgraphResolver).rewriteAndExecute(0xc0021c9b60, 0x1f8cfa0, 0xc001c9d740, 0x1fb0f20, 0xc0021c9b30, 0x0, 0xc0014cd600)
2020-11-12 11:32:11.673
/ext-go/1/src/github.com/dgraph-io/dgraph/graphql/resolve/mutation.go:243 +0x2c1
2020-11-12 11:32:11.673
github.com/dgraph-io/dgraph/graphql/resolve.(*dgraphResolver).Resolve(0xc0021c9b60, 0x1f8cfa0, 0xc001c9d740, 0x1fb0f20, 0xc0021c9b30, 0x0, 0x1f8cf00)
2020-11-12 11:32:11.673
/ext-go/1/src/github.com/dgraph-io/dgraph/graphql/resolve/mutation.go:185 +0x354
2020-11-12 11:32:11.673
github.com/dgraph-io/dgraph/graphql/resolve.(*RequestResolver).Resolve(0xc0015e3f60, 0x1f8cfa0, 0xc001c9d740, 0xc00f883500, 0x0)
2020-11-12 11:32:11.673
/ext-go/1/src/github.com/dgraph-io/dgraph/graphql/resolve/resolver.go:520 +0x5d7
2020-11-12 11:32:11.673
github.com/dgraph-io/dgraph/graphql/web.(*graphqlHandler).ServeHTTP(0xc00011e540, 0x1f863e0, 0xc0000ee0e0, 0xc00178e700)
2020-11-12 11:32:11.673
/ext-go/1/src/github.com/dgraph-io/dgraph/graphql/web/http.go:228 +0x3a9
2020-11-12 11:32:11.673
github.com/dgraph-io/graphql-transport-ws/graphqlws.NewHandlerFunc.func1(0x1f863e0, 0xc0000ee0e0, 0xc00178e700)
2020-11-12 11:32:11.673
/go/pkg/mod/github.com/dgraph-io/graphql-transport-ws@v0.0.0-20200916064635-48589439591b/graphqlws/http.go:39 +0xd5
2020-11-12 11:32:11.673
net/http.HandlerFunc.ServeHTTP(0xc0007430e0, 0x1f863e0, 0xc0000ee0e0, 0xc00178e700)
2020-11-12 11:32:11.673
/usr/local/go/src/net/http/server.go:2012 +0x44
2020-11-12 11:32:11.673
github.com/dgraph-io/dgraph/graphql/web.commonHeaders.func1(0x1f863e0, 0xc0000ee0e0, 0xc00178e700)
2020-11-12 11:32:11.673
/ext-go/1/src/github.com/dgraph-io/dgraph/graphql/web/http.go:332 +0x262
2020-11-12 11:32:11.673
net/http.HandlerFunc.ServeHTTP(0xc00011e560, 0x1f863e0, 0xc0000ee0e0, 0xc00178e700)
2020-11-12 11:32:11.673
/usr/local/go/src/net/http/server.go:2012 +0x44
2020-11-12 11:32:11.673
github.com/dgraph-io/dgraph/graphql/web.recoveryHandler.func1(0x1f863e0, 0xc0000ee0e0, 0xc00178e700)
2020-11-12 11:32:11.673
/ext-go/1/src/github.com/dgraph-io/dgraph/graphql/web/http.go:345 +0xa1
2020-11-12 11:32:11.673
net/http.HandlerFunc.ServeHTTP(0xc00011e580, 0x1f863e0, 0xc0000ee0e0, 0xc00178e700)
2020-11-12 11:32:11.673
/usr/local/go/src/net/http/server.go:2012 +0x44
2020-11-12 11:32:11.673
net/http.(*ServeMux).ServeHTTP(0x2d8e120, 0x1f863e0, 0xc0000ee0e0, 0xc00178e700)
2020-11-12 11:32:11.673
/usr/local/go/src/net/http/server.go:2387 +0x1a5
2020-11-12 11:32:11.673
net/http.serverHandler.ServeHTTP(0xc0000ee1c0, 0x1f863e0, 0xc0000ee0e0, 0xc00178e700)
2020-11-12 11:32:11.673
/usr/local/go/src/net/http/server.go:2807 +0xa3
2020-11-12 11:32:11.673
net/http.(*conn).serve(0xc00161de00, 0x1f8cee0, 0xc00f8833c0)
2020-11-12 11:32:11.673
/usr/local/go/src/net/http/server.go:1895 +0x86c
2020-11-12 11:32:11.673
created by net/http.(*Server).Serve
2020-11-12 11:32:11.673
/usr/local/go/src/net/http/server.go:2933 +0x35c
Thanks and what version of Dgraph are you running?
Dgraph v2.0.0-rc1-918-gc714c312
That’s a pretty old version. Could you try to reproduce it on the latest version and let us know if the bug is still there?