Problems with types and big datasets

[davideaimar]

I’m having problems using Dgraph with a huge dataset on a single beefy machine.
I’ve succesfully imported a dataset with 152.9G edges, but the alpha instance seems to have problems managing the schema. Every time I try to load it from ratel UI I get:

The alpha logs this:

2023/06/04 17:22:26 http: panic serving 172.24.0.1:45378: runtime error: slice bounds out of range [:8] with length 7
goroutine 46114 [running]:
net/http.(*conn).serve.func1()
        /opt/hostedtoolcache/go/1.18.5/x64/src/net/http/server.go:1825 +0xbf
panic({0x1e604e0, 0xc046638db0})
        /opt/hostedtoolcache/go/1.18.5/x64/src/runtime/panic.go:844 +0x258
github.com/dgraph-io/dgraph/x.ParseNamespaceAttr(...)
        /home/runner/work/dgraph/dgraph/x/keys.go:87
github.com/dgraph-io/dgraph/query.filterTypesForNamespace(0x0, {0xc00b629900, 0xd, 0x20?})
        /home/runner/work/dgraph/dgraph/query/query.go:2967 +0x3b9
github.com/dgraph-io/dgraph/query.(*Request).Process(0xc00c348fc8, {0x224a5b8, 0xc03404fcb0})
        /home/runner/work/dgraph/dgraph/query/query.go:2955 +0x445
github.com/dgraph-io/dgraph/edgraph.processQuery({0x224a5b8, 0xc03404fcb0}, 0xc008ee93f0)
        /home/runner/work/dgraph/dgraph/edgraph/server.go:1395 +0x590
github.com/dgraph-io/dgraph/edgraph.(*Server).doQuery(0x224a580?, {0x224a5b8, 0xc00e7990b0}, 0xc00c5d16a8)
        /home/runner/work/dgraph/dgraph/edgraph/server.go:1306 +0xc36
github.com/dgraph-io/dgraph/edgraph.(*Server).Query(0x224a5b8?, {0x224a580?, 0xc018ddd560?}, 0xc018ddd5c0)
        /home/runner/work/dgraph/dgraph/edgraph/server.go:1171 +0x2f9
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.queryHandler({0x22493d8, 0xc008e70540}, 0xc00c2d2e00)
        /home/runner/work/dgraph/dgraph/dgraph/cmd/alpha/http.go:249 +0x6c5
net/http.HandlerFunc.ServeHTTP(0x2ecd8a0?, {0x22493d8?, 0xc008e70540?}, 0x6?)
        /opt/hostedtoolcache/go/1.18.5/x64/src/net/http/server.go:2084 +0x2f
net/http.(*ServeMux).ServeHTTP(0xc00b7b69c0?, {0x22493d8, 0xc008e70540}, 0xc00c2d2e00)
        /opt/hostedtoolcache/go/1.18.5/x64/src/net/http/server.go:2462 +0x149
github.com/dgraph-io/dgraph/ee/audit.AuditRequestHttp.func1({0x22493d8?, 0xc008e70540?}, 0x45d3d105355?)
        /home/runner/work/dgraph/dgraph/ee/audit/interceptor_ee.go:91 +0xc2
net/http.HandlerFunc.ServeHTTP(0x44f44c8fb55?, {0x22493d8?, 0xc008e70540?}, 0xc027e4ee80?)
        /opt/hostedtoolcache/go/1.18.5/x64/src/net/http/server.go:2084 +0x2f
net/http.(*ServeMux).ServeHTTP(0xc00dbd9cec?, {0x22493d8, 0xc008e70540}, 0xc00c2d2e00)
        /opt/hostedtoolcache/go/1.18.5/x64/src/net/http/server.go:2462 +0x149
net/http.serverHandler.ServeHTTP({0x2243ab0?}, {0x22493d8, 0xc008e70540}, 0xc00c2d2e00)
        /opt/hostedtoolcache/go/1.18.5/x64/src/net/http/server.go:2916 +0x43b
net/http.(*conn).serve(0xc0d93fec80, {0x224a5b8, 0xc00b99bda0})
        /opt/hostedtoolcache/go/1.18.5/x64/src/net/http/server.go:1966 +0x5d7
created by net/http.(*Server).Serve
        /opt/hostedtoolcache/go/1.18.5/x64/src/net/http/server.go:3071 +0x4db

I can succesfully query data, but some queries are not working. For example, queries using expand don’t work, like this one:

{
	q(func: type(Log), first: 1)  {
      expand(_all_) {  
  	    expand(_all_)
	  }
    }
}

Even if data correctly has dgraph.type set to “Log”.

The same dataset and schema don’t give me any problem if I just import a subset of the available data, so probably the problem is correlated to the size of the data.

[MichelDiz]

Ratel can be a little annoying sometimes. Make sure you have access to localhost:8080/health?all
Also check in the console log of the browser if there is any CORs issue. Are you sure you are in the correct port? 8081, are you using port offset?

@davideaimar

Even if data correctly has dgraph.type set to “Log”.

All your objects has the type defined? without it you can query with expand all.
How many objects do you have?

If you still have issues with Type(). Try the following. Create a tree from 0x01(if you are planning to use ACL, use the UID 0x03 instead). and branch out your types from this tree. Edges in Dgraph are super reliable. Here is an example.

[
  {
    "uid": "0x1",
    "my.types": [
      { #this will be 0x02 in this example
        "type.name": "User",
        "hasNodes":  [ { "uid": "0x04"}]
      },
      {
        "type.name": "Product",
        "hasNodes":  [ { "uid": "0x07"}]
      }
    ]
  }
]

And you can just add an extra edge every new mutation. For example for users.

[ 
    {
        "uid": "0x02",
        "hasNodes":  [ { "uid": "_:George" } ]
      },
    {
        "uid": "_:George",
        "user.name": "George",
        "age":  22
      }
]

So now you can do recurse in that tree and expand your pseudo-schema tree. From the edge hasNodes you will have a count for each type and you can expand from there.

{
  var(func: uid(0x01)) {
     Users as my.types @filter(eq(type.name, "User"))
   }
  allUsers(func: uid(Users )) @filter(gt(age, 20)) {
     user.name
     age
   }
}

A GraphDB is a blank frame. You can do a lot of things.

[davideaimar]

Everything is fine with Ratel, I was using Docker mapping ports to 8081 because I had multiple instances running, I tried moving to 8080 but I have the same problem. I can access localhost:8080/health?all.

All the objects have the type set, they should be around 2.3 billions, but I can’t count them since I get another error (see: [BUG]: Allocator can not allocate more than 64 buffers · Issue #8840 · dgraph-io/dgraph · GitHub).

I think the problem is related to the amount of data I have. I tried importing just a smaller portion of data (let’s say a few millions instead of the 2.3 billions) and everything worked fine.
For context, the size of the p folder is 2.4TB. I have also other types with billions of entries.

Querying data using Type() works. For example:

{
  q(func: type(Log), first: 10) {
		count(uid)
  }
}

Gives me:

"data": {
    "q": [
      {
        "count": 10
      }
    ]
  }

It doesn’t work using expand(all) and the alpha panics when queried with:

curl localhost:8080/query -XPOST -H "Content-Type: application/dql" -d 'schema {}'

With the logs of the first message I posted.

[amaster507 (Anthony Master)]

RFC: Proposal for change in Type System

We have finally arrived at the se conclusion, lol See: Here is the main gist of my thoughts:

I really am not sure if this is aplicable to you @davideaimar as I’m not sure what version you are running.

But there is a fundamental theoretical error in how types are handled currently in Dgraph. And if you use 21.12 then you also face:

@amaster507

In v21.12, we have added a flag to forbid any key which has greater than 1000 splits by marking it forbidden in Badger. Once a key is forbidden, that key would continue to drop data and would always return an empty result.

[davideaimar]

I’m using Dgraph v22.0.2 with Docker.
I have > 5 billion dgraph.type predicates. I don’t think the problem is with the function type(), it works fine for queries like this:

{
  q(func: type(SmallerType))  {
    count(uid)
  }
}

Where the types have less then 1B entries.

I think I’m facing two separate issues:

1. Counting types with billions of entries I get:

panic serving 172.23.0.1:47114: Allocator can not allocate more than 64 buffers

See: [BUG]: Allocator can not allocate more than 64 buffers · Issue #8840 · dgraph-io/dgraph · GitHub.

2. Schema cannot be downloaded/queried, so expand don’t work.

curl localhost:8080/query -XPOST -H "Content-Type: application/dql" -d 'schema {}'

Results in:

panic serving 172.24.0.1:45378: runtime error: slice bounds out of range [:8] with length 7

I don’t know if the two problems are related.

I doubt that the second problem is related to how I imported the data. I used a locally compiled Dgraph instance, with this fix for the bulk loader: fix(bulk): removed buffer max size by davideaimar · Pull Request #8841 · dgraph-io/dgraph · GitHub, connected to a local zero instance using the same binary.

Once bulk import was completed I moved the p folder inside a docker volume and I re-run everything from there, spawning a new zero and without sharing the zw folder.

Do you think this could be the reason of the second problem? Is there a way to re-upload the schema without having to rebuild all the indexes?

[amanmangal]

Hi David,

Thanks for posting the issue on discuss. I am an engineer who work at Dgraph. Would you be interested in getting on a call with me some time? If that works, you could send me suitable times on aman@dgraph.io. We would be happy to look at the cluster for you.

Thanks

[davideaimar]

The reason of the problem was the fact that I was using two different version of Dgraph. I was running the bulk import with a locally compiled dgraph binary on v23 and running it later with Docker using v22.
Running everything on the same version (v23) solved the problem.