# Dgraph takes all disk space in windows

**URL:** <https://discuss.dgraph.io/t/dgraph-takes-all-disk-space-in-windows/12494>\
**Category:** Badger\
**Tags:** dgraph, status:accepted, kind:bug, ticket:created\
**Created:** [January 27, 2021, 5:09pm UTC](https://discuss.dgraph.io/t/dgraph-takes-all-disk-space-in-windows/12494 "2021-01-27T17:09:03Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![marcown](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/marcown/32/6069_2.png) [@marcown](https://discuss.dgraph.io/u/marcown)\
**Post date:** [January 27, 2021, 5:09pm UTC](https://discuss.dgraph.io/t/dgraph-takes-all-disk-space-in-windows/12494/1 "2021-01-27T17:09:04Z")

</div>

## I Want to Do

I want to use Dgraph with windows. I am using the latest master:

```auto
.\dgraph.exe version
[Decoder]: Using assembly version of decoder
Page Size: 4096

Dgraph version : v20.11.0-g28b75cf26
Dgraph codename : unnamed-mod
Dgraph SHA-256 : 276a70655e4c56f996a080590645d200808c7d5b71677d7f5bc3e55d68dca6fe
Commit SHA-1 : 28b75cf26
Commit timestamp : 2021-01-27 16:54:35 +0530
Branch : master
Go version : go1.15.6
jemalloc enabled : false

```

I compiled dgraph for windows with adding the following to the make file and then run make windows:

```auto
windows:
	@GOOS=windows $(MAKE) dgraph
	@mkdir windows
	@mv ./dgraph/dgraph ./windows/dgraph

```

However, when I start dgraph alpha and zero on the windows system and post the schema the space of the harddrive used up quickly (\>100GB in the “t” folder, no data, just schema posted) resulting in the following dgraph error:

(important log lines …)

```auto
I0127 18:05:50.622405 3604 log.go:34] Rebuilding index for predicate Denkbox.dockerId (1/2): Streaming about 0 B of uncompressed data (0 B on disk)
I0127 18:05:50.639406 3604 log.go:34] Rebuilding index for predicate ResearchBoxTag.name (1/2): Streaming about 0 B of uncompressed data (0 B on disk)
I0127 18:05:50.716405 3604 log.go:34] Rebuilding index for predicate User.lastSyncedAt (1/2): Streaming about 0 B of uncompressed data (0 B on disk)
fatal error: out of memory allocating heap arena metadata
fatal error: runtime: cannot allocate memory

```

## What I Did

I don’t know what to do. In linux the same setup results in 0 GB “t” folder. Maybe it has sth to do with jemalloc that is not enabled in windows?

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![chewxy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/chewxy/32/4339_2.png) [@chewxy](https://discuss.dgraph.io/u/chewxy)\
**Post date:** [January 28, 2021, 12:00am UTC](https://discuss.dgraph.io/t/dgraph-takes-all-disk-space-in-windows/12494/2 "2021-01-28T00:00:51Z")

</div>

Out of curiosity, could you share your schema?

So if I understand this right, you used Ratel, edited the schema, and then Dgraph Alpha started taking up 100GiB of data?

---

<div class="post-metadata">

**Author:** ![marcown](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/marcown/32/6069_2.png) [@marcown](https://discuss.dgraph.io/u/marcown)\
**Post date:** [January 28, 2021, 9:36am UTC](https://discuss.dgraph.io/t/dgraph-takes-all-disk-space-in-windows/12494/3 "2021-01-28T09:36:43Z")

</div>

I am not able to share the graphql schema, however, I built one with a similiar structure.

This one creates a “t” folder with ~10MB and a lot of dgraph\_index folders and DISCARD files in it. Our original schema is way bigger, ~1000 lines.

I hope you get some insight from the schema. I have no clue why the same schema produces 0MB “t” folder on linux.

```auto
############################################################
##################### CUSTOM LOGIC #########################
############################################################

type TokenPayload @remote {
  token: String!
}

type Query {
  getUserToken(username: String!, role: Role): TokenPayload!
    @custom(
      http: {
        url: "http://localhost:8085/custom"
        method: POST
        graphql: "query($username: String!, $role: Role) {getUserToken(username: $username, role: $role)}"
        forwardHeaders: ["X-Auth"]
        skipIntrospection: true
      }
    )
}

type Mutation {

  login(username: String!, password: String!): TokenPayload!
    @custom(
      http: {
        url: "http://localhost:8085/custom"
        method: POST
        graphql: "mutation($username: String!, $password: String!) {login(username: $username, password: $password)}"
        skipIntrospection: true
      }
    )

}

############################################################
####################### DATABASE ###########################
############################################################

enum Role {
  TENANT
  ADMIN
}

interface Cuid {
  id: String! @id
}

interface Ownable
{
  owner: User!
}

interface Timestamped {
  createdAt: DateTime! @search
  updatedAt: DateTime! @search
}

type User implements Timestamped
 {
  username: String! @id

  password: String!
  email: String!

  permissions: UserPermissions @hasInverse(field: owner)
  projects: [Project!] @hasInverse(field: owner)

}

type UserPermissions implements Cuid & Timestamped & Ownable {
  username: String! @search(by: [hash])

  role: Role!
}

type Project implements Cuid & Timestamped & Ownable @withSubscription {
  name: String! @search(by: [hash])
  description: String! @search(by: [fulltext])

  pinnedProjects: [Project!]
}
# Dgraph.Authorization {"VerificationKey":"thisisthesecretkey","Header":"X-Auth","Namespace":"https://thisisatest.com/jwt/claims","Algo":"HS256"}

```

I start dgraph alpha and zero via

.\dgraph alpha  
.\dgraph zero

and then I post the schema via

curl -X POST localhost:8080/admin/schema --data-binary ‘@dgraph.graphql’

---

<div class="post-metadata">

**Author:** ![chewxy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/chewxy/32/4339_2.png) [@chewxy](https://discuss.dgraph.io/u/chewxy)\
**Post date:** [January 28, 2021, 11:20pm UTC](https://discuss.dgraph.io/t/dgraph-takes-all-disk-space-in-windows/12494/4 "2021-01-28T23:20:05Z")

</div>

OK this looks like a bug. Tagging @pawan

---

<div class="post-metadata">

**Author:** ![marcown](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/marcown/32/6069_2.png) [@marcown](https://discuss.dgraph.io/u/marcown)\
**Post date:** [January 29, 2021, 9:17am UTC](https://discuss.dgraph.io/t/dgraph-takes-all-disk-space-in-windows/12494/5 "2021-01-29T09:17:11Z")

</div>

Maybe there is a problem with the os.RemoveAll line here?:

> <https://github.com/dgraph-io/dgraph/blob/4308da42e0cd1a33d768eb41f736642f9448241e/posting/index.go#L569>

---

<div class="post-metadata">

**Author:** ![marcown](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/marcown/32/6069_2.png) [@marcown](https://discuss.dgraph.io/u/marcown)\
**Post date:** [January 29, 2021, 9:37am UTC](https://discuss.dgraph.io/t/dgraph-takes-all-disk-space-in-windows/12494/6 "2021-01-29T09:37:39Z")

</div>

I created a go test program

```auto
package main

import (
	"bufio"
	"fmt"
	"io/ioutil"
	"log"
	"math/rand"
	"os"
	"path"
	"time"
)

func check(err error) {
	if err != nil {
		panic(err)
	}
}

func main() {
	fmt.Println("Checking RemoveAll with subfiles")

	if _, err := os.Stat("./tmp"); os.IsNotExist(err) {
		os.Mkdir("./tmp", 0700)
	}

	for i := 0; i < 20; i++ {

		dir, err := ioutil.TempDir("./tmp", "dgraph_index_")
		if err != nil {
			fmt.Println("err: ", err)
			log.Fatal(err)
		}
		fmt.Println("dir: ", dir)

		f, err := os.Create(path.Join(dir, "DISCARD"))
		check(err)
		defer f.Close()
		w := bufio.NewWriter(f)
		//choose random number for recipe
		r := rand.New(rand.NewSource(time.Now().UnixNano()))
		i := r.Perm(5)

		_, err = fmt.Fprintf(w, "%v\n", i)
		check(err)
		_, err = fmt.Fprintf(w, "%d\n", i[0])
		check(err)
		_, err = fmt.Fprintf(w, "%d\n", i[1])
		check(err)
		w.Flush()

		defer os.RemoveAll(dir)
	}

}

```

in linux everything in tmp gets removed, in windows nothing gets removed

EDIT:

Just realized an error in my test program:

defer os.RemoveAll should be at line 30, directly after ioutil.TempDir, this works now in windows

---

<div class="post-metadata">

**Author:** ![maaft](https://avatars.discourse-cdn.com/v4/letter/m/4af34b/32.png) [@maaft](https://discuss.dgraph.io/u/maaft)\
**Post date:** [January 29, 2021, 2:31pm UTC](https://discuss.dgraph.io/t/dgraph-takes-all-disk-space-in-windows/12494/7 "2021-01-29T14:31:57Z")

</div>

Are there any news on this? Got the same error on windows.

---

<div class="post-metadata">

**Author:** ![chewxy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/chewxy/32/4339_2.png) [@chewxy](https://discuss.dgraph.io/u/chewxy)\
**Post date:** [January 31, 2021, 3:05am UTC](https://discuss.dgraph.io/t/dgraph-takes-all-disk-space-in-windows/12494/8 "2021-01-31T03:05:10Z")

</div>

Oh right. I managed to reproduce this error. We’re gonna fix it. @ibrahim has set up some Windows test pipelines, which should catch this.

---

<div class="post-metadata">

**Author:** ![marcown](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/marcown/32/6069_2.png) [@marcown](https://discuss.dgraph.io/u/marcown)\
**Post date:** [February 10, 2021, 11:45am UTC](https://discuss.dgraph.io/t/dgraph-takes-all-disk-space-in-windows/12494/10 "2021-02-10T11:45:20Z")

</div>

Hey there @chewxy @ibrahim

is there any update on this issue? Can I help to fix it?

Thanks!

---

<div class="post-metadata">

**Author:** ![ibrahim](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/ibrahim/32/6387_2.png) [@ibrahim](https://discuss.dgraph.io/u/ibrahim)\
**Post date:** [February 10, 2021, 5:16pm UTC](https://discuss.dgraph.io/t/dgraph-takes-all-disk-space-in-windows/12494/11 "2021-02-10T17:16:11Z")

</div>

Hey @marcown, unfortunately, we haven’t been able to work on this. Please feel free to send a PR if you know what needs to be done.

---

<div class="post-metadata">

**Author:** ![maaft](https://avatars.discourse-cdn.com/v4/letter/m/4af34b/32.png) [@maaft](https://discuss.dgraph.io/u/maaft)\
**Post date:** [February 11, 2021, 3:25pm UTC](https://discuss.dgraph.io/t/dgraph-takes-all-disk-space-in-windows/12494/14 "2021-02-11T15:25:08Z")

</div>

Quick Fix here:  
dgraph → posting/index.go:590

Change to:

```auto
dbOpts := badger.DefaultOptions(tmpIndexDir).
		WithSyncWrites(false).
		WithNumVersionsToKeep(math.MaxInt32).
		WithLogger(&x.ToGlog{}).
		WithCompression(options.None).
		WithEncryptionKey(x.WorkerConfig.EncryptionKey).
		WithLoggingLevel(badger.WARNING).
		WithInMemory(true).WithDir("").WithValueDir("")

```

Note the last line. This will use InMemory for the temporary badger instance and the error does not appear.

To reproduce the bug:

```auto
package main

import (
	"os"
	"path/filepath"

	"github.com/dgraph-io/badger/v3"
)

func main() {

	tmpIndexDir := filepath.Join(".", "test")
	b, err := badger.Open(badger.DefaultOptions(tmpIndexDir))
	if err != nil {
		panic(err)
	}

	err = b.Close()
	if err != nil {
		panic(err)
	}

	err = os.RemoveAll(tmpIndexDir)

	if err != nil {
		panic(err)
	}

}

```

@Naman maybe you got an idea? I saw you’ve fixed a possible related [bug](http://discuss.hypermode.com/t/unable-to-delete-the-mem-file-when-closing-the-db-under-windows-10-20h2/12353)

---

<div class="post-metadata">

**Author:** ![marcown](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/marcown/32/6069_2.png) [@marcown](https://discuss.dgraph.io/u/marcown)\
**Post date:** [February 17, 2021, 7:33am UTC](https://discuss.dgraph.io/t/dgraph-takes-all-disk-space-in-windows/12494/15 "2021-02-17T07:33:36Z")

</div>

Sorry for asking again, but is there any update here or any plans when it can be done? At the moment it’s not usable at all

---

<div class="post-metadata">

**Author:** ![maaft](https://avatars.discourse-cdn.com/v4/letter/m/4af34b/32.png) [@maaft](https://discuss.dgraph.io/u/maaft)\
**Post date:** [February 23, 2021, 2:55pm UTC](https://discuss.dgraph.io/t/dgraph-takes-all-disk-space-in-windows/12494/16 "2021-02-23T14:55:11Z")

</div>

Should we maybe post this somewhere else where the topic is more badger related?

We’re currently using docker on windows but that’s not a long-term solution.

---

<div class="post-metadata">

**Author:** ![Naman](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/naman/32/3498_2.png) [@Naman](https://discuss.dgraph.io/u/Naman)\
**Post date:** [February 23, 2021, 6:56pm UTC](https://discuss.dgraph.io/t/dgraph-takes-all-disk-space-in-windows/12494/17 "2021-02-23T18:56:00Z")

</div>

Hey @maaft, sorry for the delays.  
I see the issue is that we were not closing the DISCARD file and windows complain about deleting the open file.  
Can you please try [https://github.com/dgraph-io/badger/tree/naman/test-windows](https://github.com/dgraph-io/badger/tree/naman/test-windows) branch and confirm if this fixes the issue for you? I will get it reviewed and merged.

---

<div class="post-metadata">

**Author:** ![marcown](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/marcown/32/6069_2.png) [@marcown](https://discuss.dgraph.io/u/marcown)\
**Post date:** [February 23, 2021, 7:01pm UTC](https://discuss.dgraph.io/t/dgraph-takes-all-disk-space-in-windows/12494/18 "2021-02-23T19:01:26Z")

</div>

@Naman

thanks for the reply und fix. I will test it tomorrow morning ( GMT+1 😉 ) and report back

---

<div class="post-metadata">

**Author:** ![marcown](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/marcown/32/6069_2.png) [@marcown](https://discuss.dgraph.io/u/marcown)\
**Post date:** [February 23, 2021, 9:40pm UTC](https://discuss.dgraph.io/t/dgraph-takes-all-disk-space-in-windows/12494/19 "2021-02-23T21:40:50Z")

</div>

Hey again,  
@Naman

i just tested it (could not wait^^):

I did a  
go get [github.com/dgraph-io/badger/v3@bfb6a35bad430f6b946cceb9a30e69fa0a88ee81](http://github.com/dgraph-io/badger/v3@bfb6a35bad430f6b946cceb9a30e69fa0a88ee81)  
in the dgraph latest master and then build windows as described in the first post.

Sadly the error is still there. But now, not only the DISCARD file is remaining, there are also  
00001.vlog  
00001.mem  
KEYREGISTRY and  
MANIFEST

Files in the temp folder.

EDIT: Files get deleted, but quite slow, during scheme posting the hdd space gets used to the maximum (\>65GB on my system).

EDIT2: Seems a bit that they only get deleted after scheme posting. Furthermore, RAM usage is really high during scheme posting → i cannot run it on my second system

EDIT3: With an empty db the ram usage remains high (7.9GB of 8GB) after scheme posting on my first system

---

<div class="post-metadata">

**Author:** ![Naman](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/naman/32/3498_2.png) [@Naman](https://discuss.dgraph.io/u/Naman)\
**Post date:** [February 24, 2021, 6:40am UTC](https://discuss.dgraph.io/t/dgraph-takes-all-disk-space-in-windows/12494/20 "2021-02-24T06:40:07Z")

</div>

> [@maaft](#):
>
> To reproduce the bug:
> 
> ```auto
> package main
> 
> import (
> "os"
> "path/filepath"
> 
> "github.com/dgraph-io/badger/v3"
> )
> 
> func main() {
> 
> tmpIndexDir := filepath.Join(".", "test")
> b, err := badger.Open(badger.DefaultOptions(tmpIndexDir))
> if err != nil {
> panic(err)
> }
> 
> err = b.Close()
> if err != nil {
> panic(err)
> }
> 
> err = os.RemoveAll(tmpIndexDir)
> 
> if err != nil {
> panic(err)
> }
> 
> }
> 
> ```

Hey @marcown, does this test works fine and deletes directory?  
For the performance issues, please have a look at [Dropping support for Windows and Mac](http://discuss.hypermode.com/t/dropping-support-for-windows-and-mac/12913).

---

<div class="post-metadata">

**Author:** ![marcown](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/marcown/32/6069_2.png) [@marcown](https://discuss.dgraph.io/u/marcown)\
**Post date:** [February 24, 2021, 7:16am UTC](https://discuss.dgraph.io/t/dgraph-takes-all-disk-space-in-windows/12494/21 "2021-02-24T07:16:02Z")

</div>

This works.

---

<div class="post-metadata">

**Author:** ![Naman](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/naman/32/3498_2.png) [@Naman](https://discuss.dgraph.io/u/Naman)\
**Post date:** [February 24, 2021, 6:41pm UTC](https://discuss.dgraph.io/t/dgraph-takes-all-disk-space-in-windows/12494/22 "2021-02-24T18:41:23Z")

</div>