I want to use Badger in Dockerfile with mounted external disk.
For that purpose I use gcs-fuse: Cloud Storage FUSE | Google Cloud
The Dockerfile is as follows:
FROM golang:1.16-buster as build-env
###############################
# gcsfuse mount of a GCS bucket
###############################
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update && apt-get install -y apt-utils
# GCS FUSE
RUN apt-get install -y lsb-release; \
gcsFuseRepo=gcsfuse-`lsb_release -c -s`; \
echo "deb http://packages.cloud.google.com/apt $gcsFuseRepo main" | \
tee /etc/apt/sources.list.d/gcsfuse.list; \
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | \
apt-key add -; \
apt-get update; \
apt-get install -y gcsfuse
# Create all mount points
RUN mkdir -p \
/mnt/badger
###############################
# Create and change to the app directory.
WORKDIR /app
COPY . ./
RUN ls -la /app/
RUN ls -la /mnt/badger
RUN CGO_ENABLED=0 GOOS=linux \
go build -a -installsuffix cgo \
server.go
# ###############################
ENTRYPOINT ["/app/run.sh"]
# ###############################
run.sh is as follows
#!/bin/sh
echo Cloud Storage FUSE
echo "mounting Cloud Storage bucket"
# gcsfuse mount of a GCS bucket
gcsfuse --debug_gcs --debug_fuse -o allow_other dcapcs /mnt/badger
cp run.sh /mnt/badger
echo "ls /app/mntdb"
ls /mnt/badger
echo "mounts completed"
echo "/app/server"
/app/server
I uploaded my Badger DB in my Google Cloud Storage bucket: dcapcs
After creating the Docker file and uploading, RUN ls -la /mnt/badger shows the database is mounted and I can see the DB files listed.
In my server.go I double check if the database files are seen:
log.Printf("badgerDBName [%s]", badgerDBName)
///////////////////////////////
// verify if db files are seen:
///////////////////////////////
files, err := ioutil.ReadDir(badgerDBName + "/")
if err != nil {
log.Printf("%s: %v\n", badgerDBName, err)
// return
} else {
for _, file := range files {
log.Printf("%s->%s\n",
filepath.Join(badgerDBName, file.Name()), file.Name())
}
}
///////////////////////////////
That means the server.go has an access to the Badger DB files I upload the
Google Cloud Storage and are accessible as regular files via gcsfuse.
I’m using:
badgerDBName := "/mnt/badger"
opt := badger.DefaultOptions(badgerDBName).WithInMemory(false).WithReadOnly(true)
db, err := badger.Open(opt)
The error I got:
2021-03-22 16:16:26.217 PDT
2021/03/22 23:16:26 badger.Open:err while opening memtables error: while opening fid: 1 error: While opening memtable: /mnt/badger/00001.mem error: while opening file: /mnt/badger/00001.mem error: truncate /mnt/badger/00001.mem: invalid argument
Default
2021-03-22 16:16:26.217 PDT
error while truncation
Default
2021-03-22 16:16:26.217 PDT
...dgraph-io/ristretto/z.OpenMmapFileUsing
Default
2021-03-22 16:16:26.217 PDT
...dgraph-io/[email protected]/z/file.go:50
3-22 16:16:26.217 PDT
....dgraph-io/ristretto/z.OpenMmapFileUsing
Default
2021-03-22 16:16:26.217 PDT
...dgraph-io/[email protected]/z/file.go:50
Default
2021-03-22 16:16:26.217 PDT
...dgraph-io/ristretto/z.OpenMmapFile
Default
2021-03-22 16:16:26.217 PDT
...dgraph-io/[email protected]/z/file.go:86
Default
2021-03-22 16:16:26.217 PDT
...dgraph-io/badger/v3.(*logFile).open
Default
2021-03-22 16:16:26.217 PDT
...dgraph-io/badger/[email protected]/memtable.go:556
Default
2021-03-22 16:16:26.217 PDT
...dgraph-io/badger/v3.(*DB).openMemTable
Default
2021-03-22 16:16:26.217 PDT
...dgraph-io/badger/[email protected]/memtable.go:130
Default
2021-03-22 16:16:26.217 PDT
...dgraph-io/badger/v3.(*DB).openMemTables
Default
2021-03-22 16:16:26.217 PDT
...dgraph-io/badger/[email protected]/memtable.go:88
Default
2021-03-22 16:16:26.217 PDT
...dgraph-io/badger/v3.Open
Default
2021-03-22 16:16:26.217 PDT
...dgraph-io/badger/[email protected]/db.go:299
It is not working.
With the option:
opt := badger.DefaultOptions(badgerDBName).WithInMemory(true).WithReadOnly(true)
db, err := badger.Open(opt)
got the error:
2021/03/22 23:35:59 badger.Open:err Cannot use badger in Disk-less mode with Dir or ValueDir set
I have a working solution when I embed the Badger DB inside the Docker container.
The question is what prevents Badger to use the remotely mounted database?