Export failed in dgraph 23.1.0

Users Dgraph
1

Hi Guys,

We are using self hosted dgraph on GKE. we just migrated from 22.0.2 to 23.1.0. We did not reload data isntead just upgraded the version. Everything is working fine except export. We are getting this error:

I0125 12:34:25.040685      45 log.go:33] Export Sent data of size 3.3 GiB
panic: runtime error: index out of range [1] with length 1

goroutine 880299 [running]:
github.com/dgraph-io/dgraph/x.ParseAttr(...)
	/home/runner/work/dgraph/dgraph/x/keys.go:120
github.com/dgraph-io/dgraph/worker.fieldToString(0x0?)
	/home/runner/work/dgraph/dgraph/worker/export.go:352 +0x4b2
github.com/dgraph-io/dgraph/worker.toType({0xc0038705a0?, 0xc01620e460?}, {{0xc0038705b0, 0xe}, {0xc016e30240, 0x8, 0x8}})
	/home/runner/work/dgraph/dgraph/worker/export.go:339 +0x127
github.com/dgraph-io/dgraph/worker.TypeExportKv({0xc0038705a0, 0x8}, {0xc01620e460?, 0x0?, 0x0?})
	/home/runner/work/dgraph/dgraph/worker/export.go:925 +0x8e
github.com/dgraph-io/dgraph/worker.exportInternal.func4(0x2)
	/home/runner/work/dgraph/dgraph/worker/export.go:857 +0x48d
github.com/dgraph-io/dgraph/worker.exportInternal({0x23d9ef8, 0xc00004c0e0}, 0xc0023d0400, 0xc00032d200, 0x0)
	/home/runner/work/dgraph/dgraph/worker/export.go:897 +0x662
github.com/dgraph-io/dgraph/worker.export({0x23d9ef8, 0xc00004c0e0}, 0xc0023d0400)
	/home/runner/work/dgraph/dgraph/worker/export.go:566 +0x1b8
github.com/dgraph-io/dgraph/worker.handleExportOverNetwork({0x23d9ef8?, 0xc00004c0e0?}, 0xc00042c190?)
	/home/runner/work/dgraph/dgraph/worker/export.go:950 +0x268
github.com/dgraph-io/dgraph/worker.ExportOverNetwork.func1(0x1)
	/home/runner/work/dgraph/dgraph/worker/export.go:1007 +0x1ae
created by github.com/dgraph-io/dgraph/worker.ExportOverNetwork
	/home/runner/work/dgraph/dgraph/worker/export.go:993 +0x2c5
[Sentry] 2024/01/25 12:34:38 Sending fatal event [317e0a48d7894186ae786e8b06269167] to o318308.ingest.sentry.io project: 1805390
[Sentry] 2024/01/25 12:34:38 Buffer flushed successfully.

Would really appreciate if anyone can help in this regard.

Thanks

2

@Damon could you please help?

3

Here is your mistake

4

@vnium Thanks for the reply. Problem is we first did it on staging environment and faced no issue so we did it on production. Now backups on production is stopped.

We think its due to some issue in schema because when we are trying to get dgraph schema the alpha pod throws error:

2024/01/26 06:48:22 strconv.ParseUint: parsing "\x00\x00\x00\x00\x00\x00\x00\x00dgraph.graphql.history": invalid syntax

Is there any way we can get data out of it and then restore it with fresh installation?

5

You can try to do the export with the old Dgraph version. If that fails, directly using badger might also work (Start from scratch without exporting to RDF(Badger)).
Between Dgraph version upgrades you need to check if the underlying disk format which is managed by Badger changes. This is usually the case when a new major version is released.

6

@vnium Thanks a lot for your valuable suggestion, we tried it as well. But I think the issue is something related to schema. Because when we are trying to get the schema either through ratel or through http query the pod throws following error:

`

2024/01/28 09:02:03 strconv.ParseUint: parsing “\x00\x00\x00\x00\x00\x00\x00\x00dgraph.graphql.persisted_query”: invalid syntax

`

Is there a way to somehow fix/rebuild the schema? alter query does not seem to have any impact.

Really appreciate your help.

Thanks

7

I missed this note a while back. V22 is not binary-compatible with v21 or v23, so requires a backup/restore or export/import to upgrade. Note that v21 and v23 actually are compatible.

So while many versions (including v21 → v23) do not require import/export or backup/restore, that is the official upgrade procedure as indicated here: Dgraph Administration - Deploy , because of cases like this where the underlying format changes.