(I’m running in Kubernetes at the moment)
I’m trying to do an upgrade from v1.1.1 to v20.07.0. I’ve exported my data which comes to about 28 million quads and I’m trying to import it into the database instance into which I’m upgrading using the live loader. I believe I’m running into some memory issues; on the clusters where the upgrade has succeeded I’ve noticed that the Alpha node tends to hit 6-7gb resident and cruise around there. In other, less-successful restorations, the pod terminates and when it restarts the live loader is unable to resume, it sits at 0 transactions committed indefinitely after.
When I run top
on the Alpha after the CPU usage is quite high but it doesn’t seem to be doing anything. The Alpha log contains these entries, approximately one a minute:
I0807 02:15:12.662504 14 draft.go:1531] Skipping snapshot at index: 36395. Insufficient discard entries: 1. MinPendingStartTs: 51043
I0807 02:15:12.663280 14 draft.go:1367] Found 3 old transactions. Acting to abort them.
I0807 02:15:12.684843 14 draft.go:1328] TryAbort 3 txns with start ts. Error: <nil>
I0807 02:15:12.684906 14 draft.go:1351] TryAbort selectively proposing only aborted txns: txns:<start_ts:51052 > txns:<start_ts:51190 > txns:<start_ts:51043 >
I0807 02:15:40.685652 14 draft.go:1370] Done abortOldTransactions for 3 txns. Error: Server overloaded with pending proposals. Please retry later