# Improve Dgraph upgrade experience by supporting in-place & rolling upgrades **URL:** \ **Category:** Dgraph\ **Tags:** dgraph, status:accepted, kind:feature\ **Created:** [January 22, 2020, 10:49am UTC](https://discuss.dgraph.io/t/improve-dgraph-upgrade-experience-by-supporting-in-place-rolling-upgrades/8821 "2020-01-22T10:49:48Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![diggy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/diggy/32/3666_2.png) [@diggy](https://discuss.dgraph.io/u/diggy)\ **Post date:** [January 22, 2020, 10:49am UTC](https://discuss.dgraph.io/t/improve-dgraph-upgrade-experience-by-supporting-in-place-rolling-upgrades/8821/1 "2020-01-22T10:49:48Z")
**Moved from GitHub [dgraph/4644](https://github.com/dgraph-io/dgraph/issues/4644)** _Posted by_ [sleto-it](https://github.com/sleto-it): ## Experience Report This is a feature request to add an additional upgrade method, the _in-place_ upgrade, which can coexist with our current _logical_ upgrade method When the _in-place_ upgrade will be available, this is a FR to support rolling upgrades (one node at a time) of Dgraph clusters, so that the upgrade can be done while the cluster remains up and running, and ideally while it keeps serving reads and/or reads and writes ### What you wanted to do - Upgrade in an easy way a running Dgraph cluster, especially when upgrading to a new patch release (e.g. from 1.1.0 to 1.1.1) - Minimize operational efforts - Minimize downtime ### What you actually did The upgrade is currently possible, but it _always_ requires an export and import (_logical upgrade_) ### Why that wasn’t great, with examples While it is understandable and acceptable that sometime a _logical_ upgrade might be required (e.g. due to some incompatible changes, to re-shard the cluster, etc), it is important to improve the upgrade experience as much as possible e.g. by introducing a new upgrade method: the _in-place_ upgrade, which would allow the installation of the new package/version and the reuse of the current data directory. The upgrade process can be responsible to run some in-place upgrade operations in the existing data directory, when needed _In-place_ upgrade will then hopefully also allow rolling upgrades, where we upgrade one node at a time, the cluster remaining up and operational in the meantime The problem related to the current upgrade approach is that, in addition to the export and import, it requires to set up a new cluster, if you want to minimize downtime (blue-green deployment for upgrades) - which, from an operational point of view, is costly in terms of required time and efforts ### Any external references to support your case - other database vendors provide this feature - increasing requests from our user base - examples: [link1](http://discuss.hypermode.com/t/better-upgrade-guide/3288), [link2](https://github.com/dgraph-io/dgraph/issues/2728) - [doc page](https://docs.dgraph.io/v1.1.0/deploy/#upgrade-database) on the upgrade, e.g. from 1.1.0 to 1.1.1 ---
**Author:** ![diggy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/diggy/32/3666_2.png) [@diggy](https://discuss.dgraph.io/u/diggy)\ **Post date:** [January 22, 2020, 4:48pm UTC](https://discuss.dgraph.io/t/improve-dgraph-upgrade-experience-by-supporting-in-place-rolling-upgrades/8821/2 "2020-01-22T16:48:41Z")
[marvin-hansen](https://github.com/marvin-hansen) _commented_ : Upvote, 100%. This is critical for operations. A preliminary way to deliver rolling updates would be a reference process in Kubernetes by setting up a second deployment and then re-route traffic to the updated version using the service. > **[Performing a Rolling Update](https://kubernetes.io/docs/tutorials/kubernetes-basics/update/update-intro/)** > > Perform a rolling update using kubectl. Obviously, not all deployments run on k8s but writing a tutorial about rolling updates in Kubernetes would already make things way better and fill the time until a proper solution is there. If possible, please update the documentation! ---
**Author:** ![diggy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/diggy/32/3666_2.png) [@diggy](https://discuss.dgraph.io/u/diggy)\ **Post date:** [January 29, 2020, 6:34pm UTC](https://discuss.dgraph.io/t/improve-dgraph-upgrade-experience-by-supporting-in-place-rolling-upgrades/8821/3 "2020-01-29T18:34:44Z")
[artooro](https://github.com/artooro) _commented_ : A side effect of not having a way to upgrade Dgraph without or at least with minimal downtime will be that users just don’t upgrade, or rarely upgrade. @marvin-hansen I’ve thought about finding a way to spin up a new deployment, and switch the kubernetes service to it, but haven’t figured out a good mechanism to keep the existing deployment up in a read-only mode, as there will be the period of time where you have to export, and import to bring the new deployment up. ---
**Author:** ![diggy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/diggy/32/3666_2.png) [@diggy](https://discuss.dgraph.io/u/diggy)\ **Post date:** [January 29, 2020, 7:16pm UTC](https://discuss.dgraph.io/t/improve-dgraph-upgrade-experience-by-supporting-in-place-rolling-upgrades/8821/4 "2020-01-29T19:16:08Z")
[marvin-hansen](https://github.com/marvin-hansen) _commented_ : @artooro Have you looked into switching traffic through ingress routing to reduce deployment risk? [https://konghq.com/blog/reducing-deployment-risk-canary-releases-blue-green-deployments-kong/](https://konghq.com/blog/reducing-deployment-risk-canary-releases-blue-green-deployments-kong/) Not perfect, but it seems practical compared to service switching. > where you have to export, and import to bring the new deployment up. Back in the days, some Linux admins used to solve that with dsh, the “Distributed Shell” by piping the export stream from one box directly into the import of another box. As long as you stay within the same K8s cluster, this might do the trick relatively fast. > read-only mode 100%, much needed. Would you fill a separate issue? I am actually having the case that I can’t upgrade much after staging and it’s already causing quite some operational headache. I guess that’s what you get when you adopt early tech. On the other hand, my experience comes down to, when it runs, it actually runs well. ---
**Author:** ![diggy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/diggy/32/3666_2.png) [@diggy](https://discuss.dgraph.io/u/diggy)\ **Post date:** [January 29, 2020, 10:30pm UTC](https://discuss.dgraph.io/t/improve-dgraph-upgrade-experience-by-supporting-in-place-rolling-upgrades/8821/5 "2020-01-29T22:30:03Z")
[campoy](https://github.com/campoy) _commented_ : Hi there, This is totally part of our roadmap, and just in case people missed that @sleto-it is part of our Customer Success team. Additionally, we recently improved our helm charts to help with initial deployments. Finally, we are working on having a Kubernetes operator too, which should help with these upgrades. All of those things will be coming up during 2020, and if you have some specific demands this is a great place to leave your feedback, as it will be useful to prioritize the many ways we can improve your experience. Thanks for all of those that already did that on this issue, btw! You rock! ---
**Author:** ![diggy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/diggy/32/3666_2.png) [@diggy](https://discuss.dgraph.io/u/diggy)\ **Post date:** [January 29, 2020, 10:38pm UTC](https://discuss.dgraph.io/t/improve-dgraph-upgrade-experience-by-supporting-in-place-rolling-upgrades/8821/6 "2020-01-29T22:38:53Z")
[artooro](https://github.com/artooro) _commented_ : @marvin-hansen yes I’m aware of the various deployment methods. Databases are a bit different than stateless web apps as well. 🙂 I’m excited to see eyeballs on this type of capability. Also glad to see @campoy part of the team. ---
**Author:** ![diggy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/diggy/32/3666_2.png) [@diggy](https://discuss.dgraph.io/u/diggy)\ **Post date:** [January 31, 2020, 8:48pm UTC](https://discuss.dgraph.io/t/improve-dgraph-upgrade-experience-by-supporting-in-place-rolling-upgrades/8821/7 "2020-01-31T20:48:47Z")
[sleto-it](https://github.com/sleto-it) _commented_ : Hi all, As always, thanks for sharing your feedback and experience Generally speaking, regarding the read-only mode, what you can do is to restart all your alpha instances using the option _–mutations disallow._ You will need to add this option and restart **all** your alphas if you want the entire cluster to be in read-only mode Depending on how you deployed Dgraph, e.g. manual start / systemd / k8s etc, the specific way to set this option can be different Option ref: [https://docs.dgraph.io/master/deploy/#restrict-mutation-operations](https://docs.dgraph.io/master/deploy/#restrict-mutation-operations) Hope this helps Thanks, ---
**Author:** ![diggy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/diggy/32/3666_2.png) [@diggy](https://discuss.dgraph.io/u/diggy)\ **Post date:** [February 1, 2020, 8:26pm UTC](https://discuss.dgraph.io/t/improve-dgraph-upgrade-experience-by-supporting-in-place-rolling-upgrades/8821/8 "2020-02-01T20:26:20Z")
[artooro](https://github.com/artooro) _commented_ : Great tip @sleto-it, I will be using that method on my next upgrade, as our live backend only does read-only transactions, while an event driven micro service handles all the writes (mutations). So if the events backlog while the upgrade is happening that’s perfectly acceptable in my case as longs as reads continue to work. ---
**Author:** ![diggy](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.dgraph.io/diggy/32/3666_2.png) [@diggy](https://discuss.dgraph.io/u/diggy)\ **Post date:** [February 24, 2020, 12:39am UTC](https://discuss.dgraph.io/t/improve-dgraph-upgrade-experience-by-supporting-in-place-rolling-upgrades/8821/9 "2020-02-24T00:39:22Z")
[marvin-hansen](https://github.com/marvin-hansen) _commented_ : Is there some progress on this issue? I’m trying to upgrade right now, and, quite frankly, the process doesn’t even make any sense. Is there a walk-trough the process because, in its current state, the documentation leaves out so many important details and raises so many questions. I filled [yet another issue](https://github.com/dgraph-io/dgraph/issues/4839), but I find it astonishing that something as basic as import/export has been made so complicated. As a preliminary workaround, can at least the documentation get some improvement so that one can do an export / import within ~20 minutes for a small DB?