Incident Report July 28, 2020
During release of gRPC changes, midway we realised that the Introspection Query was broken because of which Docs section in API Explorer was not loading and GraphQL clients like Altair was throwing a
no schema added error.
2. Why it happened?
During the release, we updated the Alpha image with the slash tag. We assumed that this would be the most recent master. Our assumption was wrong and that caused an old bug (that was fixed in the latest master) to show up in production.
3. Who was affected by it?
All users who were using the Slash GraphQL service between 3:30 pm IST - 4:40 pm IST were affected by this incident. The incident lasted a bit over an hour.
4. How do we ensure it does not happen again?
There are two things that we will do to make sure this does not happen again.
Firstly, we will check the slash image build date before we update the Alpha images, deploy it in staging and thoroughly test it for any kind of regression.
Secondly, we will be adding the introspection query as part of our API Test suite that will run on our CI. If the query fails, we will know about it.