A deploy to matrix.org encountered unexpected internal errors serving /messages API calls. The scope of those errors is yet to be identified, pending further investigation.
We noticed the problem at 11:20 AM UTC and decided to rollback the deployment at 11:41AM UTC. Doing so with the normal rolling deployment procedure unexpectedly caused restarts of all worker processes, starting at 11:50AM UTC. This has not been seen before and also needs further investigation. To restore service quickly we immediately started a bulk, non-rolling restart. This completed at 12:59 PM UTC and normal service was restored from this point onwards.