Internal server error serving /messages API calls on homeserver
Incident Report for Matrix
A deploy to encountered unexpected internal errors serving /messages API calls. The scope of those errors is yet to be identified, pending further investigation.

We noticed the problem at 11:20 AM UTC and decided to rollback the deployment at 11:41AM UTC. Doing so with the normal rolling deployment procedure unexpectedly caused restarts of all worker processes, starting at 11:50AM UTC. This has not been seen before and also needs further investigation. To restore service quickly we immediately started a bulk, non-rolling restart. This completed at 12:59 PM UTC and normal service was restored from this point onwards.
Posted Sep 28, 2022 - 10:30 UTC