Users on unable to send messages
Incident Report for Matrix
At 11:15 UTC we performed a scheduled restart of the Synapse instances serving

After a successful redeploy of our test worker, we proceeded to restart the remaining workers across various machines. By the time the last worker was restarted at 11:22 UTC we realised that users hosted on were unable to send messages from their clients.

At 11:35 UTC we identified the culprit: an exception caused by a configuration change. We completed a rollback and restarted all Synapse instances by 11:44 UTC, at which point normal service resumed.
Posted Oct 15, 2021 - 11:15 UTC