At 13:10 UTC a new deployment of Synapse was rolled out to matrix.org. We were alerted to increased failure rates and a drop in database throughput at 14:20 UTC. During investigation we identified that users started to see failing requests from 13:40 UTC onwards.
At 14:30 UTC we decided to roll back the deployment. Error rates improved starting at 14:35 UTC, and have returned to normal now that the rollback has completed (at 14:58 UTC). We are following up by investigating the root cause.