Disruption in handling a portion of incoming requests

Incident Report for Matrix

Resolved

The process has caught up with the backlog of traffic, and matrix.org Synapse should once again be fully functional.
Posted Jun 13, 2022 - 13:56 UTC

Update

We are continuing to monitor for any further issues.
Posted Jun 13, 2022 - 13:30 UTC

Monitoring

The issue was identified as related to a recent Synapse deployment (use the right virtualenv folks!). We have corrected the issue and are now monitoring the situation.

Requests should resolve, although may take longer than expected while caches warm up and lingering requests are served.
Posted Jun 13, 2022 - 13:29 UTC

Investigating

We are currently investigating the issue, starting with the load balancer.
Posted Jun 13, 2022 - 13:22 UTC
This incident affected: Matrix.org (Synapse).