We discovered an issue with our background jobs attempting to reprocess timed-out Dosespot requests excessively and frequently, leading to server resource exhaustion. Additionally, Dosespot experienced an incident tonight that caused a spike in their latency, resulting in most of our requests being redirected to them and timing out, which ultimately caused the problem on our end.

To address these scenarios and mitigate future issues, we implemented a few changes to our background jobs and adjusted our server’s scaling configuration.

We have been able to mitigate the issue and restore all systems to a working state. We are still investigating the root cause and will update this note as soon as we identify the issue.

We are currently investigating an issue affecting our API server in some regions.