System Status
Outage
Between 22:12 UTC (00:12 Berlin time) and 22:45 UTC (00:45 Berlin time), there were four periods of up to 5 minutes with elevated error rates. During those periods, between 10 and 20 % of asynchronous actions (such as adding products to a cart, placing orders) have resulted in an error. Since 80-90% worked, retrying usually resolved the problem for users.
We've been able to narrow down the problem to an update of a system component we deployed yesterday today around 9:15 UTC (11:15 Berlin time). We have rolled back this system upgrade. However, during the rollback, an unrelated error with our deployment pipeline manifested because a third-party service provider changed their API rate limits. We have implemented a workaround and were able to complete the rollback at 23:40 UTC (01:40 Berlin time).
The issue has been resolved.
We are deploying a solution. The system is mostly functional.
We see some async actions (like adding a ticket to a cart, emptying a cart) failing. We do have a suspect (a recent update of a third party library) and are investigating.
Our team has confirmed that an incident is currently impacting our system. We are now looking into solutions and will update this page as soon has we have relevant information to share.
Our automated monitoring system has detected an outage. Our team has been alerted and is looking into the issue. We will update this page if we have relevant information to share.