Load Balancer failure

Status: Resolved – Affected systems: Applications

Start: 19.12.2022 06:47 UTC – End: 19.12.2022 06:53 UTC

Description

Our load balancers are currently not functioning properly. We are investigating and will share new details as soon as we have them.

Created: 19.12.2022 06:47 UTC

Resolved

We have fixed the underlying issue and everything is back online and accessible again. Please open a support ticket if you have any remainig issues. We are very sorry for the inconvenience caused.

Created: 19.12.2022 06:53 UTC

Post Mortem

Our load balancers are available through ALIAS/ANAME/CNAME records and resolve to Floating IPs that are redirected to the currently active and available load balancers. The load balancers are connected to each other and notice if one load balancer becomes inaccessible and is not responding to requests properly by using VRRP. Our platform is built in a way that the Floating IPs are reassigned automatically to active load balancers if there is a failure. This process is robust and has served us very well in the past. The issue today was an edge case that assigned the Floating IP to an instance that didn't have the IP available, so that approximately 50% of all requests were not responding properly. This edge case has been identified immediately and is fixed by now.

Created: 19.12.2022 07:30 UTC