Private Networking issues

Status: Resolved – Affected systems: Applications

Start: 10.02.2023 08:04:00 UTC – End: 10.02.2023 08:11:00 UTC

We are currently experiencing issues with our private network. Applications are currently unavailable through our load balancers because of this and show a 503 error. We are investigating.

Resolved
This issue is now resolved as internal connectivity was restored.

Load Balancer failure

Status: Resolved – Affected systems: Applications

Start: 19.12.2022 06:47:00 UTC – End: 19.12.2022 06:53:00 UTC

Our load balancers are currently not functioning properly. We are investigating and will share new details as soon as we have them.

Resolved
19.12.2022 06:47 UTC
We have fixed the underlying issue and everything is back online and accessible again. Please open a support ticket if you have any remainig issues. We are very sorry for the inconvenience caused.

Post Mortem
19.12.2022 06:53 UTC
Our load balancers are available through ALIAS/ANAME/CNAME records and resolve to Floating IPs that are redirected to the currently active and available load balancers. The load balancers are connected to each other and notice if one load balancer becomes inaccessible and is not responding to requests properly by using VRRP. Our platform is built in a way that the Floating IPs are reassigned automatically to active load balancers if there is a failure. This process is robust and has served us very well in the past. The issue today was an edge case that assigned the Floating IP to an instance that didn't have the IP available, so that approximately 50% of all requests were not responding properly. This edge case has been identified immediately and is fixed by now.