DDoS attacks on our network, 28 to 31 October 2026
From Wednesday 28 October to Saturday 31 October, our network was the target of a series of large distributed denial of service (DDoS) attacks. This post explains how they affected you, how we stopped them, and the new automatic protection now running at the edge of our network.
From 9:35pm AEDT on Wednesday 28 October to 6:49am AEDT on Saturday 31 October, our network was the target of a series of large distributed denial of service (DDoS) attacks. They were aimed at our whole network and came back in waves about once an hour. During the larger waves, some of you saw short bursts of packet loss and higher latency.
The incident is resolved, and a new automatic protection system now filters attack traffic at the edge of our network within seconds. We know how disruptive this was, especially in the evenings. This post explains what happened and what we have changed.
What happened
A DDoS attack tries to overwhelm a network by flooding it with traffic from a very large number of machines at once, usually computers and devices that a botnet has taken over.
Our monitoring picked up the first signs at 9:35pm on Wednesday (the red box below). Our upstream provider, GSL, also detected the attack automatically and alerted us proactively. The attack they first reported was about 1 Tbps, and at 10:05pm they put our whole network into high protection mode, so that their scrubbing filters the traffic before it reaches us.
Timeline
All times are AEDT.
| When | What happened |
|---|---|
| Wed 28 Oct, 9:35pm | First signs of the attack in our traffic graphs. |
| Wed 28 Oct, 10:05pm | GSL puts our whole network into high protection mode. Waves follow about once an hour, and some are aimed at our routers. |
| Wed 28 Oct, 11:00pm | A mitigation step cuts our DHCP servers off from the public DNS resolvers 8.8.8.8 and 1.1.1.1, so they cannot reach our API. A connection that starts in the next 11 minutes may not come online. |
| Wed 28 Oct, 11:11pm | DHCP is back to normal. |
| Thu 29 Oct, 12:36am | We find that the attack also targets our routers' own addresses and their control plane (the part of a router that runs routing). |
| Thu 29 Oct, 1:30am | Control plane protection on our routers, and filtering of traffic to our routers' addresses at GSL, are in place. |
| Thu 29 Oct, 1:35pm | A large wave. Some of the routers that serve customer connections drop packets, even to destinations inside our own network. Our DNS resolvers are affected too, but their cache keeps answering. |
| Thu 29 Oct, 3:53pm | We confirm the cause: load on the forwarding plane of those routers. We add more filtering. |
| Thu 29 Oct, 8:00pm | Another wave, and reports of problems in online games and apps such as VRChat, caused by the extra filters. |
| Thu 29 Oct, 9:00pm | We remove those filters and start building a more precise approach. |
| Fri 30 Oct, 2:30am | Another wave. |
| Fri 30 Oct, 2:37pm | We confirm that the new automatic protection works: during a wave, the overload lasts under a minute. |
| Fri 30 Oct, 10:30pm | One of the largest waves: about 800 Gbps and 100 million packets per second, almost all of it kept out by GSL. About 10 million packets per second reach us and are filtered at our edge. |
| Sat 31 Oct, 6:49am | With the protections holding, we close the incident. |
Why it happened
The attack was spread across our whole network and came from a very large number of sources at once. Each source sent only a little traffic, so on its own none of it looked unusual. Added together, it reached hundreds of gigabits per second.
GSL kept the bulk of it out, but the share that still reached us carried a very large number of packets, and every packet has to be handled by the router it arrives at. During the peaks, the forwarding plane of some of our routers (the part that passes each packet on toward its destination) was busy with attack traffic, and ordinary traffic waited or was dropped. That is why the packet loss reached even services inside our own network, and why it came and went with each wave.
How we fixed it
With GSL
GSL's scrubbing lets legitimate traffic through and keeps the botnet traffic out, and it absorbed the bulk of every wave.
Automatic protection at our edge
Our first filters on Thursday helped, but they also caught legitimate traffic, so we replaced them with something more precise. The new system watches the traffic that comes into our network from GSL, all the time. When an attack starts, it picks out the attack traffic and has our edge routers drop it as it arrives, within seconds and before it reaches the routers that serve your connection. It is built to leave legitimate traffic, such as games and calls, alone, and it acts only during an attack.
The chart below shows the load on the forwarding plane of the routers that serve customers on 30 October. The 2:30am wave, before the new system, kept them busy for 5 to 10 minutes. The afternoon wave, with it, for about a minute.
With the new protection in place, you may still notice a slight latency spike when a wave starts, but nothing compared to the 40 to 50% packet loss of the earlier waves. The chart below shows the night of 29 to 30 October: the spikes around 2am and 3am are the 2:30am wave, the last one before the new protection was in place, and packet loss over the whole night was 0.04%.
What we're doing next
- DHCP without DNS. Our DHCP servers will no longer depend on DNS at all, public or our own, so a DNS problem cannot stop your connection from coming online.
- The new protection stays on, and we are extending it to every point where traffic enters our network.
- Faster alerts. Our engineers will get an alert as soon as the protection acts, with a dashboard that shows what it is doing.
- The early measures stay in place: high protection mode at GSL, filtering of traffic to our routers' addresses, and control plane protection on our routers.
Closing
Thank you for your patience while we worked through the attacks. We are pleased that our network is more resilient as a result.
We would also like to thank the team at GSL. Their engineers, spread across the globe, worked around the clock with us, 24 hours a day, from their first alert to the end of the incident. They helped us analyse the attacks, which steered our own solution in the right direction. We are grateful for their help, and we highly recommend GSL as a trusted DDoS protection partner with a unique scrubbing capability.
If you still see problems with your service, please contact us.