We've been seeing significant attack activity from Amazon over the last two months, involving apparently compromised instances that commonly send 1-10G of traffic per source and together generate Nx10G of total traffic. Even when our overall upstream capacity exceeds an attack's overall size, the nature of load-balancing over multiple 10G upstream links means that an individual link can be saturated by multiple large flows, forcing our systems to null-route the target to limit impact.
We've sent an abuse notification about every traffic source to Amazon, and specific sources seem to stop their involvement over time (suggesting that abuse teams are following up on them), but there is an endless parade of new attackers, and each source participates in many damaging attacks before it is shut down.
Is there anyone at Amazon who can help with an engineering solution in terms of programmatically detecting and rate-limiting attack traffic sources, to our networks or overall? Or applying the kludge of a rate-limit for all Amazon traffic to our networks? Or working with us on some other option?
At least one other large cloud provider has an automatic rate-limiting system in place that is effective in reducing the damage from repeat high-volume attacks.
Emails to the Amazon NOC, peering contacts (since that would be another possible solution), and abuse department have not connected me with anyone.
Yes, RTBH is used to distribute the null-routes that I mentioned.
Unfortunately, even brief saturation events lasting just 5-10 seconds (a typical amount of time to detect the loss, issue the null-route, and see the traffic start to fall off as it is distributed upstream) can cause real damage to those customers who are sensitive to latency and packet loss. So while null-routes limit the duration of the impact, they can't eliminate it entirely. And, of course, the actual target of the attack -- the now-null-routed IP address -- becomes unreachable, which was presumably the goal of the attacker.
As mentioned before, I am open to peering (where possible) but have not received a response.
My goal is to connect with someone at Amazon and work with them on a technical solution, which is why I posted asking for that here. Various factors mean that we can't just upgrade our way out of this one, and manual whac-a-mole on the sources has shown to have limited use.
These attacks also impact Amazon and the networks in between the sources and targets, and they take time to handle by the abuse teams, so there are good reasons to investigate them further and find better ways to mitigate or prevent them.
Especially considering the fact that Amazon is just a bit selective about their peers. Though, the size of our border probably had a lot to do with the fact that we didn’t get a response the first time when we requested peering for our 10s of megabits of traffic, we none the less didn’t get a response the first time we tried. Not even a “go away until you have some traffic of significance.”
In the end, we had to implore a back channel to get a peering session set up.