If you have not already seen it, experiences it, or read about it, working to head off another reflection DOS vector. This time it is memcached on port 11211 UDP & TCP. There are active exploits using these ports. Reflection attacks and the memcached is not new. We know how reflection attacks work (send a spoofed packet to a device and have it reflected back (yes please deploy source address validation and BCP 38).
Enterprises are also asked to update their iACLs, Exploitable Port Filters, and Firewalls to track or block UDP/TCP port 11211 for all ingress and egress traffic.
Deploying these filters will help protect your network, your organization, your customers, and the Internet.
Please do take a look at the cloudflare blog specifically as they name and
shame OVH and Digital Ocean for being the primary sources of mega crap
traffic
I question whether there is *any* high volume hoster out there that has a
reputation for successfully addressing abuse issues coming from their
customer base, and cuts off services... By high volume hoster I define it
as companies where anybody with a credit card can buy a $2 to $15/month
VPS/VM in a fully automated process.
OVH just happens to be one of the largest and probably ranks in the top 10
worldwide by number of hypervisors and VPS. I doubt whether any of their
30-40 competitors that are smaller than them do much better, considering
the ratio of clued and attentive staff to VMs.
OVH are worse than that. Floods of the same spam coming from the
*same IP addresses* for years at a time. Continuous probes. A total refusal
to police their network or even respond to reports of issues.
They're not a major source of abuse because of their size, it's because
they've chosen to be.
Please do take a look at the cloudflare blog specifically as they name and
shame OVH and Digital Ocean for being the primary sources of mega crap
traffic
'It should also be noted that memcached priming queries can also be directed towards TCP/11211 on abusable memcached servers. TCP is not currently considered a high-risk memcached reflection/amplification transport as TCP queries cannot be reliably spoofed.'
We also recommend implementing situationally-appropriate network access policies at the IDC edge which disallow unwanted UDP/11211 as well as TCP/11211 from reaching abusable memcached deployments.
OVH is one of the largest server providers in the world - of course they will be at the top of that list.
What exactly should they do, according to you?
Read their abuse@ alias. Shut down those customers who are being abusive.
Currently they do neither. Every so often they'll privately admit that they've been
doing an unconscionably bad job of mitigating abuse from their networks and
promise to do better, then don't.
Given some of their customers have been consistently abusive for years from the same
domain and the same IP address the problem isn't "Oh, the bad people keep
signing up with new credit cards! Oh, poor us!" or any other reasoning based on
being a large, inexpensive provider.
Why should people de-peer them?
If the overall cost of the bad traffic exceeds the benefit of the good traffic. I'm
sure it doesn't, but that people are even suggesting it is telling.
OVH is one of the largest server providers in the world - of course they
will be at the top of that list.
What exactly should they do, according to you?
They should have rough norms enforced on their traffic behavior ,
especially around udp. If they do 1tbs of udp, they should police to 3tbs
or something similar.
Why should people de-peer them?
Abuse. But abuse happens, failure to fix chronic is a reason to depeer...
not one off. Personally, i do not peer with ovh because i need my upstream
to rtbh their traffic.
Let's hope so. There are two, and only two possibilities.
1. They know what's going on in their operation. In that case, they're
fully aware of all the spam, phishing, attacks, abuse, etc., and are
choosing to actively support and encourage them because it's profitable.
2. They don't know what's going on in their operation. In that case,
they're amazingly ignorant and stunningly incompetent -- because we
all know what's going on in it and we're not even there.
But in either case, the operational impact on us is exactly the same
and so is the solution.
Of course not. The larger an operation, the greater its responsibility
to the rest of the Internet -- because the more damage it can and will do
if it's not professionally operated. Moreover, the larger the operation
the easier abuse control is and the less tolerance any of us should have
for failure. I don't want to hear any clueless whining from ignorant
newbies about how-oh-so-terribly-hard-it-is-with-a-billion-users. If you
don't know how run it, or you're too lazy, cheap, and stupid to run it,
you should never have built it: you're simply not good enough.
I ran a full scan of the internet with zmap to find vulnerable memcached servers from an AWS server. AWS received an abuse report and forwarded it to me.
I deleted the VM and the case was close...
LOL
OVH Is not dumb. Do you know how easy it is to deploy a VM today with all the automated frameworks?
Before the group takes on the pitchforks and torches and travels down to
the hosting providers' headquarters - let's take a step back and look at
the root of this issue: the memcached software has failed both the
Internet community and its own memcached users.
It is INSANE that memcached is/was[1] shipping with default settings
that make the daemon listen and respond on UDP on INADDR_ANY. Did nobody
take notes during the protocol wars where we were fodder for all the
CHARGEN & NTP ordnance?
The memcached software shipped with a crazy default that required no
authentication - allowing everyone to interact with the daemon. This is
an incredibly risky proposition for memcached users from a
confidentiality perspective; and on top of that the amplification factor
is up to 15,000x. WHAT?!
And this isn't even new information, open key/value stores have been a
security research topic for a number of years, these folks reported that
in the 2015/2016 time frame they observed more than 100,000 open
memcached instances: https://aperture-labs.org/pdf/safeconf16.pdf
Vendors need to ensure that a default installation of their software
does not pose an immediate liability to the user itself and those around
them. No software is deployed in a vacuum.
A great example of how to approach things is the behavior of the
PowerDNS DNS recursor: this recursor - out of the box - binds to only
127.0.0.1, and blocks queries from RFC 1918 space. An operator has to
consciously perform multiple steps to make it into the danger zone.
This is how things should be.
I want to add one software vendor, who is major contributor to ddos attacks.
Mikrotik till now shipping their quite popular routers, with wide open DNS recursor,
that don't have even mechanism for ACL in it. Significant part of DNS amplification attacks
are such Mikrotik recursors.
They don't care till now.
I have mixed experiences with Mikrotik, but I don't think they would do such a stupid thing. A friend of my has three offices and each one has mikrotik to form tunnels and one domain for all the company.
He is not too IP savvy, so he copy-pasted the VPN config from internet and left the rest as it was. His routers are not open DNS resolvers.
When I asked them I got no reply and their logs showed:
_drop input: in:ether1 out:(unknown 0), src-mac 00:AB:CD:81:c2:71, proto UDP, AAA.47.138.134:9082->BBB.146.251.103:53, len 51
His settings showed the DNS server ON with all the queries for the local network and he actually had a toggle "allow remote queries" on, but his routers were not open resolvers.
