I know Tier1s are blackholing traffic all the time (de-peering, congestion etc.)
but did it became a new role for Tier1s to go from transit provider to
transit blocker?
We received recently customer complaints stating they can't reach certain websites.
Investigation showed that the sites were not reachable via Tier1-T, but fine via
Tier1-L. I contacted Tier1-T and the answer was something like "yeah, this is a known phishing
site and to protect our customers we blackhole that IP" (btw - it was 2 ASes away from Tier1-T).
Huh? If I want to block something there, it should me my decision or that of my country's legal
entities by court order and not being decided by some Tier1's intransparent security department.
(Not even mentioning words like 'CGN', 'legal', 'net neutrality' or 'censorship') This might be
an acceptable policy for a cable provider but not for a Tier1.
Haven't seen something like this in many years. Did I miss a pardigm-shift here and has this
become a common "service" at Tier1s?
Ideally what should a Tier 1 or default-free network do in this
situation[1]?
1) Do nothing - They're supposed deliver any and all bits (Disregarding
a DoS or similiar situation which impedes said network)
2) Prefix filter - Don't be a party (at least in one direction) to the
bad actors traffic.
3) ?
[1] Assuming there is some sort of security and/or wrongdoing event that
isn't getting resolved via contact with their peer.
While I like that plan, there are a LOT more people who will scream about not being "protected" than those who will bitch they can't get to a phishing site.
Since networks are for-profit companies, they'll lower their costs (e.g. support calls), as long as it lowers their cost more than the "cost" of losing a customer or two (and let's be honest, that is about all they _might_ lose) who are religious about the whole "transit means everywhere" thing.
I vaguely recall having the same sort of problem many years ago with Above.net transit. IIRC, the sentiment back then was similarly that this was inappropriate behavior for a Tier1/2 transit provider. If you're going to propagate the routes, deliver the traffic. I suppose an argument could be made though that if there's phishing or malicious traffic targeting your customers from a single IP, it could be appropriate to blackhole the IP rather than reject the advertisement for an entire CIDR.
Even if said packets from an obviously compromised server on a high-speed link are attack packets causing problems for the ISP itself as well as for its customers?
Trust me, large transit ISPs don't *want* to be in the blackholing business. They only do so when they're forced into it by necessity (operational, legal, regulatory).
Also note that in the case of the server(s) you can't access, they may well be on shared hosting with thousands of sites/accounts on a single IP, one or more of which may be compromised.
A) Is it impossible for an end user or business (i.e. non-ISP) to get a direct connection to a "Tier 1" (whatever the hell that means) provider.
B) Most traffic on the Internet traverses Tier 1s today.
C) A Tier 1 has a different profit motive than a Tier 2 (whatever the hell that means) providers.
D) All Tier 1 providers are larger than all Tier 2 providers.
1) Do nothing - They're supposed deliver any and all bits
(Disregarding
a DoS or similiar situation which impedes said network)
2) Prefix filter - Don't be a party (at least in one direction) to the
bad actors traffic.
3 - Deliver all packets unless I've signed up for an enhanced security
offering?
right - I see this really as something that should be decided at the edge
of the internet (Tier2+) and not in the core.
You seem to have odd ideas about what it means to be a settlement free provider. Most of their customers are not smaller internet service providers.
I think I agree with this, and I think it can help draw a useful line.
Large DDoS attacks can and do directly affect the service that the
"tier 1" is providing to its customers (namely, moving their bits), so
filtering such attacks seems like a reasonably agreeable thing by
really anyone I think.
Phishing on the other hand will not really stop bits from moving
(except perhaps through rather long chain of unlikely things that'd
have to happen).
The last-mile consumer ISPs don't just "move bits" for their customers
really, its more about providing "internet" (which is a different
concept to normal users) -- and this is where filtering phishing sites
and blocking port 25 and such makes much more sense, because these
users will have a highly degraded experience if they become a botnet
drone or some such thing.
Granted, as Patrick says, "tier 1" isn't really a thing, and they have
a mix of customers, but I think its safe to say that these "tier 1"
providers should apply different policies for different types of
customers, because they are providering different services (even if
the underlying technology is the same/similar).
If the phishing attack is against an enterprise that is also an ISP, surely you can imagine a case where they might block traffic to prevent folks from being phished.
i think it's great that someone is blocking folks from being infected with either malware or giving up their private details improperly.
Typically these sites are hacked anyways or something else. I think that keeping the broadest set of people from being phished or compromised is a good thing(tm). Typically a site is cleaned up in a few hours or day or two without trouble. If your communication is that urgent, there are other methods like phone to communicate with the other party. not ideal, but they do exist.
Patrick, what I mean is that someone that I pay money for providing me access to
the guys I don't peer with, decides for me what's good (according to his criteria) for
me and my customers or even my customer's customers etc. If one of my peers
blackholes his customers, it's his business and not mine and I don't care.
While I eventually could vote with my wallet if I don't like that policy, my question was more,
if that behavior is already that common at 'Tier1s' (definition omitted) that it would not make
a difference anyway.
On the one hand, companies providing Internet transit are not
generally compelled by law to pass packets for any other given company
on the Internet.
On the other hand, announcing via BGP that you will carry particular
packets and then intentionally dropping them on the floor could easily
be construed as tortious interference.
The middle ground... propagating a BGP announcement but blocking a
small piece within it... I think I'd want to cover my backside by
setting a BGP community on that route which advised my peers that a
portion of it is dead-routed within my network so that they may
discard or deprioritize it if they choose.
my vendors deliver software fixes for "BGP" doesn't work in weeks, so I think that the following timeline and process I'm going to outline exceeds their BGP problems.
0 hour - Issue Reported
0-24 hours - triage; send to customer/internal customer to mitigate/remediate
25-48 hours - Customer responds, host taken down if hacked, etc..
48-96 hours+ - If no response, IP null0'ed per AUP as network security risk
48-96 hours is also where the customer freaks out and quickly fixes their problem to come in compliance with AUP.
This is a natural process. Null0 or ACLs don't stay up for days or weeks on end. That doesn't mean this catches 100% of all cases, but many ISPs get a daily report of phishing sites and malware hosted on their network each morning. You can get one too!
You can get a daily ATLAS report from Arbor as well: http://atlas.arbor.net/ (Although I can't get anyone to fix a problem with it, so anyone there can email me if you have the power to fix it).
There are other aggregators of data as well, such as SIE. If you don't know the health of your network, take a look. Many folks will email you these reports automatically, or provide you a direct feed (some in realtime, such as SIE).
1) Do nothing - They're supposed deliver any and all bits
(Disregarding
a DoS or similiar situation which impedes said network)
2) Prefix filter - Don't be a party (at least in one direction) to the
bad actors traffic.
3 - Deliver all packets unless I've signed up for an enhanced security
offering?
right - I see this really as something that should be decided at the edge
of the internet (Tier2+) and not in the core.
You seem to have odd ideas about what it means to be a settlement free provider. Most of their customers are not smaller internet service providers.
I know what it means to be a customer of $LargeGlobalISPthatsellsTransittootherISPs since
1995 and I have *never* seen one of these guys blackholing
single IPs on their own (and I'm not talking about RTB, botnet controllers that threaten to kill
the internet etc.). Now since a few weeks we get regular complaints about this. So something has changed.
The sensitive approach would really be to make this an opt-in service for their customers
and not a default service without opt-out option. In times of CGN and hundrets or thousands of
websites behind one IP, blocking addresses is not the right answer to the phishing problem.
Yes, things have changed. There are reasons that some of the transit ISPs are performing this blocking. They aren't doing it for kicks.
For example, there are non-insignificant numbers of servers/accounts which have been compromised and used to launch large-scale, high-impact DDoS attacks. The negative impact of allowing these servers to emit attack traffic far outweighs the inconvenience experienced by a few end-customers trying to access these servers (which are compromised, anyways, and therefore it isn't a good idea to try and access them in the first place).
Suggest you ask the transit ISPs in question directly. You aren't likely to get an authoritative answer on a public email list.
(de-peering, congestion etc.)