I'm hoping someone has had the same experiences, and is further toward a
resolution on this than I am. About 6 months ago, we noticed that XO was
blackholing one specific IP out of a /24. Traces to that IP stopped on
XO's network, traces to anything else out of the block went through fine.
XO finally admitted that they had a new security system that identifies
suspicious traffic and automatically blocks the IP for 30 minutes. We had
to get the IP in question "whitelisted" by their security guys. The
traffic was all legit, it was just on a high port # that they considered
suspicious.
There have several more cases like this, and XO has not been forthcoming
with information. We're either looking to be exempted from this filtering
or at least get a detailed description of how the system works. I'm not
sure how they think this is acceptable from a major transit provider.
Anybody else had similar problems?
Clayton Haydel
"transit provider". Is XO the end-access provider for either you or the
destination site? Or are both of those on some other connection, and XO
is a bystander along the way?
Cheers,
-- jra
"transit provider". Is XO the end-access provider for either you or the
destination site? Or are both of those on some other connection, and XO
is a bystander along the way?
We're a direct customer. The IP's that I've seen them block have been
both on our network and on remote networks, so I suspect their filtering
would affect any traffic that happened to pass over XO.
Clayton Haydel
Oh yes! Good lord I about went insane with this. I was working with a customer single homed to cBeyond. I spent 3 hours on the phone with cBeyond to figure out what was going on, it looks like a broken route. Come to find out it was an XO "security null". The engineer on the phone from cBeyond said to me "Well, I have learned 2 things today. 1, XO nulls for 'security purposes' at random. 2, I am no longer shocked by any ridiculous policy I will ever come across again."
In this case majority traffic was going from cBeyond to anywhere (via XO) and being eaten, however it was VERY tough to diagnose as all parties involved assumed this would not be occurring between source and destination without good public documentation or at least any record of this happening to someone else. Also I guess we all assumed that major bandwidth players don't filter anything.
I personally think its good on paper, but very bad real life until there is a way to notify the end customer of the violation quickly. This issue literally took 3 full weeks to figure out what was going on. Yes this works great in a colo datacenter as you have the customer contact info (hopefully). But in the case where my customers provider was having the IP filtered by their transit it was hell to diagnose. In my case the customer had a single infected machine that was making outbound connections on TCP3389 in the range of about 100 connections every 5 minutes and because of this was entirely being "security nulled".
Blake
So if you want to launch a DoS attack against a specific IP address you spoof TCP3389 SYNs to networks single homed to XO and they will null it for you.
While troubleshooting another issue last week, someone in the NOC at one of our ISPs mentioned that they had encountered something similar recently.
"This
looks suspiciously like another XO issue we ran across in the last few
months where they used a network security device that blocked 'suspicious'
traffic on particular ports (although it was tcp based from what I could
remember)."
In our case the symptoms looked like GBLX was eating traffic which hashed to a certain theoretical link (certain src-dst-srcport-dstport combinations) in a LAG or similar, but it was happening right near the XO-GBLX edge in the forward path so it's possible it was a security device at XO's edge.
Ah, ok. Well, that certainly gives them standing to be filtering the traffic;
whether you think their reasoning is justified becomes a different level of
question at that point.
I concur with you that their filtering probably isn't justified, but I suspect
you'd find your contract permits it.
Cheers,
-- jra