IGC (Internet Gateway Connections, an ISP in Florida that was
multihomed between NET99 and MCI) just had an unexpected outage lasting in
*The lie:*
excess of an hour. This outage was deliberately engineered by AGIS/NET99.
*The fact:*
IGC's routes were pulled from the Net99 net within 90 minutes of their being
turned off *at IGC's request*.
First, a little history:
IGC's first 'Net T1 was to NET99. Instead of connecting us
directly to their backbone (as we had expected) they connected us to
Netrunner (one of our competitors). Netrunner/NET99 only has a single T1
out of Miami (to Atlanta) and as a result, our throughput was awful and
we obtained an additional T1 to MCI.
This was common in the Net99 network. I guess it worked OK at first, but it
didn't scale as customers grew. There was a tremendous hoo-ha back when
AGIS bought Net99 because some people who were in mid-install were
redirected to AGIS. That hurt the pending customers, but it helped the
existing customers like IGC as much as it could.
*The lie:*
A few days ago, someone at Netrunner got upset with someone at
NET99/AGIS and shut off _our_ service. For some reason (we strongly
suspect a configuration error at NET99) our traffic failed to reroute
over our MCI T1 for several hours. (They have a habit of broadcasting
static routes to route around outages, thus completely screwing their
multi-homed customers.)
*The truth:*
The referenced disconnection did occur, and was restored. This kind of
problem is a very real danger of the way Net99 colocated in certain cities.
That's why AGIS only colocates in POP's and private spaces. And in the
event that a customer is dually homed and not running BGP, it's incumbent
upon the customer to make sure that *both providers* are aware of the fact
and properly setting local prefs. IGC did not do that and bears full
responsibility for any failure to route properly during an outage.
Aha: a true statement:
We asked AGIS for a one month refund to compensate us for our
losses due to the sabotage. Instead, they issued us a _one_day_ credit.
Ah, another true statement:
We got upset, both with the ability of our T to pass traffic, with
NET99's continual routing problems, with NET99's backbone being
disassembled from under us, and with NET99/AGIS's continual delays in
migrating us to the AGIS network. We asked them to discontinue our service.
True, but we certainly wouldn't have left them without a circuit. We've
tried mightily to install a new circuit to their premises, and ran into a
b8zs facilities problem. We could have installed it AMI as a temporary
measure with no problem.
*Another LIE*
They did. They also deliberately and maliciously began
broadcasting more specific routes for the individual class C networks in
our CIDR block. (We confirmed this with MCI.)
*The truth:*
Over the months various static routes have been installed by various people
on Net99 to route around some problems (a practice which predates our
purchase). You run into some curious situations like a static on a Chicago
router pointing to Phoenix. In this case, there were statics of differing
prefix lengths on different routers. The aggregate address statement got
pulled, and we didn't catch the more specifics on another router. As soon
as we found out what happened, we pulled them. In fact, we pulled them
while we were on the phone with MCI so that they could confirm that they
were gone.
The statement of malicious intent is untrue and libellous and must be
withdrawn immediately by Mr. Katz.
[...]
*Another LIE*
I contacted MCI again and they contacted AGIS/NET99 with the
ultimate threat being for MCI to stop carrying AGIS' traffic if they would
not cease broadcasting the bogus routes. Shortly after being contacted by
MCI, AGIS/NET99 stopped announcing the bogus routes to MCI, but continued
advertising these routes to their other peers at the NAPs.
Nope. Like I said, MCI didn't threaten us at all, and worked with us to see
that the routing was correct. Net99 isn't announcing those routes anywhere.
Both AGIS and Net99 announce all routes at all exchange points to all
providers in compliance with our peering contracts. If MCI had called back
and said, "we still have a problem," we'd work with them until it was taken
care of.
Fortunately, MCI was able to reconfigure our access lists at their
router to allow us to broadcast competing more specific routes for each
individual class C in the CIDR block. Thus (more or less) restoring IGC's
net connectivity.
More specifics aren't competing. They take precedence. Sometimes it's
better to be silent and thought dumb...
Ah, more libel:
NET99's deliberate, malicious actions are contrary to the very
spirit of the Internet. They resulted in DALnet (the third largest IRC
network) being split to ribbons, depriving 2,500 people of the ability to
converse with each other and cutting off DALnet's security and access
control system which is hosted here.
Another retraction required ASAP, Joel.
AND NOW, here's what Joel left out:
IGC called our NOC up, screaming, the night this occurred, and "put us on
notice" that "if we didn't disconnect our circuit to them immediately [we
all know that phone companies do nothing immediately] they were going to
flood ping [another Miami customer] until we did." We had no reason to
disbelieve them so we did turn down the port, and had the routing unwound
within a couple of hours, which isn't bad given the nonexistent notice and
other things going on.
In other words, Joel comes to the court of public opinion with unclean
hands. Joel threatened an illegal act against another customer. Joel's
righteousness rings hollow. Joel must recant immediately.
Please excuse me if this is not the appropriate forum for discussing
this event.
And this is weird. The sig and the FROM don't match. Is this all a joke?