Covad SUCKS!

Mailman List Mirror (Read Only)
1

This is a blind message. I don't even know if it's getting out of my mail
hub. Theoretically, it should. You won't be able to reply. I can route out,
but nothing can get back in.

Some of you may remember a poster of this huge eagle stooping on a mouse.
The mouse was calmly standing, facing the eagle, flipping it the bird. The
title was "Defiance". Covad is the eagle, MHSC is the mouse, and this
message is the bird.

MHSC has been down since 7Feb01 and Covad is the culprit. MHSC pays its
bills, but Covad had a problem with my upstream. On Wednesday, 7Feb01, Covad
cut off all links and feeds to my upstream. I've been down ever since.
Everyone knows how quickly telcos turn up services, that's why co-lo centers
have customers. Covad cut off MHSC access, without warning, without,
compansation, and their SafetyNet program is the worst joke. I have a /24
and the new ISPs, that Covad is forcing me to use, can't assign a new /24.
Shit, one of them can't even assign one static address. The one that this is
going out thru tis trying to convince me that I can run 62 computers, using
130 IP addrs, on a /27 <heh>. We talked earlier about how non-portable
addresses suck. Here is the reason why. I either use one of their SafetyNet
providers or suffer up to 45 days of outage. Two weeks of warning and I
would have been able to avoid the outage. (BTW, my ISDN backup feed is dead
as well). I've tried contacting Covad, thier executive staff, etc. No joy
there, Covad doesn't care.

100% of MHSC business is directly dependent on our Internet feed. 100% of
February revenue is gone. I had two product roll-outs that were scheduled
for the following week. Both are now dead issues. MHSC does CA for some of
it's customers, renumbering invalidates all those CAs. Every lead we were
working is now gone, probably forever, and so are many existing customers.
I'm looking at probable business failure. MHSC could sue, but may not
survive long enough to collect, even if we win. If MHSC survives this then
"Damn, I good!"

I urge everyone, do not;
1) Recommend Covad as a DSL provider
2) Become a Covad partner.
3) Remain on Covad service, find someone else (Rythyms sound good)

Further, if you have any Covad stock ... sell it.

The simple reason is that Covad has shown themselves unworthy of trust. They
will cut off service arbitrarily, without direct cause, without compansation
or recourse, and without warning. This is NOT the sort of behavior desired
from an outfit in charge of your key infrastructure.

We are not large, MHSC cannot afford an $80K+ revenue hit, and that's not
counting the suits for failure to meet SLA.

At this point, I'm not worried about any Covad law suits. What are they
going to do ... put MHSC out of business? They've already taken care of
that.

I hope that this comes to the attention of Covad and I hope that they don't
like it! MHSC will also be filing complaints with the CA-PUC and everywhere
else that we can. But, a four-year process will not help MHSC stay in
business. MHSC is already preparing a Marketing Communication plan to get
this story out. MHSC tried dealing with this peacfully and got ignored, by
Covad. Covad needs to be shunned by the rest of the Internet community, as a
bad net-citizen.

MHSC wasn't the ony business effected. There were 800 other busnesses
involved in the cut off, that we know of. If any of you know anyone else
that has been effected by this outage, please let me know.

2

<rant snipped>

This is a blind message. I don't even know if it's getting out of my mail
hub. Theoretically, it should. You won't be able to reply. I can route out,
but nothing can get back in.

That's strange. As far as I know, TCP requires a two-way handshake. :slight_smile:

130 IP addrs, on a /27 <heh>. We talked earlier about how non-portable
addresses suck. Here is the reason why. I either use one of their SafetyNet

Portable address wouldn't help you, since safetynet'd ISP won't talk BGP
to you. What you could do is (with agreement of your current ISP who owns
the IPs) is to go into any colo centers, put up your hardware, and tell
the colo ISP to advertise your netblocks.

would have been able to avoid the outage. (BTW, my ISDN backup feed is dead
as well). I've tried contacting Covad, thier executive staff, etc. No joy
there, Covad doesn't care.

Its going from "We don't care, we don't have to" to "We don't care, we
can't afford to". Covad (and its wholesalers) experienced major collection
problems, and they are cutting off the wholesalers who don't pay their
bills.

100% of MHSC business is directly dependent on our Internet feed. 100% of
February revenue is gone. I had two product roll-outs that were scheduled

DSL is not supposed to be bet-the-business-on product. I'm sure it'll
sound silly now, but if you have 80k$ revenue, you could afford a T1 which
has very different ETR timelines compared to DSL, and that does not
involve so many companies involved in its delivery/billing process.

for the following week. Both are now dead issues. MHSC does CA for some of
it's customers, renumbering invalidates all those CAs. Every lead we were
working is now gone, probably forever, and so are many existing customers.

I may misunderstand something, but if CA means 'X509 certificate
authority', why would it invalidate anything? CAs have CN, which is a
name, not IP address.

I'm looking at probable business failure. MHSC could sue, but may not
survive long enough to collect, even if we win. If MHSC survives this then
"Damn, I good!"

I urge everyone, do not;
1) Recommend Covad as a DSL provider
2) Become a Covad partner.
3) Remain on Covad service, find someone else (Rythyms sound good)

http://biz.yahoo.com/p/r/rthm.html, the financial situation of rhythms is
not much better than Covad or Northpoint. According their current cash
burn rate, they'll run out of cash sometime this year.

Further, if you have any Covad stock ... sell it.

Too late, its around 2 dollars now.

The simple reason is that Covad has shown themselves unworthy of trust. They
will cut off service arbitrarily, without direct cause, without compansation
or recourse, and without warning. This is NOT the sort of behavior desired
from an outfit in charge of your key infrastructure.

You are missing something. Covad did not cut off the service to you. Covad
cut off the service to DSLnetworks. DSLnetworks was not paying bills, and
it is their (dslnetworks) fault that they didn't disclose their financial
situation earlier to their subscribers.

I hope that this comes to the attention of Covad and I hope that they don't
like it! MHSC will also be filing complaints with the CA-PUC and everywhere
else that we can. But, a four-year process will not help MHSC stay in
business. MHSC is already preparing a Marketing Communication plan to get
this story out. MHSC tried dealing with this peacfully and got ignored, by
Covad. Covad needs to be shunned by the rest of the Internet community, as a
bad net-citizen.

I'm in no way trying to apologise for Covad (they got what was coming to
them), but your problems could've been averted with a T1 or redundant
connections.

3

If I read this right, you are not a Covad customer, and you didn't
choose Covad service, your upstream did. It seems to me that:

    1) You are not in any position to evaluate Covad's relationship with
your provider. You have no way of knowing what Covad's contract looks like
with your provider or whether your provider paid its bills or abused its
service.

    2) Your complaint is with your provider, regardless of Covad's action.
If your provider did something wrong in their relationship with Covad, they
are at fault. If your provider did nothing wrong in their relationship with
Covad, they still chose Covad and didn't provide for decent fall back
provisions or legal protection for their customers.

    3) You have no way of knowing how much advanced notice Covad provided to
your upstream or what alternate options they could have taken but didn't.

    Please remember to aim before you fire.

    DS

This is a blind message. I don't even know if it's getting out of my mail
hub. Theoretically, it should. You won't be able to reply. I can route

out,

but nothing can get back in.

Some of you may remember a poster of this huge eagle stooping on a mouse.
The mouse was calmly standing, facing the eagle, flipping it the bird. The
title was "Defiance". Covad is the eagle, MHSC is the mouse, and this
message is the bird.

MHSC has been down since 7Feb01 and Covad is the culprit. MHSC pays its
bills, but Covad had a problem with my upstream. On Wednesday, 7Feb01,

Covad

cut off all links and feeds to my upstream. I've been down ever since.
Everyone knows how quickly telcos turn up services, that's why co-lo

centers

have customers. Covad cut off MHSC access, without warning, without,
compansation, and their SafetyNet program is the worst joke. I have a /24
and the new ISPs, that Covad is forcing me to use, can't assign a new /24.
Shit, one of them can't even assign one static address. The one that this

is

going out thru tis trying to convince me that I can run 62 computers,

using

130 IP addrs, on a /27 <heh>. We talked earlier about how non-portable
addresses suck. Here is the reason why. I either use one of their

SafetyNet

providers or suffer up to 45 days of outage. Two weeks of warning and I
would have been able to avoid the outage. (BTW, my ISDN backup feed is

dead

as well). I've tried contacting Covad, thier executive staff, etc. No joy
there, Covad doesn't care.

100% of MHSC business is directly dependent on our Internet feed. 100% of
February revenue is gone. I had two product roll-outs that were scheduled
for the following week. Both are now dead issues. MHSC does CA for some of
it's customers, renumbering invalidates all those CAs. Every lead we were
working is now gone, probably forever, and so are many existing customers.
I'm looking at probable business failure. MHSC could sue, but may not
survive long enough to collect, even if we win. If MHSC survives this then
"Damn, I good!"

I urge everyone, do not;
1) Recommend Covad as a DSL provider
2) Become a Covad partner.
3) Remain on Covad service, find someone else (Rythyms sound good)

Further, if you have any Covad stock ... sell it.

The simple reason is that Covad has shown themselves unworthy of trust.

They

will cut off service arbitrarily, without direct cause, without

compansation

or recourse, and without warning. This is NOT the sort of behavior desired
from an outfit in charge of your key infrastructure.

We are not large, MHSC cannot afford an $80K+ revenue hit, and that's not
counting the suits for failure to meet SLA.

At this point, I'm not worried about any Covad law suits. What are they
going to do ... put MHSC out of business? They've already taken care of
that.

I hope that this comes to the attention of Covad and I hope that they

don't

like it! MHSC will also be filing complaints with the CA-PUC and

everywhere

else that we can. But, a four-year process will not help MHSC stay in
business. MHSC is already preparing a Marketing Communication plan to get
this story out. MHSC tried dealing with this peacfully and got ignored, by
Covad. Covad needs to be shunned by the rest of the Internet community, as

a

4

Wow. A telco cut off a customer who hadn't paid their bill? Amazing.
Also old news, if this is the case I'm thinking of. If you find a DSL
local loop provider that works reliably, will be in business for a while,
and doesn't require its customers to pay, please let me know. It sounds
useful.

Yes, having your DSL service cut off sucks. I know this from experience,
as my home connection is also currently down for the same reason.
However, I think the lesson to learn from this is not that Covad sucks,
but rather that keeping mission critical systems single homed on a DSL
line to an ISP that's not paying its bills is probably a bad idea.

-Steve

5

I'm no bean counter, but if you're losing this much money from a
xDSL-related outage, perhaps it might be a good idea to invest in a
carefully-planned redundant infrastructure, including some means of
backup transit.

This is by no means a revolutionary concept, and may very prevent
businesses from becoming defunct businesses in extreme cases like
this.

Some might argue that DSL (other than that which is provisioned by the
ILEC or another party with deep pockets) is going away sooner or
later, or at the very least, undergoing drastic changes in the way
it's marketed and aggregated. But that's beyond the scope of this
discussion.

Sorry if this isn't the message of sympathy you were hoping for.

-adam

6

Hmm, gee, isn't there a cliche' that fits this exact problem? Something
about eggs?

--Adam

7

2 Thumbs Up

8

:Yes, having your DSL service cut off sucks. I know this from experience,
:as my home connection is also currently down for the same reason.
:However, I think the lesson to learn from this is not that Covad sucks,
:but rather that keeping mission critical systems single homed on a DSL
:line to an ISP that's not paying its bills is probably a bad idea.

Our sales staff tells prospective clients exactly that ("If your business
will rely heavily on your connectivity, DSL is the wrong choice"). It's
not mature enough (read: its too-quick proliferation was due mostly to
largely sales-oriented, facilities-less entities flooding the market
with no forethought to quality).

And, of course, it's never a good idea to include scenarios that are out
of one's control in SLAs to one's clients. That said, I'd say Roeland's
gotten his share of crow to mange on :wink: Live and learn.

-brian

9

* Roeland Meyer <rmeyer@mhsc.com> [20010228 13:48]:
[..]

MHSC has been down since 7Feb01 and Covad is the culprit. MHSC pays its
bills, but Covad had a problem with my upstream. On Wednesday, 7Feb01, Covad
cut off all links and feeds to my upstream. I've been down ever since.

[..]

I urge everyone, do not;
1) Recommend Covad as a DSL provider
2) Become a Covad partner.
3) Remain on Covad service, find someone else (Rythyms sound good)

Further, if you have any Covad stock ... sell it.

I understand your frustration but I fail to see why you are disgusted with
Covad. What about your "upstream"? Your DSL reseller/bundler is the one
at fault, not Covad. You weren't warned by Covad because you are not a
Covad customer. Warning was _your_ providers job.

The simple reason is that Covad has shown themselves unworthy of trust. They
will cut off service arbitrarily, without direct cause, without compansation
or recourse, and without warning. This is NOT the sort of behavior desired
from an outfit in charge of your key infrastructure.

They cut off your upstream for not paying their bill. It sucks that your
company is stuck in the middle, but blame your vendor. Ask them why Covad
cut them off.

I realize you're frustrated but don't direct blame at the wrong company.
Covad is just trying to get their own customers to pay their bills. A lot
of the resellers are getting hit because their customers (ISPs) are not
paying them. Eventually it works its way up the chain. Sadly, since the
resellers are just bundling up a bunch of ISPs under one account, if
something goes wrong, all of their ISP customers can be affected.

I'm truly glad my previous employer (a regional ISP) didn't go with
DSLNetworks. We were *really* close about 18 months ago but decided to
go it on our own at the last minute. If that had not turned out to be the
case, they'd be in it deep right now and probably equally frustrated.

-jr