CAIS/Ardent and now Network Access Solutions

Mailman List Mirror (Read Only)
1

CAIS sold our account to NAS. They did this about 5 months back. They are
just now getting around to cutting us over. This involves Covad making some
changes in their switch somewhere.

Back last May, it was PSINet that was selling our account to CAIS. They sent
us an e-mail to announce that they were taking over the account and sent us
a list of the account details that they had for us. The information was
wrong.
I sent them corrections. Also made sure they knew that we had our own IP
addresses and to be sure to coordinate the BGP stuff with PSI when cutting
over.

They didn't. PSI kept announcing the routes. Ardent didnt announce them. It
took me FIVE DAYS to get them to fix it. Waited on hold for 45 minutes more
than
7 times and finally talked to brain dead drones on the other end each time.
No
luck. Finally wheedled the PSI NOCs direct number out of someone on the
phone at
PSI and called them. They dropped the route announcements for us. Then it
took three more days to get in touch with someone at Ardent. I was able to
do
that by posting a message here. One of their engineers called me and we had
it fixed
in ten minutes.

Fast forward to Dec/Jan 2001/2002: Now, CAIS (called Ardent now) sold us
to Network Access Solutions (NAS). We have been paying them ever since Dec
or so. Two months ago, they sent the same kind of e-mail: "We'll be cutting
over your connection soon, please fill out this questionnaire, etc". I did
so,
and wrote a long tome at the end warning them not to mess up the BGP stuff.
"It
couldn't be that bad twice, could it?" I though to myself. I crossed my
fingers. In
their e-mail, they told me that they would soon get back to me with a
cutover
date.

This afternoon, at 4:40PM, I got the message "Your cutover date/time is
5PM-8PM Eastern May 29, 2002 (today)" and "The date cannot be
changed".

Also they said "If you have your own IP addresses", it may take till
midnight till the routes propagate (Now I know I'm in trouble - 5pm till
midnight?).

No sooner did I read the message than the connection went dead. Not the
circuit, just the BGP announcements.

Its now 7AM Eastern time. We've been offline 13 hours now. Phone drone at
NAS says "Hmm, your not listed on our cutover sheet today". I put two and
two together: They told Ardent "drop the routes today at 5" and then lost
the paperwork internally so the cutover did not happen, but Ardent dropped
the
routes. Just got off the phone with them again after sitting on hold for 45
minutes. They dont seem to get the message that its a BGP problem and not
the circuit.

Last time, when we had problems, some kind engineer from CAIS
sent me e-mail and offered to help and, like I said, he got us back online
in, like 10 minutes. If you're out there, please let me know. I need your
help again
Please reply to jpp@well.com as the e-mail on this message is not accessible
at this time.

Sorry to bug all of you with this. Let see: 13 hours and running. Last time
it was five days. Wanna take bets on how long this time? Funny thing, when I
called
CAIS afterwards to ask them how they were going to "make it up" to me for
knocking me offline for five days, I was sent to the voicemail of some
customer
service manager who never returned my calls. Lets see how well NAS does.

John

2

NAS has been nothing but trouble. We are (or were) a Covad reseller, first
direct through Covad, then through CAIS.

The first we heard our lines had been sold was when we called CAIS for
support and were transfered to NAS. A week after that, our customers
started getting e-mail that their accounts had been sold and they now were
NAS customers. Except our CAIS explicitly stated they were NEVER to contact
our customers for any reason.

They appologized -- and a week later mailed out paper letters to all our
customers.

When our backhaul went down, it took them over 4 hours to even pick up the
ticket. It was in the same queue as regular DSL lines. It's been the same
with every circuit that goes down. One hour plus hold times for support,
e-mail to support is answered days later.

We finally have a direct rep in corporate's cell number and put all tickets
in through him, but this is no way to run things.

The best was they wanted me to sign a contract adendum stateing that if any
bill was more than 10 days late, they would take our customers -- no
mention at all of dispute resolution. I laughed at them.

So, we don't place any more Covad orders, which is fine since no one wants
to pay those prices anyway, and we sell Verizon DSL. Amazingly, Verizon DSL
has had far fewer hassels than Covad ever did.

3

We are finally back. Hmm - Level3 is one of their transit providers:

BGP routing table entry for 199.5.156.0/23, version 531543
Paths: (4 available, best #1)
  Not advertised to any peer
  3356 13953
    209.244.2.230 (metric 200201) from 165.117.1.219 (165.117.1.219)
      Origin IGP, metric 4294967294, localpref 100, valid, internal, best
      Community: 2548:172 2548:344 2548:666 3706:153
  3356 13953
    209.0.227.37 (metric 270401) from 165.117.1.144 (165.117.1.144)
      Origin IGP, metric 4294967294, localpref 100, valid, internal
      Community: 2548:172 2548:264 2548:666 3706:154
  3356 13953
    165.117.69.46 (metric 340501) from 165.117.1.140 (165.117.1.140)
      Origin IGP, metric 4294967294, localpref 100, valid, internal
      Community: 2548:172 2548:267 2548:666 3706:132
  3356 13953
    165.117.52.198 (metric 340401) from 165.117.1.155 (165.117.1.155)
      Origin IGP, metric 4294967294, localpref 100, valid, internal
      Community: 2548:172 2548:302 2548:666 3706:161

Here is a question: How are folks here handling the environment today
where you rely on your connection but you keep getting handed (sold) off
from one provider to another and each one seems to be in worse financial
shape?

Who can one rely on for connectivity? In general and in the Detroit area?
I put out a request for bids on T-1's and all the national providers were
way too high, mostly because they have no local POPs. We really need to
be as close to the top as possible, not because of bandwidth needs, but
for reliability's sake.

John

CAIS sold our account to NAS. They did this about 5 months back. They are

NAS has been nothing but trouble. We are (or were) a Covad reseller, first
direct through Covad, then through CAIS.

The first we heard our lines had been sold was when we called CAIS for
support and were transfered to NAS. A week after that, our customers
started getting e-mail that their accounts had been sold and they now were
NAS customers. Except our CAIS explicitly stated they were NEVER to contact
our customers for any reason.

They appologized -- and a week later mailed out paper letters to all our
customers.

When our backhaul went down, it took them over 4 hours to even pick up the
ticket. It was in the same queue as regular DSL lines. It's been the same
with every circuit that goes down. One hour plus hold times for support,
e-mail to support is answered days later.

We finally have a direct rep in corporate's cell number and put all tickets
in through him, but this is no way to run things.

The best was they wanted me to sign a contract adendum stateing that if any
bill was more than 10 days late, they would take our customers -- no
mention at all of dispute resolution. I laughed at them.

So, we don't place any more Covad orders, which is fine since no one wants
to pay those prices anyway, and we sell Verizon DSL. Amazingly, Verizon DSL
has had far fewer hassels than Covad ever did.

4

Who can one rely on for connectivity? In general and in the Detroit area?

    > I put out a request for bids on T-1's and all the national providers were
    > way too high...

Haven't you just answered your own question? I guess if you think
reliable service is too expensive, you're not in the market for reliable
service, no?

                                -Bill

5

Its just that they aren't local and there is no need to pay for
a circuit all the way to Chicago. It seems that so many providers
have moved out of Macomb county. Anyone have any experience with
BigNet? We are talking to them now

6

The hard facts are that carriers want to make money hopefully so they
will locate where the money is. My chances of having closely located
customers is greater in Newyork City say than somewhere distant You
either have to pull circuits to the carrier you want or settle for
someone local much smaller and in many cases much more likely to fail.
Top providers will be located in top markets, why that's where the
business is usually.

7

Clearly anyone in your market is buying from someone outside of your market.
The fees associated with reliability (if available) are a function of your
geography. Large providers are concentrating on the markets that are making
them the most money.

If you get a few networks in your area that want to save money on the cost
of reliability you can run a couple of circuits to the next large market and
try to knit together a reliable network and divide the costs that way.

My guess is that with more large providers on a profit-centered basis you
won't see the same kind of pricing equality you have been seeing between
Tier 1 and Tier N markets anymore.

Deepak Jain
AiNET

John Palmer

8

Surprised there isnt much connectivity in the Detroit area, I mean it is
Motor City and all, I would think tons of manufacturing palnts all needing
telecom of some sort or other..

  Bri

9

Nonsense. Small providers are the ones who are financially connected to
reality, and thus more likely to make money (since the owners generally
want to keep paying the bills) and remain in business.

The issue with small providers is, you don't always know what you are
going to get. You could end up with someone that doesn't have much clue
about providing good service simply because they are inexperienced. On the
other hand, you could find someone who is very experienced but who wants
to work for themselves, or on a project where they can have a significant
impract. Being small could very easily mean they are sensible, nimble, and
not burdoned with silly bureaucracy.

That said, this has very little place on NANOG.

10

As far as I know, the following national providers exist in Detroit:
(This is by no means, a complete list, and some of these aren't "Big
Name" providers, but less likely to go under than John Q Random ISP)

Verio
Winstar
XO
AT&T
Qwest
Covad
Level 3
SBC
I've heard rumors of speakeasy.net, which I think resells internap

I'm sure there's a few other national networks, these are just the ones
I have heard of off the top of my head.
-Paul

11

Agreed. As the architect for a large regional we strive to have full
transit with several large providers (and peer with folks when possible) so
that a single large provider unexpectedly 'biting the dust' or even
experiencing backbone problems won't destroy our network. {Please note:
this is not a sales plug, we aren't anywhere in the MI area.}

And yes, the smaller local and regional providers must look at margins and
cash flow constantly. They can't afford to pull the 'tricks' that are
getting the big guys in so much trouble. My one caveat is to insure that
the smaller guy is fully multihomed with at least two different upstreams.

Tim

Richard A Steenbergen

12

This assumes that "the top" is any more reliable/clued/financially solvent.

Scanning the last year's archives for "chapter 11" and/or "depeering" might
be quite enlightening....

13

Jeez. About 18 months ago I couldn't even get CAIS to return calls to
their sales department. I guess I'm glad they didn't.

14

Try to get DSL here; everyone backhauls to Chicago. And Cleveland is the
25th largest city in the USA. Lots of local providers for DS1 and Frame
and ATM, just not DSL. :slight_smile:

15

As far as I know, the following national providers exist in Detroit:
(This is by no means, a complete list, and some of these aren't "Big
Name" providers, but less likely to go under than John Q Random ISP)

The above is completely false statement in my opinion. From what I'v
seen, local "John Q Random ISP" is a lot more likely to be stable, have
good support and less likely NOT TO go under then most of the "Big Name"
providers you listed (see comments).

Verio

Dubious financial situation, although after restructuring and closing of
many offices & datacenters, they're getting better. Their network however
is fairly bad compared to majority of other national ip providers.

Winstar

No longer in business.

XO

Very very serious financial problems. Problems with their network as well.

AT&T

Fairly stable due to the fact they do not have large bonds/loans as they
used to have lots of money when they were giant monopoly long distance.
But they continue to loose share of long-distance market and although they
bought former Northpoint network, they do not seem to be using it yet (???).
Nobody is sure what their future business plans are regarding internet
or anything else...

Qwest

Serious financial problems. Customer service needs much improvement...

Covad

Out of bankrupcy but not yet stable. They buy majority of their intenet
transit from Level3 and do not really have national ip network.

Level 3

Stopped selling directly to small customers. You have to buy OC3 or
above or at least 100Mb and even that is not enough for them to take you
seriously (Note: known to still sell DS3 if you push them, but no longer
T1s).

SBC

Big bad wolf ILEC :slight_smile: Probably most stable of the list you have. Most
expensive too. I personally avoid doing business with SBC because of how
anticompetetive and anti-ISP they are.

I've heard rumors of speakeasy.net, which I think resells internap

Primarily DSL ISP. I'v heard bad things about how they tricked resellers...
Check on dslreports on opinions about their service.

I'm sure there's a few other national networks, these are just the ones
I have heard of off the top of my head.

Of course, there are other networks - Worldcom/UUNET, Sprint, CWUSA, etc

But if I were you, I'd to yellow pages for list of ISPs or the some other
list of local isps (their used to be ones from boardwatch, thelist.com,
etc; check only for the ones that actually have office in your area) and
check with them if they can provide you with T1.

P.S. I'm only expressing my personal opinion about companies above, I do
not wish to initiate any kind of discussion with others on the list that
may work for these companies or would otherwise have different view on this.