Qwest Support

Mailman List Mirror (Read Only)
1

Wow, Qwest support is indeed terrible.

Turned up the DS3 today...the connectivity seems fine. I decided to check
a couple of routeservers (nitrous); all had my much-prepended UUnet
announcement, but NONE had my Qwest announcement. Not a huge deal, but
curious to me. Is Qwest just not at the public peering points? When I
checked route-views.oregan-ix.net, I felt better, but yet annoyed. Even
with the prepends, most networks were announcing UUnet's path.

So I decided to call them and ask...man what a mistake. The guy is like,
"Ok, hold on, let me get somebody from our IP noc." 10 minutes goes by,
and he comes back with "Couldn't get anybody in the IP noc, let me try to
get somebody in your install group" (being that I turned up the DS3
today). Comes back another 10 minutes later with "Well, I left a message
for them, but there isn't much I can do. Nobody seems to be answering
their phone. If somebody doesn't call you back within 30 minutes, here's
a number to call..."

So what if my routes were actually hosed? I'd just be screwed because they
can't get anybody at the IP noc?

I wait. Nobody calls back within 30 minutes. I call the number he gave me.
Busy. You gotta be kidding me.

So I call the main number again, talk to somebody different. She has me
hold, and then brings some guy on the line "who can help me". I start to
talk about route servers, and he's immediately like "Woah, this is a BGP
problem...I can't help you. Let me try to get somebody from the IP noc."

So, I wait on hold for about 15 minutes, only to be given dial tone.

Please tell me it isn't always THIS bad?

Andy

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Andy Dills 301-682-9972
Xecunet, LLC www.xecu.net
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Dialup * Webhosting * E-Commerce * High-Speed Access

2

Wow, Qwest support is indeed terrible.

.. Long Story about being passed around and around and spending lots of
time on hold and STILL not getting anything done trimmed for brevity...

Please tell me it isn't always THIS bad?

Welcome to Qwest.

Ride the Light... Right into the darkness.

- Forrest W. Christian (forrestc@imach.com) AC7DE

3

I suppose. Except it's not even certain you were having a problem of any
kind at all.

Qwest's presence or absence from public IX's really has nothing to do with
your routes being announced. In fact, Qwest privately peers with all the
other large networks. While there are many peering sessions at the public
NAPs, most traffic is carried over private network interconnects, at least
domestically. Certain peering points in Europe (Linx), tend to run the other
way.

In fact, if Qwest were publically peering with other networks, it might be a
reason why your routes through UUNet were being prefered - private peer
originated routes are almost always assigned higher local preferences in
carrier networks, then public peer originated routes.

I'm not sure your annoyance with Qwest has any basis in their lack of
performance, as far as IP routing. BGP decision rules and other networks'
routing policies will govern which paths are used for your routes. Here is
an example...

- Network X peers with UUNet in 8 locations. Network X also peers with
Qwest, lets say in 6 locations. For whatever reason, network X chooses
UUNet's routes to you over, Qwest's. This could be due to local routing
policy, dictating that 701 routes get a higher local pref. Or AS path
lengths could be the same, and the decision could be falling to something
like router ID. Whatever.

- In general, all the UUNet peering will get treated the same by Network X's
routing policy. This won't always be the case, but let's say that none of
the peering links are congested, etc. So, a certain number of paths are
carried throughout Network X via iBGP. If UUNet's routes "won" at all those
peering points, you will not see any paths through Qwest on a single carrier
route server like Nitrous.

- Route-views, and the like are different animals. They get ebgp multihop
views from many providers, so you will tend to see paths from many different
vantage points, and are more likely to see paths from both your upstreams.

ISPs get a heavy volume of calls every day. While Qwest may not have the
greatest customer service, it's not like you were actually down or had a
qwest originated routing issue. If that were the case, my sympathy would be
greater.

- Daniel Golding

4

I would have to disagree on a lot of these points. See below.

Steven Naslund

From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of
Daniel Golding
Sent: Thursday, April 04, 2002 5:25 PM
To: Andy Dills; nanog@merit.edu
Subject: RE: Qwest Support

I suppose. Except it's not even certain you were having a problem of any
kind at all.

Qwest's presence or absence from public IX's really has nothing to do with
your routes being announced. In fact, Qwest privately peers with all the
other large networks. While there are many peering sessions at the public
NAPs, most traffic is carried over private network interconnects, at least
domestically. Certain peering points in Europe (Linx), tend to
run the other
way.

If the routes cannot be seen at the public IXs then a lot of people who are
connected to the public IXs will not see it either. Depends if you are only
talking to the "big networks".

In fact, if Qwest were publically peering with other networks, it
might be a
reason why your routes through UUNet were being prefered - private peer
originated routes are almost always assigned higher local preferences in
carrier networks, then public peer originated routes.

Local prefs are just that LOCAL. They will not matter to other networks,
they
merely show the routes I prefer in and out of my network. This should have
no
impact on AS path hop counts which is the primary method of selecting BGP
routes.

I'm not sure your annoyance with Qwest has any basis in their lack of
performance, as far as IP routing. BGP decision rules and other networks'
routing policies will govern which paths are used for your routes. Here is
an example...

- Network X peers with UUNet in 8 locations. Network X also peers with
Qwest, lets say in 6 locations. For whatever reason, network X chooses
UUNet's routes to you over, Qwest's. This could be due to local routing
policy, dictating that 701 routes get a higher local pref. Or AS path
lengths could be the same, and the decision could be falling to something
like router ID. Whatever.

What I would wonder here is : If network X prefers UUnet over Quest then
maybe
UUnet offers better performance than Quest. I think that most networks will
not set a local pref unless there is a reason to override the default BGP
behavior
which is to use AS path length. If service providers are avoiding Quest
there is
probably a good reason for it. I don't think many people would try to give
UUnet
more preference that they already get by default, more likely networks lower
the
UUnet pref in order to balance their traffic.

- In general, all the UUNet peering will get treated the same by
Network X's
routing policy. This won't always be the case, but let's say that none of
the peering links are congested, etc. So, a certain number of paths are
carried throughout Network X via iBGP. If UUNet's routes "won" at
all those
peering points, you will not see any paths through Qwest on a
single carrier
route server like Nitrous.

Not true. Nitrous shows all routes it knows about whether they are
preferred or not.

- Route-views, and the like are different animals. They get ebgp multihop
views from many providers, so you will tend to see paths from
many different
vantage points, and are more likely to see paths from both your upstreams.

ISPs get a heavy volume of calls every day. While Qwest may not have the
greatest customer service, it's not like you were actually down or had a
qwest originated routing issue. If that were the case, my
sympathy would be
greater.

How would Quest have known if he actually had a problem since they never
really
talked to him ? What if he had a real routing problem ?

5

I'm not sure your annoyance with Qwest has any basis in their lack of
performance, as far as IP routing. BGP decision rules and other networks'
routing policies will govern which paths are used for your routes. Here is
an example...

No, my annoyance was definitely with their support, not with their
network. I recognize that my experience with complex inter-AS routing is
minimal...I think my post indicates that. Anyway, check the Subject.

ISPs get a heavy volume of calls every day. While Qwest may not have the
greatest customer service, it's not like you were actually down or had a
qwest originated routing issue. If that were the case, my sympathy would be
greater.

True, but my point was basically "Had I had a _real_ problem, I would have
been screwed." They didn't know if I had am emergency issue or not...they
knew enough to know I was asking a BGP question, and they couldn't find
anybody for me to talk to. If, say, somebody was announcing routes from my
blocks...I would have been just as out of luck.

I'm not asking for sympathy here...I've been reading nanog for several
years and reading/posting to inet-access since 95...I know that these
lists are the LAST place to go for sympathy :slight_smile:

Glad I posted though, I've receieved several helpful notes on avoiding
qwest's nonsense and getting somebody clued on the phone.

Andy

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Andy Dills 301-682-9972
Xecunet, LLC www.xecu.net
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Dialup * Webhosting * E-Commerce * High-Speed Access

6

> Qwest's presence or absence from public IX's really has nothing to do with
> your routes being announced. In fact, Qwest privately peers with all the
> other large networks. While there are many peering sessions at the public
> NAPs, most traffic is carried over private network interconnects, at least
> domestically. Certain peering points in Europe (Linx), tend to
> run the other
> way.

If the routes cannot be seen at the public IXs then a lot of people who
are connected to the public IXs will not see it either. Depends if you
are only talking to the "big networks".

I'm not aware of any tier 1's (those without transit) who rely on only
public IX connections.

> In fact, if Qwest were publically peering with other networks, it
> might be a
> reason why your routes through UUNet were being prefered - private peer
> originated routes are almost always assigned higher local preferences in
> carrier networks, then public peer originated routes.

Local prefs are just that LOCAL. They will not matter to other
networks, they merely show the routes I prefer in and out of my network.
This should have no impact on AS path hop counts which is the primary
method of selecting BGP routes.

No, see http://www.amazon.com/exec/obidos/ASIN/157870233X/

I think that most networks will not set a local pref unless there is a
reason to override the default BGP behavior which is to use AS path
length.

That would not be my experience.

> carried throughout Network X via iBGP. If UUNet's routes "won" at all
> those peering points, you will not see any paths through Qwest on a
> single carrier route server like Nitrous.

Not true. Nitrous shows all routes it knows about whether they are
preferred or not.

Yes true. Once the path selection is made only the "best" path is passed
on.

I'd suggest you listen to Dan, in my experience he tends to know what he
is talking about.

7

You totally missed the point. Had this been a real emergency, he would be unable to get resolution since Qwest was unable to dredge up a clue within their customer support machine.

Greg U

8

Yeah. I got that part of it, and I don't disagree that many large carriers
have less than stellar support.

Some things to consider when looking at the support you get from a large
ISP...

- Most problems you will have are going to be circuit problems - i.e. "my
T-1 is down, AGAIN". Therefore, most carrier's support operations are built
around fixing Layer 1 problems. You give them the circuit ID, and then, in
theory, you get your circuit fixed.

- Routing problems occur, but most are global, across a carrier's entire
network, or regional, at a POP or POPs. There are many things that can cause
this - bad router code, misconfiguration, etc. In a properly designed
network, routing problems that impact a single user are rare.

- Resources to assist customers in diagnosing routing problems are scarce,
for a variety of reasons, some good, some very bad. This is compounded by
the fact that the "hit rate" for customer routing problems is low. Most
times when a customer calls and says that their T-3 is down, it really is.
Most of the time when a customer calls and says they are having a BGP
problem, it's rarely originated by the carrier, and is usually a customer
misconfiguration or misunderstanding.

- Daniel Golding

9

I think the main point here isn't the fact that the poster's routing was, in fact,
not set up properly; it was the fact that he was unable to get a live body at Qwest
to check it out.

-C

10

Interestingly enough, there was indeed a problem. I'm not satisfied with
the answer; it doesn't really make any sense. Not that I'm too concerned
about knowing what actually happened, but hopefully somebody can suggest a
possible explanation.

I got a call this morning from my install manager (who is very nice...I
like Christina) who told me that for some reason, when she came in this
morning, and found my voice mail, she looked and their router wasn't
getting our routes. Which seems really weird, because they were in plenty
of views on the oregon route server...and our traffic graphs show plenty
of ingress traffic since the turnup.

So anyway, I checked nitrous. And yes, my Qwest routes are there now...so
it looks like I did have a legit beef after all. Once qwest's router
started seeing my routes (or whatever got fixed), my ingress UUnet traffic
dropped to nil, just like I wanted...

Strangeness, but probably related to global router configs that needed to
be auto-updated I'm guessing...

Andy

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Andy Dills 301-682-9972
Xecunet, LLC www.xecu.net
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Dialup * Webhosting * E-Commerce * High-Speed Access

11

And here we go, down the rabbit hole... (see below)

Steve Naslund Said...

I would have to disagree on a lot of these points. See below.

Steven Naslund

> Daniel Golding Said...
>
>
>
>
> I suppose. Except it's not even certain you were having a problem of any
> kind at all.
>
> Qwest's presence or absence from public IX's really has nothing
to do with
> your routes being announced. In fact, Qwest privately peers with all the
> other large networks. While there are many peering sessions at
the public
> NAPs, most traffic is carried over private network
interconnects, at least
> domestically. Certain peering points in Europe (Linx), tend to
> run the other
> way.
>

If the routes cannot be seen at the public IXs then a lot of
people who are
connected to the public IXs will not see it either. Depends if
you are only
talking to the "big networks".

How is this so? There are plenty of routes that are not announced at public
exchance points, that are floating around in the global routing table.
That's because almost all of the folks who are at public peering points, are
also buying transit from the "big networks", in one way or another. Needless
to say, this is how the internet works.

> In fact, if Qwest were publically peering with other networks, it
> might be a
> reason why your routes through UUNet were being prefered - private peer
> originated routes are almost always assigned higher local preferences in
> carrier networks, then public peer originated routes.
>

Local prefs are just that LOCAL. They will not matter to other networks,
they
merely show the routes I prefer in and out of my network. This
should have
no
impact on AS path hop counts which is the primary method of selecting BGP
routes.

I think you are missing the point. Almost all large networks have similar
sorts of routing policies, where routes recieved through private
interconnect, get pref-ed higher than routes through public exchanges.
Because of this confluence of routing policies, private peer-originated
routes tend to be prefered. Look at the NANOG presentation archives for good
examples of how this works.

> I'm not sure your annoyance with Qwest has any basis in their lack of
> performance, as far as IP routing. BGP decision rules and other
networks'
> routing policies will govern which paths are used for your
routes. Here is
> an example...
>
> - Network X peers with UUNet in 8 locations. Network X also peers with
> Qwest, lets say in 6 locations. For whatever reason, network X chooses
> UUNet's routes to you over, Qwest's. This could be due to local routing
> policy, dictating that 701 routes get a higher local pref. Or AS path
> lengths could be the same, and the decision could be falling to
something
> like router ID. Whatever.

What I would wonder here is : If network X prefers UUnet over Quest then
maybe
UUnet offers better performance than Quest. I think that most
networks will
not set a local pref unless there is a reason to override the default BGP
behavior
which is to use AS path length. If service providers are avoiding Quest
there is
probably a good reason for it. I don't think many people would
try to give
UUnet
more preference that they already get by default, more likely
networks lower
the
UUnet pref in order to balance their traffic.

This is simply not true. There is no incentive to balance outbound traffic
amongst peers, EXCEPT to preserve inbound/outbound ratios. Almost all
backbones set local prefs on the vast majority of routes, in order to
establish a routing policy. Also, backbones do not tend to alter their local
prefs or routing policies based on the perceived quality of their peers -
only on things like congestion across a specific peering link. It is often
tempting to think that because one manages traffic in a certain way, in a
smaller or non-transit AS, that one would do it in the same way in a large
transit AS, with significant peering. However, in reality, these are much
different animals.

>
> - In general, all the UUNet peering will get treated the same by
> Network X's
> routing policy. This won't always be the case, but let's say
that none of
> the peering links are congested, etc. So, a certain number of paths are
> carried throughout Network X via iBGP. If UUNet's routes "won" at
> all those
> peering points, you will not see any paths through Qwest on a
> single carrier
> route server like Nitrous.

Not true. Nitrous shows all routes it knows about whether they are
preferred or not.

Looking Glasses like Nitrous, show the routes on the routers that they look
into. If paths from a specific provider are selected on all peering routers,
for a given route, than no paths to alternative providers will appear. This
is how BGP works. I suggest reading the appropriate RFCs, and perhaps,
Routing TCP/IP volume II by Jeff Doyle. Remember: large carriers tend to
have dedicated peering routers, where numerous private peers are terminated.

>
> - Route-views, and the like are different animals. They get
ebgp multihop
> views from many providers, so you will tend to see paths from
> many different
> vantage points, and are more likely to see paths from both your
upstreams.
>
> ISPs get a heavy volume of calls every day. While Qwest may not have the
> greatest customer service, it's not like you were actually down or had a
> qwest originated routing issue. If that were the case, my
> sympathy would be
> greater.

How would Quest have known if he actually had a problem since they never
really
talked to him ? What if he had a real routing problem ?

If this was a real routing problem, it would be a much more interesting
conversation.

12

Did they ask you to update RADB or IRR entries? Many times these are used to
build prefix lists for customers at some ISPs, and they get updated
periodically (basically a CRON job). Typically, stuff like RTConfig is used
for this, or home-grown Perl scripts. There can sometimes be a delay in
getting this pushed out (maybe the script died?)

- Daniel Golding

13

One final post on this...

I called today because I needed to announce a couple of new routes. I
called their 877-37-LIGHT number...I get an amazing array of people when I
call this line.

The first lady is very nice, and even goes to check with somebody else
("BGP, I've heard of that"), and she tells me to email my filter change
request to support@qwestip.com. Ok, sounds fair. I do so, at around 2:30.
I wanted to get it in by 5, so I was in no rush. I also find out from the
nice lady that Qwest has a webpage that will submit the request for me
(qwestsource.net). I didn't have a username and password, so I filled out
the form and waited for a response.

At 3:30, I get the username and password for qwestsource. My luck, it
doesn't work.

Around 4:15, with not even an autoresponse from my email to support, I
call the LIGHT number again. I get the most clued-sounding person I've
talked to with Qwest yet. He's all crisp and Johnny TalkShow, and
authoritively states "There's a special address for BGP requests. Email
bgp@qwestip.net." I mention the bad username and password on qwestsource,
and he says "Ah yes, that sometimes happens. Here's a number to call
for the group that manages that..."

So I hang up the phone, satisfied. I email bgp@qwestip.net, and pick up
the phone to call about my qwestsource access. As I'm listening to the
phone tell me that I've called the telco provisioning group, I see the
bounce show up from bgp@qwestip.net.

I'm had to laugh at that...some dude in this mass queue in a cubicle
somewhere in Qwest is just sitting there talking like a tv character
issuing bum information authoritively, with a strange hint of pleasure
now that I think about it.

So I call the LIGHT number one more time. Third time's the charm,
apparently. I got a nice lady who put me right through to an engineer who
had access to the filters. He added the filters, we cleared, and he was
seeing all but one of the routes. We scratched our heads for a few
minutes and cleared again, which of course fixed it. He then proceeds to
log into the qwestsource backend to fix my account. All taken care of.

So, I guess my bottom line is that Qwest has good people. But not a good
setup. The problem isn't that the people I'm talking to aren't qualified
to do their jobs, it's just that Qwest's setup makes me talk to the wrong
people.

I like UUnet's way. I call up, hit 2-1-1, enter a series of digits, #,
another series of digits, #, and then I talk to the people who have
enable. With Qwest, I still have to enter our account number, but I'm sure
I end up in the same place the dsl customers go. At least, it seems like
it. One thing I must give Qwest (this goes for UUnet as well) is that I
have yet to wait on hold initially. I've spent a fair number of minutes on
hold, but not until after talking to somebody first.

If Qwest was setup like UUnet, I would think their IP customers would be
much more content. What am I going to call Qwest about? Two things.
Circuit down, routing change. Now that I have qwestsource working, I can
submit requests. The circuit really shouldn't go down. Regardless, it
would certainly be nice to be able to quickly talk to the people in charge
of running the network without trying to con my way through mostly good
intentioned front line phone jockeys.

Network seems pretty solid though, and I can live with the occaisional
support escapades when cutting my price in half and increasing my
available bandwidth significantly...overall, I'm satisfied so far. It's
not like I have a lot of time to give to this craziness, but if I have to
choose two from "bandwidth, really inexpensive, and my time", I'd have to
be spending somebody else's money to not pick the first two...

Andy

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Andy Dills 301-682-9972
Xecunet, LLC www.xecu.net
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Dialup * Webhosting * E-Commerce * High-Speed Access