T3 Latency

Mailman List Mirror (Read Only)
1

Chuck,

The question posed by Chris, "how long is your access line"?, is a rather
important data point before answering your question. I have a T3 that is
about 15 miles long. It runs between two 7500 routers. Its minimum ping
round trip time with 100 byte pings is 2 ms. It is not very heavily loaded
with peaks of about 10 Mb/s today and the max round trip time was 21 ms.

So that should give you some minimal bounds of what you might expect. As
the saying goes your mileage may vary. Speed of light does play in here,
if the circuit is longer. Also intervening electronics such as a frame cloud
or telco muxes also add latency.

Walt

2

The rule of thumb I use is that the speed of light in fiber-optic cable is
roughly 2x10^8 m/sec.

2x10^8 m/sec = 200,000,000 m/sec = 200,000 km/sec = 200 km/msec =~ 130 mi/sec

I once worked with a customer whose first hop out was ~30ms, regardless of the
load on the line (a t3, iirc). Sure enough, he was on a very large SONET ring
that travelled the north-south length of the US roughly twice before his
traffic went elsewhere.

              ......Matthew

3

Matthew:
  Appears to be a typo in your final number of 130 mi/sec, but I get where
you're going with this. I'm just having a problem trying to figure out how
I end up with a couple thousand fiber miles from Northern Michigan to
Chicago. Should be interesting to sort this one out.

Thanks,

Chuck

4

Yup. Dropped a letter there. 130mi/msec.

              .....Matthew

5

Or, perhaps a more simplified and easily remembered figure...

RTT on a straight line run from sfo to dc would be ~63ms. (Seems to be
roughly 100 to 120 for most real circuits.)

We all know, however, that telcos rarely use straight lines. Still, I
would not expect more than 6 to 7ms. Perhaps your telcos equipment,
through some fluke, has you operating on the backup path?

6

Chuck,

should read 130mi/msec I guess. Which would end up with ~7msec per
1000miles.

Arnold

7

Packet size of your pings may have something to do with it, but assuming
you are pinging with the same size packet across the board, the data
should be reliable. If you are using unix boxes your pings will have a
different level of resolution than say from a windows box. One test that
should be reasonably conclusive is the following.

The hop that is 22ms and 7 hops away from your upstream should ping you on
the other side of the new T3. If it approaches 42 ms, you have a 20ms
T3. Is it ATM? Your upstream could be running on a congested ATM cloud. If
the latency drops in the wee hours of the morning, even for a few minutes,
its congestion (which is obvious).

I have a 27 mile (measured by air or telco :slight_smile: ) DS3 that even under 30mb/s
load pings minimum 1 ms average 1-2 ms. (cisco<->cisco). At 1400 bytes the
average goes to 3ms.

Your upstream may be putting you on a channelized interface on a router
that has a very busy VIP (if its a cisco). Or is running you through a lot
of electronics on the way.

The point here is that any kind of aggregation he is doing could eat up
10-20ms, by design or oversubscription. Bigger providers are more likely
to aggregate DS3s into bigger access methods whereas small providers
usually don't have the operational necessity.

Deepak Jain
AiNET

8

Wayne:
  Now there's a thought that didn't occur to me. They may be running me
all over creation for some reason rather than taking some more direct
route. Perhaps they ran out of somthing here in Michigan and had to run it
down to Texas and back to get it connected correctly. I think I'll have
them inventory the path and let me know what it is.

Thanks to all the responses on this.

Chuck Scott

9

Arnold:
  Yep, that's about what I figured, which means that we're seeing
something like 3000 miles of fiber, or somewhat less depending on the
latency of any interconnecting equipment. Funny for a path that's about
300 miles line of site and perhaps 500-600 they way I think they have it
running.

Chuck

10

Packet size of your pings may have something to do with it, but assuming
you are pinging with the same size packet across the board, the data
should be reliable. If you are using unix boxes your pings will have a
different level of resolution than say from a windows box. One test that
should be reasonably conclusive is the following.

The hop that is 22ms and 7 hops away from your upstream should ping you on
the other side of the new T3. If it approaches 42 ms, you have a 20ms
T3. Is it ATM? Your upstream could be running on a congested ATM cloud. If
the latency drops in the wee hours of the morning, even for a few minutes,
its congestion (which is obvious).

  Have been testing from our 7204 router and from various linux boxes. No
matter how small the packet size, we always see the 20 ms.
  Yes, the total ping time to that other system is a minumum of 42 ms. Not
running ATM and have not seen any times when the latency drops below 20ms.

The point here is that any kind of aggregation he is doing could eat up
10-20ms, by design or oversubscription. Bigger providers are more likely
to aggregate DS3s into bigger access methods whereas small providers
usually don't have the operational necessity.

  This may be the key. More of an effect of the way they're bringing me
in. I guess I need to ask them some specific questions.

Chuck

11

What is your confidence level that your test packets are taking
the path that you expect?

Are you multihomed? Does your addressing and routing setup allow
for the return packets to be taking a different path other than the new DS3?
What address are your test pings being sourced with, and what is the
destination's return path to that address?

                                                 dave

12

That was something I was considering, but when I also get 20 ms pinging
in the same /30), I don't think the responses are going anywhere but
straight back to my interface.
  No, this location is not multi-homed.

chuck

13

Also sprach Charles Scott

Have been testing from our 7204 router and from various linux boxes.
No matter how small the packet size, we always see the 20 ms. Yes, the
total ping time to that other system is a minumum of 42 ms. Not running
ATM and have not seen any times when the latency drops below 20ms.

This may be the key. More of an effect of the way they're bringing me
in. I guess I need to ask them some specific questions.

I've been watching this thread with some interest...from reading it all,
my gut instinct is that some circuit provider has your circuit routed
all over creation, or there's some other sub-optimal aspect of your
circuit (well...I guess that's pretty obvious).

The question then really becomes...how important is the latency *to
you*. Clearly, 20ms latency on a 300-400 mile (as the crow flies) T3 is
sub-optimal somehow. You might very well have customers and potential
customers question that, because it *does* affect performance, however
slightly, and it might be an indicator of other, more in depth problems.
Also, if you're looking at several thousand fiber miles on a 300 mile
line of sight run...that gives more opportunity for backhoe fade and
other problems.

Honestly, if I were in your situation, I would give some very serious
thought to refusing the circuit. I would be very likely to demand that
they get the circuit down to 10ms, and might even consider a demand of
around 7ms before I would accept it. That, of course, is my thinking,
and I don't know all the details of your situation.

I believe you posted looking for input and what others thought on it
though, so, FWIW, there's my $.02.

Another possibility is to look into what the provider gives as far as
SLA's are concerned. (I know some on the list have heard about this
before...forgive me for sharing an old, though still ongoing, story
again) I have a T3 between Louisville, KY and Indianapolis, IN (around
3ms latency, unloaded) that is carried by 6 different telco's. It took
7 months from time of order to get that circuit turned up. When it has
gone down, it has taken no less than 18 hours to get it fixed, and the
conference calls involved have typically been to the point of outright
hilarity (depending on your perspective I guess). The circuit was
ordered by UU.Net, and when we found out how the circuit was routed, we
were sure to sign up for their SLA agreements. We have had very good
luck getting SLA credits when the circuit goes down...and with the
circuit typically being down for 24-30 hours...we typically gets a
free month of service when it dies. I'm dumbfounded that UU.Net hasn't
kicked WCOM's butt into getting this circuit groomed onto WCOM
facilities in order to cut down the number of carriers to 2, but they
haven't done it yet, and we keep collecting the SLA credits when it goes
down, so I can deal with that...I'd rather it stay up and get fixed
quicker, but a day of credit per hour of outage does tend to take the
sting out of having the circuit down.

Anyway...my point in that story...if the provider has SLA's that
include latency provisions, it might be worthwhile to try to get it
fixed by pounding on the SLA issue until they realize that its in
*their* best interest to get the circuit fixed. :slight_smile:

14

The question then really becomes...how important is the latency *to
you*. Clearly, 20ms latency on a 300-400 mile (as the crow flies) T3 is
sub-optimal somehow. You might very well have customers and potential
customers question that, because it *does* affect performance, however
slightly, and it might be an indicator of other, more in depth problems.
Also, if you're looking at several thousand fiber miles on a 300 mile
line of sight run...that gives more opportunity for backhoe fade and
other problems.

  I fully agree. Aside from the marketing handycap this introduces, the
"backhoe fade" factor (love that term) is more significant. Did I mention
that this line has 13 cross connects and they first tried to turn it up on
the 13th of this month? Yes, I'm concerned about reliability and
repairability of the line.

Honestly, if I were in your situation, I would give some very serious
thought to refusing the circuit. I would be very likely to demand that
they get the circuit down to 10ms, and might even consider a demand of
around 7ms before I would accept it. That, of course, is my thinking,
and I don't know all the details of your situation.

  I have already posted a message to the marketing and provisioning
people requesting that they at least explain why the latency is
there. Something better than low priority responses to ICMP. At this point
we're not in a position to refuse the circuit, we need it, but I can
probably work this diplomatically with them to get this worked
out. Considering this line was ordered in July, and has missed repeated
deadlines, they owe me a few favors.

Another possibility is to look into what the provider gives as far as
SLA's are concerned.

  There are some service guarantees, but it's been so long since I placed
the order, I don't remember what they are. I'll have to go pull the file
and look. Good point though.

Thanks,

Chuck

15

I know one of my circuits runs from Seattle to Los Angeles to El Paso (fiber
cut a few weeks ago) to Washington DC and finally to New York City.

Nice to know my packets get to visit almost 15 states before hitting their
destination.