Is it fair to say that most traffic drops happen in the access layers, or
the first and the last miles, and the % of packet drops in the core are
minimal? So, if the packet has made it past the first mile and has
"entered" the core then chances are high that the packet will safely get
across till the exit in the core. Sure once it gets off the core, then all
bets are off on whether it will get dropped or not. However, the key point
is that the core usually does not drop too many packets - the probability
of drops are highest in the access side.
Youre saying that the probability of packet drop at peering points would
roughly match that at the edge. Is it? I thought that most core switches
have minimal buffering and really do cut-through forwarding. The idea is
that the traffic that they receive is already shaped by the upstream
routers.
It a capacity question. Several core networks [cough Verizon cough]
intentionally under-provision the "settlement-free" peering links to
other core networks. You can't cut-through when the destination
interface already has a queue of packets waiting to be sent.
I would say that the probability of a packet drop at any particular peering
point is less than the probability at one of the two edges.
However, given that most packets are likely to traverse multiple peering
points between the two edges, the probability of a packet drop along
the way at one of the several peering points overall is roughly equal
to the probability of a drop at one of the two edges.
If you first list the causes of a dropped packet, then you can figure out
how likely they are at different points in time (first\last\peer\etc) by
making some assumptions.
Here's an **example**:
*Cause | Location | Likelihood*
Congestion | Last mile | Low
Congestion | First mile | Low
Congestion | Peering | Medium
Layer 1 | First mile | Low
Layer 1 | Core | Low
Layer 1 | Last mile | High
You can even go as far as drawing a cause and effect diagram for each
location. Then you can collect real world data and fine tune your
assumptions.
Generally speaking there are zero or one settlement free peering
points in the active path between any two edges. Not always, but close
enough to it to discount the exceptions.
Speaking for my own experience, I almost never see loss on my Verizon
FiOS edge but see loss at the various Verizon borders with other
networks -all the time-. They keep pitching me on upgrading to 75/75
but that isn't the upgrade I need.
Is there a paper or a presentation that discusses the drops in the core?
If i were to break the total path into three legs -- the first, middle and
the last, then are you saying that the probability of packet loss is
perhaps 1/3 in each leg (because the packet passes through different IXes).
That sounds too aggressive for the middle mile. Dont you think so?
That was just an example, that list has to be completed on a specific
network or scenario, it changes dramatically. Imagine you were to create a
list for a DoD network instead of public peering based network, it would
change dramatically.
Is there a paper or a presentation that discusses the drops in the core?
Hi Glen,
Probably, but I don't know where to point you.
If i were to break the total path into three legs -- the first, middle and
the last, then are you saying that the probability of packet loss is perhaps
1/3 in each leg (because the packet passes through different IXes). That
sounds too aggressive for the middle mile. Dont you think so?
Break it in to five legs:
1. Your immediate last mile
2. The set of networks you directly or indirectly pay to transmit and
receive packets
3. The border link between your networks and the remote user's networks
4. The set of networks the remote user directly or indirectly pays to
transmit and receive packets
5. The remote user's immediate last mile
In some cases, your packets meet on a network which both you and the
remote user pay for. In those cases, leg 3 does not exist. However,
those cases are less common than the one where neither of you pays the
same networks.
Legs 1 and 5 are often over noisy copper wire suspended from a street pole.
Leg 3 is routinely under-provisioned (too little bandwidth for the
traffic demand).
Legs 2 and 4 rarely exhibit loss for long.
Quite the inverse, I'd say; most of the capacity
headaches center around the handoff between
networks, and most of the congestion points
I come across are with private peering links
where one party or the other is unwilling or
unable to augment capacity. The first and
last mile are fine, but the handoff between
the networks is where congestion and drops
occur.
As others have noted, this will vary greatly
depending on the network in question--so
asking a broad community like this is going
to yield a broad range of answers. You
aren't going to find one single answer, you'll
find a probability curve that represents the
answers from many people running different
networks.
You'll find the location of packet drops tends
to shift depending on where companies are
willing to spend money; some companies
will spend money on the access layer to
ensure no drops happen there, but are
less willing to pay for capacity upgrades
at peering handoffs. Other networks will
short-change their access, but maintain a
well-connected peering edge.
So--short answer is there is no one answer
to your question. Collect the different answers,
plot the curve, and decide where along the
curve you want *your* network to land,
and build accordingly. Nobody has infinite
money, so nobody builds to a level to ensure
zero loss probability to every destination around
the planet.
1. TCP (and most other IP protocols) depends on, and forces packet congestion and drops. Packet drops alone are not necessarily a measure of network quality. Other than some laboratory conditions,
there must always be some congestion somewhere.
2. Packet queuing and drops are most likely at network transition points. Usually speed or latency transition points, but also network administration transition points.
3. Packet queuing and drops are less likely between network transition points, i.e. across the same network (LAN, WAN, ISP, etc).
That's why some ISPs claim they have 0% packet loss on their network. They don't include network transition points in their statistics; but have worse end-to-end performance than another network which includes 0.1%
packet drops in their reported statistics.
Generally I don't believe ISPs that claim 100% uptime or 0% packet loss.
Is there a paper or a presentation that discusses the drops in the core?
If i were to break the total path into three legs -- the first, middle
and the last, then are you saying that the probability of packet loss
is perhaps 1/3 in each leg (because the packet passes through
different IXes).
It is unlikely packets pass through an IXP more then once.
“Unlikely”? That’s putting it mildly.
Unless someone is selling transit over an IX, I do not see how it can happen. And I would characterize transit over IXes far more pessimistically than “unlikely”.
There is another scenario (which unfortunatly is not that uncommon)
where packets could traverse two IXPs, and no transit is sold over any
of those two IXs.
Imagine the following:
Network A purchases transit from network B & network C. Network B &
Network C peer with each other via an IXP. Network A announces a /16 to
network B but 2 x /17 to network C. Network D peers with B via an IX
(and not with C) and receives the /16 from B, but note that internally
network B has two more specifics covering the /16 received from C and
the /16 itself. Network B will export the /16 (received from customer)
but not the /17s (received over peering) to its peers.
Because of longest prefix matching, network B will route the packets
received from network D over an IXP, towards network C, again over an
IXP.
This phenomenon is described extensively in the following
Internet-Draft:
Although I have trouble believing it is very common, in the sense that I do not believe it is a large number of packets or percent of traffic.
To be clear, I fully believe people are doing the more specifics to provider B but not C. Sometimes there is even a good reason for it (although probably not usually). However, most of the Internet will send traffic directly to B, or even A - especially since most packets are sourced from CDNs[*].
I'm told it happens relatively often in networks supporting a lot of
schools. Being an unpaid pass-through for schools paying other ISPs
functions as a loss-leader that attracts more schools as customers.
Lots of people have mentioned “but XXX happens” to me. And you are all correct. XXX happens.
My point was not “this never happens”. Just those other topologies are a tiny, tiny fraction of the packets flowing on the Internet.
Most packets flow from CDNs to broadband. And those packets flow mostly direct (on-net or PNI), or over a _single_ IXP. Corner cases exist, but they are just that - corner cases.