Point to Point Ethernet

Mailman List Mirror (Read Only)
21

The fundamental disconnect here is that a bunch of Layer 3 guys are
trying to define Layer 2.

History shows us that Layer 2 winds up being IEEE, and Layer 3 IETF.

ITU-T and others write long "standards" that wind up not being so, due
to too many "options", while spending lots of money and keeping the
airlines, rental car companies, and meeting space vendors in business.

If you want "Real Ethernet" (IE multiple access, not point to point) in
a metro framework, then why increase the likelihood of collisions by
using jumbo frames?

If you want to use Ethernet in point to point, then do it, just make
sure your optics are up to the task, and the endpoints are configured
properly.

If what you're looking for is carrier Ethernet with the sort of "craft"
interfaces and features you're used to from the telco world, then you
may want to talk to Ipitek. (I've done some consulting for them, but am
in no way affiliated or compensated for sales.)

From: Mikael Abrahamsson [mailto:swmike@swm.pp.se]
Sent: Wednesday, July 08, 2009 11:33 AM
To: Ricky Beam
Cc: nanog@nanog.org
Subject: Re: Point to Point Ethernet

Ethernet is not a point-to-point technology. It is a multi-point
(broadcast, bus, etc.) technology with DECADES of optimization and
adoption. No one has gotten IEEE to adopt a larger frame size, and

you

want to drop *fundamental* elements of ethernet?!?

I think the latest suggestion was to do away with the mechanisms, not
change the frame format. It's like when you take a /30, run isis/ospf
over
it and tell the routing protocol it's a point to point link so it
doesn't
have to create a node for the "multi access network" that really isn't
there.

Same way here, putting the ethernet link in "p2p" mode would mean it
wouldnt do arp anymore, didn't care about source or destination MACs,

it

just installed static ARP entry for other end and sent out packets,
other
end would be in promisc mode and accept anything.

I don't see much gain from this though, and it's another way things can
be
configured wrong and cause havoc if you connect this interface to a

LAN.

22

History shows us that Layer 2 winds up being IEEE, and Layer 3 IETF.

mpls

23

Best case, you blow 12 bytes on IFG in gig, 20 bytes on fast-e/slow-e.

As far as I know Gig and 10 Gig (with LAN PHY) are exactly the same
as 10 and 100 Mbps in this respect, i.e. 8 bytes of preamble and 12
bytes of IFG. So you always have an overhead of 20 bytes, no matter
what.

10 Gig with WAN PHY is a whole different ballgame, of course.

Steinar Haug, Nethelp consulting, sthaug@nethelp.no

24

Overhead shmoverhead.

Seriously, we're fighting over the non-issue. It's not the "wasted"
0.02% of bandwidth (@ 1Gbps) that's the issue. It's the utility of a
"come as you are" "plug and play" network that "Ethernet" (which really
loosely means all IEEE 802 protocols) provides, which the current
carrier networks do not.

If I read the thread correctly, what you really are asking for is the
ability to plug your IEEE compliant gig/10gig switch into a carrier port
and just have it ARP and respond for valid IP addresses on the segment,
as opposed to all the back and forth provisioning, truck rolls, and
interaction with bell-head union workers that the current system
requires.

Now, HOW to accomplish that is an interesting discussion, and the first
valid result will probably be a great business.

That doesn't require breaking Ethernet, using promiscuous mode, or much
except the carriers stopping trying to throw their legacy
circuit-switched requirements onto a packet switched network.

There's plenty of fiber in the ground. Light dark stuff with the new
network, plug it into IEEE 802* compliant layer 2, and IETF compliant
layer 3 infrastructure; and leave the dying Bellcore/ITU network on the
old copper and SONET.

From: sthaug@nethelp.no [mailto:sthaug@nethelp.no]
Sent: Wednesday, July 08, 2009 11:34 PM
To: tkapela@gmail.com
Cc: nanog@nanog.org
Subject: Re: Point to Point Ethernet

Best case, you blow 12 bytes on IFG in gig, 20 bytes on

fast-e/slow-e.

25

For the sake of my knowledge (and perhaps that of some others on the list),
I would like to ask if the current work on standards by IETF, ITU and IEEE
not a step to address the issue of seamlessly using Ethernet in the
metro/core?
IETF is working on GMPLS Ethernet Label Switching (GELS), which proposes to
replace the Ethernet control plane (MAC learning, spanning tree etc) by the
GMPLS control plane. This would provide explicitly routed Ethernet LSPs. ITU
seems to be working on Transport MPLS (T-MPLS), and IEEE seems to be at work
on the Provider Backbone Bridge (PBB) standard.
Granted the difficulties and faults with the standardization process, my
question is more concerned with the technical nature.
Thanks and best regards

26

Have you built an nationwide greenfield network based on dark fibre and ethernet? I have.

Don't misrepresent the problems that might arise from this. It might be easier to do compared to a SONET network, but it's still a can of worms and you definitely want a lot of the "bellhead" stuff you're ridiculing.

27

I frequently run into scenarios where two devices (two routers, or a
router and a host) need a point-to-point connection to each other with
a capacity of (much) more than 10 Gbps.

For cost reasons, Ethernet is often used.

Since more than 10 Gbps is needed, we end up with multiple parallel
10GE point-to-point connections.

Because the devices often don't support LAG or have limitations on the
number of links in a LAG, we often cannot use LAG at all or cannot put
all 10GE links in a single LAG group.

So, we end up with multiple parallel layer-3 point-to-point
connections where each connections is either an Ethernet or a LAG
group.

Furthermore, in order to conserve IP addresses, there is a desire to
make these interfaces unnumbered.

The involved devices have a numbered loopback interface whose address
is used as the "donor" for the unnumbered Ethernet / LAG interfaces.
Most router vendors already support unnumbered point-to-point
Ethernet, see for example:

http://www.juniper.net/techpubs/en_US/junos9.5/information-products/topic-collections/config-guide-network-interfaces/interfaces-configuring-an-unnumbered-interface.html#id-10432956

However, there are some interoperability issues / open questions
related to point-to-point unnumbered Ethernet, see for example:

http://forums.juniper.net/jnet/board/message?board.id=JUNOS&message.id=130

http://forums.juniper.net/jnet/board/message?board.id=switch&thread.id=835

I would be very interested in some standards (i.e. an IETF BCP) to
describe the best current practices for these applications of
Ethernet.

I am not particularly interested in re-inventing a new flavor of
Ethernet for this application. All that is needed, in my opinion, is
some clarifications or best practices on how to use the existing
standards to create point-to-point unnumbered Ethernet connections.

PS -- I am also aware of some esoteric BRAS applications of Ethernet
where one side is numbered and the other side is unnumbered.

28

Cayle,

This may be partial hijack of the thread or even a trivial query but I ask
this since you mentioned "For cost reasons, Ethernet is often used". We hear
this argument all the time. The standard unabridged reason I have learned is
the ubiquity of Ethernet devices, whatever that means. Can you say why
precisely the cost of Ethernet is low compared to other viable alternatives?

Zartash

29

The components going into ethernet devices are cheaper because of high volume, but it's also that the SONET/SDH stuff is grossly overpriced "because we can" by short sighted vendors. There are cheap ethernet ports for cheap platforms, there are basically no cheap SONET/SDH ports anywhere.

POS is technically better compared to Ethernet for carrier links imho, but for instance Cisco price their SPA-TENGE-XFP at 1/6 the cost of SPA-OC192-XFP.

I know quite a lot of people who would gladly pay more for POS, but not that much more.

30

Volume. Economies of scale. Etc.

Ethernet is cheap because it's everywhere, and built into almost everything. (however, the likes of Cisco and Juniper still charge insane amounts for line cards, be they ethernet, T1, or OC48.) Given the choice of buying a $4k DS3 card or just plugging into an existing, builtin ethernet port, which do you think most people will choose? And it doesn't take a multi-thousand dollar Router(tm) to deal with ethernet -- a 200$ "trash" PC can handle routing (and NAT) for a great deal of traffic. (in many cases, *better* than the high priced kit.)

Case in point, our voice/data line from TW enters the building as fiber (along with about 4000 other circuits), crawls up the inside of the building as HDSL (single pair) to a box in my "closet" where 8 POTS line and an ethernet are handed to me. The ethernet runs to a switch and then to the firewall. If it wasn't handed to me as ethernet, I'd need a router to turn it into ethernet. (On the other wall... $8k worth of gear to turn a DS3 into ethernet. Yes, the Optera Metro shelf at the other end of that DS3 could just as easily be an ethernet port -- but that would require TW and VZB to play nice with each other; it was enough of a pain to get the DS3 to work. But that's miles off topic.)

--Ricky

31

Zartash Uzmi wrote:

Can you say why precisely the cost of Ethernet is low compared to other viable alternatives?

Becuase there's a lot of it?

Gigabit ethernet ports cost less than 9600bps terminal server ports.

32

Once upon a time, Ricky Beam <jfbeam@gmail.com> said:

Ethernet is cheap because it's everywhere, and built into almost
everything. (however, the likes of Cisco and Juniper still charge insane
amounts for line cards, be they ethernet, T1, or OC48.) Given the choice
of buying a $4k DS3 card or just plugging into an existing, builtin
ethernet port, which do you think most people will choose?

Also, if you are plugging in a lower-speed link, you can plug ethernet
in a <$1000 switch and trunk it to a router, while a mux for T1/T3/OCx
circuits costs a lot more.

33

Chris Adams wrote:

Once upon a time, Ricky Beam <jfbeam@gmail.com> said:

Ethernet is cheap because it's everywhere, and built into almost
everything. (however, the likes of Cisco and Juniper still charge insane
amounts for line cards, be they ethernet, T1, or OC48.) Given the choice
of buying a $4k DS3 card or just plugging into an existing, builtin
ethernet port, which do you think most people will choose?

Also, if you are plugging in a lower-speed link, you can plug ethernet
in a <$1000 switch and trunk it to a router, while a mux for T1/T3/OCx
circuits costs a lot more.

I just ordered a circuit to be delivered over Ethernet - Verizon just
plugged a pair of STM-1's into an ISG5100 and it's suddenly ridiculously
cheaper.

~Seth

34

It should be noted that this usually isn't recommened. Dropping an
ethernet circuit directly into a switch (even if it is laywer 3) can
create design issues, esecially later when you need to scale the
network. One big issue that is often overlooked is many swithces do not
support traffic shaping. Some support policing, but shaping can be far
more efficient. There are some nortel switches that do this, but I
haven't seen many in the wild.

35

Hate to say it, but also some of the cost on the circuits can be blamed
on uncle Sam. ATM circuits are currently tariffed that same way are
voice circuits. These tariffs are not charged to Ethernet because it is
a 'data circuit'. At least that was the case a little while back.

36

Brian Raaen wrote:

Hate to say it, but also some of the cost on the circuits can be blamed
on uncle Sam. ATM circuits are currently tariffed that same way are
voice circuits. These tariffs are not charged to Ethernet because it is
a 'data circuit'. At least that was the case a little while back.

Are you sure it's "Uncle Sam"? My experience is that voice tariffs are
always cheaper than data; telco's mantra is still "I Smell Dollars Now".

The telcos were mightily pissed when we redesigned protocols to pass over
voice circuits instead of requiring data circuits.

Usually, non-tariffed lines seem to be much more expensive, as the account
manager says "Oh, that special order will have to be approved by HQ".

37

Brian Raaen wrote:

Hate to say it, but also some of the cost on the circuits can be blamed
on uncle Sam. ATM circuits are currently tariffed that same way are
voice circuits. These tariffs are not charged to Ethernet because it is
a 'data circuit'. At least that was the case a little while back.

?
Are you sure it's "Uncle Sam"? My experience is that voice tariffs are
always cheaper than data; telco's mantra is still "I Smell Dollars Now".

The telcos were mightily pissed when we redesigned protocols to pass over
voice circuits instead of requiring data circuits.

Usually, non-tariffed lines seem to be much more expensive, as the account
manager says "Oh, that special order will have to be approved by HQ".

Strictly speaking, it's not Uncle Sam, but the PUCs who review the tariffs.

I would view it fundamentally as a lack of competition. Who can provide ATM backhaul from central offices? In many cases just the incumbent.

Roderick S. Beck
Director of European Sales
Hibernia Atlantic

38

Prices of terrestrial SDH/SONET cards are very low for transport providers. For customers I believe there is a greater divergenc between the Ethernet and SONET/SDH costs.

A strong hunch based on what clients tell me Cisco charges for SONET/SDH interfaces.

Roderick S. Beck
Director of European Sales
Hibernia Atlantic
13-15, rue Sedaine, 75011 Paris
http://www.hiberniaatlantic.com
Wireless: 33+6+8692+5357.
French Landline: 33+1+4355+8224
AOL Messenger: GlobalBandwidth
rod.beck@hiberniaatlantic.com
info@globalwholesalebandwidht.com
``Unthinking respect for authority is the greatest enemy of truth.'' Albert Einstein.

39

Prices of terrestrial SDH/SONET cards are very low for transport providers. For customers I believe there is a greater divergenc between the Ethernet and SONET/SDH costs.

A strong hunch based on what clients tell me Cisco charges for SONET/SDH interfaces.

I doubt a lot of people would think that SDH/SONET cards for *routers*
are inexpensive. And yes, I have a reasonable idea of what kind of
discounts are available out there...

Steinar Haug, Nethelp consulting, sthaug@nethelp.no

40

Prices of terrestrial SDH/SONET cards are very low for transport providers. For customers I believe there is a greater divergenc between the Ethernet and SONET/SDH costs.

A strong hunch based on what clients tell me Cisco charges for SONET/SDH interfaces.

I doubt a lot of people would think that SDH/SONET cards for *routers*
are inexpensive. And yes, I have a reasonable idea of what kind of
discounts are available out there...

Steinar Haug, Nethelp consulting, sthaug@nethelp.no

Transport in my world is not Layer 3. It's Layer 2: Ethernet, SDH, SONET, waves.

But my clients are mostly Layer 3.

So you and I are in fundamental agreement.

Roderick S. Beck
Director of European Sales
Hibernia Atlantic