We've run into an issue with a customer that has been confounding us for a few
months as we try to design what they need.
The customer has a location in the relative middle of nowhere that they are
trying to build a protected OC3 to. Ultimately, their traffic on it will be
packet data (IP/ethernet, not channelized/voice). But they seem to be
absolutely 100% set on the idea that they build with Cisco ONS boxes and that
they run and control the D1-D12 bytes in order to manage protection switching
on the OC3 (and have their DCC channel for management).
Since this is the middle of nowhere, we are having to piece it together from a
few runs of dark fiber here and there and lit services from about 3 other
providers to get from the desired point A to the desired point B. The issues
we seem to be hitting are:
-We seem to be unable to find anyone who sells lit OC3 with D1-D12
transparency for the client. Sometimes we can get D1-D3, but that's it.
-lit OC3/12/48 is ridiculously expensive comapred to 1g ethernet waves or 10g
waves (choice LAN/WAN ethernet or OC192)
10g waves are cheap enough that we have entertained the idea of buying them and
putting OC-192/muxponders on the ends to provide the OC-3, but even then I'm
having trouble finding boxes that will do D1-D12 transparency for client OC-3.
Building the whole thing on dark fiber so that we could specify the exact
equipment on every hop isn't going to happen, as the "protect" path is about
1000 miles and the geography is such that we don't really have a market for all
the other wasted capacity there would be on that path.
Having much more experience with ethernet/packet/MPLS setups, we are trying to
get the client to admit that 1g/10g waves running ethernet with QoS would be as
good as or better in terms of latency, jitter, and loss for their packet data.
So far they will barely listen to the arguments. And then going the next leap
and showing them that we could work towards <50ms protection switching with
MPLS/BFD/etc packet-based protocols is another stretch.
Am I missing something here that my customer isn't, or is it the other way
around?
Not sure if I see the problem here. Show them the bill for an OC3 service,
and then show them the bill for the equivalent ethernet service. This
usually works for me. If they want to pay for OC3 when there's no
compelling reason to, who are you to argue?
Yes, of course the response is, "figure out how to make the OC3 cheaper".
So we're rapidly approaching a response of "you've engineered yourself into
a corner, take it or leave it".
I just can't see how to get OC3 D1-D12 tunneled through without doing it as
a mix of OC-48 and dark fiber for the entire path and specifying lots of
complicated boxes just to get those bytes through. That cost is an order of
magnitude more than just buying OC3 from multiple carriers (who can't tunnel
D1-D12), which is a magnitude more than buying mpls-based gig-e or gig-e
wave.
I wasn't around (well, I was just a T1/DS3 customer) when all the OC3/12/48
SONET networks were built 10-15 years ago. I suppose they were all built
directly on the fiber (maybe with WDM but no layer1.5-2 muxing) and the
provider was always the one who handled protection switching?
I've considered using J's PE-4CHOC3-CE-SFP (OC3 emulated SAToP), then I
could do it all with gig-e underneath. Does anyone make a cheaper OC3
circuit emulation module or box? Most likely the customer wouldn't believe
such a thing is possible and we'd have to put something in the contract
allowing them SLA credit if their OC3 suffers too many timing slips or
something.
A few of the engineers at $DAYJOB still try and claim SONET is easier to
troubleshoot, but that hasn't been my practical experience.
With ethernet it's something like:
- Layer 1 - light levels (DoM on nearly everything)
- Layer 1 - link pulse
- Layer 2 - can I send frames
SONET it's, in practice:
- Layer 1 - light levels (DoM on newer kit, SOL on older)
- Layer 2 - Seemingly random collection of alarms
- Layer 2 - Is PPP up?
Sure being able tell a provider that our kit is showing a particular
alarm can sometimes be helpful, but for every time that's been helpful
I've seen multiple circuits down for timing, often for no explainable
reason (I have atomic standards at home, and still can't explain a few
of the cases I've seen).
As others have said doing a "diverse 1/10g ethernet" quote and a
"protected SONET" quote may be worthwhile, and adding a 20% pad to the
SONET one for staff training may also be perfectly justifiable.
Also other then the ONS15300 series (which I'm amazed haven't been
discontinued) I can see nothing that actually offers OC3 line ports,
building something new using those today seems like using a 2500 as a
console server, sure for a lab or demo perhaps, but otherwise I'm
staying well away from them.
A few of the engineers at $DAYJOB still try and claim SONET is easier to
troubleshoot, but that hasn't been my practical experience.
With ethernet it's something like:
- Layer 1 - light levels (DoM on nearly everything)
- Layer 1 - link pulse
- Layer 2 - can I send frames
SONET it's, in practice:
- Layer 1 - light levels (DoM on newer kit, SOL on older)
- Layer 2 - Seemingly random collection of alarms
- Layer 2 - Is PPP up?
Just the fact that BFD had to be reinvented shows that there is ample
reason to prefer the steady-train-of-frames-with-status of SONET/SDH over
perhaps-nobody-sent-a-packet-or-the-line-is-dead quagmire of Ethernet --
I have run pretty large (for sweden) networks over SDH (POS linecards on
top of waves, not a full SDH system) and essentially similar networks as
GE over waves, and I truly prefer the failure modes and analysis tools
in SDH to the guesswork and afterthought patches of alohanet..
Still, the stupid f€%&€/# that make prices for linecards made me go GE
instead of OC48 for the most recent deployment. In Sweden, both vendors
claim about 6 times as much, per megabit, for SDH line cards.
This can't really make sense.
As others have said doing a "diverse 1/10g ethernet" quote and a
"protected SONET" quote may be worthwhile, and adding a 20% pad to the
SONET one for staff training may also be perfectly justifiable.
Maybe training is more expensive (it takes some CPU to parse SLOF/SLOS
and PLOP etc) but it leads to lower OPEX since the "Maybe" factor is
essentially gone. Operationally it is quite worthwhile to say "I have
SLOS in my far end, which means somebody pulled a patch worngly in
your just terminated maintenance window." instead of "The line is dead,
can you please check something?" to your circuit provider.
Yeah, SDH and similar probably will die, but cheap aint good. Only.
Not all Ethernet switching implementations are necessarily equal;
there are 802.3ah OA&M and 802.1ag connectivity fault management /
Loopback (MAC ping) / Continuity Check Protocol / Link Trace. (Which
aren't much use without management software, however.)
There /are/ reasons to prefer SONET for certain networks or
applications; so it might (or might not) be a reasonable requirement,
it just depends.
Price is not one of those reasons; all the added complexity and use
of less common equipment has some major costs, not to mention risks,
involved if mixing many different service providers' products. SONET
comes at a massive price premium per port and switching table entry on
hardware modules that are much more expensive than 10g switches, and
providers often charge a big premium regardless...
Therefore; it is not the least bit surprising that a 10g wave would be
massively less expensive in many cases than an OC3 over a long
distance between point A and point B.
As I see it... if you are thinking of 1000 miles of dark fiber to
nowhere to support an OC3, then forget the "wasted" capacity; the
cost of all that dark fiber needed just for them should get added to
the customer's price quote for the OC3.
Same deal if instead you need an OC48 at various hops to actually
carry that OC3 and be able to end-to-end and tunnel the DCC bytes over
IP or restrict equipment choices so you can achieve that D1-12 byte
transparency....
> Subject: Re: Are people still building SONET networks from scratch? Date:
> Just the fact that BFD had to be reinvented shows that there is ample
> reason to prefer the steady-train-of-frames-with-status of SONET/SDH over
> perhaps-nobody-sent-a-packet-or-the-line-is-dead quagmire of Ethernet --
Not all Ethernet switching implementations are necessarily equal;
there are 802.3ah OA&M and 802.1ag connectivity fault management /
Loopback (MAC ping) / Continuity Check Protocol / Link Trace. (Which
aren't much use without management software, however.)
Of course.
There /are/ reasons to prefer SONET for certain networks or
applications; so it might (or might not) be a reasonable requirement,
it just depends.
Yes.
Price is not one of those reasons; all the added complexity and use
of less common equipment has some major costs, not to mention risks,
involved if mixing many different service providers' products. SONET
comes at a massive price premium per port and switching table entry on
hardware modules that are much more expensive than 10g switches, and
providers often charge a big premium regardless...
Yes. The 6x difference I alluded to was a comparison of line cards for
OC192 and 10GE on major league routers, like CRS or T-series. Most of
the bits are the same, yet the price \delta is insane.
Therefore; it is not the least bit surprising that a 10g wave would be
massively less expensive in many cases than an OC3 over a long
distance between point A and point B.
Especially since it might be possible to get it provisioneed e2e.
As I see it... if you are thinking of 1000 miles of dark fiber to
nowhere to support an OC3, then forget the "wasted" capacity; the
cost of all that dark fiber needed just for them should get added to
the customer's price quote for the OC3.
Yup.
Same deal if instead you need an OC48 at various hops to actually
carry that OC3 and be able to end-to-end and tunnel the DCC bytes over
IP or restrict equipment choices so you can achieve that D1-12 byte
transparency....
I'm a simple man. I just want the bitpipe to do IP over. It so happens
that the combined engineering of the telco business made for a nice
set of signalling bells and whistles that tend to work well on long
point-to-point circuits. If not perfectly well, then at least orders of
magnitude better than a protocol that was designed to sometimes convey
frames over one nautical mile of yellow coax.
Then again, the yellow coax has evolved, significantly.
The "once-in-a-lifetime" that happened here (took a while though) was WAN PHY for 10GE. All the benefit of SDH at Ethernet cost.
When I approached the 40GE/100GE working group about getting a few of the benefits of this into that standard, I was instantly shut down by people who thought WAN PHY was a huge mistake that shouldn't be repeated.
The only thing I wanted was to have frames sent all the times so one would get a basic bit error reading, plus having the PHY send some basic information such as "I am seeing light and my world looks ok", so we could get PHY based interface-down when there was a fiber cut.
But that wasn't to be, so the only way to get this will be to OTN-frame the whole thing I guess. *sigh*
I've considered using J's PE-4CHOC3-CE-SFP (OC3 emulated SAToP), then I
could do it all with gig-e underneath. Does anyone make a cheaper OC3
circuit emulation module or box? Most likely the customer wouldn't believe
such a thing is possible and we'd have to put something in the contract
allowing them SLA credit if their OC3 suffers too many timing slips or
something.
And so you find yourself at the intersection of two timeless maxims:
1) The customer is always right, but not everyone needs to be our customer.
2) Don't say no to the customer, let the customer say no thanks.
Time to model the cost/benefit/profit margin of having these folks as
a customer at all (I'd imagine that this circuit is not the only thing
that they buy from you or you'd be running away even today). What are
your engineering costs for this trick? Are you passing that on to the
customer?
You may find it advantageous to do a pricing model where you do
circuit emulation on a hope-for-the-best basis and count on a maximum
SLA payout every month (and still make money). Then if you fail to
pay SLA credits from time to time, that's pure gravy.
In my neck of the woods, critical locations often exist "in the middle of
nowhere", resulting in underserved facilities, where best effort networks
such as metro Ethernet cannot be trusted to remain available 24x7x365. Many
times, during prime business hours, I will see a telco metro Ethernet
spanning tree convergence which results in my traffic re-routing for 20-30
seconds over my private backup network path, then switching back to the
metro Ethernet path after the telco technicians have finished their
maintenance. Several times when I have called in a trouble ticket, the
telco tech has asked "what is the big deal, it was only a 20 second
outage?". In the Enterprise environment, a planned spanning tree
convergence in the middle of business hours is one of the quickest ways for
a network engineer to be relieved of their duties, but apparently the bar
is considerably lower in the telco environment.
Not only that, but the telco SLAs associated with metro Ethernet are
totally bogus, with a best round trip SLA of 20 milliseconds, ranging up to
50 milliseconds for "bronze" service. For short distances of 100 miles or
less (rule of thumb is that light travels over fiber at 0.80 x speed of
light, or 1000 miles in 10 milliseconds), an SLA of 20-50 milliseconds
amounts to fraud, just another way for the telcos to scam the consumer.
The tone of many of the entries on this thread where the user is depicted
as being unreasonable, underscores the need for a coordinated national
broadband policy in the USA, based upon the Australian model in which the
government is building out fiber to every residence and business, no matter
where they are located.
In my neck of the woods, critical locations often exist "in the middle of
nowhere", resulting in underserved facilities, where best effort networks
such as metro Ethernet cannot be trusted to remain available 24x7x365. Many
times, during prime business hours, I will see a telco metro Ethernet
spanning tree convergence which results in my traffic re-routing for 20-30
seconds over my private backup network path, then switching back to the
metro Ethernet path after the telco technicians have finished their
maintenance. Several times when I have called in a trouble ticket, the
telco tech has asked "what is the big deal, it was only a 20 second
outage?". In the Enterprise environment, a planned spanning tree
convergence in the middle of business hours is one of the quickest ways for
a network engineer to be relieved of their duties, but apparently the bar
is considerably lower in the telco environment.
Not only that, but the telco SLAs associated with metro Ethernet are
totally bogus, with a best round trip SLA of 20 milliseconds, ranging up to
50 milliseconds for "bronze" service. For short distances of 100 miles or
less (rule of thumb is that light travels over fiber at 0.80 x speed of
light, or 1000 miles in 10 milliseconds), an SLA of 20-50 milliseconds
amounts to fraud, just another way for the telcos to scam the consumer.
The tone of many of the entries on this thread where the user is depicted
as being unreasonable, underscores the need for a coordinated national
broadband policy in the USA, based upon the Australian model in which the
government is building out fiber to every residence and business, no matter
where they are located.
Regards,
David
If service is critical enough to me that 20 second hiccups make
a difference, I'll find two providers to provide connectivity to the
location via relatively cheap waves, and I'll run link-node protection
at my layer to get fast reconvergence in the sub-second range.
And I'd warrant it'll still come out cheaper than the government-built
costs we see in Australia.
I really, really don't think more government intervention is the right
answer.
*heh* Fair enough. I guess it's really a question of
what scale you're looking at. Even when building a
greenfield datacenter in the middle of an empty field
in an out-of-the-way corner of a state with cheap
electricity, you can generally find two fiber providers
that will run fiber into the location based on the
premise of the long-term payback a 50MW datacenter
brings with it.
At smaller scales, I could see how it could become
more challenging to convince a second provider it's
worth their while to build out to your location.
In my neck of the woods, critical locations often exist "in the middle of
nowhere", resulting in underserved facilities, where best effort networks
such as metro Ethernet cannot be trusted to remain available 24x7x365. Many
times, during prime business hours, I will see a telco metro Ethernet
spanning tree convergence which results in my traffic re-routing for 20-30
seconds over my private backup network path, then switching back to the
metro Ethernet path after the telco technicians have finished their
maintenance. Several times when I have called in a trouble ticket, the
telco tech has asked "what is the big deal, it was only a 20 second
outage?". In the Enterprise environment, a planned spanning tree
convergence in the middle of business hours is one of the quickest ways for
a network engineer to be relieved of their duties, but apparently the bar
is considerably lower in the telco environment.
Not only that, but the telco SLAs associated with metro Ethernet are
totally bogus, with a best round trip SLA of 20 milliseconds, ranging up to
50 milliseconds for "bronze" service. For short distances of 100 miles or
less (rule of thumb is that light travels over fiber at 0.80 x speed of
light, or 1000 miles in 10 milliseconds), an SLA of 20-50 milliseconds
amounts to fraud, just another way for the telcos to scam the consumer.
The tone of many of the entries on this thread where the user is depicted
as being unreasonable, underscores the need for a coordinated national
broadband policy in the USA, based upon the Australian model in which the
government is building out fiber to every residence and business, no matter
where they are located.
The NBN is to be delivered over a mixture of fibre (93% of homes), wireless
and satellite services[1].
