Hello folks,
Simple question: does “routed optical networks” have a clear meaning in the metro area context, or not?
Put differently: does it call to mind a well-defined stack of technologies in the control and data planes of metro-area networks?
I’m asking because I’m having some thoughts about the clarity of this term, in the process of carrying out a qualitative survey of the results of the metro-area networks survey.
Cheers,
Etienne
Maybe some clarification as to what you’re asking for would help. You’re mixing fiber, networks, and a MAN. Fiber is just the medium. It could be for IP switching or projecting a light show. Are you asking if there are diverse paths throughout a metro area?
Hi Etienne
In short, the idea is that optical networks are wasteful and routers do a better job making more use of a network’s capacity than ROADMs. Take the extra router hop (or 3 or 8) versus short-cutting it with an optical network because the silicon is so low-latency anyway that it hardly makes a difference now. Putting more GBs per second on fewer strands means saving a lot of money on infrastructure costs.
400G ZR comes to mind as a foundational technology since it basically made active optical muxponder equipment obsolete in the metro. The savings here means telcos/enterprises can afford more router ports, which we’ve already established can utilize paths more efficiently anyway. Otherwise, this is more of a concept and can be executed with a variety of pre-existing technologies, or someone’s new secret sauce that bakes everything together like SD-WAN did to its constituent technologies.
-Matt
Hi Etienne,
It depends on who is the owner of the fiber.
The incumbent carrier typically has enough fiber strands to avoid any colored interfaces (that are 3x expensive compare to gray) in the Metro.
Metro ring typically has 8-10 nodes (or similar). 16-20 strands of fiber were not possible to construct anyway – any cable is bigger.
It is the same cost to lay down fiber on 16 strands or 32.
Hence, PTT just does not need DWDM in Metro, not at all. Hence, the DWDM optimization that you are talking about below is not needed too.
If you rent a single pair of fiber then you need colored interfaces to multiplex 8-10 nodes into 1 pair on the ring.
Then the movement of transponders from DWDM into the router would eliminate 2 gray interfaces on every node (4 per link): one on the router side, and another on the DWDM side.
Overall, it is about a 25% cost cut of the whole “router+DWDM”.
It is still 2x more expensive compare to using additional fiber strands on YOUR fiber.
By the way, about “well-defined stack of technologies”:
NMS (polished by SDN our days) should be cross-layer: it should manage at the same time: ROADM/OADM in DWDM and colored laser in Router.
It is a vendor lock up to now (no multi-vendor). Hence, 25% cost savings would go to the vendor that has such NMS, not to the carrier.
Technology still does not make sense because no multivendor support between the NMS of one vendor and the router or DWDM of another.
Looking at the NMS history, it would probably never be multi-vendor. For that reason, I am pessimistic about the future of the colored interfaces in routers (and alien lambdas in DWDM). Despite a potential 25% cost advantage in eliminating gray interfaces.
PS: “routed optical networks” is proprietary marketing. Nobody understands what you mean. I did google to understand.
Eduard
Josh, thank you, your remarks (and those of Matt and Eduard) are helping me to understand better.
For some context, please look at this graphic that shows the results of the question
“Which of the following best describes your current dominant form of metro-aggregation?”
The left bar chart shows *NOG reponses;
the right bar chart shows responses obtained through market research among Tier 1 and regional operators.
Operator group respondents overwhelmingly selected “routed optical networks over Ethernet without ROADMs”.
Now, during an interview I held to assess the answers,
the interviewee (an experienced network engineer) questioned the meaning.
I realized that I may have been conditioned by Cisco marketing (I attend a few webinars), and
I wanted to understand what respondents understood.
Summarizing an answer to your observation, I was conditioned by Cisco marketing.
Cheers,
Etienne
Hi Etienne,
Below is our (Cisco) definition of the Routed Optical Network. The goal, metro or long haul or subsea, is to reduce the number of control planes. By migration TDM traffic using CEM or PLE to the IP layer, you eliminate the OTN control plane and management. Eventually, when standards are settled the ultimate goal is to have a single control plane for the network. I’m not trying to be a commercial here, but you can read more in the resources section on this page: https://www.cisco.com/c/en/us/solutions/service-provider/routed-optical-networking/index.html
HTH,
Eve
Routed optical networking, is an architecture that delivers improved operational efficiencies and simplicity. The solution works by merging IP and private line services onto a single layer where all the switching is done at Layer 3. Routers are connected with standardized 400G ZR/ZR+ coherent pluggable optics.
With a single service layer based upon IP, flexible management tools can leverage telemetry and model-driven programmability to streamline lifecycle operations. This simplified architecture integrates open data models and standard APIs, enabling a provider to focus on automation initiatives for a simpler topology.
This is a very convoluted way of backing into the ole packet-switched
vs. circuit switched decision.
Hello Eve,
Thank you for weighing in; I’m eager for feedback from the field.
This eagerness stems from my work, over the past two years,
to form my understanding of where current- and next-gen metro area networks are heading.
I need this understanding to help academics in my field of specialization
to better understand energy consumption in metro-area networks.
Your observation about elimination of OTN resonates well
with what I’ve heard from webinars, and what I’ve read in studies.
It also matches what I’ve shown in the graphic I linked to in an earlier post in this thread (this graphic).
However, the larger operators are less inclined to drop OTN as a server layer network
(layer network used as defined in G.805).
Indeed, part of the scope of the question leading to the results shown,
actually was to try to understand the prevalence of OTN in operators’ current networks.
As regards greenfield, the *NOG results are a bit more nuanced.
IP/MPLS over Ethernet over DWDM with ROADMs for node bypass gets 34% of the vote,
up from about 13% of what is currently in their networks.
Cheers,
Etienne
Very helpful observations, Matt, thank you.
How comfortably does the phrase “routed optical networks over Ethernet without ROADMs” sit with you?
I mean: would you accept a limitation of “optical network” to the case of
a network without optical layer switching (of the type done by add-drop multiplexers)?
Cheers,
Etienne
In short, the idea is that optical networks are wasteful and routers do a
better job making more use of a network’s capacity than ROADMs. Take the
extra router hop (or 3 or 8) versus short-cutting it with an optical
network because the silicon is so low-latency anyway that it hardly makes a
difference now. Putting more GBs per second on fewer strands means saving a
lot of money on infrastructure costs.This is a very convoluted way of backing into the ole packet-switched
vs. circuit switched decision.
I don’t follow.While ROADMs can be thought of as circuit-switchers,
the number of concurrent clients and switching latency put ROADMs on a different operational level than packet switchers, right?
Cheers,
Etienne
I’ve seen proposals for an LSR MPLS/ROADAM type solution, where imagine you are at a hop where in a long distance system solution, you would end up with OEO, but instead you get directionality capability with an IP/MPLS capable device. As mentioned previously, the 400-ZR/ZR+/ZR-Bright/+0 optics are the latest example of that.
I know of a few companies that have looked at solutions like this, and can expect there to be some interesting solutions that would appear as a result. Optical line systems tend to have pretty low power requirements compared to a router, but some of the routers are getting pretty low power as well when it comes to the power OPEX/bit, and if you have the ability to deliver services as an integrated packet optical you could see reduced costs and simplified components/sparing.
I’ll also say that I’ve not yet seen the price compression that I had expected in the space yet, but I figure that’s coming. We are seeing the bits/watt ratio improve though, so for the same or less power consumption you get more bits. Some of this technology stuff is truly magical.
- Jared
So right Jared…magic has been in the NPU capacity increase that’s driven the cost per 100G down on 1RU routers; and the integration of DSPs and more into QSFP-DD form factors at much lower power than expected. The standards for optical links are maturing as well, but we still have work to do on the management side for the electrical interfaces.
Eve
I’ve seen proposals for an LSR MPLS/ROADAM type solution, where imagine you are at a hop where in a long distance system solution, you would end up with OEO, but instead you get directionality capability with an IP/MPLS capable device. As mentioned previously, the 400-ZR/ZR+/ZR-Bright/+0 optics are the latest example of that.
Jared, I understand your point in the above statement to be that
directionality is cost-effectively implemented through label-switched paths,
rather than (ROADM-enabled) optical path switching.
Do I understand right?
Thank you.
Etienne
This may or may not always be the case. Especially for large carriers, where there could be a requirement to sell some of those dark fibre pairs to large customers (think the content folk coming into town, e.t.c.), they may no longer have the priviledge of having plenty of free fibre in the metro. Or if they did, the rate of traffic expansion means they burn through those fibre pairs pretty quick. 10Gbps isn’t a lot nowadays, and 100Gbps may start to push the limits depending on the size of the operator, the scope of the Metro-E ring and the level of service that needs to be maintained during a re-route (two available paths in the ring could balance 100Gbps of traffic, but if one half of that ring breaks, the remaining path may need to carry a lot more than 100Gbps, and then packets start to fall flat on the floor). At that scale, DWDM in the metro will make sense, at least more sense than 400G-ZR, at the moment. Some operators would also be selling Transport services in or along the metro, and customers paying for that may require that they do not cross a router device. There are plenty of DWDM pizza boxes that cost next to nothing. At scale, the price of these is not a stumbling block. And certainly, the price of these would be far lower than a router line card. OpenROADM is a good initiative. But it seems it’s to be to Transport equipment vendors what IPv6 and DNSSEC is to the IP world :-). Mark.
A fight that will never go away.
There has been some compromise in recent years, with Transport-heavy customers accepting standard Ethernet services, but only if they are carried by a Transport device.
Mark.
I’ve seen proposals for an LSR MPLS/ROADAM type solution, where imagine you are at a hop where in a long distance system solution, you would end up with OEO, but instead you get directionality capability with an IP/MPLS capable device.
My memory is rather fuzzy, but didn't Juniper attempt something like this in their PTX's after they picked up BTI? I think the plan was to co-locate the ROADM at the bottom of the PTX chassis, or something along those lines.
I know Cisco (and Juniper) tried by integrating GMPLS into their code as a starting point, but that didn't go very far with customers. It just seemed impossible for the Transport teams to allow the IP/MPLS teams that level of access into their line system :-).
As mentioned previously, the 400-ZR/ZR+/ZR-Bright/+0 optics are the latest example of that.
A rather high barrier to entry for most operators, but we have to start from somewhere.
I know of a few companies that have looked at solutions like this, and can expect there to be some interesting solutions that would appear as a result. Optical line systems tend to have pretty low power requirements compared to a router, but some of the routers are getting pretty low power as well when it comes to the power OPEX/bit, and if you have the ability to deliver services as an integrated packet optical you could see reduced costs and simplified components/sparing.
The main problem is distance.
If you need to move that kind of capacity more than 50km, it's hard to avoid DWDM.
I’ll also say that I’ve not yet seen the price compression that I had expected in the space yet, but I figure that’s coming. We are seeing the bits/watt ratio improve though, so for the same or less power consumption you get more bits. Some of this technology stuff is truly magical.
I think for long spans, DWDM will not only be cheaper, but the only feasible solution.
For the metro, it will come down to what motivates the business... plenty of features, or plenty of speed.
Also, DWDM vendors are adding speed and distance faster and cheaper than the IP/MPLS vendors can. So they will always be one step ahead in that respect; and we have the submarine cable systems to thank for that.
Mark.
To be honest, I’ve been hearing about this since as long as I can remember. IPoDWDM was another attempt at trying to make the above a reality. But for some reason, operators prefer to keep these networks separate, and many customers, especially very large ones, prefer to bypass routers for their Transport services. I think the effort will be appreciated, but if history is anything to go by, vendors are going to struggle to strip operators and customers away from some degree of separation. Mark.
So right Jared....magic has been in the NPU capacity increase that's driven the cost per 100G down on 1RU routers;
But that has only mainly solved for speed. Features have taken a hit, especially if the operator is motivated by the costs of merchant silicon.
There has been a marked improvement of features from merchant silicon, both from their vendors as well as the router OEM's that implement them in clever ways to work around their restrictions, but there are still some things only in-house silicon can do, at a price point most operators are not comfortable to pay anymore.
I think that as more of the Internet collapses into the hands of a few public cloud and content providers, operators are likely to place less and less importance on features, and just focus on speed, since the public Internet offers very little guarantees, if not none at all. I'm keen to see how this pans out.
and the integration of DSPs and more into QSFP-DD form factors at much lower power than expected.
Coherent has certainly changed the game, no doubt. If you grow steadily, you can wait for the evolution to make it into the IP/MPLS. If you need to move faster, you can't ignore the importance of Transport options in your network.
Mark.
At that scale, DWDM in the metro will make sense
I would risk to say a little more on this.
Indeed, maybe the situation (in many countries) when the Carrier sells a lot of TDM services.
But in general, packet services are enough these days for many carriers/regions.
Additionally, I am sure that in many countries/Metro it is cheaper to lay down a new fiber than to provision DWDM, even if it is a pizza box. The colored interface is still very expensive.
Of course, there are some Cities (not “towns”) where it is very expensive or maybe even impossible to lay down a new fiber.
Yes, in the majority of cases, it is cheaper to lay down fiber.
Hence, the importance of DWDM for the Metro is overestimated.
Use only routers. Provision enough fiber. Have always 1 router hop to the aggregation (hub-spoke topology), no routers chaining in the ring.
If fiber is not enough – then use normal DWDM with an external transponder. Routers would be still in hub-spoke topology.
Ed/
There aren’t enough TDM services to warrant DWDM, nowadays. The reason for DWDM is mainly being driven by Ethernet, and IP. At any reasonable scale, it’s actually pretty hard to buy a TDM service, in most markets. I disagree. Existing fibre may be cheap because it was laid down a decade or more ago, en masse, by several operators. So the market would be experiencing a glut, not because it is cheap to open up the roads and plant more fibre, but because there is so much of it to begin with. At worst, there is still enough duct space that the operator can blow more fibre. But when that duct gets full, and there are no more free ducts available, or another route needs to get built for whatever reason, it is a rather costly affair to open up the roads and trunk some fibre, in any market. So no, DWDM is not more expensive, if you are delivering services at scale. It is actually cheaper. It is only more expensive if you are small scale, because in some markets, the fibre glut means you can buy dark fibre for cheaper than you can light it with DWDM. But this is a situation unique to small operators, not large ones. This only matters for the line side. For client-facing, it’s not a drama. And you typically buy more optics for the client side than you do the line side. I think what you mean to say is that in the majority of cases where there is fibre glut, and dark fibre is a market option, buying fibre is cheaper than lighting it with DWDM. This is true. But I think that on a global scale, this is the exception, not the rule. In general, you are not likely to be able to buy dark fibre, cheaply or otherwise, if you look at all markets in the world. Again, only if you are small scale. If you are a large scale operator with as many IP/Ethernet customers as you have Transport, DWDM is essential. Yeah, you sound like an equipment vendor whose main customers are incumbent telco’s in a few rich markets :-). The life of the average operator, around the world, is far less glamorous. Mark.