400G link L2 flakiness over DWDM

Mailman List Mirror (Read Only)
1

We have a number of new 400G provider waves with odd flakiness that we
can't seem to sort out.

These links appear good - light levels are fine, links will be up with
working LLDP. But if we admin down / up the link on the IP side, many
of them won't come back up on their own. Our provider sees "loss of
alignment" alarms on the optical equipment, and we can eventually get
the links back up if they bounce them a few times on the DWDM side.

We're using Juniper PTX's, with Juniper 400G FR4 and 400G LR4 optics.
Same issue on different circuits/devices/paths with this provider, but
no issues with other transport providers using the same optics on our
end. Believe our provider is using Ciena optical equipment.

Has anyone seen issues like this before? Our provider has been
looking into this for months without much progress. One tantalizing
clue was another customer using Cisco Nexus had better results after
they enabled "link transmit reset-skip"
(Bug Search Tool). Not
clear exactly what that option does, and there doesn't seem to be an
obviously equivalent config knob in Junos.

Thanks,
Oliver

2

This sounds like a Ciena-specific issue.

I have found the below on their knowledge portal:

https://my.ciena.com/CienaPortal/s/article/6500-Submarine-How-to-clear-Loss-of-Alignment-Marker-alarm-on-M2M-port

https://my.ciena.com/CienaPortal/s/article/Waveserver-5-Intermittent-errors-BIP-errors-or-Loss-of-Alignment-Marker-on-the-equipment-connected-to-WS5-client-ports

We don't run Ciena, but I have reached out to a good mate at Ciena to see if he is aware about this and if their kit is implicated. I'll let you know when I hear back.

Mark.

3

No feedback yet from my Ciena contact, but I finally got full access to those knowledge articles.

The 6500 issue appears to be an AOC cable length difference between M2M (mate-to-mate) ports. Ensuring the AOC cables are the same clears the LOA alarm.

On Waveserver, the issue appears to be a software defect that appears when certain ports are configured with both Ethernet and OTU4 profiles for 400G services. Resolution is based on recreating the client service or hard rebooting the line card. There is no mention of a software upgrade, but since the article is from 2023, I'd expect that should already have been done.

Both the 6500 and Waveserver have this issue only when 400G services are delivered.

The articles do point to "submarine" applications, but I'd ignore that since the equipment used for terrestrial and submarine is largely the same (very minor differences in the line cards).

Will ping if my Ciena contact reaches back.

Would be good to validate the above with your provider, and let us know please. Thanks.

Mark.

4

Mark is the undisputed DWDM wizard!

5

On Waveserver, the issue appears to be a software defect that appears
when certain ports are configured with both Ethernet and OTU4 profiles
for 400G services. Resolution is based on recreating the client service
or hard rebooting the line card. There is no mention of a software
upgrade, but since the article is from 2023, I’d expect that should
already have been done.

This is a not uncommon bug on Ciena stuff. Thankfully it doesn’t happen often and is a straightforward fix, although bouncing a card is obv not ideal in a lot of cases.

6

One tantalizing
clue was another customer using Cisco Nexus had better results after
they enabled “link transmit reset-skip”
(https://quickview.cloudapps.cisco.com/quickview/bug/CSCvi45168). Not
clear exactly what that option does, and there doesn’t seem to be an
obviously equivalent config knob in Junos.

I have a vague recollection of that option being there to basically ignore certain events that would normally cause link drop in order to keep an unstable link up. It was a long time ago, I sent a message to the guy I used to work with who might remember more.

7

Indeed.

My Ciena contact responded. He will check with their support team on what they know about this issue. He deals mostly with subsea (SLTE) design.

Mark.

8

I've been looking at the Waveserver line, so this is somewhat relevant to my interests.

Given that this is a "not uncommon", do you or anyone else happen to know if there are other options such as e.g. configuring the port for only one of Ethernet or OTU service?

9

Based on how the issue was described in the report, the problem seems to occur if there are multiple ports on the line card configured with different mixes of Ethernet and OTN on ports 3 and 11, as well as on ports 4, 8, 12, and 16.

The 2nd scenario is if on Waveserver, 400G Ethernet is enabled on ports 6 and 12, but a mix of both Ethernet and OTN are configured on ports 6, 7, 12, and 13 without 400G.

The 3rd scenario is 400G Ethernet and OTN mixed between ports 4 and 10.

It's all very confusing.

I'd certainly raise this with your Ciena account team before shelling out any cash, if you are keen to use them.

Mark.