How are you configuring BFD timers?

Mailman List Mirror (Read Only)
1

Hey,

For those running BFD on your land-based point-to-point links, I’m interested in hearing about what factors you consider when deciding how to configure your timers and multiplier.

On paper, BFD between two devices over a local or metro dark fibre or wave seems pretty trivial: Assuming your gear can a) support echo mode b) hardware offloads echo processing c) automatically treats echos as vital and puts them into the appropriate high priority queue, then setting the timers down to their lowest possible values (3ms on some of the gear that I’ve seen) and some low multiplier seems more than reasonable. But?

From another angle, your link isn’t dark fibre or a wave but, for example, ethernet over some sort of IP based L2 Transport, and is still a low (sub 1ms) one-way latency local or metro link. How do you set your timers, and what do you base that on?

From yet another angle, what if your link is a long-haul wave, or for that matter a wave of any distance that imposes a one-way latency that is higher than the minimum tx and rx timers that are supported by your gear? We’ll assume an unprotected wave, because I’m sure if it’s protected, you have no choice but to consider the one-way latency of the longest of the two segments.

I made some assumptions above about support for echo mode and hardware offload, but what if (some of) your gear doesn’t support some or all of that stuff? How do you factor your configuration decisions?

Thanks!

2

In practice, the vendor's recommendations regarding Routing Engine HA provide a lower bound. I'm just starting out with 1000ms x 3 multiplier, but my network is not national or global. I believe I could go as low as 500ms to keep HA happy.

3

Using 250ms x 3 on fiber connecting Pennsylvania to Florida...

Best regards,

Alex

4

Using 200 ms / 200 ms / x3 on either metro dark fiber or longhaul waves
(Paris / Frankfurt / Amsterdam) successfully.

Best regards.

Y.

5

Silly question perhaps, but why would you do BFD on dark fiber?

Kind regards,

Job

6

To speed up BGP routing convergence. The (2x) dark fiber links from PA to FL are being used as Layer3 datacenter interconnects, where each datacenter has its own AS. The DF is also carrying FCIP traffic, so we need failover to be as fast as possible.

Best regards,

Alex

7

Wouldn't any tangible problem on a dark-fiber link result in an interface shutdown, ostensibly creating the trigger one would need to begin re-convergence?

8

He's asking because if it was dark the interface would go down when the link was lost and the router would pull routes. But PA to FL would lead me to believe it'll be a wave from some type of DWDM gear which brings us to BFD.

Luke Guillory
Vice President – Technology and Innovation

Tel: 985.536.1212
Fax: 985.536.0300
Email: lguillory@reservetele.com

Reserve Telecommunications
100 RTC Dr
Reserve, LA 70084

9

Correct, Luke.

Best regards,

Alex

10

Right, BFD on a dark fiber link (should) be immediately detected and the
detecting end should send a cease/stop/whatever message to the remote peer
to drop the neighbor relationship.

BFD really comes into it's own in a derived circuit (such as metro-E or
other type setup) where you can have an indirect failure (traffic does not
pass, but the last mile link remains up).

Ken

11

A few years ago I did some testing and found that the time between the transceiver detecting LOS and the routing protocol (ISIS in this case) being informed that the link was down (triggering the recalculation) took longer than it took BFD to signal ISIS to recalculate.

12

Which platform ? What context ?

Best regards.

13

They were ME3600s. AFAIR it was two of these things in a lab connected back to back with two links between them, one metric higher than the other. Some sort of traffic generator running between the two that would generate fixed size UDP frames at some tens of milliseconds interval, yanking the preferred link, counting how many packets were lost and doing some math, correlating with various logs debugs on the boxes.

14

Simple paranoia perhaps? Just because you only have layer 1 equipment
(transponders) right now, doesn't guarantee that you won't need to stick
in a layer 2 switch in the path tomorrow, and if you already have BFD
running, you won't forget to add it at that time.

As a real-life example: we have dark fiber from our main campus to
another campus in the neighbouring city, and from there we have dark
fiber to a partner a few kilometers further away. We run wavelengths
the entire way from the main campus to our partner (being demuxed and
muxed back at the sub-campus). At one point, the fiber needed to be
rerouted, and then the attenuation became to high for one of the CWDM
wavelengths. Solution: put an ethernet switch at our sub-campus to
act as a kind of amplifier for that wavelength.

(This was using 120km CWDM gigabit transceivers directly in the routers
at each end. We have since retired those and use 10 gigabit DWDM with
transponders and EDFA amplifiers.)

Yes, it was a duct-tape solution, but it was cheap and got the work
done. :slight_smile:

  /Thomas Bellman

15

He's asking because if it was dark the interface would go down when the link was lost and the router would pull routes. But PA to FL would lead me to believe it'll be a wave from some type of DWDM gear which brings us to BFD.

Could it not also help with a unidirectional failure of the fibre?
E.g. Loss of signal in only one direction might not bring the link
down, but if one BFD peer stops receiving BFD packets it'll bring the
link down.

A few years ago I did some testing and found that the time between the transceiver detecting LOS and the routing protocol (ISIS in this case) being informed that the link was down (triggering the recalculation) took longer than it took BFD to signal ISIS to recalculate.

Have you looked at testing and adding this command to your IOS devices:

ip routing protocol purge interface

Also have you tried to set the carrier delay to zero?

carrier-delay down 0

I haven't compared them to BFD explicitly, but I would expect the two
commands together to have the same effect as you're seeing with BFD
(the link down is signaled to the IGP ASAP).

Cheers,
James.

16

Why does DWDM imply need of BFD?

DWDM has no problem propagating loss-of-signal and asserting remote failure.

It is of course possible to configure DWDM so that this does not
happen, particularly if you buy protected circuit you might not want
it to happen. But as per usual, you should test that vendor delivers
what you buy.

I think the problem probability of not detecting failure on waves,
dark fibres and copper connections are smaller than the probability of
having issue caused by presence of BFD. I think you need very
unreliably failing link to capitalise on BFD. Radio is good candidate,
switch masked liveliness is good candidate.

Of course you also need to be sure you don't kill your BGP's fast
external failover, by using eBGP multihop on your point-to-point
circuit.

17

Ethernet handles unidirectional failure natively through autonego asserting RFI.

A======B

B sees loss-of signal
A does not
B asserts RFI
A receives RFI and goes down

Some devices, such as JunOS can also assert RFI for external factors,
like you can ask JunOS to assert RFI to client port, when pseudowire
goes down, so client experiences link down when you experience
pseudowire down.

18

Hey,

For those running BFD on your land-based point-to-point links, I’m interested in hearing about what factors you consider when deciding how to configure your timers and multiplier.

On paper, BFD between two devices over a local or metro dark fibre or wave seems pretty trivial: Assuming your gear can a) support echo mode b) hardware offloads echo processing c) automatically treats echos as vital and puts them into the appropriate high priority queue, then setting the timers down to their lowest possible values (3ms on some of the gear that I’ve seen) and some low multiplier seems more than reasonable. But?

From another angle, your link isn’t dark fibre or a wave but, for example, ethernet over some sort of IP based L2 Transport, and is still a low (sub 1ms) one-way latency local or metro link. How do you set your timers, and what do you base that on?

From yet another angle, what if your link is a long-haul wave, or for that matter a wave of any distance that imposes a one-way latency that is higher than the minimum tx and rx timers that are supported by your gear? We’ll assume an unprotected wave, because I’m sure if it’s protected, you have no choice but to consider the one-way latency of the longest of the two segments.

I made some assumptions above about support for echo mode and hardware offload, but what if (some of) your gear doesn’t support some or all of that stuff? How do you factor your configuration decisions?

Thanks!

Going back to the original question;

From another angle, your link isn’t dark fibre or a wave but, for example, ethernet over some sort of IP based L2 Transport, and is still a low (sub 1ms) one-way latency local or metro link. How do you set your timers, and what do you base that on?

Personally I don't care if it's a wavelength, dark fibre or L2 VPN
service. I don't treat them differently based on the underlying
connectivity type. The SLAs are probably more important. But if we are
paying for say 10G of capacity on a link which is say a 10G pseudowire
from another carrier, I treat it the same as a dark fibre connected to
10G transceivers at each end. Wave lengths are generally more stable
in my opinion, we did have a 10G L2 Ethernet circuit from a carrier
that was a pseudowire from them essentially, and their PE was under a
DDoS attack so our L2 VPN service was affected (because the pseudowire
was flapping up and down). But once the circuit is up and running for
a while, if you're regularly pushing somewhere near the max circuit
bandwidth and monitoring circuit latency, you'll get a feel for "how
good" the carrier is and then adjust from there. Generally speaking
though, if the carrier is "good" I treat DF/lamda/L2 circuits the same
with regards the BFD/IGP tuning.

I made some assumptions above about support for echo mode and hardware offload, but what if (some of) your gear doesn’t support some or all of that stuff? How do you factor your configuration decisions?

Elsewhere in the thread you have mentioned that you are using Cisco
ME3600 devices. If you disable BFD echo mode you will be able to get
low timers on these devices. Echo mode is enabled by default on IOS
when you enable BFD under an interface, which these devices don't
support, so you need to explicitly disable it. See the min/max/avg BFD
timers below between two ME devices when the interfaces are configured
with "bfd interval 50 min_rx 50 multiplier 3":

ME3600#show bfd neighbors interface te0/2 details
...
Session state is UP and using echo function with 50 ms interval.
Session Host: Software
...
Rx Count: 72, Rx Interval (ms) min/max/avg: 1/4976/4323 last: 2348 ms ago
Tx Count: 74, Tx Interval (ms) min/max/avg: 1/4968/4217 last: 1436 ms ago

If you add the command "no bfd echo" to the interface you should see
the following min/max/avg BFD timers:

ME3600#show bfd neighbors interface te0/2 details
...
Session state is UP and not using echo function.
Session Host: Software
...
Rx Count: 3314443, Rx Interval (ms) min/max/avg: 1/72/47 last: 36 ms ago
Tx Count: 3310865, Tx Interval (ms) min/max/avg: 1/72/47 last: 40 ms ago

We have a mixture of devices and they don't all support BFD echo mode.
We have for example Cisco ASR9000s that support both echo / no echo
mode, so it may have one interface towards a Juniper MX running BFD
echo mode and one interface towards a Cisco ME which runs no echo
mode. It's working fine for us.

Cheers,
James.

19

I was thinking about this as I wrote that post. I've not had a chance
to test this across our various devices types, I will have to try and
find the time to test which devices support this and how effectively
it works. For now we just use BFD because it's tested as working on
all our device types. We've seen issues where LOS isn't working
properly across third party DWDM platforms so for now, we have no BFD
bugs and it catches these issues for us.

I think it might also be worth testing Ethernet RFI to check the
delays that Jason was talking about; which is quicker to signal link
down and begin the re-convergence process, RFI / LOS / BFD?

Cheers,
James.

20

RFI is corollary to LOS.

Both are essentially speed of light limited, there in no uncertainty
or detection margin. After one side sees LOS other side will see RFI,
if it's not seeing LOS as well.

I personally would consider BFD for: radio, pseudowire, L2 switch and
equivalent poorly failing links. But I wouldn't run it on dark,
copper or wave. Purely an anecdote, but I have far more BFD caused
problems than BFD solved problems, spanning multiple vendors.
(CAT7600, ASR9k, MX).

Regarding BFD echo and control mode, neither is guaranteed to be HW or
SW implementation, both can be both. Standard intends echo mode to be
HW, but does not guarantee.