Hi all.
So I finally decided to start messing around with FRR for a native IS-IS deployment for some of our FreeBSD-based Anycast services.
I hit an issue that I posted to the FRR list that hasn’t progressed beyond identifying a bug:
2020/03/21 03:12:36 ISIS: isis_send_pdu_bcast: sock_buff size 8192 is less than output pdu size 9014 on circuit em0
2020/03/21 03:12:36 ISIS: [EC 67108865] ISIS-Adj (1): Send L2 IIH on em0 failed
This is being addressed here:
But my main question was if there was a command or setting in zebra.conf and/or isisd.conf that I can use to define the MTU IS-IS should use to set itself up, rather than being informed by what the interface currently runs at. I’ve tried everything that is documented as well as stuff that isn’t, but nothing is accepted or recognized. Either no one runs IS-IS on FRR, or much of the implementation is still being developed and/or hasn’t been tested in the wild, i.e., no traction. I’m hoping there is someone on this list that has played with IS-IS on FRR to point me in the right direction. The setup is FRR 7.3 on FreeBSD-12.1. Thanks. Mark.
Mark,
I suggest you ask this directly on the FRR slack:
https://frrouting.slack.com/
I’m also interested to know who’s been trying FRR IS-IS in the wild. At last check your former guess seemed to be correct and it wasn’t under active development.
Regards
Jeremy Austin
So the problem really isn't FreeBSD itself. IS-IS looks at MTU in order
to work, and if nothing is manually set, it infers the MTU from the
physical interface over which it is enabled.
While the current IS-IS implementation in FRR is buggy to the extent
that it does not assume MTU can be larger than 8,192 bytes, that does
not prevent an operator from telling it what MTU it should use for IIH
messages, provided the physical interface can support it.
Suffice it to say, I already have a number of Sysctl and kernel knobs in
FreeBSD to tune the system for the Anycast services we run on them. I'd
be disinclined to mess about with that as I don't think it has any
bearing on an over-the-top service such as IS-IS.
Quagga runs well (OSFP though, I'll admit), and I'll keep looking for
answers on IS-IS in FRR until it stops making sense.
Mark.
Mark, Thanks for sharing your experiences with FRR. I don’t work with IS-IS, but have found the development to be very active and fixing reproducible bugs quickly.
It look like FRR put a patch in for the bug you referenced and have a test build from 3/21 available which allows for up to 16k MTU @ https://ci1.netdef.org/browse/FRR-FRRPULLREQ-11364/artifact
I hope this helps and please continue to share your progress with the community.
Thanks for the feedback, John. It's great to hear this issue is being fixed.
So I'm not really trying to get IS-IS to run at a high MTU, as that
doesn't really do anything for the data plane. What I am really looking
for is a way to tell IS-IS to run a specific MTU.
In my network, even though we can carry up to 9,192 bytes in the data
plane, we manually set IS-IS to operate at 8,000 bytes. This is due to
VMware's limitation to address an MTU larger than 9,000, and we use it
to run CSR1000v for our route reflector. So globally, our IS-IS domain
runs 8,000 bytes.
I am just looking for a way to tell IS-IS, in FRR, to run at 8,000
bytes, which can be done without needing to fix the current bug in 7.2.
I just can't seem to find any configuration settings to do this, or
anyone that knows what they are.
Mark.
ok, if IS-IS is kinda working on FRR, at least enough to get loopbacks
and external interfaces around a pop, i gotta ask.
anyone running a ubiquity edgerouter infinity with frr, is-is, and four
or so full bgp feeds?
randy
I'm not sure what 'globally our IS-IS domain runs 8000 bytes' means.
Your LSP MTU is like 1492B, there isn't a mechanism to fragment and
reassemble LSP in-transit. ISIS network doesn't support different MTU
sizes and I've not heard anyone being brave enough to increase LSP MTU
above 1492B.
The only thing that is larger in your network is hellos, and I'm not
even sure how that works, considering 802.3 cannot signal larger
frames than 1500B.
Mikael Abrahamson and I quickly looked at this, some platforms (IOS,
IOS-XR, EOS) use ethernetII + 8870, but only for hellos and revert
back to standard 802.3 after. Some platforms (Junos) accept 8840
hello, but do not send it.
So after 1500B MTU detection does not work by-standard and everyone
has been forced to implement CiscoISIS to just get ISIS up, and Cisco
is forced to accept subMTU hello equally to interoperate with non8840
hosts.
It's really time to kill 802.3 and CLNS and do pure ISIS over
ethernetII+IPv6, no one really understands how all this works, it's
complex and fragile.
I'm not sure what 'globally our IS-IS domain runs 8000 bytes' means.
Your LSP MTU is like 1492B, there isn't a mechanism to fragment and
reassemble LSP in-transit. ISIS network doesn't support different MTU
sizes and I've not heard anyone being brave enough to increase LSP MTU
above 1492B.
I won't speak for Mark, but NO, when you're carrying somebody's else's traffic you do your best to have the MTU on each and every backbone link "high enough" : preferably in the 9200(bytes) range, so you can easily transport 9000(bytes) client packets, and by no means so small that you need to fragment 1500(IP)/1514(Eth) byte packets. If things are really-really bad, 1600 bytes towards the edge.
The only thing that is larger in your network is hellos, and I'm not
even sure how that works, considering 802.3 cannot signal larger
frames than 1500B.
Ethernet cannot signal MTU. But if you have equipment at both sides of a P2P link, you don't need any signalling. You check the MTU suits your needs and put it in statically. Same for NNIs : the MTU is signalled in a document called "contract" or "agreement". And no, the guy that is being woken up at 3am for an issue, if he/she doesn't know that we run Jumbo, then he/she should start updating the CV.
Back to the original question, I would expect FRR to be able to manually specify a MTU/frame-size, like any other decent NOS (even if it's not a full NOS).
You might want to have read the message and have a basic understanding
of ISIS before posting. ISIS is not ethernetII, ISIS is 802.3, which
in fact does signal packet size, and cannot signal larger than 1500B.
And to your original question, no, nothing in Mark's ISIS network is
above 1500, for the same reason.
Not for me. I can't get it to even start, i.e., no link adjacencies due
to an inability to agree on MTU.
So if anyone has IS-IS on FRR running at all, please reach out.
Mark.
I mean that every router and switch interface in our network runs at
9,178 bytes. Ordinarily, we wouldn't need to do anything to IS-IS as it
would just take that and run with it. However, VMware ESXi does not
support anything larger than 9,000 bytes, and that is where we run our RR's.
So to make it all harmonious, while the actual data plane in our routers
and switches can forward up to 9,178 bytes, we set IS-IS MTU to 8,000
bytes on all of them, as a lowest common denominator so that our RR's
can participate in the IS-IS domain.
It's got nothing to do with trying to transport CLNS frames or fragment
them, since as you rightly point out, fragmentation is not supported at
Layer 2.
Mark.
I won't speak for Mark, but NO, when you're carrying somebody's else's traffic you do your best to have the MTU on each and every backbone link "high enough" : preferably in the 9200(bytes) range, so you can easily transport 9000(bytes) client packets, and by no means so small that you need to fragment 1500(IP)/1514(Eth) byte packets. If things are really-really bad, 1600 bytes towards the edge.
Yes, this.
Within our core, we run 9,178 bytes (which translates to 9,192 bytes on
Junos and IOS XR), to support the transport of Jumbo frames for
customers that need it, typically l2vpn type services.
If VMware ESXi did not limit itself to 9,000 bytes, we wouldn't have
needed to set the IS-IS MTU at all, as it would have been inferred from
the physical interface of each device.
Back to the original question, I would expect FRR to be able to manually specify a MTU/frame-size, like any other decent NOS (even if it's not a full NOS).
One would have thought.
Even if no one is running it, I'd have assumed the developers would have
said something. My Google-fu is generally okay, but not on this one :-).
Mark.
and switches can forward up to 9,178 bytes, we set IS-IS MTU to 8,000
bytes on all of them, as a lowest common denominator so that our RR's
can participate in the IS-IS domain.
And the only thing this 'ISIS MTU' (think you mean CLNS MTU) does, is
for some cases make ISIS hellos larger. If you don't pad ISIS hellos,
or if you have standard compliant ISIS, it doesn't do anything past
1500B.
It's got nothing to do with trying to transport CLNS frames or fragment
them, since as you rightly point out, fragmentation is not supported at
Layer 2.
More accurately ISIS doesn't have a way to fragment LSP into two LSP,
so the whole ISIS network needs the same LSP-MTU, which has nothing to
do with 'clns mtu', LSP-MTU does not increase with increasing CLNS
MTU. There is no utility of having a harmonised CLNS MTU, so if you
want to speak to FRR, you can just do 1500B and it's fine.
Ummh, not really. As mentioned, we run 9,178 bytes on IOS and IOS XE,
and 9,192 bytes on Junos and IOS XR, in our network.
The 8,000 bytes we run is done only on the IOS, IOS XE and CSR1000v
boxes. It doesn't mean that we are trying to transport IS-IS PDU's at
8,000 bytes. It just means we can, and are capping it to just about what
VMware ESXi can support.
Mark.
And the only thing this 'ISIS MTU' (think you mean CLNS MTU)...
Yes, I mean "clns mtu".
does, is
for some cases make ISIS hellos larger. If you don't pad ISIS hellos,
or if you have standard compliant ISIS, it doesn't do anything past
1500B.
Agreed.
But in the network devices, IS-IS will infer the interface MTU to setup
adjacencies. It doesn't matter what it is, it just needs to either be
the same or low enough for the interface to support it.
More accurately ISIS doesn't have a way to fragment LSP into two LSP,
so the whole ISIS network needs the same LSP-MTU, which has nothing to
do with 'clns mtu', LSP-MTU does not increase with increasing CLNS
MTU. There is no utility of having a harmonised CLNS MTU,...
Do you run CSR1000v on ESXi?
so if you
want to speak to FRR, you can just do 1500B and it's fine.
Which is what this whole thread is about. How do I set that, manually,
without changing the physical interface MTU?
Mark.
Ummh, not really. As mentioned, we run 9,178 bytes on IOS and IOS XE,
and 9,192 bytes on Junos and IOS XR, in our network.
Quite and then you specified but for ISIS you run 8000. I'm only
talking about your ISIS here, not rest. And that 8k doesn't do
anything useful, so you can run 1500B to your FRR and not lose
anything.
The 8,000 bytes we run is done only on the IOS, IOS XE and CSR1000v
boxes. It doesn't mean that we are trying to transport IS-IS PDU's at
8,000 bytes. It just means we can, and are capping it to just about what
VMware ESXi can support.
As I explained this is not true, by standard ISIS cannot do this for
1500B MTU. As ISIS is 802.3 and 802.3 cannot send larger frames.
Cisco has proprietary hack to do EthernetII for hellos, but even that
doesn't guarantee anything, because ISIS will still come up, even if
it receives subMTU hello as 802.3.
You want to run your physical at high MTU and you also like ISIS to come up?
FRR doesn't seem to support Cisco proprietary EthernetII based ISIS,
so it'll only do 802.3 standard. Have you tried to just use what ever
interface MTU, and NOT pad on far-end cisco? If I read the code right,
this should be then standard max_size 802.3 and it should come up.
If you force Cisco to pad, it'll use the EthernetI hack, which is not
supported by FRR, and it won't work. I noticed in lab just now, if I
set 'pad sometimes' on IOS-XR, it won't pad at all for >1500B MTUs, so
seems like 'sometimes' stops Cisco from using the EthernetII hack.
Quite and then you specified but for ISIS you run 8000. I'm only
talking about your ISIS here, not rest. And that 8k doesn't do
anything useful,
Except for our CSR1000v on ESXi. We only ever needed it when we went
that route. We didn't need it prior to that.
so you can run 1500B to your FRR and not lose
anything.
Which is my point - there is nowhere in FRR to set this, that is
documented or has worked from experiment/general knowledge.
As I explained this is not true, by standard ISIS cannot do this for
1500B MTU. As ISIS is 802.3 and 802.3 cannot send larger frames.
Cisco has proprietary hack to do EthernetII for hellos, but even that
doesn't guarantee anything, because ISIS will still come up, even if
it receives subMTU hello as 802.3.
Again, this is only required on our CSR1000v ESXi deployments, because
our experience was that IS-IS would not establish because neighbors were
running at a different MTU from the RR's, and IS-IS on the RR's was
trying to setup adjacencies at a physically higher MTU than ESXi could
support.
So lowering it fixed it, and we just picked 8,000. We could have picked
2,000, or 1,500, or whatever. The actual number doesn't matter, as long
as it's the same on the boxes that need it.
We spent about 6hrs troubleshooting this back in 2014, so we didn't put
it in just for fun.
Mark.