At some point over night on 30th September (i.e. the night going into 1st October), we saw a number of Spectrum (Charter) customers stop handling fragmented UDP packets. This has manifested itself in such that the phones of affected customers are no longer receiving UDP SIP INVITE packets which exceed whatever their WAN side MTU is. We've so far had 6 customers report the issue - we can see that the last call on 30th September worked and the first and subsequent calls on 1st October failed.
Is anyone aware of an update to CPE devices pushed out that night which may have broken their ability to handle IP fragmentation?
Hey Phil,
At some point over night on 30th September (i.e. the night going into 1st October), we saw a number of Spectrum (Charter) customers stop handling fragmented UDP packets. This has manifested itself in such that the phones of affected customers are no longer receiving UDP SIP INVITE packets which exceed whatever their WAN side MTU is. We've so far had 6 customers report the issue - we can see that the last call on 30th September worked and the first and subsequent calls on 1st October failed.
Is anyone aware of an update to CPE devices pushed out that night which may have broken their ability to handle IP fragmentation?
I don't know anything specific to this case, but you'd serve your best
interest to send small enough packets that do not need fragmentation,
particularly in the backbone. Even devices often considered SP
quality, such as ASR9k, fragment in the linecard CPU, giving very poor
service quality compared to sending two packets not fragmented.
While we can say this should just work, the reality is, it's not very
reliably true and I would not build product or business on the
assumption that it works well.
While we can say this should just work, the reality is, it's not very reliably true and I would not build product or business on the assumption that it works well.
Yup. Understood. We can't get away from sending multi-packet messages. We try our best to keep SIP messages as small as possible though sometimes certain optional features required by customers push it beyond their MTU. We're also starting to see decreasing MTUs as customers deploy various SD-WAN solutions and it's tough to keep up with these when you're already teetering on the edge of what used to be considered a fairly common minimum MTU value.
We can, of course, get away from using UDP. We can and do run SIP over TCP and indeed over TLS on TCP though the stateful nature of TCP often makes this undesirable. We see a lot of SIP phone implementations that do not handle TCP connection failures very well and result in a loss of calls for a period of minutes if this happens, as they take a while to notice the connection has dropped and should be re-established. Pros and cons of each.
If anyone has any specific information about Spectrum CPE changes or indeed any contacts who may be able to interrogate this internally within Spectrum that would be appreciated.
hey,
I don't know anything specific to this case, but you'd serve your best
interest to send small enough packets that do not need fragmentation,
particularly in the backbone.
In this case the SIP invite is already sent fragmented from the source and no fragmentation is required in transit. Someone probably thought "we see a lot of DDOSes with UDP fragments, better drop them altogether".
Wait till STIR/SHAKEN is enabled. Were going to see real quickly who isn’t handling fragmentation correctly…
Hi Phil,
Contact me off list with the locations impacted and I will look into it.
Jim
At some point over night on 30th September (i.e. the night going into 1st October), we saw a number of Spectrum (Charter) customers stop handling fragmented UDP packets
To bring this thread to a close, Charter kindly investigated and fixed the issue. It was caused by a change to their network that occurred on 1st October. Thanks to those involved for their help.