Hi,
I have 1G Level3 ethernet dedicated internet service as one of my ISP's at my company based in the Los Angeles (Inland Empire) area. After seeing strange application behavior while using this circuit, I failed it out of service and have been troubleshooting it with a directly connected machine (publically addressed, no firewall, nothing between this machine and our Level3 router). I have taken several packet captures while accessing various sites and have noticed large numbers of out of order packets which are wreaking havoc with TCP connections and other traffic. In my experience, per-packet load balancing across various different links can cause this issue. I do not see this behavior with my other ISP's. I have had several tickets opened with Level3 but have had no success. Any help here? Anyone out there seen this and have any contacts that may be able to help?
FYI - we own our own public IP space and advertise via BGP to Level3. Currently I am using a dedicated /24 of our space advertised to Level3 only to ensure that the return path is through Level3 and not another ISP. Also, everything is single linked from a layer 2 and 3 perspective from the router to the test machine to ensure that the cause of any out of order packets is not on our end.
Thanks,
Hi Mark,
I'm going to throw out a guess here. By any chance, is the first octet of
your router's MAC address a 4 or a 6?
In general, modern routers do not load balance per-packet, which is what
caused out-of-order issues in days gone. Load balancing is usually done
based on a hash of the source and/or destination IP of the packet, MPLS
label, or Ethernet (on a switched interface). The most common cause for
actual unordering of packets/frames in a modern service provider network,
in my experience, is actually this hashing mechanism. Many vendor's
hashing implementations assume, based on position in the frame, that a
frame with a MAC address beginning with 4 or 6 is an IPv4 or IPv6 frame,
not an MPLS frame. This can result in out of order packets. The most
common fix is control word being applied on a pseudowire (assuming you are
being carried across a pseudowire in the SP network), but if this *is* what
is occurring, you could also resolve the issue by changing your MAC address.
Hi Mark,
I'm going to throw out a guess here. By any chance, is the first octet of
your router's MAC address a 4 or a 6?
In general, modern routers do not load balance per-packet, which is what
caused out-of-order issues in days gone. Load balancing is usually done
based on a hash of the source and/or destination IP of the packet, MPLS
label, or Ethernet (on a switched interface). The most common cause for
actual unordering of packets/frames in a modern service provider network,
in my experience, is actually this hashing mechanism. Many vendor's
hashing implementations assume, based on position in the frame, that a
frame with a MAC address beginning with 4 or 6 is an IPv4 or IPv6 frame,
not an MPLS frame. This can result in out of order packets. The most
common fix is control word being applied on a pseudowire (assuming you are
being carried across a pseudowire in the SP network), but if this *is* what
is occurring, you could also resolve the issue by changing your MAC address.
Hi Jason,
I think that this was it. The first octet of our router’s Level3 facing interface MAC is a 6. Changed it to 0 and it does look much better now. Thanks!