evil ipv6 bit?


After some apparently unrelated changes, one of my routers stopped routing traffic to a few IPv6 destinations. After a lot of experimentation, including rebooting (did not help), I found this:

archive.ubuntu.com: 2001:67c:1360:8001::17

"ping6 vrf internet 2001:67c:1360:8001::17" from the router shell works.

ping6/traceroute from a customer connection has the packet dropped by the router. Traceroute gets nothing back at all.

2001:67c:1360:7fff:: is ok. Does not reply to ping because I just made up that address. But I get a valid traceroute all the way to the destination.
Anything between 2001:67c:1360:8000:: and 2001:67c:1360:ffff:ffff:ffff:ffff:ffff is dropped.

My route table looks like this:

albertslund-edge1#show ipv6 forwarding route vrf internet 2001:67c:1360:8001::17
IPv6 Routing Table:
Headers: Dest: Destination, Gw: Gateway, Pri: Priority;
Codes : K: kernel, I1: isis-l1, SFN: sf-nat64, R: ripng, AF: aftr, B: bgp,
          D: direct, I2: isis-l2, SLN: sl-nat64, O: ospfv3, D6: dhcp, P: ppp,
          S: static, N: nd, V: vrrp, A: address, M: multicast, UI: user-ipaddr,
          GW-FWD: PS-BUSI,GW-UE: PS-USER,LDP-A: LDP-AREA, UN: user-network,
          US: user-special;
Dest Owner Metric
   Interface Pri Gw
2001:67c:1360::/48 B 0
   xgei-0/0/0/6 200 ::ffff:
::/0 B 0
   xgei-0/0/0/6 200 ::ffff:

Notice how this is a /48 route and one bit at the /49 level changes how it is routed. That is not right.

I tried adding a /128 static route but that does not do anything. The packet is still dropped.

I just now discovered this:

google.com: 2a00:1450:400e:807::200e

That address works fine. But then I changed that one bit in the address: 2a00:1450:400e:8807::200e and voila, the router drops the packet.

Now I am stumbled. What could the 49th bit in the destination IPv6 address field in a packet mean to the router, that would make it drop the packet?

Some extra information about the network: We are using MPLS with l3vpn (vrf) and l2vpn (vpls). The traffic is qinq tagged before being transported in a l2vpn towards the router in question. The l2vpn does not transport the outer vlan tag. The l2vpn is then terminated on a loopback cable. On the other end of that loopback cable we receive the traffic as ordinary qinq tagged without MPLS tagging. It is on this interface the router apparently drops the packet. It might conceivably also drop the packet on the way out of the l2vpn.

I have a similar setup, but instead of a loopback cable, the l2vpn is terminated on another MPLS switch, which then connects to a router of the same model. This setup does not have the problem.

The change I introduced was changing from an internal interface called "bvi" to the loopback cable. The bvi interface is a simulated loopback cable construct. We are dropping the bvi interface because it is very buggy. We did not have this problem with the bvi interface however.

The hardware is ZTE M6000-S V3.00.20(3.40.1).



Are you sure it is dropped? Can you see some drop counter increase?
Have you observed nothing coming out from any port?

My guess is bad memory, and that bit is statically set or statically
unset and cannot be changed. Might cause CRC error, IP checksum error
or just mangled packet coming out of the router

There was also this example from a while ago:

*Juniper MX80 strange dst MAC address behavior*

And then this:

*Forwarding issues related to MACs starting with a 4 or a 6 (Was: [c-nsp] Wierd MPLS/VPLS issue)*

Those are both related to _MACs_, though, rather than IPs.

Not sure if this is related or not, but in this[1] thread over on pdns-users, there was talk about subdomains under clouds.archive.ubuntu.com only having a single NS, which may have been having connectivity issues as recently as yesterday.

1: https://mailman.powerdns.com/pipermail/pdns-users/2018-January/025197.html

Theodore Baschak - AS395089 - Hextet Systems
https://bgp.guru/ - https://hextet.net/
http://mbix.ca/ - http://mbnog.ca/