Today I was investigating strange unknown unicast traffic in LAN of IX which I operate. It was about 200 kbps of constant unknown unicast load.
Unknown unicast is a rare ocasion in IX LAN as participants MAC addresses are almost persistent.
I added a server in the vlan and started sniffing:
all the unicast was coming from one participant MAC. One of the frames:
XX:XX:XX:XX:ee:98 > 5e:5c:ab:31:c0:cb, ethertype IPv4 (0x0800), length 1434: XX.XX.16.30.80 > XX.XX.76.137.41934: Flags [.], seq 1619512599:1619513967, ack 3595347045, win 235, options [nop,nop,TS val 3650267181 ecr 1841068329], length 1368: HTTP
Okay, destination MAC 5e:5c:ab:31:c0:cb isn't really in switches FDB table.
I looked up the routing table and figured out that router announcing the bestpath for XX.XX.76.137 has MAC 5e:5e:ab:31:c0:cb.
So customers sent a frame to: 5e:*5c*31:c0:cb
The right address should be : 5e:*5e*31:c0:cb
I tried next packet in dump, the same story:
customer sends to: *90*:c6:9a:*e5*:2f:c1
right mac is : *b0*:c6:9a:*e4*:2f:c1
I converted differencing bytes to binary representation:
I guess all the unknown unicast frames MAC addresses were having the same slightly difference from the right ones.
So the router in general works fine (it has several gigabits of traffic in the IX) but sometimes it changes one or two bits in frame's destination MAC address.
My guess it is caused by a large ARP table on customer's router. The router may have some tricky lookup algorithm preceding constant calculating speed over accuracy.
I called their NOC, they said it is Juniper MX80 and also confirmed that it has more than 4k ARPs.
Have anybody encountered with that kind of issue?