We're seeing a bit of a weird one on our network at the moment, and wondering
if anyone else has seen it.
Since Friday we're seeing Apple devices (we believe it's both laptops and
iPhones) responding to ARP requests for the default gateway IP with their own
MAC address (i.e. ARP spoofing / MITM type attack). We're only seeing it on
Apple devices, but what's more strange is that we're only seeing it where
those Apple devices are connected to Cisco 1810 and 1815 APs, and where those
APs are connected to a Cisco WLC running v8.5 software. If we downgrade the
WLC to v8.2 the problem goes away (but v8.2 doesn't support 1815 APs, so we
can't roll that out globally). We're engaged with Cisco TAC, but they're
trying to deny it's their problem. Apple support are investigating, but aren't
admitting to having seen it before.
Has anyone else seen or heard of similar issues over the last few days?
I was a network engineer on the WiFi network at SFO, for both passengers and baggage scanners, with several hundred APs. Several times we were misled by packet captures that seemed to show client traffic causing network problems, such as packet storms, but which ultimately always had some more mundane cause, like a failed DHCP server or flapping switch interface.
The particular SFO network I worked on has Juniper switching and Aruba APs, so it’s not directly applicable to your ecosystem. But the complexities of interpreting packet captures may apply.
I have some packet captures, but as they're from a live network, I'd rather
not post them publicly.
I was a network engineer on the WiFi network at SFO, for both passengers and
baggage scanners, with several hundred APs. Several times we were misled by
packet captures that seemed to show client traffic causing network problems,
such as packet storms, but which ultimately always had some more mundane
cause, like a failed DHCP server or flapping switch interface.
Sure - we're rattling every possible other cause we can think of, including
using alternative DHCP server software vendor, etc. The only thing that's
reliably making the problem go away is running the APs against WLC version 8.2.
The particular SFO network I worked on has Juniper switching and Aruba APs,
so it???s not directly applicable to your ecosystem. But the complexities of
interpreting packet captures may apply.
I'm the sort of person who has copies of RFCs printed out on his desk. I'm
fairly experienced at interpreting packet captures
You asked if anyone else has seen this. It’s possibly going on in other networks but nobody is noticing. What symptoms brought the problem to your attention?
You can sanitize the packet captures by limiting them to just the headers. The payloads are likely not useful for troubleshooting anyway, since this seems to be a Layer 2 problem. You asked for help, and sanitized packets would help people help you
Apple's Bonjour protocols include something called Apple Bonjour Sleep Proxy
for Wake on Demand --- When a device goes to sleep, the Proxy that
runs on various
Apple devices is supposed to seize all the IP and MAC addresses that
device had registered,
so it can wait for an incoming TCP SYN, (and if one's received, then
signal the
sleeping device to wake up and process the connection.)
Bonjour and the related mDNS protocols used for AirPlay/AirPrint/etc
are built on Link-Local
multicast. I wonder what would happen if some random Wireless LAN
controllers malfunctioned,
and decided that it would like to ignore that Link-Local restriction
and proxy those packets b/w
subnets anyways, as if they had been unrestricted multicast or
Unicast, Possibly with the
source IP address on registration Mangled to or "gateway'd" from the
router's IP address.
(Or perhaps they wanted to have a feature to let someone AirPlay from
a different VLAN than
another device?)
Either way.... playing around with the source IP address on the
Link-local m/c packets
might accidentally get them a Default Gateway IP address registered
with other workstations
as a mobile device that's gone to sleep and needs a proxy.
Apple's Bonjour protocols include something called Apple Bonjour Sleep Proxy
for Wake on Demand --- When a device goes to sleep, the Proxy that runs on
various Apple devices is supposed to seize all the IP and MAC addresses that
device had registered, so it can wait for an incoming TCP SYN, (and if one's
received, then signal the sleeping device to wake up and process the
connection.)
That's a very interesting observation - when we talk to the users of the
Apple devices, they quite often say that the device was 'asleep' when it
was sending these 'spoofed' ARP responses.
(Or perhaps they wanted to have a feature to let someone AirPlay from a
different VLAN than another device?)
Cisco Wireless does claim to have some features to 'help' Bonjour / mDNS
to work better. I wonder if one of those features is misbehaving.
We are running 8.5 and 1815s and I don’t think we are seeing this problem.
We do have a very small number of 1810s and did see some strange behavior but it doesn’t seem to match this problem description.
Is proxy arp disabled on the default gateway device? That could potentially interact strangely with the features mentioned in earlier posts and mentioned below.
Apple's Bonjour protocols include something called Apple Bonjour Sleep Proxy
for Wake on Demand --- When a device goes to sleep, the Proxy that runs on
various Apple devices is supposed to seize all the IP and MAC addresses that
device had registered, so it can wait for an incoming TCP SYN, (and if one's
received, then signal the sleeping device to wake up and process the
connection.)
That's a very interesting observation - when we talk to the users of the
Apple devices, they quite often say that the device was 'asleep' when it
was sending these 'spoofed' ARP responses.
The "Information About Passive Clients” section of this document
says:
"Wireless LAN controllers currently act as a proxy for ARP requests. Upon receiving an ARP request, the controller responds with an ARP response instead of passing the request directly to the client. This scenario has two advantages:
• The upstream device that sends out the ARP request to the client will not know where the client is located.
• Power for battery-operated devices such as mobile phones and printers is preserved because they do not have to respond to every ARP requests."
Perhaps that function on version 8.5 is interacting incorrectly with the Apple Sleep Proxy feature on the Apple devices.
"When a sleep proxy sees an IPv4 ARP or IPv6 ND Request for one of the sleeping device's addresses, it answers on behalf of the sleeping device, without waking it up, giving its own MAC address as the current (temporary) owner of that address.”
