I encountered an interesting issue today and I found it so bizarre so I
thought I would share it.
I brought online a spare server to help offload some of the recent VMs that
I have been deploying. Around the same time this new machine (we¹ll call it
Server-B) came online, another machine which has been online for about a
year now stopped responding to our monitoring (and we¹ll name this
Server-A). I logged into the switch and saw that the machine that stopped
responding was in the same VLAN as this newly deployed, and then quickly
noticed that Server-A¹s MAC address was now on Server-B¹s switch port.
³What the ...² was my initial response.
I went ahead and moved Server-B¹s to another VLAN, updated the switchport,
cleared the ARP, and Server-A came back to life. Happy new year to me.
So here is the interesting part... Both servers are HP Proliant DL380 G4s,
and both of their NIC1 and NIC2 MACs addresses are exactly the same. Not
spoofd and the OS drivers are not mucking with them ... They¹re burned-in
I triple checked them in their respective BIOS screen. I acquired these two
machines at different times and both were from the grey market. The ³What
the ...² is sitting fresh in my mind ... How can this be?
In the last 15 years of being in IT, I have never encountered a ³burned-in²
duplicated MACs across two physically different machines. What are the
odds, that HP would dup¹d them and that both would eventually end up at my
shop? Or maybe this type of thing isn¹t big of deal... ?
There may be some setting you're overlooking or a bug which needs an update to fix, or you may simply have purchased HP ProLiant *cases*, rather than actual *servers*.
;>
Note that search engine results for 'proliant dl380 duplicate mac' returns multiple links.
I know HP has a disc they put out which updates all the firmware/bios in a specific server model, its not too far fetched that a vendor might have a modified version that also either purposely or accidentally changes the MAC address. Off the top of my head, I'm not sure where the MAC is stored - maybe an eeprom or a portion of the bios flash. Or, it could be botched flashing that blew away the portion of memory where that was stored and the system defaulted to a built in value.
Excellent example is, IIRC, the older sparc stuff, where the ethernet cards didn't have MAC addresses as part of the card, but were stored in non-volatile or battery backed memory. Memory goes poof, and you'll have problems. Some WRT54G/WAP54Gs suffer from the same problem when throwing third party firmware on there.
So along simlar lines, Ubiquiti sell routerstation pro boards with
sequential MAC addresses.
The trouble is they've allocated a single MAC for the first port - the
second ethernet port (also attached to the bridge) doesn't get a second
MAC.
So in a purchase of a few hundred boards, we had plenty that were sequential.
Since the FreeBSD driver allocated MAC+1 to the second NIC, this caused
duplcate MAC addresses and this caused hilarity to ensue.
The "fix" was to just get this company to apply for some MAC space and then
use -that- on the second NIC and the bridge interfaces.
This was actually the intended way to use "MAC" addresses, to used as
host addresses rather than as individual interface addresses, according
to the following paper -
That paper also discusses why 48 bits were chosen as the size, despite
"Ethernet systems" being limited to 1024 hosts.
I think things evolved into MAC per NIC because when add-in NICs
were invented there wasn't any appropriate non-volatile storage on the
host to store the address.
Had a fun one with D-Link ADSL modems a few years ago. The MAC address used to source PPPoE frames from the ADSL interface was the same in a batch of modems. This wasn't a problem when using our wholesaler's ATM based DSLAMs, but when we moved users to our own Ethernet based systems, we ran into some fun.
The DSLAMs were configured to rewrite the user MAC address into a constructed address including their DSLAM port and PVC details toward the BRAS. Even though this rewrite was occurring correctly, within a single DSLAM unit two users with the same real MAC address would have five minutes of connectivity each, alternating between the two ports.
None of the HP servers I have contain duplicate MAC addresses. (I just
looked through all the iLO2 cards to make sure I wasn't lying.) I'll
send you some details offlist.
I don't use the iLO port, so I didn't look at it's MAC within the BIOS,
however my issue isn't that the MACs are the same within a physical machine.
They're different, just like all the other HP gear ... It's that I have two
machines that the MACs are identical. Like Server-A's NIC1 matches
Server-B's NIC1 ... And the same goes for NIC2. Heck, maybe even their iLO
matches too. I just re-read my post and I can see where maybe I didn't
explain it properly. Yesterday was a long day ...
I guess it's not that big of deal now, I resolved it rather quickly by
putting Server-B on another VLAN.
So here is the interesting part... Both servers are HP Proliant DL380 G4s,
and both of their NIC1 and NIC2 MACs addresses are exactly the same. Not
spoofd and the OS drivers are not mucking with them ... They¹re burned-in
I triple checked them in their respective BIOS screen. I acquired these two
machines at different times and both were from the grey market. The ³What
the ...² is sitting fresh in my mind ... How can this be?
From the same grey market supplier?
I know HP has a disc they put out which updates all the firmware/bios in
a specific server model, its not too far fetched that a vendor might
have a modified version that also either purposely or accidentally
changes the MAC address. Off the top of my head, I'm not sure where the
MAC is stored - maybe an eeprom or a portion of the bios flash. Or, it
could be botched flashing that blew away the portion of memory where
that was stored and the system defaulted to a built in value.
Excellent example is, IIRC, the older sparc stuff, where the ethernet
cards didn't have MAC addresses as part of the card, but were stored in
non-volatile or battery backed memory.
This was actually the intended way to use "MAC" addresses, to used as
host addresses rather than as individual interface addresses, according
to the following paper -
That paper also discusses why 48 bits were chosen as the size, despite
"Ethernet systems" being limited to 1024 hosts.
I think things evolved into MAC per NIC because when add-in NICs
were invented there wasn't any appropriate non-volatile storage on the
host to store the address.
On really old Sun gear, the MAC address was stored on a separate ROM chip; if the
motherboard was replaced, you'd just move the ROM chip to the new board.
I'm not sure what you mean, though, when you say "when add-in NICs were
invented" -- the Ethernet cards I used in 1982 plugged into Unibus slots
on our VAXen, so that goes back quite a ways...
About 11-12 years ago, we ghosted Compaq Prosignia 330? desktops with
Intel NICs. When we ghosted them, some of the desktops ended up with
the same MAC addresses on the NICs. It turned out that there were two
different models of Intel NICs in the desktops and ghosting the
desktop with the second type of NIC resulted in the MAC address from
the original Ghost computer put on that computer. Updating the NIC
driver resolved the issue.
I should note -- this isn't that surprising. The IPv6 stateless autoconfig
RFCs have always assumed that this could happen, which is why duplicate
address detection is mandatory.
>>
>> Excellent example is, IIRC, the older sparc stuff, where the ethernet
>> cards didn't have MAC addresses as part of the card, but were stored in
>> non-volatile or battery backed memory.
>
> This was actually the intended way to use "MAC" addresses, to used as
> host addresses rather than as individual interface addresses, according
> to the following paper -
>
> "48-bit Absolute Internet and Ethernet Host Numbers"
> Yogan K. Dalal and Robert S. Printis, July 1981
> http://ethernethistory.typepad.com/papers/HostNumbers.pdf
Yup.
>
> That paper also discusses why 48 bits were chosen as the size, despite
> "Ethernet systems" being limited to 1024 hosts.
>
> I think things evolved into MAC per NIC because when add-in NICs
> were invented there wasn't any appropriate non-volatile storage on the
> host to store the address.
>
On really old Sun gear, the MAC address was stored on a separate ROM chip; if the
motherboard was replaced, you'd just move the ROM chip to the new board.
I'm not sure what you mean, though, when you say "when add-in NICs were
invented" -- the Ethernet cards I used in 1982 plugged into Unibus slots
on our VAXen, so that goes back quite a ways...
More that as add-in cards supplied their own "storage" for the MAC
address, rather than expecting it from the host (e.g. something like
MAC addresses set by init scripts at boot or the ROM chip you
mentioned on Suns), this has now evolved into an expected model of a
MAC address tightly bound to an Ethernet interface and supplied by the
Ethernet interface e.g. by an add-in board if one is added. Now that
this model as been around for a long time, people find it a bit strange
when MAC addresses aren't as tightly bound to a NIC/Ethernet interface.
This is all speculation on my part though, I'd be curious if the
reasons are different.
When I first read that paper, it was really quite surprising that "MAC"
addresses were designed to be more general host addresses/identifiers
that were also to be used as Ethernet addresses. One example they talk
about is using them as unique host identifiers when sharing files via
floppy disk.
We got a batch of NICS that had duplicate MACs in several pallets of
IBM desktops, about 15 years back. We noticed this only when two of the
machines were shipped to the same field office location.
I've heard other state agencies talk about the same sort of problem with
IBM and several other vendors.
