[Fwd:] Nvidia NICs with duplicate mac addresses

Forwarded to NANOG in the interests of wider awareness... having been
there and torn out my already scarce hair, duplicate MAC addresses can
really mess up your day...

FWIW, IEEE's OUI lookup page says that's not assigned to *anyone*.

  http://standards.ieee.org/regauth/oui/index.shtml

Cheers,
-- jra

This reminds me of a story I was told a while back that there was a batch
of 3com NIC's that all went out with the same MAC from the factory. I
never found out if that was a rumor/urban legend or the truth. Anyone
know firsthand or have an article about that?

  -Scott

Does this MAC present itself all the time, or just during boot?

Marvel makes a NIC prevalent in some Dell systems, that presents MAC
0c00.0000.0000 during its startup process. If you run port security, and
several people boot their computer within the cam table expiration period,
port security will disable the port. You can work around it but it is time
consuming in large networks where port security are enabled.

Robert D. Scott Robert@ufl.edu
Senior Network Engineer 352-273-0113 Phone
CNS - Network Services 352-392-2061 CNS Receptionist
University of Florida 352-392-9440 FAX
Florida Lambda Rail 352-294-3571 FLR NOC
Gainesville, FL 32611 321-663-0421 Cell

it was real. (I still ahve some 3c503's with the problem :slight_smile:
this is one reason why it is so important to be able to override the MAC.

--bill

The Nvidia NIC on the Asus motherboard on my desktop computer
spontaneously changed its MAC maybe a year ago from 00:13:d4:fe:04:ee
to 00:0b:e0:f0:00:ed. It can still be overridden in software,
but the default MAC address -- the one stored in "ROM" -- simply
made a one-time, spontaneous, permanent change.

Nvidia NICs ... as my mother said, if you can't say anything nice,
don't say anything at all. So the rest is silence.

Jim Shankland

Out of curiosity, does this happen often enough we might want to
consider an automated means to negotiate out of the problem state
(e.g. detect collisions and negotiate MAC address by DHCP)?

It would take years to deploy but might save thousands of hairs.

Robert E. Seastrom wrote:

Just when you thought this couldn't happen any more...

Copying from a different email list...

mac address 04:4b:80:80:80:03, was showing up in multiple places
across the network.

That's why I stick to ARCNET :slight_smile:

...
alec

- --
`____________
/ Alec Berry \______________________________

Senior Partner and Director of Technology \
PGP/GPG key 0xE8E9030F |
http://alec.restontech.com/#PGP |
-------------------------------------------|
            RestonTech, Ltd. |
       http://www.restontech.com/ |
         Phone: (703) 234-2914 |

\___________________________________________/

Robert D. Scott wrote:

Does this MAC present itself all the time, or just during boot?

Marvel makes a NIC prevalent in some Dell systems, that presents MAC
0c00.0000.0000 during its startup process. If you run port security, and
several people boot their computer within the cam table expiration period,
port security will disable the port. You can work around it but it is time
consuming in large networks where port security are enabled.

I know that this doesn't apply here, but a year or so ago I had a client
that had issues with port security continually dropping an end user's
PC. The problem was the MAC address kept changing from Realtek to Cisco.
Sometimes the same NIC would present both MACs at the same time.

It turned out the box was apparently infected with something. Never
could find anything specific (even when booting the Windows box from
Knoppix and scanning for unusual files) except for some large ADS files
that where apparently encrypted. A clean wipe and complete rebuild of
the box fixed the problem.

Jon
- --
Jon R. Kibler
Chief Technical Officer
Advanced Systems Engineering Technology, Inc.
Charleston, SC USA
o: 843-849-8214
c: 843-224-2494
s: 843-564-4224

My PGP Fingerprint is:
BAA2 1F2C 5543 5D25 4636 A392 515C 5045 CF39 4253

The Marvel NIC presents the MAC as what we believe to be part of dot1x
negotiation. These were new Dells out of the box, not yet infected. If we
disable dot1x on the NIC the problem goes away.

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750e_3560e/software/re
lease/12.2_44_se/release/notes/OL14629.html#wp885689

Cisco's Recommendation: Replace the NIC

Robert D. Scott Robert@ufl.edu
Senior Network Engineer 352-273-0113 Phone
CNS - Network Services 352-392-2061 CNS Receptionist
University of Florida 352-392-9440 FAX
Florida Lambda Rail 352-294-3571 FLR NOC
Gainesville, FL 32611 321-663-0421 Cell

"David W. Hankins" <David_Hankins@isc.org> writes:

Forwarded to NANOG in the interests of wider awareness... having been
there and torn out my already scarce hair, duplicate MAC addresses can
really mess up your day...

Out of curiosity, does this happen often enough we might want to
consider an automated means to negotiate out of the problem state
(e.g. detect collisions and negotiate MAC address by DHCP)?

It would take years to deploy but might save thousands of hairs.

The same DHCP server (ip helper-address blah) serves my office, my
home, and the colo. Can you give me an idea of a good heuristic for
telling the difference between moving my laptop around and finding MAC
address collisions? Or are you suggesting that you hand out a MAC
address along with an IP address when the client DHCPs and the client
then changes it?

-r

The same DHCP server (ip helper-address blah) serves my office, my
home, and the colo. Can you give me an idea of a good heuristic for
telling the difference between moving my laptop around and finding MAC
address collisions?

The theoretically conforming client supplies two pieces of
information;

Its DHCPv4 client-identifier-option (per RFC 4361), which will be
different even on systems with identical MAC addresses (unless you
are very lucky).

The 'chaddr' BOOTP header contents ("its current MAC address"), which
is a return-address for unicast replies (before the client has an
IP address, ARP doesn't work).

The DHCPv4 client-identifier tells us it's a specific interface on
your laptop, no matter where it boots. Context from the packet
(giaddr (relay-agent address on that network), the interface it was
received upon) is used in normal DHCPv* operation (since its dawn) to
find a broadcast domain. From there, we find subnets on that domain,
and dynamic addresses within that subnet that are appropriate. This
is how we locate configuration when your laptop connects to different
wires in normal operation.

The new trick is to detect two clients with the same MAC address, but
with different client identifiers, that are active on the same
broadcast domain. It's just a simple database lookup to detect a
collision.

The solution is to:

Or are you suggesting that you hand out a MAC
address along with an IP address when the client DHCPs and the client
then changes it?

Precisely. Negotiate a new address in a broadcast reply. I suppose
you could just as easily always allocate a MAC dynamically, but this
seems invasive.

An alternative solution is to deny the client from receiving a config,
signalling an error to the operator, so the first client remains
operational. But this can have false positives.

It'd take years to deploy tho. Many clients today send no identifier
and just use their chaddr contents. Their MAC is their ID, so there
is no way to detect collisions. Most others send a client-id that
is identical to their chaddr contents, which is approximately useless.

You'd also be suffering MAC changes on systems that boot multiple
operating systems without releasing their previous lease (because
the clients send inconsistent identifiers, and even with DUID-based
id's, I think this is not going to change). This is in addition to
today, where such clients consume multiple IPv4 addresses. Insult
to injury.

Hi Jim,

My mother wasn't quite so adamant, she just said "don't cuss", so I'll
try to keep it clean as I relate my experiences with these "NICs of
our affliction".

As you're probably aware, Nvidia doesn't really have it together in
terms of playing nicely with open source folks. The Linux driver is
horribly reverse-engineered and the execution of the implementation is
even worse - it couldn't figure out the card's actual MAC address so
it assigned a random (and until recently often invalid, with an OUI
from bogus space) address. This resulted in some interesting stuff in
our switch logs.

My recollection is that FreeBSD was no better. We had one batch that
refused to pass traffic at 1000/full even when forced. Left me pining
for nice reliable stuff like counterfeit Cisco hardware bought from a
shady eBay store out of Hong Kong.

But I digress. Eventually we voted with our feet; I gave strict
instructions to our build guys to stop wasting our time with crummy
NICs with no support, and insisted that they pay the small amount of
extra money it takes to go with Intel. If you're looking for a funny
"prank" to play on your tech staff, speccing a batch of these, sitting
back, and watching the fun would get two thumbs up from me... but
otherwise steer clear.

Drive Slow,
Paul Wall

Oh, it's the truth - trust me. There was an Interop show (back when it really was an Interoperability event) that was made quite enjoyable for the network staff by that set of NIC cards....

  Chris