Multiple vendors' IPv6 issues

Mailman List Mirror (Read Only)
1

Hi NANOG,

The company I work for has no business case for being on the IPv6-Internet.
However, I am an inquisitive person and I am always looking to learn new
things, so about 3 years ago I started down the IPv6 path. This was early
2012.

Fast forward to today. We have a /44 presence for our company's multiple
sites; All our desktop computers have been on the IPv6 Internet since June,
2012 and we have a few AAAAs in our external DNS for some key services —
and, there have been bugs. *Lots* of bugs.

Now, maybe (_maybe_) I can have some sympathy for smaller network companies
(like Arista Networks at the time) to not quite have their act together as
far as IPv6 goes, but for larger, well-established companies to still have
critical IPv6 bugs is just inexcusable!

This month has just been the most disheartening time working with IPv6.

Vendor 1:

Aruba Networks. Upon adding an IPv6 address to start managing our WiFi
controller over IPv6, I receive a call from our Telecom Lead saying that or
WiFi VoIP phones have just gone offline. WHAT? All I did was add an IPv6
address to a management interface which has *nothing* to do with our VoIP
system or SSID, ACLs, policies, roles, etc.

Vendor 2:

Palo Alto Networks: After upgrading our firewalls from a version which has
a nasty bug where the IPv6 neighbor table wasn't being cleaned up properly
(which would overflow the table and break IPv6), we now have a *new* IPv6
neighbor discovery bug where one of our V6-enabled DMZ hosts just falls of
the IPv6 network. The only solution: clear the neighbor table on the Palo
Alto or the client (linux) host.

Vendor 3:

Arista Networks: We are seeing a very similar ND bug with Arista. This one
is slightly more interesting because it only started after upgrading our
Arista EOS code — and it only appears to affect Virtual Machines which are
behind our RedHat Enterprise Virtualization cluster. None of the hundreds
of VMware-connected hosts are affected. The symptom is basically the same
as the Palo Alto bug. Neighbor table gets in some weird state where ND
breaks and the host is unreachable until the neighbor table is cleared.

Oh, and the final straw today, which is *almost* leading me to throw in the
IPv6 towel completely (for now): On certain hosts (VMs), scp'ing a file
over the [Arista] LAN (10 gigabit LAN) takes 5 minutes over IPv6 and <1
second over IPv4. What happened?

It really saddens me that it is still not receiving anywhere near the kind
of QA (partly as a result of lack of adoption) that IPv4 has.

Oh, and let's not forget everybody's "favorite" vendor, Cisco. Why is it,
Cisco, that I have to restart my IPv6 OSPF3 process on my ASA every time my
Palo Alto firewall crashes and fails over, otherwise none of my VPN clients
can connect via IPv6?

Why do you hurt me so, IPv6? I just wanted to be friends, and now I just
want to break up with you. Maybe we can try to be friends again when your
vendors get their shit together.

-David

2

Had ipv4 ever hurt you ?

Me too.

CB

3

David,

While I agree with you that there is no excuse for the general IPv6 brokenness across all vendors, they are just doing what participants on lists like this one tell them. Name&Shame may help a little, but until a large number of people get serious and stop prioritizing IPv4 in their purchasing demands, the vendors are not going to prioritize IPv6. Until the vendors clearly hear a collective "we are not buying this product because IPv6 is broken", everyone will get exactly the behavior you are witnessing.

While I appreciate the challenges you are facing, it is likely that you will be helped by documenting the percentage of IPv6 traffic you see when things do work. While it may not be much now, that can change quickly and will provide internal ammunition when you try to take a stand about refusing to use a product. If your IPv6 percentage grows anywhere near the 2x/yr rate that Google has been seeing it won't take long before IPv6 is the driving protocol. For fun, project this
http://www.google.com/intl/en/ipv6/statistics.html forward 4 years and hand it to the vendors that can't get their IPv6 act together. Then ask them how they plan to still be in business at that point ......

Tony

4

IPv4 still hurts me (in some ways, worse than IPv6), and it's 2015.
Figures...

You just need to open cases with your vendors and help them fix these
issues. Sadly, no way around this. Software is not perfect. The humans
that write it, even less so.

Mark.

5

Is VMWare contributing somehow to the problem?

Marcin

6

My current favorites are:

https://tools.cisco.com/bugsearch/bug/CSCut62344

Which doesn't allow you to see the neighbors on an interface. this is fun
when diagnosing qemu/kvm issues with the macvtap and hosts with ipv6.
turns out you to 'fix it' you need to make the macvtap interface promisc
as the icmpv6 messages don't make it through the macvtap driver to the VM
breaking neighbor discovery.

  You can guess how we saw the first bug with the second one.

  This isn't as bad as a colleague who told me he is taking
classes at a university whose professor said that a /20 is neither a class A
or class B allocation but in the middle, not knowing that CIDR has existed
for the past 20 years. Turns out we need a few more SMEs to teach people
about CIDR and IPv6 addressing to prevent univeristy professors from
teaching the next generation something that doesn't apply anymore.

  - Jared

7

You don't need full promisc mode, just the (poorly documented) allmulticast option (ip link set dev $macvtap allmulticast on)

8

David,

While I agree with you that there is no excuse for the general IPv6 brokenness across all vendors, they are just doing what participants on lists like this one tell them. Name&Shame may help a little, but until a large number of people get serious and stop prioritizing IPv4 in their purchasing demands, the vendors are not going to prioritize IPv6. Until the vendors clearly hear a collective "we are not buying this product because IPv6 is broken", everyone will get exactly the behavior you are witnessing.

While I appreciate the challenges you are facing, it is likely that you will be helped by documenting the percentage of IPv6 traffic you see when things do work. While it may not be much now, that can change quickly and will provide internal ammunition when you try to take a stand about refusing to use a product. If your IPv6 percentage grows anywhere near the 2x/yr rate that Google has been seeing it won't take long before IPv6 is the driving protocol. For fun, project this
http://www.google.com/intl/en/ipv6/statistics.html forward 4 years and hand it to the vendors that can't get their IPv6 act together. Then ask them how they plan to still be in business at that point ......

Tony

  I like this page even better for that purpose. It does the forward projecting for you and projects 33% in one year and above 90% in 4 years.

https://www.vyncke.org/ipv6status/project.php?metric=q&country=us

  This says that 45% of web pages viewed by people worldwide are available via IPv6 (It does not say that 45% of web pages are available via IPv6, it says that since Facebook and others, which are IPv6 enabled, have more page views than some less popular sites that are IPv4 only and that results in 45% of web pages viewed being available via IPv6.)

http://6lab.cisco.com/stats/

http://6lab.cisco.com/stats/information.php#content

  It is also interesting to sort this page by IPv6 percent.

http://www.stateoftheinternet.com/trends-visualizations-ipv6-adoption-ipv4-exhaustion-global-heat-map-network-country-growth-data.html#networks

9

Hi,

If someone from google is listening it would be really nice to
spend a few minutes t oavoid flash for displaying this graph, it doesn't
work on my Google Nexus 4 and my flash-less chrome/chromium desktops :).

Sincerely,

Laurent

10

...And poorly supported on some real hardware (notably Wi-Fi adapters), where the hash filter on each NIC's MAC is not guaranteed to support "ALLMULTI".

It's a prerequisite for software multicast forwarding, and, it might be argued, a good litmus test for IPv6 readiness.