IPv6 migration steps for mid-scale isp

Mailman List Mirror (Read Only)
1

Hello,

Recently we have decided to start IPv6 migration in our network. We
have ~1K BNGs and connecting our customers to network using PPPoE.
I'd be interested in hearing from the technical community about their
experiences and recommendations on this process. I'm wondering:

Shall I go for IPv6-only deployment or dual stack?
Where to start with IPv6? (core, edge or ...)
What are the best practices for ISPs?
What are the costs and return on investment?
How to identify address CPE and legacy application issues?

Fredrik

2

Hello,

Recently we have decided to start IPv6 migration in our network. We
have ~1K BNGs and connecting our customers to network using PPPoE.
I'd be interested in hearing from the technical community about their
experiences and recommendations on this process. I'm wondering:

Shall I go for IPv6-only deployment or dual stack?

For PPPoE with existing IPv4, go dual stack.

Where to start with IPv6? (core, edge or ...)

Core, peering, work outwards towards end users.

What are the best practices for ISPs?
What are the costs and return on investment?
How to identify address CPE and legacy application issues?

There is a lot written and presented about IPv6 deployment. People have been doing this in volume since around 2010, and if you search for IPv6 deployment experience you'll find lots of presentations.

Some I found that seem relevant:

https://www.ipv6council.be/experiences-de-deploiements-ipv6/

If you prefer video form, there are lots of presentations from conferences, available on youtube as well.

3

We are small but we are just about out of IPv4 and aren't going to get
or buy any more. We have been testing for a while.

Shall I go for IPv6-only deployment or dual stack?

You should plan for adding customers eventually that are IPv6-only,
unless you have all the v4 you will ever need, and you will need to
reserve IPv4 address blocks for translation.

How to identify address CPE and legacy application issues?

Legacy application issues can be solved (for the most part) with
464XLAT, which also solves IP-literal-in-HTML problems. You need PLAT at
the core and CLAT at the client. Unfortunately so far the only good way
we've found to do CLAT is OpenWRT on the CPE or router. We are getting
ready to bundle Linksys routers flashed with OpenWRT.

For PLAT at the core we are running jool. It's actually quite simple to
set up and we are currently using OSPF to do anycast, but we will
probably be migrating to a single set of HA servers in the core. The
good news is that even if it goes down, Netflix and Facebook will still
work as they are fully functional on v6.

We have tested this in my home and at my office with IPv6-only
VLANs/wireless SSIDs, mostly without a hitch.

If you run this setup without the CLAT on the client side you get NAT64
so it still will work for most things.

I would be very, very happy if larger ISPs would put pressure on router
manufacturers to support CLAT, we have no clout. I would also love to
hear if any of this is stupid or if there's a better way.

--Brock

4

Thank you all for your Ideas. AFAIK one of the main decisions for IPv6
transition and deployment is the choice of IPv6 IGP. I read somewhere
that its a good practice to use different IGP protocol for IPv6 and
IPv4. For example if IGP for IPv4 is IS-IS then use OSPFv3 for IPv6.
any comments on this?
Additionally I will appreciate it if you share your suggestions on
products and their performance? For example If I go for NAT64+DNS64 to
handle IPv4 traffic, What sort of carrier grade products are you
recommending and can you share your experience on their
performance/pitfalls? currently we have ~150Gbps of IPv4 traffic, so
we need a solution to
support such scale and future growth.

Regards,

5

Hi,

My advice is not to mix IGPs. If you are already running ISIS, then go for multi-topology and dual-stack it.

I've done that several times, running different backbones, works like a charm. Less overhead as you'd only be running one IGP.

My 2 cents.

6

Thank you all for your Ideas. AFAIK one of the main decisions for IPv6
transition and deployment is the choice of IPv6 IGP. I read somewhere
that its a good practice to use different IGP protocol for IPv6 and
IPv4. For example if IGP for IPv4 is IS-IS then use OSPFv3 for IPv6.
any comments on this?

I've never heard this promoted as good practice - why on earth would
you want to make life more difficult for yourself by running two IGPs
if you can run one?

Yes, if you use OSPF for IPv4 you *have* to use something else for
IPv6. But if you already run IS-IS there is no reason to change, just
remember to enable multi-topology.

Steinar Haug, Nethelp consulting, sthaug@nethelp.no

7

if IGP for IPv4 is IS-IS then use OSPFv3 for IPv6.

this is nuts. one is-is instance carries both

randy

8

Hi,

My advice is not to mix IGPs. If you are already running ISIS, then go for multi-topology and dual-stack it.

I've done that several times, running different backbones, works like a charm. Less overhead as you'd only be running one IGP.

My 2 cents.

Adding few cents of €. :wink:

mh

9

Yes, if you use OSPF for IPv4 you *have* to use something else for
IPv6. But if you already run IS-IS there is no reason to change, just
remember to enable multi-topology.

Well... sort of.

The reality is that from a configuration and management perspective, there's very little difference between OSPFv2 for IPv4 and OSPFv3 for IPv6.

I've run OSPF 2 and 3 on multiple networks and it's not difficult or problematic in any way.

Owen

10

16/09/2017 kl. 10.56 sthaug@nethelp.no:

Yes, if you use OSPF for IPv4 you *have* to use something else for
IPv6. But if you already run IS-IS there is no reason to change, just
remember to enable multi-topology.

Steinar Haug, Nethelp consulting, sthaug@nethelp.no

We somehow manage to run just OSPFv2 and still deliver both IPv4 and IPv6... :slight_smile:

The trick is MPLS with Internet in a VRF for both IPv4 and IPv6.

Regards,

Baldur

11

Hi Fredrik,

Running two different IGPs for IPv4 and IPv6 is a recipe for disaster even
if it’s a short-term goal.

Here are a few things to consider;

OSPF is good for small ISPs with small routing tables (10 to 15K routes).
It will support more routes but configuration of your network becomes more
complex hence an increase in human error (network engineers)

EIGRP is more suitable for mid-size say 50K subscriber base but you are
really stretching your luck if you go beyond the 50K subscriber base.

EIGRP is more susceptible to flap when adding a new device with an MTU
mis-match etc. You can google up some stories about EIGRP flap issues….

My recommendation is to use iBGP for both IPv4/IPv6, you can use OSPF or
EIGRP for link layer connectivity and iBGP to carry the traffic.

I prefer OSPF over EIGRP because of its equal cost load balancing if you
have multiple interfaces from PE devices to your core.

iBGP is scalable, you can introduce router reflectors to avoid full mesh
peering between PE routers – and the sky if your limit!

Hope this helps

Thanks

Ahad

12

Hello,

for my point of view, the start question is do you control CPEs (can
re-configure and re-flash it), or users buy and own CPEs themself?

13

Fully agree, 464XLAT is the way to go.

We have tested this in many IPv6-only access deployments, non-cellular networks (cellular is well tested by T-Mobile and others, that have got it in production for years).

We always have the issue of the CPEs support, but this is the same problem if you want to go to lw4o6, MAP, etc. In general, newer transition mechanisms, aren’t well supported.

So, you either use OpenWRT if you can re-flash the CPEs, or you push your vendors to make sure they provide a firmware upgrade.

This is the reason I started to work on an update of the RFC7084 (https://datatracker.ietf.org/doc/draft-ietf-v6ops-rfc7084-bis/ and https://datatracker.ietf.org/doc/draft-palet-v6ops-rfc7084-bis-transition/) and see also the related discussion in v6ops.

Also, I run a panel with CPE vendors in the last week APNIC meeting, and the interesting thing is that they confirmed there is no any technical issue to support those (hardware is ok), and they have already developed it, just waiting for customers to ask for it.

https://conference.apnic.net/44/program/schedule/#/day/6/bof-discussion-with-ipv6-ce-vendors

I will compile a report out of this panel ASAP.

So please, keep pushing your vendors for it!

Regards,
Jordi

14

Hi Fredrik,

Running two different IGPs for IPv4 and IPv6 is a recipe for disaster even
if it’s a short-term goal.

Here are a few things to consider;

OSPF is good for small ISPs with small routing tables (10 to 15K routes).
It will support more routes but configuration of your network becomes more
complex hence an increase in human error (network engineers)

Another perfectly workable alternative is to divide your network up into OSPF
instances which each have an internal ASN and linking them together with BGP.

EIGRP is more suitable for mid-size say 50K subscriber base but you are
really stretching your luck if you go beyond the 50K subscriber base.

You also have the problem that EIGRP locks you into an all-Cisco network.

EIGRP is more susceptible to flap when adding a new device with an MTU
mis-match etc. You can google up some stories about EIGRP flap issues….

My recommendation is to use iBGP for both IPv4/IPv6, you can use OSPF or
EIGRP for link layer connectivity and iBGP to carry the traffic.

I prefer OSPF over EIGRP because of its equal cost load balancing if you
have multiple interfaces from PE devices to your core.

iBGP is scalable, you can introduce router reflectors to avoid full mesh
peering between PE routers – and the sky if your limit!

I think in general most serious networks consider this a question of OSPF
vs. ISIS for IGP and BGP is the only choice for EGP.

I find it interesting that you don’t even mention ISIS in your discussion.

I don’t know of any substantial networks running EIGRP these days. I’m not
saying they don’t exist, but they are certainly rare exceptions.

Owen

15

> iBGP is scalable, you can introduce router reflectors to avoid full mesh
> peering between PE routers – and the sky if your limit!

I think in general most serious networks consider this a question of OSPF
vs. ISIS for IGP and BGP is the only choice for EGP.

I find it interesting that you don’t even mention ISIS in your discussion.

I don’t know of any substantial networks running EIGRP these days. I’m not
saying they don’t exist, but they are certainly rare exceptions.

The fact that we'll be running dual-stack for perhaps another decade and
that there are no 36-hour days available makes the choice very simple;
IS-IS is my preferred choice. One routing instance less.

But, I'd rather limit the IS-IS scope to "links and loopbacks" -- there
is no need to have link-state flooding for a customer network that will
always be originated from one specific access router. iBGP is much more
appropriate for that. As long as I'll have one working path up to that
router I can rely on BGP to tell me where the network is.

The key is the time domain. If the topology is likely to be changing
slowly (customer moves premises or commissions new connection), use
BGP to signal it. If the topology is potentially unstable, i.e. subject
to backhoes and similar, use IS-IS.

Oh, by the way; I concur with Owen: EIGRP is not done. I've stumbled
on it once the last decade, and it was a PABX network engineer who
insisted.

16

On 9/13/17, 8:08 AM, "NANOG on behalf of Fredrik Sallinen"

Hello,

Recently we have decided to start IPv6 migration in our network. We
have ~1K BNGs and connecting our customers to network using PPPoE.
I'd be interested in hearing from the technical community about their
experiences and recommendations on this process. I'm wondering:

Shall I go for IPv6-only deployment or dual stack?

Dual-stack is the best way to get to IPv6-only. You’ll need IPv6-only in
order to solve the IPv4 runout problem, though “only” is likely to include
translation.

Where to start with IPv6? (core, edge or ...)

Web site.
Then core and peering.
Then push toward regional networks. You’ll need IPv6 into at least the
part of your data center does provisioning and monitoring.
Then start pushing to customers.

What are the best practices for ISPs?

Lots of documents exist. Here’s one: RFC 6782 - Wireline Incremental IPv6

What are the costs and return on investment?

Oh, I have so much to say on this topic. Search for “TCO of CGN” and “TCO
of IPv6” to find it.
You should have very little incremental capital cost; that is, you
shouldn’t have to buy new hardware, because practically anything less than
ten years old can support IPv6. Whether it *does* is a question.
You’ll have some opex network engineering costs in testing and deployment,
which might be high(ish) if you have a lot of different equipment in your
network. Your biggest cost is likely to be the development labor to update
your IPAM, provisioning systems, management, logging, and tech support
tools to be able to store and use the new address format. Development is
likely to be what takes longest, so that’s really the place to start.

The return on investment is that you don’t have to spend $15 to buy an
IPv4 address for every new user you have to sign up. Or $25 per address in
2019, if trends continue. [1]
Depending on how happy you are with your transition mechanism (NAT64,
464xlat, MAP, etc.) you could, instead, sell off those IPv4 addresses.
“Hey, boss, we’ll turn up 10,000 new customers in 2019, right? We can
either spend $250,000 to buy addresses, or we can put 10% of our customers
behind NAT64 (or whatever) and sell their IPv4 addresses for $25 each.”
(Dozens of NANOGers laugh, a few cock their heads and think “maybe…”).

How to identify address CPE and legacy application issues?

How do you do it now?
I’m guessing you test CPE that you provide, at least for basic
functionality.
Some bugs still get past you. A few customers call, you notice a trend
among customers with X CPE that they have a problem in the area where you
just rolled out IPv6. You roll back IPv6 in that area or (hopefully) just
from those devices, and put that CPE in the lab and test the heck out of
it.

For legacy applications, it depends on the application. If you can update
it, that’s the best answer. Or you can put a NAT64 box in front of it (on
a VM, router, firewall, or load balancer—you don’t necessarily need new
hardware). But there’s also the entire rest of the old Internet: you will
probably want to have some kind of transition mechanism in place.

I know folks who specialize in transition mechanisms; I’ll follow up
privately so this doesn’t look like a sales pitch on the list.

Fredrik

Lee

[1] Charts, using IPv4auctions.com (Hilco Streambank) data:
http://www.wleecoyote.com/blog/2017prices.htm

17

CPE's are owned by our customers but we have control over ~60% of them
using TR069.

18

Please correct me If I'm wrong, AFAIK 464XLAT works best with mobile
networks and its not suitable for fixed broadband. right?

19

It's most widely deployed in mobile networks, yes. There is nothing that says it couldn't work anywhere else.

However, in fixed networks with PPPoE the most commonly used model is dual stack with RFC7084 style routers.

20

There are several ISPs doing trials (thousands of users).

RFC6877 (464XLAT), section 4. Network Architecture, indicates clearly “Wireline Network Architecture can be used in situations where there
   are clients behind the CLAT, regardless of the type of access service
   -- for example, fiber to the home (FTTH), Data Over Cable Service
   Interface Specification (DOCSIS), or WiFi.”

Vendors confirmed two weeks ago they have implementations in CEs.

RFC7084 was created before all the new transition technologies (including 464XLAT and MAP, for example, or even lw4o6 that has many advantages compared to DS-LITE, being the same but requiring a much simpler CGN), so that’s why I’m working to update it (most probably as an “accompanying document” only for the transition part:

https://datatracker.ietf.org/doc/draft-ietf-v6ops-rfc7084-bis
https://datatracker.ietf.org/doc/draft-palet-v6ops-rfc7084-bis-transition

New versions to be publish this week hopefully …

Regards,
Jordi