Problems with removing NAT from a network

Mailman List Mirror (Read Only)
1

I've got a customer that is looking to multihome with upstreams in two POPs. Currently they multihome in one POP and utilize a single edge router for some one to one NAT and some PAT for their users.

Before they turn up the BGP peer in the new POP I've advised them to abolish NAT once and for all in order to avoid issues with non-stateful NAT between network edges and possible asymmetric routing of their Internet traffic.

The PAT can be removed easily enough. The tricky part is the one-one NAT. They have quite a few systems which have 1918 IPs which they claim "cannot be changed". At least not without some painful rebuilds of criticals systems which have these IPs deeply embedded in their configs.

Has anyone here had to fix this kind of problem before? Is there a solution that would allow NAT to offloaded to a smaller device hanging off each edge router that can communicate state between each other in case traffic is asymmetrically routed?

2

They shouldn't be using IP addresses in configs, they should be using DNS names. Time to bite the bullet and get this fixed prior to their eventual forced migration to IPv6.

3

The devil's in the details (obviously), and someone that reads into the
scenario better than me might have a more direct suggestion, but...

I'd start by moving the NAT at least one hop into the AS so that routing
symmetry can be enforced there. This allows for multi-homing (asymmetric
routing at the edge) without (before) dealing with the NAT issue (if there
is one at that point).

4

You didn't mention, but are you introducing a second border router? Is
the new upstream circuit from a new provider, or is it a second,
redundant circuit to the same provider in a different POP? Does your
customer have their own portable address space, or are they using
provider address space?

I'll make some presumptions: yes, it is a different provider, and no,
they don't have their own address space.

Based on those guesses/presumptions, I'd push to acquire portable
address space. Advertise it to both providers, carve a chunk of that
address space off and route it to a firewall(s) to perform border NAT.
Migrate old, provider dependent external NAT space to new, portable
address space.

-M

5

Somebody should tell the nytimes.com about this being a bad practice,
many of their images are linked to ip addresses directly and will
certainly fail in the future (this year, mobile) networks that will
use NAT64/DNS64. I am sure users will find other places to view their
news when nytimes.com fails to work in these ipv6-only networks.

Small summary of the problem of IPv4 literals and how they will break
in certain IPv6 environments that will be deployed this year
http://groups.google.com/group/ipv4literals

Cameron

6

>
>
>> At least not without some painful rebuilds of criticals systems which ha=
ve these IPs deeply embedded in their configs.
>
> They shouldn't be using IP addresses in configs, they should be using DNS=
names. =A0Time to bite the bullet and get this fixed prior to their eventu=
al forced migration to IPv6.
>

Somebody should tell the nytimes.com about this being a bad practice,
many of their images are linked to ip addresses directly and will
certainly fail in the future (this year, mobile) networks that will
use NAT64/DNS64. I am sure users will find other places to view their
news when nytimes.com fails to work in these ipv6-only networks.

Which is one of the reasons why DS-lite is a better solution for
providing legacy access to the IPv4 Internet than NAT64/DNS64.
DS-lite only breaks what NAT44 breaks. DS-lite doesn't break new
things.

7

>
>
>> At least not without some painful rebuilds of criticals systems which ha=
ve these IPs deeply embedded in their configs.
>
> They shouldn't be using IP addresses in configs, they should be using DNS=
names. =A0Time to bite the bullet and get this fixed prior to their eventu=
al forced migration to IPv6.
>

Somebody should tell the nytimes.com about this being a bad practice,
many of their images are linked to ip addresses directly and will
certainly fail in the future (this year, mobile) networks that will
use NAT64/DNS64. I am sure users will find other places to view their
news when nytimes.com fails to work in these ipv6-only networks.

Which is one of the reasons why DS-lite is a better solution for
providing legacy access to the IPv4 Internet than NAT64/DNS64.
DS-lite only breaks what NAT44 breaks. DS-lite doesn't break new
things.

Thanks for the tip. But, there are legitimate business reason in
various different types of networks for various strategies, thanks for
plugging the one your organization makes. I am tired of the IPv6
transition flavor of the day war. The reality for content folks is
that there will be IPv4 host, IPv6 hosts, and dual stack hosts.
Content needs to be dual-stack to reach everyone the best way
(native), but if they lack dual-stack and they use IPv4 literals, they
are going to lose eyeballs. End of story.

Content folks-- do yourself a favor and follow Roland's advice (also
in RFC 1958) and don't use address literals, use names.

And, you will notice that the list at
http://groups.google.com/group/ipv4literals shows only a few web site,
because there are only a few that have this design flaws. If you know
others, strengthen your case and add them to the list so that all
parties can benefit. Otherwise, it is just a few poorly designed
internet services that will be in a rush to fix services when users
complain.... or there web pages hits start trending down while their
competitors trend up.

Cameron

8

And the list looks like it does because the list only shows a *few* web sites. Other surveys have shown significantly more cases. ( http://tools.ietf.org/html/draft-wing-behave-http-ip-address-literals-02#appendix-B "An examination of Alexa's top 1 million domains [Alexa] at the end of August, 2009, showed 2.38% of the HTML in their home pages contained IPv4 address literals."

And the list looks like is does because the list only shows a few *web sites*. Quite a few network protocols, particularly peer-to-peer protocols, rely on moving around the IP address literals of peers via mechanisms other than DNS. This includes BitTorrent, Adobe's RTMFP, and Skype's proprietary protocol, and every VoIP system using STUN and/or ICE, to name just a few. Once users figure out that none of those will work when they use you as an ISP, they'll find one that's chosen a better transition technology.

Also note that DNSSEC end-to-end and DNS64/NAT64 are mutually exclusive. Now that DNSSEC is actually getting some traction, that's just one more reason to chose a different way to transition.

Matthew Kaufman

9

Or just run a dual-stack network, with centralized NAT44, and avoid the headache of tunneling. If you're going to run two protocol families on the end host, and deal with the issues that causes, why require tunneling to make it work? Is it so hard to forward IPv4 packets natively?

Cheers,
-Benson

10

>
>> On Wed, Jan 5, 2011 at 6:42 PM, Dobbins, Roland <rdobbins@arbor.net> wro=
te:
>> >
>> >
>> >> At least not without some painful rebuilds of criticals systems which=
ha=
>> ve these IPs deeply embedded in their configs.
>> >
>> > They shouldn't be using IP addresses in configs, they should be using =
DNS=
>> names. Time to bite the bullet and get this fixed prior to their=
eventu=
>> al forced migration to IPv6.
>> >
>>
>> Somebody should tell the nytimes.com about this being a bad practice,
>> many of their images are linked to ip addresses directly and will
>> certainly fail in the future (this year, mobile) networks that will
>> use NAT64/DNS64. I am sure users will find other places to view their
>> news when nytimes.com fails to work in these ipv6-only networks.
>
> Which is one of the reasons why DS-lite is a better solution for
> providing legacy access to the IPv4 Internet than NAT64/DNS64.
> DS-lite only breaks what NAT44 breaks. DS-lite doesn't break new
> things.

Thanks for the tip. But, there are legitimate business reason in
various different types of networks for various strategies,

Indeed. I just which DS-lite was thought of about the same time
as NATPT was. That way network operators would have the code in
things like cell phones today rather than the next gen resulting
in them being forced to use NAT64 and with that all the additional
problems it causes.

The network operator is going to be running a big/distributed NAT
box of one description or another to share the available IPv4
addresses. It just a matter of which packets its processing. The
end choice may come down to which DHCP options the client requests
or doesn't. i.e. return DNS64 nameservers if the client doesn't
request the DS-lite configuration parameters.

thanks for plugging the one your organization makes.

We also ship a DNS64 implementation.

                 I am tired of the IPv6
transition flavor of the day war. The reality for content folks is
that there will be IPv4 host, IPv6 hosts, and dual stack hosts.
Content needs to be dual-stack to reach everyone the best way
(native), but if they lack dual-stack and they use IPv4 literals, they
are going to lose eyeballs. End of story.

Agreed they will loose eyeballs. HTTP and IPv4 literals is one of the
easier problems to be mitigated. Its the rest of the places where IPv4
addresses are passed that causes problems.

11

And, you will notice that the list at
http://groups.google.com/group/ipv4literals shows only a few web site,
because there are only a few that have this design flaws.

And the list looks like it does because the list only shows a *few* web
sites. Other surveys have shown significantly more cases. (
http://tools.ietf.org/html/draft-wing-behave-http-ip-address-literals-02#appendix-B
"An examination of Alexa's top 1 million domains [Alexa] at the end of
August, 2009, showed 2.38% of the HTML in their home pages contained IPv4
address literals."

And the list looks like is does because the list only shows a few *web
sites*. Quite a few network protocols, particularly peer-to-peer protocols,

I understand my users pretty well, they only go to a few web pages ...
its the nature of the net. I assure you, i am not taking any undue
risk with regards to web. Try our friendly user trial and give me
your feedback, thats why i am running it.

rely on moving around the IP address literals of peers via mechanisms other
than DNS. This includes BitTorrent, Adobe's RTMFP, and Skype's proprietary

Ah Skype. According to your web page you work at Skype. Skype is a
well known IPv6 spoiler application. In fact, in the IETF and many
other circles, Skype is the only app that we can't seem to get to work
with IPv6. Are you here to help with that or to tell us that we need
to keep IPv4 around indefinitely?

In fact, we were just talking about how Skype as a spoiler this morning

http://www.ietf.org/mail-archive/web/v6ops/current/msg06837.html

Here is a pointer to IPv6-only users who would love to use Skype on
IPv6-only handsets.

http://talk.maemo.org/showthread.php?t=60320&page=24

Looking at the last post, it looks like they were able to NAT46 Skype
to then talk out the NAT64 ... ugly. But serious, get with me off
list and lets collaborate. I can help from the networks side, and i am
eager to make progress. Skype should not be the IPv6 spoiler app when
NEARLY EVERYTHING ELSE WORKS. Read the thread i mentioned, real
users, real developers, real network that is IPv6-only. Notice that
things generally work, those folks have hacked their way to perhaps
even making Skype work.

protocol, and every VoIP system using STUN and/or ICE, to name just a few.
Once users figure out that none of those will work when they use you as an
ISP, they'll find one that's chosen a better transition technology.

Seriously, 95+% of my traffic is web and email, and STUN and ICE don't
matter much to grandma as long as m.v6.facebook.com loads.

Also note that DNSSEC end-to-end and DNS64/NAT64 are mutually exclusive. Now
that DNSSEC is actually getting some traction, that's just one more reason
to chose a different way to transition.

Strategy is done. Implementation is on-going. 3GPP and IETF joint
meeting said dual-stack and IPv6-only + NAT64 are the 2 paths forward
for mobile.

http://www.3gpp.org/ftp/workshop/2010-03-01_IPv4-to-IPv6_with-IETF/Docs./IPW100060.zip

Without going into too much detail, ds-lite does not live in mobile
and likely never will. Any solution that requires IPv4 is not
strategic. IPv4 addresses simply are not available at the scale of
large mobile network operators, public, private, or bogon. Mobile
must move to IPv6, and IPv6-only + NAT64 pushes the envelope in ways
that dual-stack never has, and hence it just might work to
*transition* to v6.

As long as dual-stack is around, the app vendors don't have to move
and network guys have to dream up hacks to support these legacy apps
(CGN ....).

Cameron

12

As long as dual-stack is around, the app vendors don't have to move
and network guys have to dream up hacks to support these legacy apps
(CGN ....).

NAT64 is CGN expecially when it is being implemented by the cellular
carriers.

13

As long as dual-stack is around, the app vendors don't have to move
and network guys have to dream up hacks to support these legacy apps
(CGN ....).

NAT64 is CGN expecially when it is being implemented by the cellular
carriers.

Agreed. And, the NAT44 that 99% of my customer use to today is also a CGN.

It's status quo, all v4 flows require state in my network, NAT44 or NAT64.

But, NAT64 has an exit strategy. With every new AAAA that comes out,
that is one less destination requiring state in my network.

Cameron

14

I will give you that it is easy to see with NAT64 when the target
space has moved. It also forces you to upgrade all client applications
to support IPv6 from the start.

Anyway its horses for courses.

Mark

15

I understand my users pretty well, they only go to a few web pages ...
its the nature of the net. I assure you, i am not taking any undue
risk with regards to web. Try our friendly user trial and give me
your feedback, thats why i am running it.

I'm not particularly surprised that a mobile client platform has a different access pattern than desktop users... not a whole lot of mobile BitTorrent clients yet, for instance.

Ah Skype. According to your web page you work at Skype. Skype is a
well known IPv6 spoiler application. In fact, in the IETF and many
other circles, Skype is the only app that we can't seem to get to work
with IPv6. Are you here to help with that or to tell us that we need
to keep IPv4 around indefinitely?

Indeed, I work at Skype now and Adobe (developing RTMFP) before that.

At this point (because not everyone has IPv6) this class of applications (along with BitTorrent and ICE-using VoIP apps) needs to be able to use your NAT64 to talk to peers that are IPv4-only. To do that, they need to be able to discover your NAT64 even though they're not doing DNS lookups to find the IPv4 addresses of peers.

This will take 1) a way to do this and 2) upgrades of the apps to take advantage of it. It is impossible to do #2 until #1 is solved.

There's been discussion in BEHAVE about this topic... draft-korhonen-behave-nat64-learn-analysis for instance. I even proposed a solution that wasn't raised in that or previous documents here: http://www.ietf.org/mail-archive/web/behave/current/msg09050.html (which I suppose, since it hasn't been mentioned elsewhere, should be written up as a draft if/when I have some free time)

  Skype should not be the IPv6 spoiler app when
NEARLY EVERYTHING ELSE WORKS. Read the thread i mentioned, real
users, real developers, real network that is IPv6-only. Notice that
things generally work, those folks have hacked their way to perhaps
even making Skype work.

There's lots of other apps that don't work. Skype is just the squeaky wheel because it is so popular.

Seriously, 95+% of my traffic is web and email, and STUN and ICE don't
matter much to grandma as long as m.v6.facebook.com loads.

See my above comment about how I'm not surprised, given the specific client population.

As long as dual-stack is around, the app vendors don't have to move
and network guys have to dream up hacks to support these legacy apps
(CGN ....).

Dual-stack + NAT44 has a lot fewer unsolved corner cases *and* doesn't require apps to be upgraded to do discovery of the NAT64 prefix(es) (which, for some legacy apps that are no longer under development will never happen).

NAT64/DNS64 is an interesting experiment that works for >95% of the web. But it isn't really a solution unless "the web" is all you care about.

Matthew Kaufman

16

I understand my users pretty well, they only go to a few web pages ...
its the nature of the net. I assure you, i am not taking any undue
risk with regards to web. Try our friendly user trial and give me
your feedback, thats why i am running it.

I'm not particularly surprised that a mobile client platform has a different
access pattern than desktop users... not a whole lot of mobile BitTorrent
clients yet, for instance.

Ah Skype. According to your web page you work at Skype. Skype is a
well known IPv6 spoiler application. In fact, in the IETF and many
other circles, Skype is the only app that we can't seem to get to work
with IPv6. Are you here to help with that or to tell us that we need
to keep IPv4 around indefinitely?

Indeed, I work at Skype now and Adobe (developing RTMFP) before that.

At this point (because not everyone has IPv6) this class of applications
(along with BitTorrent and ICE-using VoIP apps) needs to be able to use your
NAT64 to talk to peers that are IPv4-only. To do that, they need to be able
to discover your NAT64 even though they're not doing DNS lookups to find the
IPv4 addresses of peers.

This will take 1) a way to do this and 2) upgrades of the apps to take
advantage of it. It is impossible to do #2 until #1 is solved.

There's been discussion in BEHAVE about this topic...
draft-korhonen-behave-nat64-learn-analysis for instance. I even proposed a
solution that wasn't raised in that or previous documents here:
http://www.ietf.org/mail-archive/web/behave/current/msg09050.html (which I
suppose, since it hasn't been mentioned elsewhere, should be written up as a
draft if/when I have some free time)

Skype is not defined in an IETF RFC, so saying you need an RFC to move
forward is bit confusing. There are several methods that just work
today,

I am all for standards, but a closed platforms generally find ways to
progress without or in spite of standards. Skype is a closed
platform.

Skype should not be the IPv6 spoiler app when
NEARLY EVERYTHING ELSE WORKS. Read the thread i mentioned, real
users, real developers, real network that is IPv6-only. Notice that
things generally work, those folks have hacked their way to perhaps
even making Skype work.

There's lots of other apps that don't work. Skype is just the squeaky wheel
because it is so popular.

Please make a list and let us know. Otherwise, this is just hand
waving like the IPv4 literals sites. I have had people on various
mailing list tell me all things they think won't work, but in my own
experience and in the experience of my beta users, who are publicly
documenting their efforts on the support boards, things work well.
Yes, some things don't work due the fact that it is a beta and
something don't work due to the fact that it is IPv6-only, but they
are not show stoppers for the business.

As a user, i have been using IPv6-only on Symbian for 9 months without
an issue. The service works as i would expect it to with IPv4, no
user perceived differences. Skype is a squeaky wheel, but most (99%)
things users do works fine. As mentioned, in mobile, 95+% of the
traffic is web and email. And, most "apps", are also just web and
email.

My advice to Skype is to come up with a solution to work for IPv6-only
clients. That is my advice to all apps and all content. IPv6-only
clients are an obvious reality in an IPv4 exhausted world.

You cannot seriously come to a network operators support mailing list
and say that the network guys have to keep investing in network tweaks
while you wait for a standards body to solve a problem for your closed
non-standard applications.

I won't go into my diatribe about why dual-stack is not an obvious
choice in mobile, you can check the video of it at the google IPv6
conference in 2010. If you have further questions, i am glad to help
you understand and entertain your ideas off list. The main point is
the dual-stack and substantially more expensive and the IPv4 part of
dual-stack is run out of addresses....private, public, bogon.

http://www.youtube.com/watch?v=1GlRgaFriYU#t=15m38s

Seriously, 95+% of my traffic is web and email, and STUN and ICE don't
matter much to grandma as long as m.v6.facebook.com loads.

See my above comment about how I'm not surprised, given the specific client
population.

As long as dual-stack is around, the app vendors don't have to move
and network guys have to dream up hacks to support these legacy apps
(CGN ....).

Dual-stack + NAT44 has a lot fewer unsolved corner cases *and* doesn't
require apps to be upgraded to do discovery of the NAT64 prefix(es) (which,
for some legacy apps that are no longer under development will never
happen).

Dual-stack cost the mobile network operator 2x in the packet core,
check the Youtube video. It is not acceptable to ask the network
operators to increase their cost by 2x while the apps sit and wait.

NAT64/DNS64 is an interesting experiment that works for >95% of the web. But
it isn't really a solution unless "the web" is all you care about.

Experiment? Yes, the experiment part is over. As i mentioned, we are
deploying. The 3GPP has adopted IPv6-Only + NAT64 as an official path
nearly a year ago. I have tried to make it clear to folks that
upgrading to IPv6 is the only way to ensure your apps and content
continue to work as you expected. Otherwise, you traverse NAT64 CGN
and some things won't work. The same is true for DS-lite and others,
but in different ways. To that end, lets partner up and make Skype
work with IPv6.

I also assure you, many mobile operators are pursuing this NAT64 path
for the same reason I am.

Cameron

17

Skype is not defined in an IETF RFC, so saying you need an RFC to move
forward is bit confusing.

I don't see a disconnect at all. Skype also uses TCP and UDP, which are both subjects of RFCs.

That said, it doesn't need to be an RFC... just *a reliable way* of discovering the appropriate NAT64 prefix.

  There are several methods that just work
today,

Of the methods proposed in the survey draft, only one - the one that doesn't require the DNS64 spec or operator to make any changes (making an AAAA lookup for something you know only has an A record) - works but *only if* the mapping scheme is such that it is possible to successfully derive a functional prefix and the scheme from the results of that query.

So in other words, *if* the query results in an AAAA where, by inspection, you can guess where you'd need to stuff the IPv4 address bits *and* the resulting address causes the "right" NAT64 (if there's >1) to be used, then you're set.

I am all for standards, but a closed platforms generally find ways to
progress without or in spite of standards. Skype is a closed
platform.

No question. And for all you know we might be working on other ways around this problem, but none of them as elegant as a defined specification for how to discover the presence of a NAT64 and the mapping.

There's lots of other apps that don't work. Skype is just the squeaky wheel
because it is so popular.

Please make a list and let us know. Otherwise, this is just hand
waving like the IPv4 literals sites.

I'll start with "peer to peer connectivity using RTMFP in Flash Player" and "BitTorrent". Both Flash Player and BitTorrent are fairly popular on desktop platforms.

I'm sure there's more.

My advice to Skype is to come up with a solution to work for IPv6-only
clients. That is my advice to all apps and all content. IPv6-only
clients are an obvious reality in an IPv4 exhausted world.

That's not the problem... the problem is reaching the existing base of IPv4 clients from those IPv6-only clients without making Skype relay all the traffic via servers somewhere, as I'm sure you know.

You cannot seriously come to a network operators support mailing list
and say that the network guys have to keep investing in network tweaks
while you wait for a standards body to solve a problem for your closed
non-standard applications.

I've been on this list since approximately the time it was formed, so I'm not coming here to ask for something. Just pointing out what will break.

I also assure you, many mobile operators are pursuing this NAT64 path
for the same reason I am.

Randy Bush would encourage his competitors to do just as you've done, I'm sure.

Matthew Kaufman

18

Doesn't all of this become moot if Skype just develops a dual-stack capable client
and servers?

Owen

19

Skype is an interesting case because of its peer-to-peer nature. Given the
state of v6 deployment and operational experience[1], and especially given the
fragility of tunneled IPv6 connections[2], there is potential for a lot of
broken links.

    --Steve Bellovin, http://www.cs.columbia.edu/~smb

[1] All concerned are mystified about why my office machines can't reach [major
site] via v6, even though traceroute6 from both ends shows the packets reaching
the same network. A couple of weeks earlier, I couldn't get anywhere via IPv6
because someone had connected a NAT+v6 tunneler in bridge mode, and my machine
believe its RA messages.

[2] See http://www.google.com/intl/en/ipv6/faq.html#tunnel

20

Doesn't all of this become moot if Skype just develops a dual-stack capable client
and servers?

Really, only some fraction of the supernodes and the login servers need
to be dual stack.