Best practice for ptp/loopback numbering for "small" enterprise multihome setup

Mailman List Mirror (Read Only)
1

Hello,

tl;dr - If I only have a /24 PI - is there any way to use this and not “chop it up / deagg” to use for ptp/loopbacks ?

Hopefully I can explain this in a manner that makes sense.

Say I have a vanilla dual router/dual upstream setup (think enterprise internet edge).

It’s basically an “H” shape:

- Two ISPs
- Two routers (“crosslink” is the middle of the H - iBGP)
- Each router has at least a link downstream into my public “outside” segment. I run an FHRP here. This is where my DMZ firewalls, VPN endpoints etc. have their outside interfaces.

Let’s also say I only have a /24 of PI.

I need to number the crosslink and the loopbacks. The upstreams will use their own /30 / /31 let’s say for the top of the H. My downstream interfaces will have my /24 (or parts of it) on the bottom of the H.

My understanding and instinct is that at least the crosslink should be numbered with public addresses. One scenario where this might matter is if router2 loses his inside interface (ex: switch failure) - he still needs to be able to deliver traffic via crosslink via router1 and onward downstream. When this traffic traverses here, if there is any kind of error (i.e. router is sourcing an ICMP packet of some kind) - this will be sourced from router1’s crosslink yes (assuming router1 is generating the error of course) ?

Loopback’s may be negotiable, as only router1/2 are using these to pin up iBGP. Nevertheless, my instinct would be to also use public addresses here.

As I said in the tl;dr - my main point of contention here is breaking up my /24 I.e. use the very top /30s / /31s for ptp/loop. I would then have at most the bottom /25 to use contig. on my “lan” - and I would need to use the next /26, /27 and so on in some manner for the space to be useable...

Here are some other options, and my understanding of the pros/cons:

- Use RFC1918
  Makes my eye twitch out of the gate. Not to mention packets sourced from here *should* get blocked by my upstreams by way of uRPF. Likely to be filtered at other points and directions as well…

- Get a /29 from one of the ISPs for ptp/loops
  Better than RFC1918. Kind of weird. If this is from isp1 - should have no issues sourcing toward them. Might have issues (uRPF) toward isp2. Announce this via BGP / no-export to both ISPs ? Now it’s getting even weirder…

- Use my own PI space
  Should have no filtering issues at all. Now I have to deagg my /24 and use the pieces (largest /25).

Am I making too big of a deal of this ? If you’ve read this far - I do appreciate it. Anxious to hear feedback on this.

PS: I would likely ACL on my upstream interfaces to block direct packets to my routers themselves as well.

2

tl;dr - If I only have a /24 PI - is there any way to use this and not
“chop it up / deagg” to use for ptp/loopbacks ?

i take real addresses out of the /24 for p2p
i take 1918 addresses for ibgp loopbacks

randy

3

Couple things come to mind that might be a more efficient use of address space: First, you don't need two routers in order to have dual upstreams. Have you considered multi-homing using a single router? If you need redundancy, it could be built into a single chassis. Another option is that some routers can perform active/standby failover without the need for extra public addresses. For example, two Cisco ASAs would have a cross-link, but this link is limited to keeping state and HA heartbeat between the two units and can be numbered with either an IPv6 link local or RFC-1918 address. Other platforms may have the option for Virtual Chassis, VSS, stacking, or similar technology that can conserve address space compared to two independent and traditionally addressed routers.

4

If you feel like getting fancy...

Use /32 routes to reclaim the unused base and broadcast address in any /30s

Pick the next largest size block that has your /24 neither at the
start or end and assign that to your lan.

Use proxy arp and more specific routes to grab traffic that doesn't
fall in the /24 or is part of the loopback and point to point
numbering and move it off the lan and towards its destination
(including upstream).

This way, every address in the /24 that you don't specifically use
elsewhere is usable as a unicast address on your LAN segment,
including the .0 and .255 addresses.

e.g.

You have 10.0.1.0/24

Put 10.0.0.0/22 on your LAN
Add proxy arp and route 10.0.0.0/24 upstream
Add proxy arp and route 10.0.2.0/23 upstream
Add proxy arp and route 10.0.1.254/32 to your first router loopback
etc.

If you're really clever you can convince the stations that 10.0.0.1 is
the default gateway but convince the router that 10.0.0.1 is upstream
so that the router doesn't even need a dedicated IP address facing the
LAN.

Regards,
Bill Herrin

5

i take real addresses out of the /24 for p2p
i take 1918 addresses for ibgp loopbacks

This.

6

And here I almost went as far as to suggest unnumbered IPs.... you're plan is... well... diabolical in comparison.

7

Hi Blake,

I aim to please. I really wish the router vendors supported a
statically configured "ICMP error from" address overriding source
address selection for ICMP error messages so that you could just put
RFC1918 on the router to router links instead of wasting global IP
addresses on them.

Regards,
Bill Herrin

8

Yes, in such a case, it would be extremely useful. However for bigger
transit networks, it is not nice if you only see loopbacks in
traceroutes, as opposed to interfaces.

In this particular case:

The only P2P interface that needs addressing is the link between the
two routers. For an enterprise stub-AS, I would certainly compromise
and use RFC1918 here as well (and use the entire /24 on the user
facing interface). Other than traceroutes, this does not realistically
cause an issue, *IN THIS PARTICULAR* scenario: you will not have
different MTU's requiring proper PMTUD handling in this direction and
some broken traceroutes won't be a problem for this business case. We
can't stop huge eyeball networks from using RFC1918 addressing on P2P
links, I guess we can live with a stub-AS doing it.

Another alternative is to use the actual user interface to get your
iBGP across, which is publicly addressed. Of course think about
different failure scenarios (partitioning), but it can work too. Maybe
use a different RFC1918 addressed P2P link with higher IGP costs, so
the 1918 dedicated link is used for in case of LAN partitioning only,
and usually you see public IP's in the traceroute.

I would avoid single control plane redundancy with proprietary
solutions like VSS and the likes. At some point they will tell you:
for this OS upgrade, you need to take BOTH devices out, because you
can't upgrade the cluster from major release X to Y, without
completely bringing it down.

Lukas

9

I think this may actually be the best solution proposed so far. iBGP (as spec'd by the op) can be setup between the WAN, LAN, loopback, or tunnel interfaces just as easily as using a dedicated p2p link. No reason to add complexity by throwing in an unnecessary p2p link in the mix.

10

> As I said in the tl;dr - my main point of contention here is breaking up my /24 I.e. use the very top /30s / /31s for ptp/loop. I would then have at most the bottom /25 to use contig. on my “lan” - and I would need to use the next /26, /27 and so on in some manner for the space to be useable...

If you feel like getting fancy...

Use /32 routes to reclaim the unused base and broadcast address in any /30s

Pick the next largest size block that has your /24 neither at the
start or end and assign that to your lan.

Use proxy arp and more specific routes to grab traffic that doesn't
fall in the /24 or is part of the loopback and point to point
numbering and move it off the lan and towards its destination
(including upstream).

In a lab, sure.

In production, you may be able to troubleshoot this a few months from
now, but how will the on-duty junior engineer handle this at 03 AM?

This way, every address in the /24 that you don't specifically use
elsewhere is usable as a unicast address on your LAN segment,
including the .0 and .255 addresses.

e.g.

You have 10.0.1.0/24

Put 10.0.0.0/22 on your LAN
Add proxy arp and route 10.0.0.0/24 upstream
Add proxy arp and route 10.0.2.0/23 upstream
Add proxy arp and route 10.0.1.254/32 to your first router loopback
etc.

If you're really clever you can convince the stations that 10.0.0.1 is
the default gateway but convince the router that 10.0.0.1 is upstream
so that the router doesn't even need a dedicated IP address facing the
LAN.

What you are suggesting is to configure public IP address space that
isn't yours, this should be a big nono. At the very least you can't
reach the public IP addresses 10.0.0.0 and 10.0.3.255 from the hosts,
because they won't be sending ARP requests for subnet and broadcast
addresses. So you break real public IP's of other people. In the worst
case scenario, some of this goes wrong and you cause entire public
subnets to be unreachable from this network.

We are talking about how to avoid 1918 addressing on P2P links. I'd
say NOT configuring other people's public IP addresses on your network
is more important than NOT configuring 1918 addressing on your
network.

Lukas

11

In production, you may be able to troubleshoot this a few months from
now, but how will the on-duty junior engineer handle this at 03 AM?

Hi Lukas,

In the network Vom describes, he is surely the only network engineer.
I would agree that it is not something anyone should try at scale --
the configuration complexity is higher than any ordinary network
architecture.

Vom's question was how to carve off some addresses without being stuck
at 1/2 the allocation as his maximum subnet size. At the sacrifice of
some complexity, it can be done. As described, you can even recapture
3 addresses that would normally be lost to you were you not attempting
to carve off addresses.

What you are suggesting is to configure public IP address space that
isn't yours, this should be a big nono.

That's one way of looking at it. Here's a different one: It is an
entirely legitimate network configuration to give your LAN a 0.0.0.0
netmask and rely on proxy arp to route off of it for non-local
addresses. Nobody does it this way, it's inefficient and gets very
complex when there's more than one router, but it in no way implies
configuring yourself address space which is not yours.

At the very least you can't
reach the public IP addresses 10.0.0.0 and 10.0.3.255 from the hosts,
because they won't be sending ARP requests for subnet and broadcast
addresses.

In the described configuration, those addresses are almost guaranteed
to be base addresses or broadcast addresses of someone else's network
which you wouldn't be able to reach or access anyway. There is a tiny
chance that someone else did the same thing you did or decided to use
a /32 route to capture and use those two addresses as unicast, but
you've a better chance of winning the lottery or being hit by
lightning than finding those two addresses in use.

Regards,
Bill Herrin

12

Hi Bill,

> In production, you may be able to troubleshoot this a few months from
> now, but how will the on-duty junior engineer handle this at 03 AM?

Hi Lukas,

In the network Vom describes, he is surely the only network engineer.

Actually I think it's more likely that he's a contractor/consultant,
but either way, contractor/consultant or employee ... all of them
change over time.

This falls into the kind of duct-tape "solutions" that inevitably
cause issues down the line, which then have to be diagnosed by
engineers at other networks. I'm that engineer at the other network,
diagnosing the issue because quote "it must be your fault because we
only have the issue with that single IP address of yours. Also Youtube
works just fine here."

Vom's question was how to carve off some addresses without being stuck
at 1/2 the allocation as his maximum subnet size. At the sacrifice of
some complexity, it can be done. As described, you can even recapture
3 addresses that would normally be lost to you were you not attempting
to carve off addresses.

Almost anything can be done by increasing complexity. But here the
cure is worse than the disease.

> What you are suggesting is to configure public IP address space that
> isn't yours, this should be a big nono.

That's one way of looking at it. Here's a different one: It is an
entirely legitimate network configuration to give your LAN a 0.0.0.0
netmask and rely on proxy arp to route off of it for non-local
addresses. Nobody does it this way, it's inefficient and gets very
complex when there's more than one router, but it in no way implies
configuring yourself address space which is not yours.

You are configuring a prefix that is not assigned to you and not
specifically reserved for local connectivity like 1918 (instead it is
almost certainly partially assigned to another AS for IPv4 unicast use
on the public Internet). That is the very definition of configuring
address space which is not yours, whether you're successful or not at
reducing the impact with proxy-arp.

Sure, in this specific case 0.0.0.0 and 255.255.255.255 truly will
never be used by anyone on the Internet (and also your hosts/router
will almost certainly crash, because ARP tables are not designed for
hundreds of thousands of entries), but that's not the point.

It's not legitimate in my book, when we are talking about hosts on the
public Internet which are required to connect to the rest of the
Internet, maybe even host services. If the bar is "can I be sued for
this?" than you are certainly right.

> At the very least you can't
> reach the public IP addresses 10.0.0.0 and 10.0.3.255 from the hosts,
> because they won't be sending ARP requests for subnet and broadcast
> addresses.

In the described configuration, those addresses are almost guaranteed
to be base addresses or broadcast addresses of someone else's network
which you wouldn't be able to reach or access anyway. There is a tiny
chance that someone else did the same thing you did or decided to use
a /32 route to capture and use those two addresses as unicast, but
you've a better chance of winning the lottery or being hit by
lightning than finding those two addresses in use.

Eyeball networks assign /32 to end users including .0 and .255. Of
course the likelihood that those two addresses are actually requiring
end-to-end connectivity to this AS is not huge.

But the fact of the matter is that you are knowingly breaking a valid
configuration in other people's networks with a non-null likelihood of
it causing problems, and for what? To avoid 1918 addressing on a
single P2P link of an enterprise stub-AS? This cure truly is worse
than the disease and it would certainly be unacceptable in my book.

Now let's talk about the likelihood of the OP leaking the /22 to the
transits (which hopefully filter strictly) in the process of setting
this up. I won't need lottery or lightning analogies for that.

Lukas

Lukas