Hi folks,
Anyone knows what is used for the AWS Elastic IP? is it LISP?
Thanks.
Regards,
-lmn
Hi folks,
Anyone knows what is used for the AWS Elastic IP? is it LISP?
AWS does not really talk about things like this, but i highly doubt it is
LISP.
i sort of doesn't matter right? it is PROBABLY some form of
encapsulation (like gre, ip-in-ip, lisp, mpls, vpls, etc) ...
something to remove the 'internal ip network' from what is routed in a
datacenter and what is routed externally on the tubes.
Maybe a better question: "Why would lisp matter here?" (what makes
lisp the thing you grabbed at as opposed to any of the other possible
encap options?)
I don't know how the public blocks get to the datacenter (e.g. whether they are using MPLS) but after that I think it is pretty straightforward. All of the VMs have only one IPv4 address assigned out of 10/8. This doesn't change when you attach an Elastic IP to them.
All that is happening is that they have some NAT device somewhere (maybe even just a redundant pair of VMs?) that has a block of public IPs assigned to it and they are static NAT'ing the Elastic IP to the VM. They control the allocation of the Elastic IPs, so they just pick one that is routed out of that datacenter already. They probably don't need to do anything out of the ordinary to get it there.
From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Christopher
Morrow
Subject: Re: AWS Elastic IP architecture
[...]
i sort of doesn't matter right? it is PROBABLY some form of encapsulation
(like gre, ip-in-ip, lisp, mpls, vpls, etc) ...
[...]I don't know how the public blocks get to the datacenter (e.g. whether they are using MPLS) but after that I think it is pretty straightforward. All of the VMs have only one IPv4 address assigned out of 10/8. This doesn't change when you attach an Elastic IP to them.
right, so they encap somwhere after between 'tubez' and 'vm'. and
likely have a simple 'swap the ip header' function somewhere before
the vm as well.
All that is happening is that they have some NAT device somewhere (maybe even just a redundant pair of VMs?) that has a block of public IPs assigned to it and they
i'd question scalability of that sort of thing... but sure, sounds
like a reasonable model to think about.
What I am trying to get at is yeah, you still need the l2 extension
encapsulation, but on top you need something for disaster recovery, machines
mobility between data centers, sort of like Vshield Edge using NAT – you can
probably what the vm mobilty looks like is a change in the L2 path,
right? why make it anymore complicated than that? inside a single
availability domain I would expect the L2 domain a vm sees doesn't
change, even if the VM itself is moved from physical machine to
physical machine.
making it more complex at the vm level is probably a bunch of work
that doesn't have to happen.
change the NAT pool and update the DNS record, but the internal would remain
that sounds like a bunch of work though, which I don't think is really
necessary. I'm just a plumber, though so I don't actually know what
anyone does with this stuff.
the same no matter where you move it to. LISP seems like a simple
solution…so as specific host route injection, which for enterprise shouldn’t
lisp wasn't really finalized (still sort of isn't) when aws/ec2
started going like gang busters. They might have changed technology
under the hood, but it doesn't seem like they would have had to (not
in a drastic 'change encap type' sort of way at least).
be much of a problem, but DRaaS cloud provider, this could ballooning the
routing table pretty quickly.
how so? does the external and internal view from the vm have to be the
same? do the public /32's have to be individually routed ? inside what
scope at the datacenter?
What does Google use?
no idea, probably rabbits with different colored carrots?
I can tell you that EC2 Classic and VPC EIPs come from separate
netblocks...if that gives you any hints whatsoever.
There's no crossover between the two platforms in IP space.
I agree it appears ugly from a traditional network service provider perspective, but to my understanding much of the large scale cloud stuff is built using the cheapest, dumbest switching you can find and as little rich L3 routing gear (e.g. ASR/MX) as you can get away with.
The more functionality you can pack into software (with the universal building block being a VM), the less you have to worry about buying network hardware to any particular requirement other than "forwards Ethernet most of the time." It gives more control and agility to the developers of the platform, and spending a few gigabytes of RAM for every /23 and adding a little more latency and jitter ultimately becomes an economical trade off.
You can also move the network stuff up to the hypervisor layer (which I am sure they have done for things like Security Groups), but it makes rolling out updates harder and increases the general hack-level.
it's not clear to me that you need ram at all for this... there are
multiple dimensions to the scaling problem I was aiming at, this is
but one of them.
anyway, unless an EC2/aws/etc person speaks up I bet we can
'conjecturebate'(tm) forever without success.
At re:Invent they started releasing a surprising amount of detail on how
they designed the VPC networking (both layering/encapsulation itself and
distributing routing data). Like Michael mentioned, they really stuff as
much as possible into software on the VM hosts. That presentation is
https://www.youtube.com/watch?v=Zd5hsL-JNY4
While looking for that video I stumbled on a couple others that look along
those same lines:
https://www.youtube.com/watch?v=HexrVfuIY1k (all the connectivity options)
https://www.youtube.com/watch?v=YoX_frLHbEs (talks about public IP options)
Yeah, if it were LISP, they could probably handle IPv6.
Owen
From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Christopher
Morrow
Subject: Re: AWS Elastic IP architecture
[...]
i sort of doesn't matter right? it is PROBABLY some form of encapsulation
(like gre, ip-in-ip, lisp, mpls, vpls, etc) ...
[...]I don't know how the public blocks get to the datacenter (e.g. whether they are using MPLS) but after that I think it is pretty straightforward. All of the VMs have only one IPv4 address assigned out of 10/8. This doesn't change when you attach an Elastic IP to them.
right, so they encap somwhere after between 'tubez' and 'vm'. and
likely have a simple 'swap the ip header' function somewhere before
the vm as well.
It doesn’t sound like they have to encap/decap anything.
Sounds like the packet comes in, gets NAT’d and then gets routed to the 10/8 address by normal means.
Why do you assume some encap/decap process somewhere in this process?
All that is happening is that they have some NAT device somewhere (maybe even just a redundant pair of VMs?) that has a block of public IPs assigned to it and they
i'd question scalability of that sort of thing... but sure, sounds
like a reasonable model to think about.
They are known to be running multiple copies of RFC-1918 in disparate localities already. In terms of scale, modulo the nightmare that must make of their management network and the fragility of what happens when company A in datacenter A wants to talk to company A in datacenter B and they both have the same 10-NET addresses, the variety of things that are inherently broken by NAT or multi-layer NAT, and a few other relatively well-known problems, the biggest scalability problem I see in such a solution is the lack of available public IPv4 addresses to give to elastic IP utilization.
However, this is a scale problem shared by the entire IPv4 internet.
The solution is to add IPv6 capabilities to your hosts/software/etc.
Unfortunately, if you build your stuff on AWS, said solution is not possible and Amazon, despite repeated public prodding, has not announced any plans, intention, or date for making IPv6 available in a meaningful way to things hosted on their infrastructure.
Suggestion:
If you care about scale or about your application being able to function in the future (say more than the next couple of years), don’t build it on AWS… Build it somewhere that has IPv6 capabilities such as (in no particular order): Linode, Host Virtual[1], SoftLayer, etc.
Owen
[1] Full disclosure: I have no affiliation with any of the companies listed except Host Virtual (vr.org <http://vr.org/>). I have done some IPv4 and IPv6 consulting for them. I have no skin in the game promoting any of the above organizations, including Host Virtual. To the best of my knowledge, all of the organizations have ethical business practices and offer excellent customer service.
why can't they do v6 with any other encap?
the encap really doesn't matter at all to the underlying ip protocol
used, or shouldn't... you decide at the entrance to the 'virtual
network' that 'thingy is in virtual-network-5 and encap the packet...
regardless of ip version of the thing you are encapsulating.
why do you think they have a single 10/8 deployment per location and
not per customer? if it' sper customer, they have to provide some
encap (I'd think) to avoid lots and lots of headaches. I don't imagine
that if aws/ec2 is 'millions of customers' running on 'cheapest
ethernet reference platform possible' they can do much fancy stuff
with respect to virtual networking. I'd expect almost all of that to
have to happen at the vm-host (not the guest), and that there's just
some very simple encapsulation of traffic from the 'edge' to the
vm-host and then 'native' (for some sense of that word) up to the
'vm'.
Yeah, if it were LISP, they could probably handle IPv6.
why can't they do v6 with any other encap?
That’s not my point.
the encap really doesn't matter at all to the underlying ip protocol
used, or shouldn't... you decide at the entrance to the 'virtual
network' that 'thingy is in virtual-network-5 and encap the packet...
regardless of ip version of the thing you are encapsulating.
Whatever encapsulation or other system they are using, clearly they can’t do IPv6 for some reason because they outright refuse to even offer so much as a verification that IPv6 is on any sort of roadmap or is at all likely to be considered for deployment any time in the foreseeable future.
So, my point wasn’t that LISP is the only encapsulation that supports IPv6. Indeed, I didn’t even say that. What I said was that their apparent complete inability to do IPv6 makes it unlikely that they are using an IPv6-capable encapsulation system. Thus, it is unlikely they are using LISP. I only referenced LISP because it was specifically mentioned by the poster to whom I was responding.
Please try to avoid putting words in my mouth in the future.
Owen
i love that you are always combative, it makes for great tv.
Yeah, if it were LISP, they could probably handle IPv6.
why can't they do v6 with any other encap?
That’s not my point.
sort of seemed like part of your point.
the encap really doesn't matter at all to the underlying ip protocol
used, or shouldn't... you decide at the entrance to the 'virtual
network' that 'thingy is in virtual-network-5 and encap the packet...
regardless of ip version of the thing you are encapsulating.Whatever encapsulation or other system they are using, clearly they can’t do IPv6 for some reason because they outright refuse to even offer so much as a verification that IPv6 is on any sort of roadmap or is at all likely to be considered for deployment any time in the foreseeable future.
it's totally possible that they DO LISP and simply disable ipv6 for
some other unspecified reason too, right? Maybe they are just on a
jihad against larger ip numbers? or their keyboards have no colons?
So, my point wasn’t that LISP is the only encapsulation that supports IPv6. Indeed, I didn’t even say that. What I said was that their apparent complete inability to do IPv6 makes it unlikely that they are using an IPv6-capable encapsulation system. Thus, it is unlikely they are using LISP. I only referenced LISP because it was specifically mentioned by the poster to whom I was responding.
Please try to avoid putting words in my mouth in the future.
you have so many words there already it's going to be fun fitting more
in if I did try.
have a swell weekend!
Because that’s what one of their engineers told me at one point in the past.
Admittedly, it may have changed.
My understanding was along the lines of a very large flat L2 space among the VM Hosts with minimal routing on the hosts and a whole lot of /32 routes.
Again, my information may be incomplete, obsolete, or incorrect. Memories of bar conversations get fuzzy after 12+ months.
Owen
i love that you are always combative, it makes for great tv.
Yeah, if it were LISP, they could probably handle IPv6.
why can't they do v6 with any other encap?
That’s not my point.
sort of seemed like part of your point.
I swear, it really wasn’t.
the encap really doesn't matter at all to the underlying ip protocol
used, or shouldn't... you decide at the entrance to the 'virtual
network' that 'thingy is in virtual-network-5 and encap the packet...
regardless of ip version of the thing you are encapsulating.Whatever encapsulation or other system they are using, clearly they can’t do IPv6 for some reason because they outright refuse to even offer so much as a verification that IPv6 is on any sort of roadmap or is at all likely to be considered for deployment any time in the foreseeable future.
it's totally possible that they DO LISP and simply disable ipv6 for
some other unspecified reason too, right? Maybe they are just on a
jihad against larger ip numbers? or their keyboards have no colons?
I suppose, but according to statements made by their engineers, it has to do with the “way that they have structured their backend networks to the virtual hosts”.
I’m pretty sure that I’ve ruled the last two out based on discussions I’ve had with their engineers, but you’re right, I was probably a little more glib about it than was 100% accurate.
Bottom line, however, is it doesn’t matter what the reason, they are utterly incapable of doing IPv6 and utterly and completely unrepentant about it.
So, my point wasn’t that LISP is the only encapsulation that supports IPv6. Indeed, I didn’t even say that. What I said was that their apparent complete inability to do IPv6 makes it unlikely that they are using an IPv6-capable encapsulation system. Thus, it is unlikely they are using LISP. I only referenced LISP because it was specifically mentioned by the poster to whom I was responding.
Please try to avoid putting words in my mouth in the future.
you have so many words there already it's going to be fun fitting more
in if I did try.
LoL
have a swell weekend!
You too.
Owen
They could do 6rd by just flipping a switch on one of their routers.
Granted it is not native IPv6 but maybe better than nothing.
Regards
Baldur
Perhaps if that energy which was spent on raging, instead was spent on
a Google search, then all those words would've been unnecessary.
As it turns out that IPv6 is already available on ELBs since 2011:
https://aws.amazon.com/blogs/aws/elastic-load-balancing-ipv6-zone-apex-support-additional-security/
Official documentation:
http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/elb-internet-facing-load-balancers.html#internet-facing-ip-addresses
Netflix is using it already as per their techblog since 2012:
http://techblog.netflix.com/2012/07/enabling-support-for-ipv6.html
Regards,
Andras