Matt Harris
Infrastructure Lead
816‑256‑5446
Direct
Looking for help?
We build customized end‑to‑end technology solutions powered by NetFire Cloud.
We use SonicWall TZ series for just this purpose. The IPSec VPN endpoints can be behind NAT, and we just use DYNDNS to map whatever is current to a FQDN. Each side thus has the public IP of the other side and can connect as long as you pass through GRE.
-mel via cell
Meraki may be considered expensive, requires perpetual license to operate and is difficult to get currently (very long lead times) but is dead.stupid.simple to install and maintain. I have yet to find a business or home network that it does not work on out of the box, but if you find one it would be an issue to overcome for any solution, right? i.e. open some ports on the up stream device one time.
I think my experience is unique, but wanted to put it out there anyway. I’ve actually had quite a few problems with Meraki equipment during the one instance I worked with them. After a few hours to days, the switches would stop functioning. You could still access them through the webgui and issue a reboot to resolve the issue, but the problem persisted even after many resets and calls with Cisco.
Again, likely some bonk hardware, but in case anyone else has had a similar experience I wanted this to be known.
Thank you,
– Ryland
ZeroTier is not a free-as-in-freedom project. Running it in Linux boxes
or network appliances to provide a VPN to paying customers may be
prohibited (at least for some customers, and before 2025) by its
convoluted license:
https://github.com/zerotier/ZeroTierOne/blob/master/LICENSE.txt
I recommend using something that doesn't have litigious companies
nitpicking about what you can and can't use it for.
John Gilmore
tailscale is 3-clause BSD.
there is a reverse engineered version of the rendezvous protocol also.
I work in a large oil company and we have S2S VPNs every where. Any modern Cisco or Juniper router will meet your requirements. An off the shelf security appliance will do the job to i.e ASA, Palo Alto, Fortinet or Juniper. Meraki is great if you want to manage from the cloud or vpn as a service. Good luck.
Sean P Kelly
Howdy,
I just want to say thank you to everyone who responded. It was very
helpful and I now have a bunch of leads to chase. I'll let you know
what I end up doing. Given the lead times on some of the equipment it
may be a while...
Warm regards,
Bill Herrin
We like pfSense.
I believe they sell Netgate appliances.
Mark.
Take a general purpose OS, strip down the userspace a bit,
stick the whole thing in a box, and call it an appliance. They'll never
know the difference. 
I don't know how people around here feel about Mikrotik, but they have included Wireguard support in their latest operating system.
dave
I don't know how people around here feel about Mikrotik, but they have included Wireguard support in their latest operating system.
They've also included fq_codel and sch_cake:
https://forum.mikrotik.com/viewtopic.php?t=179307
For a site to site, kernel mode vpn such as ipsec or wireguard (but
not openvpn), we successfully FQ+AQM packets entering the tunnel.
If that's the bottleneck link, for a mixture of, say low rate voip and
high rate file transfer traffic, the results are a pretty marvellous
reduction of jitter and latency through the tunnel.
Before: http://www.taht.net/~d/ipsec_fq_codel/oldqos.png
After: http://www.taht.net/~d/ipsec_fq_codel/newqos.png
I know some Tik heads here that are happy about this.
I am running ROS 7.1.2 on my home router, but I don't use it.
Mark.
Sabri Berisha <sabri@cluecentral.net> writes:
I read on some mailing list that Meraki likes to ping 8.8.8.8 every
second...
That's probably to be fair with the quad-x dns providers since they
alrady were abusing 1.1.1.1.
Makes me wonder what Meraki uses 9.9.9.9 for
Bjørn
Mikrotik with RouterOS v7 with WireGuard or ZeroTier were the first things I thought of, but it might be a a bit premature for a production environment. In a year, I’d have no problem recommending that.
Thank you Joy for de-lurking. I actually was not familiar with ZeroTier, and this is a space that I thought I was quite familiar with, so I’m glad you brought it to everyone’s attention. I will look further at ZeroTier, it looks very interesting.
I am also a very long-time lurker (although I was a NANOG list admin ~10 years ago) who is emerging to join this conversation.
I have recently been doing some work to evaluate and develop VPN solutions for connecting multiple data center cloud environments, including low-power small edge sites, and I have some thoughts about the current state of the art to share.
Until recently a very strong proponent of IPSEC. I liked the way IPSEC was placed within the OSI model directly at layer 3, unlike some of the VPN technologies which operate above or below layer 3. However I do not believe that IPSEC is future-proof, for the following two reasons:
-
IPSEC does not lend itself to dynamic routing or dynamic configuration. It is very much a static set-it-and-forget-it technology, but that doesn’t work in a dynamically changing environment.
-
IPSEC does not always lend itself to hardware offloading in the way some other technologies do. Some NICs do support hardware acceleration for IPSEC, but this does not always integrate well with kernel or user space when you are integrating virtual network functions (VNFs) like routers/firewalls/load-balancers.
Wireguard works well in dynamic environments. TLS using something like OpenSSL does as well. Both provide key advantages, particularly on top of Linux.
-
Support for hardware offloads such as TCP segmentation provide vast improvements in performance on higher-end x86 hardware. Some recent testing I have been shown proves that TCP segmentation offload can provide more than a 5X speedup compared to other HW offloads without TCP segmentation (from 5Gb/s to above 25Gb/s in some tests).
-
With the right encryption algorithm CPU acceleration for cryptography reduces CPU load and increases performance.
-
Integration with kernel routing provides the ability to integrate with dynamic routing such as BGP daemons (e.g. FRRouting, etc.).
-
In recent Linux kernels eBPF/XDP provide a hardware interface to the kernel which accelerates network throughput to near line-rate, while minimizing CPU impact.
This may not apply to William Herrin’s (OP) use case of a VPN appliance for 100mbps to 1gbps speeds, but it is something to keep in mind for building higher performance solutions or for planning for increasing bandwidth in the future. For the 100mbps+ use case I have had success building appliances using OpenVPN on top of certain ARM based platforms like Marvell Armada, or single-board computers with Intel CPUs with AES-NI acceleration. I am currently looking at implementing Wireguard on the same platforms. For a simple low-power ARM router appliance the Turris Omnia has been a great fully open platform running a custom LEDE/OpenWRT OS. The Turris Mox provides a modular hardware platform for expandability, albeit with slightly less performance. Both of these platforms are developed by the engineers at CZ.nic, the TLD registrar for the Czech Republic.
https://secure.nic.cz/files/Turris-web/Omnia/Omnia2020_datasheet.pdf
https://www.turris.com/en/mox/overview/
-Dan Sneddon
Dan,
One point you didn’t touch on is that IPSec is integrated into IPv6, typically hardware-accelerated on the NIC, enabling device-to-device VPNs, mitigates most of the dynamic issues associated with network-to-network IPSec over IPv4.
Yes, I realize IPv4 is hanging around longer than most expect, but in some cases I think you can make a case for deploying IPv6 just on the VPN benefits alone. With no public-facing services, IPv6 is already deployed in most LANs as a direct result of its use by modern OSes for inter-LAN communication. All you typically need to do is enable IPv6 at the gateway.
-mel
1) IPSEC does not lend itself to dynamic routing or dynamic configuration. It is very much a static set-it-and-forget-it technology, but that doesn’t work in a dynamically changing environment.
Hi Dan,
Depending on how you configure it, IPSEC can work fine with dynamic
routing. The thing to understand is that IPSec has two modes:
transport and tunnel. Transport is between exactly two IP addresses
while tunnel expects a broader network to exist on at least one end.
"Tunnel" mode is what everyone actually uses but you can deconstruct
it: it's built up from transport mode + a tunnel protocol (gre or ipip
I don't remember which) + implicit routing and firewalling which
wreaks havoc on dynamic routing. Now, it turns out that you can
instead configure IPSec in transport mode, configure the tunnel
separately and leave out the implicit firewalling.
This may not apply to William Herrin’s (OP) use case of a VPN appliance
It's not relevant to my situation, no. I need the VPN to establish a
statically addressed clean layer 3 on top of dynamically addressed and
natted endpoints to support the next appliance in the chain where
dynamic addressing is not possible. I don't actually care if it adds
security; it just needs to establish that statically addressed layer.
Oh yeah, and it has to be listed under "virtual private network" on
the government NIAP list.
Regards,
Bill Herrin
Intriguing. This week I started to look around for new wireguard implementation tools and appliances. I've used openvpn and ipsec in the main although last month put together a 10x and IPv6 wireguard net in my home and out to two vps hosts which is handy. For my own use this is ok -ish, but I am not so sure about keeping track of the configs, managing users and adding configs as a network grows. In other words I want help when scaling wg and handling change particularly if I am managing nets for other projects or delegating.
Tailscale, ZeroTier and some others are doing a great job I feel and no doubt have a handle on that. I've not tried them as yet.
Because I do like to have options that are not mediated I have kept looking as much for my own curiousity and education as for deploying a service in anger. But having a toolset that can support the latter capability has to be the aim to work towards.
I've found a few potentially interesting more recent projects and am intending to start to test deploy some of these in sequence to see how I get on. I think I'll start wth
https://github.com/gravitl/netmaker Please note I've only reviewed the documentation. I've not yet played with it.
This seems to offer at an early stage in its development a webappliance (optionally) with CoreDNS if you want naming support and IPv6 and at least some client management features. It claims to be fast but that can be tested. It also is deployable as a docker/kubernetes k8 which is intriguing when deploying and managing containers between multiple hosts across data centres. It uses a mongodb licence which may or may not be a problem.
If one plays with IPSEC then I guess one could run wg through IPSEC but is there any point unless you already have an IPSEC branch and don't want to take it down whilst adding wg for a new class of devices/userbase?
I'd be interested in sharing experiences and advice (offlist) and delighted to learn from wireguard and vpn's clueful folk.
thank you for an interesting discussion.
Christian
William Herrin <bill@herrin.us> writes:
The thing to understand is that IPSec has two modes: transport and tunnel. Transport is between exactly two IP addresses while tunnel expects a broader network to exist on at least one end.
That is (syntactically) correct. However, it is possible to NAT many LAN IPs (say RFC 1918) to one single Internet IP (say from a SOHO ISP) and use IPSec /Transport/ Mode to a single remote IP. The IPSec sees exactly two IPs.
"Tunnel" mode is what everyone actually uses
I may be enough of an outlier that I'm a statistical anomaly. But I'm using IPSec /Transport/ Mode between my home router and my VPSs. I have a tiny full mesh of IPSec /Transport/ Mode connections.
Using the aforementioned many-to-one NAT, my home LAN systems access the single globally routed IP of each of my VPSs without any problem.
Aside: I did have to tweak MTU for LAN traffic going out to the VPS IPs.
So -1 for '"Tunnel" mode is what everyone actually uses', and +1 for /Transport/ Mode
but you can deconstruct it: it's built up from transport mode + a tunnel protocol (gre or ipip I don't remember which) + implicit routing and firewalling which wreaks havoc on dynamic routing.
I question the veracity of that statement. It may be that's what many implementations / administration systems do. But I really thought that IPSec /Tunnel/ Mode was more than just IPSec /Transport/ Mode combined with some tunneling protocol.
Now, it turns out that you can instead configure IPSec in transport mode, configure the tunnel separately and leave out the implicit firewalling.
Agreed. I feel like this speaks to implementation / management systems that are built on top of IPSec.
It's not relevant to my situation, no. I need the VPN to establish a statically addressed clean layer 3 on top of dynamically addressed and natted endpoints to support the next appliance in the chain where dynamic addressing is not possible. I don't actually care if it adds security; it just needs to establish that statically addressed layer.
It sounds to me like you don't even actually need encryption of a typical VPN and might be able to use something like GRE+key or IPSec /Tunnel/ Mode with AH without ESP.
Oh yeah, and it has to be listed under "virtual private network" on the government NIAP list.
NIAP: Product Compliant List
Oh joy. Layer 8 - politics