Dear NANOG,
Some friends of mine are operating a nonprofit (on shoe string) and looking
to connect some CDN caches to an IX fabric. A BGP speaking device is needed
between the caches and the BGP peers connected to the fabric. The BGP
speaker is needed to present the peers on the IX with a unified view of the
assemblage of CDN nodes.
I was wondering whether anyone was experience with the "EdgeRouter Infinity
XG" device, specifically in the role of a simple peering router for a
couple of tens of thousands of routes. (I'd point default to the left and
take just the on-net routes on the right to reduce the table size
requirement).
I hope the device can do at least 2xLACP trunks, has a sizable FIB, is
automatable (supports idempotency), can forward IMIX at line-rate, *flow,
and exposes some telemetry via SNMP.
Any note sharing would be appreciated!
Kind regards,
Job
I'm Ubiquiti's biggest critic. I'll check with my colleagues.
EdgeRouter is... meh. If I was looking at that class of gear I'd go with a Mikrotik.
~Seth
Job,
There is a bit of a price differential here, depending on whether you need
SFP+; the Infinity is "dead cheap", and has fairly opaque BGP
daemon+debugging tools. Also still technically a beta product. Not sure if
it meets your automation requirements. I wouldn't want to be deploying them
in a redundant pair, myself, but just when you say something can't be done…
Mikrotik's CCR1072: 10-gig router (shipping, not anything that's just been
announced) has an API, can certainly handle a few tens of thousands of
routes fine (single core BGP though), but I can't vouch for its ability to
do IMIX or *flow at line rate. This has probably been stress tested by
somebody. I doubt the sampling is in hardware.
If you don't need 10G ports then your options expand considerably. Do you
have a target throughput?
Specs...
- MIPS64 16 Core 1.8 GHz
- 16 GB DDR4 RAM
- 8 MB NOR Flash 4 GB eMMC NAND Flash
- Data Ports: (1) RJ45 Serial Port, (8) SFP+ Ports (1) RJ45 Gigabit
Ethernet Port
- 2 hotswap power supplies
No LACP. ECMP is currently broken. MPLS/VPLS is currently broken and not
done in hardware - this may eventually change. As far as the other stuff,
"telemetry" etc - no.
As far as BGP crunching, plenty of routes, etc - it would easily and
happily be fine with that.
As far as automation, it's a JunOS-like CLI originally based on vyatta,
which AT&T now owns - and one of the main reasons is it's scriptability,
use of Ansible and other tools right on the device, python, etc.
Technically I believe it's based on VyOS rather than Vyatta. Same base, but just delineating that VyOS is open source and I don't believe AT&T wields any control over it.
EdgeOS was forked from Vyatta well before (around Vyatta Core 6.2?) VyOS
took up the last public Vyatta release. It has therefore diverged
somewhat from current VyOS releases, but the two are still
mostly-compatible.
Paul
EdgeOS was forked and employees were poached from Vyatta before it was
purchased by Broadcom, when it was open source. I think a few things
came down from VyOS after that, but not many.
Why not use a Linux or BSD computer for this? It is cheap and you know
exactly what you are getting. It will forward 10 gig at line rate at least
for normal traffic.
Regards
Baldur
I kinda feel the same way. I wish FRR was a big more mature at this
point though.
The RAM is upgradeable but it can support quite a few full tables out of
the box. The routing software under the hood got upgraded by Ubiquiti to
ZebOS https://www.ipinfusion.com/products/zebos/ from the VyOS code.
There is a Cavium bug regarding UDP packets though that can be nasty if you
hit it.
https://community.ubnt.com/t5/EdgeMAX/UDP-packet-loss-with-EdgeRouter-Lite/m-p/1343012
Even though the thread starts by talking about the Lite, all of the Cavium
EdgeRouters currently have the problem. The beta work around is to
restrict packet forwarding to only use one of the CPU cores. This is with
or without hardware offloading enabled. Hopefully Cavium will have a real
fix soon. I have two of these I'm itching to put into production once the
bugs are worked out.
BTW... At Fandor (before I left) we got one of the last /24s that ARIN
had. Our transit providers at the office were Monkey Brains (wireless)
and Zayo (fiber). We purchased a ER Pro, upgraded the memory and were
peering v4 with both on this box. MB didn't have V6 at that point. We
did nail up our V6 announcement with Zayo and got it that way.
If folks need config examples.
Tim
Hello Jeremy,
can certainly handle a few tens of thousands of
routes fine (single core BGP though),
It can take multiple full views. It's also faster than an MX104.
but I can't vouch for its ability to
do IMIX or *flow at line rate
I wouldn't load one to 80g, but at 10-20G, it creates no bottleneck.
The entire packet-pipeline is in software. IPFIX is not sampled, it's
1:1 only AFAIK. It's also lacking some features, meaning you'd need to
filter through pmacct to add BGP informations.
Best regards,
Tried the Infinity, unsuccessfully. Several of them. Ended up pulling them
all, sitting in my homelab for now. Multiple full tables, nothing fancy for
firewall or QOS, but ran into issues with random ribd/bgpd crashes and
kernel panics. I've submitted a lot of logs and core dumps to UBNT. I would
personally stay away from them until they are out of beta, and possibly
even another 6-12 months after that.
The current stable EdgeMax version (1.9.1.1) is relatively stable, but
using an outdated ZebOS (1.2.0?) with a number of issues (MPLS, OSPF, BGP)
- nothing too major, but can be annoying. Probably okay for what you
described. Depending on how much throughput you need, an ERPro, or Mikrotik
would probably be fine. If you need 10G, load up VyOS on some cheap servers
with an Intel or Solarflare card... probably cheaper than a beta Infinity
or Mikrotik.
Forgot reply all...
That does not apply to the infinity. Those shipped with 1.9.8dev.
Ah, okay. I haven't used one yet.
Also, I don't talk about beta outside of beta. 
When I lasted checked in with Ubiquiti on these issues for that and the
ER-Pros - they told me that everything was to be resolved in 2.0....
We shall see...
1.9.7 definitely applies to Infinity:
ER-8-XG:
https://dl.ubnt.com/firmwares/edgemax/v1.9.7/ER-e1000.v1.9.7+hotfix.1.5005858.tar
(SHA256:b1a16900e3fbe1eef3876548ac7eda12a95ef849d4328f22b478459e2a506b92)