I came across this over the weekend. Given that the project was abandoned six years ago, are there any other efforts with a similar goal (more intelligently placing routes into FIBs of low-FIB capacity devices?
It’s not a problem, due to cheap, plentiful high-speed memory and rapid prefix search silicon in backbone routers. The entire Internet routing table consumes at most a few gigabytes when fully structured (and only a few hundred Mbytes stored flat). That’s less memory than your average laptop sports.
Even in the worst case scenario, where every network decides to announce only its most specific prefixes, the BGP backbone would temporarily enter an oscillating state that generates a large number of routing updates into the inter-domain routing space. In this case, BGP route damping will quickly suppress the crazies while the backbone stabilizes.
Small routers should not be taking full tables, since there is no point to them being in the default free zone. For large routers, neither memory nor CPU speed are an issue. High-speed routers operating in the default-free zone have a critical path in the forwarding decision for each packet: it needs to take less than the inter-packet arrival time for minimum-sized IP packets.
This is easy to achieve with today’s hardware. A router line card with an aggregate line rate across all of its point-to-point interfaces of 10Tbps (readily available in today’s gear) can process packets with just a handful of cycles in the FIB Ternary Content Addressable Memory (TCAM) using ASIC-assisted lookups. TCAM is the most expensive component you’re paying for in such a router. It’s not cheap, but backbone routers don’t need to be cheap. They just need to not be memory-constrained.
-mel via cell
https://github.com/sflow-rt/active-routes
Inspired by SIR, but uses Bird multi-table capability to separate RIB/FIB routes.
The problems of today are the same as the problems of yesterday, just with bigger numbers.
There are likely more networks with 10 gigabit or less total external capacity than there are with more.
Creating imaginary lines in the sand doesn’t really help anyone.
If your BGP router supports sflow or netflow and includes ASN in those, you can use
akvorado, as-stat, or elastiflow.
Disagree that it’s a line in the sand. It’s use the right tool for the job.
If a device is low FIB, it’s that way for a reason. There are plenty of ways to massage that with policy and software, depending on capabilities , but at the end of the day, trying to sort 10 pounds of shit to store in a 5 pound bag is eventually going to end up the same way.
Some of the reasoning behind ‘i need/want to do SDN things’ is ‘low fib device’ sort of reasonings. Some is: “I just want a forwarding decision that’s not entirely LPM oriented”
(or to be fair: “My LPM is not JUST longest prefix, but also some other data”)
For folk that are looking for software alternatives you might look at faucet:
https://github.com/faucetsdn/faucet
which still seems to be actively developed.
“The right tool for the job” gets into a religious argument in assuming that one’s way to do the job is the only reasonable way to do the job.
Large networks historically have a very poor (IMO) model of gigantic iron in a few locations, which results in sub-optimal routing for the rest of their network between those large POPs. I’ve heard time and time again that someone buying service from a major network in say New Orleans has a first hop of Dallas or Atlanta. I agree that full-route capable routers need to be in the large, central locations, but it isn’t cost effective to have them at every POP, especially if you’re a last-mile provider.
I’d go into more examples of where it doesn’t make sense to have full-route routers everywhere, but I’m afraid that the Internet would then focus on the examples instead of the core idea of intelligently putting routes into the FIBs of low-FIB routers throughout my network.
“The right tool for the job” gets into a religious argument in assuming that one’s way to do the job is the only reasonable way to do the job
I disagree that it’s religious. I completely agree there are locations in networks that having full DFZ capable routers doesn’t make technical or economic sense. But there have long been different products for those different use cases.
To perhaps explain my viewpoint better,(and perhaps I didn’t properly comprehend the problem you’re aiming to solve) :
If you are trying to use SDN stuff to shuffle routes on and off a box because you have the wrong sized routers in place, then I would argue you’re doing it wrong.
If you are trying to use SDN stuff to (as Christopher mentioned) make decisions that are not strictly LPM, and the equipment you have cannot do that, then that’s different and entirely reasonable.
If the second use case is more of what you were asking, then I apologize for misunderstanding.
Then please bless the world with the right way.
You acknowledge that not every router in a network needs to be fully DFZ capable, but then crap on my desire to have more than a default route in one.
Initially, my thought was to use community filtering to push just IXes, customers, and defaults throughout the network, but that’s obviously still sub-optimal.
I’d be surprised if a last mile network had a ton of traffic going to any more than a few hundred prefixes.
I think in a low-fib box at the edge of your network your choices are:
“the easy choice, get default, follow that”
“send some limited set of prefixes to the device, and default, so you MAY choose better for the initial hop away”
you certainly can do the second with communities, or route-filters (prefix-list) on the senders, or…
you can choose what prefixes make the cut (get the community(ies)) based on traffic volumes or expected destination locality:
“do not go east to go west!”
these things will introduce toil and SOME suboptimal routing in some instances… perhaps it’s better than per flow choosing left/right though and the support calls related to that choice.
In your NOLA / DFW / ATL example it’s totally possible that the networks in question do something like:
“low fib box in tier-2 city (NOLA), dfz capable/core devices in tier-1 city (DFW/ATL), and send default from left/right to NOLA”
Could they send more prefixes than default? sure… do they want to deal with the toil that induces? (probably not says your example).
SDN isn’t really an answer to this, though… I don’t think. Unless you envision that to lower the toil ?
I’m not concerned with which technology or buzzword gets the job done, only that the job is done.
Communities certainly work. I could tag each of my peers (not a bad practice in the first place) with a different community, or set of communities and only allow say non-route server peers and customers to put routes into the limited-FIB boxes. However, how many routes can a given router hold in its FIB? 10K? 100k? 87,500? I’d have to make some manual choices about what gets included and what doesn’t. That’s not that big of a deal.
I’m getting almost 11k prefixes from my Amazon, Cloudflare, and Google peers. How many of those 11k do I have significant bits to? 1k at most? Less than 10% of the routes I’m putting in via communities are ones I actually care about. I get 142k from my HE peer (oddly, I just noticed my HE IPV6 peer is down, so time to remedy that. See, something good does come from arguing on the Internet!). That would be at or exceeding the FIB size of many routers, yet to what gain? I of course do have to make an administration somewhere to decide what gets included and what doesn’t. Of course whatever is done manually will result in sub-optimal routing. I’ll be concerning myself with useless prefixes and ignoring ones from transit (excluded from this whole thing) altogether.
Does something exist to make more intelligent choices than I can? Yes, at least two of them are in the public domain. I’ve had offlist responses about others home-brewing their own, similar solutions. Are they reliable? I don’t know. Are they worth it? I’d assume so, but that depends greatly on if they’re reliable.
Looking briefly at the couple of things out there, they’re evaluating the top X prefixes in terms of traffic reported by s-flow, where X is the number I define, and those get pushed into the FIB. One recalculates every hour, one does so more quickly. How much is appropriate? I’m not sure. I can’t imagine it would NEED to be done all of that often, given the traffic/prefix density an eyeball network will have. Default routes carry the rest. Default routes could be handled outside of this process, such that if this process fails, you just get some sub-optimal routing until repaired. Maybe it doesn’t filter properly and sends a bunch of routes. Then just have a prefix limit set on the box. Maybe it sends the wrong prefixes. No harm, no foul. If you’re routing sub-optimally internally, when it does hit a real router with a full FIB, it gets handled appropriately.
I would just be looking for solutions that influence what’s in the FIB and let the rest of the router work as the rest of the router would.
Lots of 1M tcam fib limits in older gear.......
So yeah, its the same problem, bigger numbers and still not solved in any sort of non-painful or expensive way.
I think Ill explore the google path and paper on it again.
Joe
Mike Hammett wrote:
Very true.
“big enough” equipment from not that long ago couldn’t carry a full table today (or tomorrow).
Mike Hammett wrote:
I'm not concerned with which technology or buzzword gets the job done, only that the job is done.
Looking briefly at the couple of things out there, they're evaluating the top X prefixes in terms of traffic reported by s-flow, where X is the number I define, and those get pushed into the FIB. One recalculates every hour, one does so more quickly. How much is appropriate? I'm not sure. I can't imagine it would *NEED* to be done all of that often, given the traffic/prefix density an eyeball network will have. Default routes carry the rest. Default routes could be handled outside of this process, such that if this process fails, you just get some sub-optimal routing until repaired. Maybe it doesn't filter properly and sends a bunch of routes. Then just have a prefix limit set on the box. Maybe it sends the wrong prefixes. No harm, no foul. If you're routing sub-optimally internally, when it does hit a real router with a full FIB, it gets handled appropriately.
Unless it loops.
The rest sounds nice. But flow caching got a bad rap back in the early worm days. But thats because the situation was a little worse back then. Cache the wrong routes or run out of cache, router dies. So long as thats not the case automating optimization is an extremely valuable goal.
I would just be looking for solutions that influence what's in the FIB and let the rest of the router work as the rest of the router would.
The problem comes when the router wont work at all without the FIB routes, like in the olden days.
I guess I wasn’t around for those days.
As far as running out, again, assuming the tooling works correctly, I’d think to target fewer routes than you could hold. Maybe 1k routes is all one would need to get a significant percent of the traffic. A lot of room to mess up if you can hold 100k, 500k routes.
Mike,
I’m not sure I understand what you mean by “suboptimal“ routing. Even though the Internet uses AS path length for routing, many of those path lengths are bogus, and don’t really represent any kind of path performance value. For example, a single AS might hide many hops in an MPLS network as a single hop, obscuring asymmetric routing and other uglies. Prepending also occurs when destinations are trying to enforce their own engineering policies, which often conflict with yours or mine.
So what do you mean by “suboptimal“? Are you thinking that the “best” path in BGP tables actually meant you were getting a performance benefit? Because that’s definitely not the case in today’s Internet. Were were you thinking that you would be going along less congested paths? That’s really at the mercy of the traffic engineering of backbone providers over which we have no control.
I generally populate local router FIBs to merel choose an exit point for purposes of load balancing, and nothing more.
-mel
I hesitated to get too specific in examples because someone is going to drag the conversation into the weeds.
Let’s take the the Dallas - New Orleans - Atlanta example where I have a connection from New Orleans to Dallas and a connection from New Orleans to Atlanta.
Let’s say I peer with Netflix in both markets. Netflix chooses to serve me out of Atlanta, for whatever reason. Say my default route sends my traffic to Dallas. That’s not where Netflix wanted it, so now I have to go from Dallas to Atlanta, whether that’s my circuit or across the public Internet. Potentially, it’s on MPLS and it rides back through the New Orleans router to get back to Atlanta. That’s a long trip when I already had a better path, the less-than-full-fib router just didn’t know about it. Given that Netflix is a sizable amount of traffic in an eyeball ISP, that’s a lot of traffic to be going the wrong way. If the website for Viktor’s Arctic Plunge in Siberia was hosted in Atlanta, I wouldn’t give two craps that the traffic went the wrong way because A), I’ll probably never go there and B) when someone does, it won’t be meaningfully enough traffic to accommodate.
Someone’s going to tell me to put a full-table router in New Orleans. Maybe I should. Okay, so maybe I have a POP in Ashford, Alabama. It has transport to New Orleans and Atlanta. There aren’t enough grains of sugar in Ashford, Alabama to justify a current-generation, full table router. Now I’m even closer to Atlanta, but default may point to New Orleans.