I am inheriting a WISP network with Mikrotik equipment throughout. One of
my first duties is to make the network multihomed. We have our first
internet connection at one location and our second internet connection will
be delivered at a second location in a week or so.
I understand all of the steps I need to go through with ARIN in terms of
getting an ASN and so forth.
My question is about BGP on the Mikrotik platform. The guy who I am
supplanting swears that we are supposed to be bringing the second internet
link to the same place as the first internet link for BGP to work properly.
Obviously that is not true with major brand routers which would do the BGP
job just fine. (And he's the same guy that has bridged this whole network,
so it is easy to disbelieve his opinion.) But maybe he knows that Mikrotik
can't perform BGP in the same way that other routers can.
So here's the question. Is there something about running BGP on a Mikrotik
platform that precludes having the internet connections come in at different
locations?
Sincerely,
Lorell Hathcock
OfficeConnect.net | 832-665-3400 x101 (o) | 832-782-4656 (c)
713-992-2343 (f) | lorell@officeconnect.net
Texas State Security Contractor License | ONSSI Certified Channel Partner
Axis Communications Channel Partner | BICSI Corporate Member
Leviton Authorized Installer
job just fine. (And he's the same guy that has bridged this whole network,
so it is easy to disbelieve his opinion.)
ew. nasty.
So here's the question. Is there something about running BGP on a Mikrotik
platform that precludes having the internet connections come in at different
locations?
I will refrain from making any smart-ass comments about Mikrotik and BGP,
but no: there is no reason whatever that you can't take your internet feeds
from different locations, so long as you have a good quality interior
network link between those two locations, and your two routers talk iBGP to
each other. Just make sure your boxes have enough RAM to cope with a full
dfz feed.
I.e. it's just the same as using any other router in this regard.
Nick
I've used Mikrotiks for everything except BGP, but we don't use Mikrotiks for BGP only because we already had BGP on a different platform...personally, when it comes to BGP, I think people are better off running it on devices they are familiar with rather then trying to learn the idiosyncrasies of a new platform.
Bret
note that you do NOT have to have a full feed on either location, if
your goal is simply primary/backup links... getting default from both
providers and sending your prefixes out to both (potentially
preferring one with an intentionally longer aspath, or other normal
tricks/config) will accomplish primary/backup just fine.
Don't use a sledghammer when a push pin works.
-chris
I am inheriting a WISP network with Mikrotik equipment throughout. One of
my first duties is to make the network multihomed. We have our first
internet connection at one location and our second internet connection will
be delivered at a second location in a week or so.
[snip]
My question is about BGP on the Mikrotik platform. The guy who I am
supplanting swears that we are supposed to be bringing the second internet
link to the same place as the first internet link for BGP to work properly.
Obviously that is not true with major brand routers
And it is not true with Mikrotik either... I work for a WISP that uses
Mikrotik almost exclusively, everything from our core to customer CPEs. We
have multiple Mikrotik edge routers at diverse locations, with 200+Mbs
internet connections thru different providers, all running full BGP feeds,
and all sharing those feeds between each other. A simple 1U box with a good
MB, 1-2GB RAM, flash drive for booting, and good multi-port Gb ethernet cards
for each is all that is needed.
We are a small ISP by most standards, but we have had no problem running
180Mbs and 40,000pps in/out on just one of our edges, while carrying on with
multiple BGP feeds and exchange between our internal routers.
Adrian
While Mikrotik's BGP implementation isn't very sofisticated, there is no
reason, why you can't have your feeds in different places. As Nick
outlined, you need to set iBGP up between the boxes.
I'm running myself a ISP on mainly Mikrotik basis (basestations and
clients, approx 2500 users) and I've been extensively testing Mikrotik's
BGP stack in the last 4 years (from 2.9 and up).
Mikrotik wrote the whole routing stack from scratch in 3.x, which
resultet in tons of problems and bugs. In my opinion, it still isn't
where it should be. Don't get me wrong, but there are several pitfalls.
- Mikrotik still has some memory leaks in the BGP stack somewhere,
causing funny issues at times.
- Filters aren't adequate for my use, and lacking a lot on IPv4, but
even more on IPv4.
First of all, you will need at least a RB1000, RB1100 or a PC based
Mikrotik router to get enough ram, to accomodate one full-table or more.
Anything less and you can forget it.
I'm running a mix of Quagga boxes, Cisco and recently Juniper instead
for BGP. For our internal routing OSPF on Mikrotik definatly does the job.
Just my 2c.
Kind regards,
Martin List-Petersen
Dear Lorell,
My question is about BGP on the Mikrotik platform. The guy who I am
supplanting swears that we are supposed to be bringing the second internet
link to the same place as the first internet link for BGP to work properly.
Obviously that is not true with major brand routers which would do the BGP
job just fine. (And he's the same guy that has bridged this whole network,
so it is easy to disbelieve his opinion.) But maybe he knows that Mikrotik
can't perform BGP in the same way that other routers can.
So here's the question. Is there something about running BGP on a Mikrotik
platform that precludes having the internet connections come in at different
locations?
That depends on the netwoek in between this two locations.
There could be a lot of good reasons why this is no good idea; please bring some light into this.
Kind regards,
Ingo Flaschberger
We are putting a private PTP metro ethernet (fiber based) link between the
two locations. And both locations will have one internet connection.
I am reading that Mikrotik has a memory leak in its BGP implementation. Any
more info about this?
Sincerely,
Lorell Hathcock
OfficeConnect.net | 832-665-3400 x101 (o) | 832-782-4656 (c)
713-992-2343 (f) | lorell@officeconnect.net
Texas State Security Contractor License | ONSSI Certified Channel Partner
Axis Communications Channel Partner | BICSI Corporate Member
Leviton Authorized Installer
Dear Lorell,
We are putting a private PTP metro ethernet (fiber based) link between the
two locations. And both locations will have one internet connection.
this network between should be no problem,
what routing protocols do you use in your network? ospf?
Kind regards,
Ingo Flaschberger
We will implement OSPF.
Sincerely,
Lorell Hathcock
OfficeConnect.net | 832-665-3400 x101 (o) | 832-782-4656 (c)
713-992-2343 (f) | lorell@officeconnect.net
Texas State Security Contractor License | ONSSI Certified Channel Partner
Axis Communications Channel Partner | BICSI Corporate Member
Leviton Authorized Installer
I haven't seen either of those issues running the v4.x stream of RouterOS. The memory leak was solved a while ago and Mikrotik has fairly short release cycles.
We have extensive inbound and outbound filters on our eBGP doing most of the normal things that you would do on a cisco. The IPv6 filters must be built via the terminal to avoid limitations with the current GUI but they also work very well
Dear Lorell,
We will implement OSPF.
so what arguments speak against 2 bgp upstreams?
Kind regards,
Ingo Flaschberger
Dear Lorell,
We will implement OSPF.
so what arguments speak against 2 bgp upstreams?
It's not an either or proposition...
ospf carries your internal routes, ibgp carries you external routes between internal routers. you can carry default around in either in fact you probably should since routers that don't need a nuanced view of the outside world don't need to carry such a big table.
Well, I believe the original poster said that one of his colleagues
swore that BGP multihoming wouldn't work unless both feeds terminated on
the same router. I suppose said colleague has never heard of iBGP
between two routers of the local AS. Those two routers should probably
take a full table and exchange them between the two but going inside the
network, yeah, they should probably simply originate a default into the
the ospf routing. But I am making some assumptions here. I am assuming
the two routers have connectivity between them sufficient to handle the
required traffic in case one of the upstreams fails (backhaul bandwidth
is at least equal to upstream bandwidth). Maybe the colleague knew that
the links between the sites were insufficient and that is why both links
were desired on the same physical unit or something. It is impossible
to sort out other people's networking from short blurbs on a mailing
list.
George
* George Bonser:
Well, I believe the original poster said that one of his colleagues
swore that BGP multihoming wouldn't work unless both feeds terminated on
the same router. I suppose said colleague has never heard of iBGP
between two routers of the local AS. Those two routers should probably
take a full table and exchange them between the two but going inside the
network, yeah, they should probably simply originate a default into the
the ospf routing.
Does this really work that well? Won't you still get loops or
blackholes unless the eBGP routes on all border routers are identical?
I think you also need iBGP speakers along all feasible paths between
eBGP speakers.
None in my mind.
The legacy network operator was unfamiliar with actual best practice
enterprise/carrier networking policies that he thought that for BGP to work
on a two internet feed network, both internet connections have to be
delivered to the same location. I thought since he has more insight into
Mikrotik, that he knew about a bug with Mikrotik that made the argument
true. Feedback from NANOG list members that also run Mikrotik has proven
that there is no problem with running current rev levels of the Mikrotik
RouterOS and BGP with internet feeds at two different locations.
Sincerely,
Lorell Hathcock
OfficeConnect.net | 832-665-3400 x101 (o) | 832-782-4656 (c)
713-992-2343 (f) | lorell@officeconnect.net
Texas State Security Contractor License | ONSSI Certified Channel Partner
Axis Communications Channel Partner | BICSI Corporate Member
Leviton Authorized Installer
From: Florian Weimer
Sent: Monday, May 24, 2010 2:35 AM
To: George Bonser
Cc: joel jaeggli; Ingo Flaschberger; nanog@nanog.org
Subject: Re: Mikrotik BGP Question
* George Bonser:
Does this really work that well? Won't you still get loops or
blackholes unless the eBGP routes on all border routers are identical?
As opposed to what, injecting the entire BGP table into your igp? That
generally doesn't work well.
I think you also need iBGP speakers along all feasible paths between
eBGP speakers.
I was assuming the eBGP speakers were directly connected over some sort
of interconnecting backhaul. Again, you can't really figure out what
someone's topology is from a short blurb on a mailing list. Yes, if
there are intervening hops, they will need to speak iBGP as well and
possibly configured as route reflectors if it isn't practical to fully
mesh everything.
Maybe there is a reason the legacy operator said both uplinks must be
connected to the same router. If the two locations are not
interconnected, that would be one reason. I don't believe the original
poster described their internal connectivity.
George
* George Bonser:
Does this really work that well? Won't you still get loops or
blackholes unless the eBGP routes on all border routers are identical?
As opposed to what, injecting the entire BGP table into your igp?
As opposed to just injecting defaults.
Maybe there is a reason the legacy operator said both uplinks must be
connected to the same router. If the two locations are not
interconnected, that would be one reason. I don't believe the original
poster described their internal connectivity.
There was a follow-up that mentioned that there's a direct connection,
so they just have to make the other paths infeasible.
In some ways, I find the MikroTik RouterOS routing filter syntax a little
more powerful than Cisco's route-maps. As routing filters work the same
way as firewall filters, you can group rules in "chains" and reuse parts
of your filters in other filters by jumping to another chain. This could
be used, for instance, on a peering setup, where you have a number of
rules per peer but also some common filtering for all peers, or to handle
specific and generic filtering for your customers.
I haven't yet found anything that I missed being able to with filters, at
least with BGP. With other routing protocols, it's another story.
Regards,
Allan Eising