Best way to deal with bad advertisements?

  Hi Matt,

> > In this case, the very first thing you should probably do is to
> > start announcing the more specific /24s to match their advertisements!
> > Depending on AS-PATH length (how various nets hear your announcements
> > vs. theirs) this may solve the immediate problem, allowing you to hunt
> > them down and kill them at your leisure.
>
> The downside to this is that we go from advertising /16's
> out, to advertising a fleet of /24's out, most of which
> would be filtered by Sprint's ever-lovin' CIDR-forcing
> wall.

  If your more specific networks are filtered, then wouldn't the
  evil ISP's be filtered as well?

  This would be a large problem only if you gain transit from Sprint....

Bingo. We buy transit from Sprint.

> I agree with Sprint, and Sean, but in this case
> it pretty much makes it hard for us to force the issue
> by dropping to the same or smaller sized announcement.

  Well, I'm not sure that the two entities can be put in the
  same sentence any more, but you can always leave the less specific
  /16 in there while you attempt to advertise the more speciic.

Good point; I'd forgotten you can have both advertised, and
those that hear the /16, and no /24's will honor it, those
that hear the /24's get to worry about the weights independantly
of the /16's advertisement.

> Good thought, though! Even if it does result in going
> from 2 /16 announcements to 512 /24 announcements in
> the process, growing the routing tables, and generally
> making everyone else unhappy as well.

  I'd rather have happy workable customers and an unhappy
  community in the short run, than unhappy unworkable customers and
  a happy community.

Agreed; if this were something I felt would be resolved within
24 hours, it wouldn't even really be worth mentioning. When
it starts moving on 48 hours, I start worrying about whether
or not we're going to start showing up in the Top 50 lists. :slight_smile:

  I think your letter will raise the awareness of this kind of
  problem. Of course we all know it's possible, but it's not a
  problem that we've had to deal with on a malicious level.

? I do assume that there's no doubt the evil-isp is doing this
  maliciously?

This is the third time they've done this. The first two times
we chalked up to ignorance and stupidity.

This time, though, we're not as willing to give them
the benefit of the doubt.

> *sigh* There really MUST be some nice way of handling
> lame ISP's like this.

  One thing you could do is coordinate with largerish ISPs to filter
  the incorect network from the affected peering sessions. While
  this is a stopgap fix, and not one to be repeated, I don't think
  you'd have problems getting it done w/ MCI, UUnet, AGIS, etc...

> > 1) Announce *your own* routes more specifically.
> > This may lose you ANS connectivity, though.
>
> And Sprint, and anyone else that filters small specifics.

  Again, not if you leave the /23 or /16s in place... Then you
  just revert to the pre-action situation. It is again important to
  note that if your announcement would be blocked by mask length
  policies, then the evil-isp would be as well.

Not since we connect through sprint, and they don't. :frowning:

> > 2) Announce *their* routes more specifically.

  Ouch, that's playing as dirty as them. Can't recommend it unless
  it's life or death...

I know. Tempting though it is...

> I took that step last night, and was advised to remove it by
> those more in tune with legal issues. I guess it's not
> considered "nice" to sink to the same level as your
> attacker, and play dirty. :-}

  Aiyeee.

In thinking about it, I realized I didn't want to be the
one listed as having escalated the cold war prefix race
resulting in a flood of GIFS illustrating the death of
the net. :slight_smile:

> > 3) You can post to NANOG and other lists in an attempt to embarrass/
> > get someone who knows the jokers to poke them.

  I recommend this, show traceroutes, RR entries, InterNIC assignments,
  routing table dumps, and state the problem clearly. You can bet
  the appropriate folks will poke them.

We did this last time, in complaining to MCI, their upstream
provider, and MCI responded in record time, putting in a
temporary filter for those blocks in less than 36 hours.

That helped for about 30 seconds, before we found that they
then announced the same blocks through a second connection
which hadn't shown up as a path previously when we did
a 'show ip bgp 205.158.193.0 255.255.255.0 l'

With the advent of more and more multihomed networks, there's
more and more paths that need to be filtered to stop an
invalid announcement like this; perhaps the concept of a
routing database that can only be updated by authoratative
contacts isn't that unreasonable anymore. Gone are the
days when you can just trust that everyone else will
"do the right thing" in maintaining the well-being of the
net. Just as the Guardian project came to the InterNIC,
perhaps a similar check-and-verify proceedure needs to
be put in place before new routing announcements are
added or believed.

I miss the older, more democratic days of the net, but it
seems the overall level of knowledge and skill is dropping,
forcing more and more levels of checks and balances to
prevent abuse either through stupidity and ignorance, or
malicious intent.

  In summary, whatever they do to hit you, do for yourself in
  self-defense. Don't advertise their networks, just advertise your
  networks as specifically as needed. Continue to raise the ante by
  involving more appropriate folks, and provide specific
  documentation to those involved of what happened when. It sounds
  like a war to me. Try to find middle ground with them, there must
  be SOME reason they are after your cidr space. Perhaps you can
  negotiate a fix?

  I doubt it will be to long before a standard as-path list looks
  like this:

  ....
  ip as-path 10 deny EVIL-ISP1-AS
  ip as-path 10 deny EVIL-ISP2-AS
  ip as-path 10 deny EVIL-ISP3-AS
  ....

  In this age of global routing, with no central body, politicking
  and negotiation are your best tools for solution. There's no
  overseeing body to go to. You can gain allies, but it's up to
  you. Good luck, and count most folks here as allies.

  $.02

  -alan

Thanks! It's a new ballgame for geeks and engineers more
at home in a telco closet hunched over a laptop cabled into
a console port; I'm not as familiar with politics and the
subtle tactics of diplomacy, negotiation, and power brokering.

I can see it's time I started learning some new tricks, and
prepared for the new way of doing business on the net.

Thanks again for the support. I'm sorry to see that the
days of simple trust may be coming to a close, but we've
all invested money into this creation, and we can't really
afford to let others bring it down, whether by SYN attacks,
domain name theivery, or IP theivery.

Matt

> I think your letter will raise the awareness of this kind of
> problem. Of course we all know it's possible, but it's not a
> problem that we've had to deal with on a malicious level.
>
> ? I do assume that there's no doubt the evil-isp is doing this
> maliciously?

This is the third time they've done this. The first two times
we chalked up to ignorance and stupidity.

This time, though, we're not as willing to give them
the benefit of the doubt.

I don't believe you. If you were as confident as you say you are that this
is an evil ISP you would have just said:

    Evilnet Inc. is blackholing my routes. I've sent mail to
    techie@evilnet.inc, bigboss@evilnet.inc and chambermaid@evilnet.inc
    and nobody returns my mail. I phoned them at 1-888-555-2222
    and left voicemail, I faxed them at 1-888-555-1111 and I don't get
    any response.

In this way you accomplish the following:

1) clear identification of the problem, i.e. blackholed routes
  
2) clear identification of who seems to be causing the problem

3) clear identification of the contact means that you tried and the
   results or lack thereof obtained.

As a result, somebody who happens to know that Joe Bloe is the techie
at EvilNet can call Joe at home and say, "Hey Joe, did you know that
so-and-so doesn't like what you are doing and can't get a hold of you by
email or telephone. Maybe you better fix this...". Or it could be
Evilnet's upstream who contacts them. Or somebody could email you
Evilnet's secret "human" NOC phone number, or whatever.

We did this last time, in complaining to MCI, their upstream
provider, and MCI responded in record time, putting in a
temporary filter for those blocks in less than 36 hours.

That helped for about 30 seconds, before we found that they
then announced the same blocks through a second connection
which hadn't shown up as a path previously when we did
a 'show ip bgp 205.158.193.0 255.255.255.0 l'

Trying to solve a social problem with technology often results in this
kind of thing.

I miss the older, more democratic days of the net, but it
seems the overall level of knowledge and skill is dropping,
forcing more and more levels of checks and balances to
prevent abuse either through stupidity and ignorance, or
malicious intent.

I think you are jumping to conclusions here by assuming it is due to
stupidity, ignorance or malicious intent. I strongly suspect that it is
due to lack of information and work overload. Lack of information is
subtly but significantly different from stupidity and ignorance and you
yourself are contributing to Evilnet's lack of information by withholding
important information about the problem.

Shine the light of day on the problem and it will soon clear up. Throw all
the relevant information into the "public" NANOG mailing list pool and
numerous avenues for action will open up.

Michael Dillon - ISP & Internet Consulting
Memra Software Inc. - Fax: +1-604-546-3049
http://www.memra.com - E-mail: michael@memra.com