operational: icmp echo out of control?

Mailman List Mirror (Read Only)
1

Hello,

I've observed that our border routers are getting pinged a fair bit.
I measured on one router and saw:

  5 per second, seems consistent throughout the day,
  roughly 40 different sources every 15 seconds

I took a look at the varied sources and discovered that the sites
are well connected and those that resolve resolve to akamai names.

This isn't more than a nuisance for me, but I run a small net.
Should I conclude that an ISP with a population 10 times bigger
would have their border routers getting pinged at 10 times the
rate I see? If so, should we care, or just ignore it?

Thanks,
-mark

2

I can't speak for the specific rate at which our devices get pinged,
but the issue doesn't really worry me.

On Juniper (and probably on Cisco, but I can't speak for it) you can
apply a policer to ICMP on the loopback interface and it will take
effect for traffic addressed to any interface on the router. So if
it gets to be out of control, some will start to get dropped. No
need to watch over it and worry about adding filters and such.

-c

3

Congratulations!

You've just discovered akamai load balancing.

Block it if you want your customers to get akamai content from the less
optimal source. ;-}

-bryan bradsby

NOC: 512-475-2432
Texas State Government Net

4

I can't speak as to what exactly Akamai is doing, but this kind of probing
for "performance" reasons is becoming increasingly common as more people
jump on the "optimized routing" bandwagon.

Not only do you have operational networks originating these probes on
their own (InterNAP, Digital Island, Akamai, others), but you now have
companies making boxes which "optimize routing" in part by doing these
probes from every one of their customers.

Right now it's mostly noise, but it has the potential to get way out of
hand. A packet or two an hour probably wouldn't hurt anyone, but 5
packets/sec is personally what I would consider to be an acceptable amount
of data to be directed at any specific host or router. Not only can this
many probes trigger ICMP rate limiting and ruin the data for the prober
and others, it is just plain unnecessary.

Path latency doesn't change much, you can determine this with very few
probes. Reachability does not need to be continuously probed, you can take
cues from other data to decide if you need to re-probe. Packet loss cannot
be reliably determined without a lot more packets than it is reasonable to
send.

Much like web spidering, some simple common sense can help keep probes
from becoming a hassle:

* Control the rate of your probes to a given destination.
* Don't allow your probes to continuously hit a destination.
* If you are using traceroute-style probes, extra care must be taken
   as if you were pinging every host along the path.
* If at all possible, only target destinations you actually exchange
   traffic with. For example, get a netflow feed.
* Make sure a DoS attack cannot provoke your system into probing innocent
   third parties.
* Consider what is the smallest unit of "distinct network topology" you
   need to map. A very reasonable number would be a /24.
* Source your probes from an IP which resolves to something that can
   explain what your probe is doing, and a webpage for people to read
   more about what you are doing and why (such as how it benefits them).
* Have an "opt out" option for networks who REALLY don't like probes.

5

Date: Thu, 23 May 2002 16:36:23 -0400
From: Richard A Steenbergen

[ moderate snipping throughout ]

I can't speak as to what exactly Akamai is doing, but this
kind of probing for "performance" reasons is becoming
increasingly common as more people jump on the "optimized
routing" bandwagon.

Perhaps most maddening is that ICMP echo/response hardly reflects
real-world performance. (At least I don't usually tunnel my
HTTP, SMTP, and FTP packets through ICMP, but perhaps I'm just
being weird again.)

Not only do you have operational networks originating these
probes on their own (InterNAP, Digital Island, Akamai,
others), but you now have companies making boxes which
"optimize routing" in part by doing these probes from every
one of their customers.

I'd hope that they're having the IP stack communicate timing info
to the apps. The information is superior, and it doesn't require
any additional packets.

Path latency doesn't change much, you can determine this
with very few probes. Reachability does not need to be
continuously probed, you can take cues from other data to
decide if you need to re-probe. Packet loss cannot be
reliably determined without a lot more packets than it is
reasonable to send.

Much like web spidering, some simple common sense can help
keep probes from becoming a hassle:

Hmmmm.... anyone recall the number of the RFC that says many of
the same things?

* If at all possible, only target destinations you actually
   exchange traffic with. For example, get a netflow feed.

Which, IMHO, is the sane way anyhow. Why spend bandwidth and CPU
munching on a point that exchanges 0.00000001% of one's traffic?
It's silly. Now, if it's an outlier that shows performance worse
than, say, 3 sigma slower than average, that might be another
story.

6

I can't speak as to what exactly Akamai is doing, but this

I should add that Akamai contacted me with minutes of my initial
post to ask for more data and they said that they are looking
into it... leaving me with the impression that what I was seeing
was not typical.

-mark

7

Its important to note a point entioned here that vendors are building
boxes to do this as well. I ran a 3dns pair for a while and wow the
mail that came in from people with firewalls or simply watching for
probes. F5 was opening all sorts of half opened connections and wierd
ports other than ones involving dns. I believed they called it iquerry.

8

Its important to note a point entioned here that vendors are building
boxes to do this as well. I ran a 3dns pair for a while and wow the
mail that came in from people with firewalls or simply watching for
probes. F5 was opening all sorts of half opened connections and wierd
ports other than ones involving dns. I believed they called it iquerry.

draft-ietf-dnsext-obsolete-iquery-03.txt

randy

9

The problem here is that other types of probes raise IDS alarms on way too many
networks - the next-best method is to probe HTTP ports, but we don't want to
have to pull down thousands of web pages just to get performance stats. So,
they send a SYN, wait for the ACK, record the latency and send a FIN.
Sounds benign, but you'd be surprised how klaxons go off in response to this.

-C

10

Chris Woodfield wrote:
>...the next-best method is to probe HTTP ports, but we don't want to

have to pull down thousands of web pages just to get performance stats.

Why not just passively measure the time it takes to send actual
traffic to actual clients? It shouldn't take too much talent
to use the aggregate data for an evolving best guess for each client request.

KL

11

So what, someone sneezes on an ethernet cable and IDS alarms go off. :slight_smile:

Theoretically, ICMP Echo should be less intrusive for performance
measuring since it is clearly only for this purpose, whereas doing an
actual TCP handshake could easily be mistaken for a port scan. But for so
many network admins, all they know is "ICMP bad".

Besides, if you thought ICMP was inaccurate, a TCP handshake is far worse.
First, you have no way of knowing if the TCP replies are the originals, or
if they have been retransmitted, so any single sample is pretty useless
for measuring reply time. Second, for most cases with TCP opens, you are
measuring the interaction time of the application process, not response
time from the kernel. Depending on how the application is written, and
what event model it uses, it is very likely to be doing other things
before and after it gets notified that there is a new connection, then
does the accept().

Internet path measurement isn't simple, you're probably best off trying to
get as much data as possible passively. :slight_smile:

12

Date: Tue, 28 May 2002 14:43:25 -0400
From: Richard A Steenbergen

Theoretically, ICMP Echo should be less intrusive for
performance measuring since it is clearly only for this
purpose, whereas doing an actual TCP handshake could easily

And less accurate for asymmetric traffic, under-representing the
return flow.

be mistaken for a port scan. But for so many network admins,
all they know is "ICMP bad".

That'll be the day when someone calls abuse saying "I'm being
attacked by ICMP unreachables!" :wink:

Besides, if you thought ICMP was inaccurate, a TCP handshake
is far worse. First, you have no way of knowing if the TCP
replies are the originals, or if they have been
retransmitted, so any single sample is pretty useless for
measuring reply time. Second, for most cases with TCP opens,
you are measuring the interaction time of the application
process, not response time from the kernel. Depending on how

Ah, yes. Reminds me of the "benchmark" where a company that
writes mailserver software tested HELO response time. Gee, my
MXen spend more time actually sending and receiving messages!

the application is written, and what event model it uses, it
is very likely to be doing other things before and after it
gets notified that there is a new connection, then does the
accept().

If it gets that far. I suspect that accept filters would give
artificially fast results.

Then there are issues such as delayed ACK. And if one is
measuring throughput, does the session ever make it past slow
start? And just how many frames do they send during slow start?
(Tunable under FreeBSD...)

Once ECN becomes more common, does one consider those points
outliers or useful data?

Internet path measurement isn't simple, you're probably best
off trying to get as much data as possible passively. :slight_smile:

Less intrusive, and chances are that anything too small for
statistical validity is insignificant, anyway.

13

Also sprach E.B. Dreger

be mistaken for a port scan. But for so many network admins,
all they know is "ICMP bad".

That'll be the day when someone calls abuse saying "I'm being attacked
by ICMP unreachables!" :wink:

"That'll be..."? Future tense? Hrmm...

14

Here is a sample posting from the "future"... It was much worse when my email address was the ARIN contact and I would get people screaming at me asking why I was hacking their machine :frowning:

---------begin annoying auto-robot-program----------
The attempt was detected by the personal firewall running on my machine,
and I am quite concerned about it.

If you are in fact responsible for this network, please do the following:

1) Research the access attempt(s),
2) Inform the responsible parties to discontinue access attempts,
3) Reply to me with your findings.

If you are not responsible for this network, please forward this message to
the person who is, or, if you do not know who this person is, please get
back to me with that information as well. Thank you.

The access attempt(s) are shown below, including the date and time, port
number, TCP or UDP indicator, and, if known, a service name associated with
the port.

     Jeu 09 mai 2002 15:30:22, Port 3, ICMP, Destination Unreachable
     Jeu 09 mai 2002 15:30:21, Port 3, ICMP, Destination Unreachable
     Jeu 09 mai 2002 15:30:10, Port 3, ICMP, Destination Unreachable
     Jeu 09 mai 2002 15:30:09, Port 3, ICMP, Destination Unreachable

15

I don't know whats worse, those crappy personal firewalls that make every
packet look like a life or death assault, or the idiots who send abuse
email demanding that you do something for them or they will sue and/or
hax0r you.

I've seen supposed "security professionals" for theoretically clued places
like NASA send abuse complaints over traceroutes they've originated, and
people complain about "port 80 hacking attempts" then flatly refuse to
admit they visited website.

At best, it's annoying clutter. Is it any wonder that legitimate emails
about ongoing DoS attacks are completely ignored or responded to a week
later? At worst, it can get innocent people in trouble and cost them a lot
of time, effort, and potentially money.

These false abuse reports are FAR too common, and the net equivilent of
crying wolf. In my opinion, it is the responsability of these personal
firewall makers to at least make an EFFORT to warn their users about this.
So far, I havn't seen it.

16

It's common enough that it's got it's own acronym. IWF - Idiot With Firewall.

17

We call them OZZADs and here is how we respond:

http://condor.depaul.edu/~jkristof/technotes/incident-response.html

John

18

[deleted]

The access attempt(s) are shown below, including the date and time, port
number, TCP or UDP indicator, and, if known, a service name associated with
the port.

     Jeu 09 mai 2002 15:30:22, Port 3, ICMP, Destination Unreachable
     Jeu 09 mai 2002 15:30:21, Port 3, ICMP, Destination Unreachable
     Jeu 09 mai 2002 15:30:10, Port 3, ICMP, Destination Unreachable
     Jeu 09 mai 2002 15:30:09, Port 3, ICMP, Destination Unreachable

19

> We call them OZZADs and here is how we respond:

Hmm.. 3 people have asked already "What's an OZZAD?" :wink:

So I don't have to keep answering this, forwarded to the group:

Over Zealous Zone Alarm Dork

John

20

[ On Tuesday, May 28, 2002 at 19:17:32 (+0000), E.B. Dreger wrote: ]

Subject: Re: operational: icmp echo out of control?

> be mistaken for a port scan. But for so many network admins,
> all they know is "ICMP bad".

That'll be the day when someone calls abuse saying "I'm being
attacked by ICMP unreachables!" :wink:

You mean that hasn't happened to you yet? Where have you been? :slight_smile: