I just took a closer look at something odd I'd noticed several days ago. One of our DNS servers was sending crazy amounts of ARP requests for IPs in the /24 its main IP is in. What I've found is we're getting hit with DNS requests that look like they're from "typical internet traffic for someone in China" hitting this DNS server from IPs in its /24 which are currently not in use (at least on our local network). It would appear someone in China is using our IP space, presumably behind a NAT router, and they're leaking some traffic non-NAT'd.
20:53:41.361734 IP 209.208.121.66.41755 > 209.208.121.126.53: 15939+ A? ns5.z.lxdns.com. (33)
20:53:43.523210 IP 209.208.121.95.39393 > 209.208.121.126.53: 15939+ A? www.nanhutravel.com. (37)
20:53:48.411805 IP 209.208.121.66.33390 > 209.208.121.126.53: 15939+ A? test.csxm.cdn20.com. (37)
20:53:50.557680 IP 209.208.121.135.40056 > 209.208.121.126.53: 15939+ A? rextest2.lxdns.com. (36)
20:53:56.918993 IP 209.208.121.135.37291 > 209.208.121.126.53: 15939+ A? www.51seer.com. (32)
20:54:20.033902 IP 209.208.121.95.37544 > 209.208.121.126.53: 15939+ A? image.dhgate.cdn20.com. (40)
20:54:21.900295 IP 209.208.121.66.35144 > 209.208.121.126.53: 15939+ A? static.xn-app.com. (35)
20:54:27.711853 IP 209.208.121.66.33518 > 209.208.121.126.53: 15939+ A? oa.hanhe.com. (30)
20:54:29.642938 IP 209.208.121.135.41723 > 209.208.121.126.53: 15939+ A? pic1.kaixin001.com. (36)
20:54:32.357414 IP 209.208.121.95.38564 > 209.208.121.126.53: 15939+ A? rr.snyu.com. (29)
20:54:38.901315 IP 209.208.121.95.37840 > 209.208.121.126.53: 15939+ A? edu.163.com. (29)
20:54:39.807385 IP 209.208.121.95.36069 > 209.208.121.126.53: 15939+ A? image.dhgate.cdn20.com. (40)
20:54:40.833778 IP 209.208.121.66.34949 > 209.208.121.126.53: 15939+ A? uphn.snswall.com. (34)
20:54:42.070294 IP 209.208.121.95.38405 > 209.208.121.126.53: 15939+ A? zwgk.cma.gov.cn. (33)
20:54:42.189939 IP 209.208.121.135.36637 > 209.208.121.126.53: 15939+ A? btocdn.52yeyou.com. (36)
20:54:45.767299 IP 209.208.121.95.41405 > 209.208.121.126.53: 15939+ A? img1.kaixin001.com.cn. (39)
20:54:48.595582 IP 209.208.121.66.40099 > 209.208.121.126.53: 15939+ A? rextest2.cdn20.com. (36)
20:54:49.480147 IP 209.208.121.95.42363 > 209.208.121.126.53: 15939+ A? www.dameiren.com. (34)
20:54:50.714200 IP 209.208.121.135.41497 > 209.208.121.126.53: 15939+ A? pic1.kaixin001.com.cn. (39)
20:54:54.116841 IP 209.208.121.135.36828 > 209.208.121.126.53: 15939+ A? i.jstv.com. (28)
I hope they got a good deal on the IP space...and a better deal on their buggy router.
In message <Pine.LNX.4.61.1006162044210.5148@soloth.lewis.org>, Jon Lewis write
s:
I just took a closer look at something odd I'd noticed several days ago.
One of our DNS servers was sending crazy amounts of ARP requests for IPs
in the /24 its main IP is in. What I've found is we're getting hit with
DNS requests that look like they're from "typical internet traffic for
someone in China" hitting this DNS server from IPs in its /24 which are
currently not in use (at least on our local network). It would appear
someone in China is using our IP space, presumably behind a NAT router,
and they're leaking some traffic non-NAT'd.
Why was this traffic hitting your DNS server in the first place? It should
have been rejected by the ingress filters preventing spoofing of the local
network.
We've been seeing the same thing since 2010-06-10:
22:13:19.687981 IP 72.236.167.197.41789 > 72.236.167.138.domain: 38783+ A? jkl.cnr.cn. (28)
22:13:19.773076 IP 72.236.167.124.33327 > 72.236.167.138.domain: 38783+ A? i10.aliimg.com. (32)
22:13:19.855750 IP 72.236.167.169.33381 > 72.236.167.138.domain: 38783+ A? www.vrp3d.com. (31)
22:13:19.941155 IP 72.236.167.200.33005 > 72.236.167.138.domain: 38783+ A? www.51seer.com. (32)
22:13:20.026342 IP 72.236.167.141.36652 > 72.236.167.138.domain: 38783+ A? img1.kaixin001.com.cn. (39)
22:13:20.102540 IP 72.236.167.188.39525 > 72.236.167.138.domain: 38783+ A? pic.kaixin001.com.cn. (38)
22:13:20.204403 IP 72.236.167.103.37838 > 72.236.167.138.domain: 38783+ A? pic.kaixin001.com. (35)
22:13:20.791201 IP 72.236.167.186.38958 > 72.236.167.138.domain: 38783+ A? pic1.kaixin001.com. (36)
22:13:20.876527 IP 72.236.167.121.33000 > 72.236.167.138.domain: 38783+ A? pic1.kaixin001.com.cn. (39)
22:13:20.971393 IP 72.236.167.203.33726 > 72.236.167.138.domain: 38783+ A? logo.kaixin001.com.cn. (39)
22:13:21.051831 IP 72.236.167.120.35298 > 72.236.167.138.domain: 38783+ A? qqtest.cdn20.com. (34)
22:13:21.132215 IP 72.236.167.196.34862 > 72.236.167.138.domain: 38783+ A? upload.elle.cn. (32)
22:13:21.218372 IP 72.236.167.116.35073 > 72.236.167.138.domain: 38783+ A? www.elle.cn. (29)
Spoofed, all with a TTL of 3. Given that all of the domains in question appear to have nameservers in common, I assumed someone was trying to make us participate in a DDoS attack, and started dropping all of the traffic.
When I ran a smaller simpler network, I did have input filters on our transit providers rejecting packets from our IP space. With a larger network, multiple IP blocks, numerous multihomed customers, some of which use IP's we've assigned them, it gets a little more complicated to do.
I could reject at our border, packets sourced from our IP ranges with exceptions for any of the IP blocks we've assigned to multihomed customers. The ACLs wouldn't be that long, or that hard to maintain. Is this common practice?
In message <Pine.LNX.4.61.1006162237180.5148@soloth.lewis.org>, Jon Lewis write
s:
> Why was this traffic hitting your DNS server in the first place? It should
> have been rejected by the ingress filters preventing spoofing of the local
> network.
When I ran a smaller simpler network, I did have input filters on our
transit providers rejecting packets from our IP space. With a larger
network, multiple IP blocks, numerous multihomed customers, some of which
use IP's we've assigned them, it gets a little more complicated to do.
One can never do a perfect job but one can stop a large percentage
of the crap. You should know the multi-homed customers and their
address ranges so they become exceptions. You also run filters on
internal routers. There are internal ingress/egress points as well
as interconnects.
urpf doesn't work as well for stopping inbound traffic to your network, because most people aren't totally defaultless, so the default route makes all traffic valid.
i've received several questions about the context of this mail, so i
thought it would be worth posting to clear up the reference.
for those who missed it, i presented a lightning talk at nanog 49 in
san francisco yesterday on some very early conceptual work on a really
interesting strategy to dramatically extend the useful life of v4
prefixes. the talk is linked from: http://nanog.org/meetings/nanog49/agenda.php and i encourage people to
take a look at it.
if you like the general idea (Probabilistic Assignment of Prefixes: a
System for Managing and Extending Address Resources is what some
people are starting to call it), i'd encourage you to take the
suggestion made at the mic by mark kosters, cto of arin, and work to
help refine the proposal and establish a useful policy framework
around its implementation.
work is needed especially in collision domain modeling and count of
resource implications for the operational overhead per prefix.
experience with high flow rate instrumentation is likely to be needed
in the near future as well.
i wanted to thank everyone for the kind words and suggestions after
the presentation and look forward to productively exploring this idea.
For those that missed the presentation, it was a real eye-opener on just
how important it is for you to move forward with IPv6 before something like
this actually starts getting implemented.
Hah, given the number of times people I have worked with have said "oh, I'll just use apnic space if we run out of IPs, i don't need to talk to them anyway", I think it's humorous that someone in China felt the same way about ARIN space.
oh, i think this has several important advantages aver carrier-grade
nat (which i believe to be mostly dead, anyway, no? someone who knows
more can chime in with references to the contrary should this not be
the case).
firstly: cgn puts reachability in the hands of a single organization.
with the PAP System you have a set of distributed choices about
reachability: different people can assess their different tolerance
to certain kinds of unreachability.
as i said in the presentation, the probability that there will be
positive operational overhead for a prefix is related the the count of
reuse within an association domain for a prefix ( p(Oop) = Cr(Ap) ).
We need to work out how to subdivide which parts of the internet
actually want to communicate directly with each other reliably and
make sure that they are within association domains.
in any case, i think this is more the subject of future work (and
possibly future nanog presentations) so i'll leave this here.
Reverse path filtering + asymmetric routing = epic fail. Jon did say
Multihomed customer.
Refer to RFC 3704 (BCP84). Note section 2.2 (Strict Reverse Path
Forwarding) last part of the final sentence: "in particular, when
applied to multihoming to different ISPs, this assumption may fail."
What RPF can do in this case though, is pro-actively prevent possible
future problems.
If all IP blocks are tied down to null, and urpf is enabled in loose
mode on an interface, it will catch cases where someone is sourcing
traffic to you using IPs from the unassigned space that you have in your
free pools.
Every month or so I re-route my blackholed traffic to a sinkhole, and
more often than not, I see some ingress traffic from my unassigned space.
Once upon a time, Steve Bertrand <steve@ipv6canada.com> said:
If all IP blocks are tied down to null, and urpf is enabled in loose
mode on an interface, it will catch cases where someone is sourcing
traffic to you using IPs from the unassigned space that you have in your
free pools.
That's not true on JUNOS devices - discard routes still count as valid
routes for loose-mode uRPF.
Reverse path filtering + asymmetric routing = epic fail. Jon did say
Multihomed customer.
If all IP blocks are tied down to null, and urpf is enabled in loose
mode on an interface, it will catch cases where someone is sourcing
traffic to you using IPs from the unassigned space that you have in your
free pools.
Hi Steve,
I'm not sure what that accomplishes. It doesn't close any doors. With
loose-mode RPF he can still forge packets from any address actually in
use.
Every month or so I re-route my blackholed traffic to a sinkhole, and
more often than not, I see some ingress traffic from my unassigned space.
You'd be better off pointing the forward routes at a packet logger so
you can gain some insight into who is scanning the network,
particularly when the scanner actually is internal.
yes, that is correct. However, it stops someone from outside sending
your network packets with a source address that currently resides in one
of your free pools.
What it does, is prevents packets with the illegal IP address from
actually being delivered to the intended destination within your network
preserving some (perhaps a very small amount) of bandwidth/router resources.
For instance, if I send your mail server a packet with a source of one
of your IPs that you currently do not have in use and you don't have rpf
enabled, the forged packet will make it to the server, be sent back to
it's next-hop, and then be discarded (if you have tie downs).
With urpf enabled, the packet is discarded upon the first ingress into
the network, thereby preventing it from going any further. This is what
I use loose mode for anyway.
Are you saying that JUNOS will not drop on source even if the only valid
route for an IP address is to null? On any other router I've used,
null/disc etc is a valid route, but it is considered special in that if
the route is to null, discard it, even on source.
