Open Letter to D-Link about their NTP vandalism

Mailman List Mirror (Read Only)
1

Well, this is at least marginally on topic, and I think it deserves a wider audience. It is written by Poul-Henning Kamp (the affected party). Please read it.

http://people.freebsd.org/~phk/dlink/

It ends with the following:

Didn't something like this happen before?

Yes, D-Link is not the first vendor to make a hash of the NTP protocol. Some years back NetGear products blasted University of Wisconsin off the net. I have repeatedly pointed D-Link's lawyer at this case. Fortunately, in my case it is not that bad.

The NetGear incident caused the NTP protocol designers to add a "kiss of death" option to the Latest (S)NTP standard but D-Links devices does not respect that option. I have tried.

2

GPS.dix.dk service is described as:

DK Denmark GPS.dix.dk (192.38.7.240)
Location: Lyngby, Denmark
Geographic Coordinates: 55:47:03.36N, 12:03:21.48E
Synchronization: NTP V4 GPS with OCXO timebase
Service Area: Networks BGP-announced on the DIX
Access Policy: open access to servers, please, no client use
Contacts: Poul-Henning Kamp (phk@FreeBSD.org)
Note: timestamps better than +/-5 usec.

I think he should use dns views to answer the queries to gps.dix.dk and either:
( a ) answer 127.0.0.1 to all queries from outside his service area
( b ) answer a D-Link IP address to all queries from outside his
service area (which could lead to getting their attention; dunno if
from their engineers or from their lawyers).

Rubens

3

Rubens Kuhl Jr. wrote:

GPS.dix.dk service is described as:

DK Denmark GPS.dix.dk (192.38.7.240)
Location: Lyngby, Denmark
Geographic Coordinates: 55:47:03.36N, 12:03:21.48E
Synchronization: NTP V4 GPS with OCXO timebase
Service Area: Networks BGP-announced on the DIX
Access Policy: open access to servers, please, no client use
Contacts: Poul-Henning Kamp (phk@FreeBSD.org)
Note: timestamps better than +/-5 usec.

I think he should use dns views to answer the queries to gps.dix.dk and either:
( a ) answer 127.0.0.1 to all queries from outside his service area
( b ) answer a D-Link IP address to all queries from outside his
service area (which could lead to getting their attention; dunno if
from their engineers or from their lawyers).

Neither of which would solve the problem of his bandwidth being used by these, although (b) might actually serve to get their attention.

Perhaps as a thanks to him for the public service he provides the DIX, all of the users at DIX could set their external routers to reject incoming NTP packets from networks other than their own? Or even combine that with (b), although it might be more effective if it targeted, oh, www.dlink.com instead of an IP address.

Then at least it would not be taking up internal DIX bandwidth capacity.

By no means am I encouraging legally actionable activity, however, and as noted, (b) just might be.

4

Hi,

    Should not be hard to fix...

    Its clearly a missuses of dix.dk services.

Couple of thinks:

    Since its bgp and DIX customers surely have to provide a list of subnets to announce (filter and such), add those the the ntp server,

    or use ipf/ipfw/iptables to filter in the dix customers

    and I would redirect the others traffic to a dummy clock with a messed up time... after a few complaints DLINK would wake up.
    (Dont try to pin any legal issues to this ... its DIX servers/bandwidth/ressources, DLink (and its customers) has no regard on what DIX does with its ressources)

5

> I think he should use dns views to answer the queries to gps.dix.dk and either:
> ( a ) answer 127.0.0.1 to all queries from outside his service area
> ( b ) answer a D-Link IP address to all queries from outside his
> service area (which could lead to getting their attention; dunno if
> from their engineers or from their lawyers).

Neither of which would solve the problem of his bandwidth being used by
these, although (b) might actually serve to get their attention.

This reduces the bandwidth, as instead of dropping NTP packets, they
would never come to him in the first place.

Perhaps as a thanks to him for the public service he provides the DIX,
all of the users at DIX could set their external routers to reject
incoming NTP packets from networks other than their own? Or even combine

Which still would require him to answer DNS requests for gps.dix.de.

that with (b), although it might be more effective if it targeted, oh,
www.dlink.com instead of an IP address.

Answering with CNAME instead of A is a good enhancement of the
original idea... :slight_smile:

Then at least it would not be taking up internal DIX bandwidth capacity.

It still would require him to answer the DNS requests. Only way to
addres that is everybody outside DIX declare gps.dix.de as
www.dlink.com in their resolvers.

By no means am I encouraging legally actionable activity, however, and
as noted, (b) just might be.

Motion granted.

Rubens

6

Rubens Kuhl Jr. wrote:

<big snip>

It still would require him to answer the DNS requests. Only way to
addres that is everybody outside DIX declare gps.dix.de as
www.dlink.com in their resolvers.

Oh, I see two things here - the first is that he's in charge of his DNS, which he probably isn't. DIX likely is, but that's minor. They'll probably support him in this.

The second is that I was concatenating this letter and the also referenced Netgear letter, where they were doing refs by IP address instead of DNS like the D-Link is.

Combine both of them - reject outside the DIX DNS requests outside the service area (or send them to a DLink CNAME as mentioned) and as a backup reject/redirect all NTP from outside to the gps.dix.de IP address at the edge.

Belt and Suspenders as such.

As for the bogus NTP data idea... how many people buying a consumer grade router like this even have a clue what NTP is, much less notice what it's doing to that box over in the corner? It won't affect their computer, therefore they won't care. It's just buzzwords on the box.

7

*sigh* Yes yes everyone loves a good "large stupid company screws the
little guy by sticking their small/free service into a commercial product"
story, but unfortunately none of these solutions are very pragmatic. If I
hosted an NTP server and dlink put it in a default query list of a default
firmware, and then I asked them to pay my Equinix bill for the next 5
years, I would fully expect them to provide a nice little ascii diagram of
exactly where I could stick it.

Its just NTP, I can't imagine that it is *really* enough traffic to care
all that much. There are probably a hundred people on this list who could
donate free transit for this and not give it a second thought (hell if I
had a pop anywhere close to .dk I would donate a gigabit solely to end
this nanog thread before it turns into a bunch of self-righteous whining).
There are probably an equal number of people who could donate hardware for
this, either for filtering or for the IX (if they REALLY don't have the
resources to handle it without charging, which I highly doubt). I'm sure
you could probably pick out the dlink queries with sufficient packet
inspection too, which I'm also sure you can achieve with a FreeBSD box and
a couple hours of spare time. :slight_smile:

Seriously now, there are a million viable solutions here, ranging from
mild inconvenience to attempting to screw dlink for being dumbasses, all
of which are free. Point the A record else where and have people who care
change to a new record, it's not worth $62k.

Oh and one more thing, if the goal was restricting the traffic to only
people who participated at this IX (as per the description), please add
this to the list of reasons why announcing your IX subnet over the global
internet is a BAD IDEA!

8

Did you read the posting? His ISP is charging him. He's also put in
a fair amount of time trying to get this resolved. As for transit --
NTP works much better with short RTTs, which is precisely why it's
good to have a server in Denmark.

    --Steven M. Bellovin, http://www.cs.columbia.edu/~smb

9

Its just NTP, I can't imagine that it is *really* enough traffic to care
all that much.

You're kidding, right? Do you know what happened to wisc.edu:

  Human Verification

10

+<20060407224918.GM45591@overlord.e-gerbil.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20060407224918.GM45591@overlord.e-gerbil.net>
User-Agent: Mutt/1.5.9i

11

According to Richard Clayton (who helped Poul-Henning track the problem
down) it's about 37pps continuously for each stratum-1 NTP server.
(Remember there are many other servers configured into the D-Link
firmware that are also being abused.)

http://www.lightbluetouchpaper.org/2006/04/07/when-firmware-attacks-ddos-by-d-link/

Tony.

12

Correct me if I'm wrong, but... That was only "really" a problem for them because there was a flaw in the Netgear code that caused the devices to make requests every second. That's not (as far as I'm aware) happening here, so we're not talking huge amounts of bandwidth.

We intentionally run public NTP servers, which are even in the pool.ntp.org pool, as well as on some NTP lists. I've pegged about 35,000 unique IPs using our North America server in the last 24 hours, or about 175pps. Bandwidth usage is about 100Kbps per second on average. The occasional burst up to 250Kbps+, but those are pretty rare.

This link here: http://www.lightbluetouchpaper.org/2006/04/07/when-firmware-attacks-ddos-by-d-link/ says he's getting 37pps. NTP uses 76 byte packets. 37pps * 76 byte packets = 22.4Kbps, or less than the amount of traffic a dialup user can spew. If you're running a semi-public server on the internet, and it can't handle a dialup user flooding it - you need a firewall anyway. :slight_smile:

I can see how unwanted NTP traffic could be a nuisance, but not how it could possibly cost US$8,800 per year. Nor requiring the use of a US$5000 "external consultant" to track down the source of the traffic. Nor worthy of invoking the Slashdot masses in outrage. Let alone why an additional traffic load of less than a dialup user accessing your server in any way is worthy of caring. Bad on D-Link for what they've done, but total overreaction on the other side as well.

I think the lesson here is that any service you make available to the public (NTP, DNS, IRC, SMTP, whatever) is going to be used in ways that do not match with your desires. If you're not willing to ACL/police the service, you're going to have to accept that people are going to use it in ways you'd rather they didn't.

13

yeah, i went and dug through our netflow records and even pegged
the ip up for a few seconds on a router to collect a few requests and saw
very little traffic to this IP.

  My suggestion is rename from gps -> gps1 and drop the gps
dns name. That combined with some bind/whatever views that
scope the dns responses are effective since it's a DNS name.

  While it's similar to the wiscnet stuff, it's not identical
and can be [easily] mitigated.

  - jared

14

That isn't what he asked. He explained the problem to them, asked
them to rectify it, and their response was "here, talk to our lawyers".
Which is merely a discrete way of saying "f*** off, we're bigger".

Infer at will, please, if you will.

I know phk personally (give or take a little, we're from the same
country, and have both participated vigorously in the same UNIX user
group [yes, there have been such entities]); for those who are unaware
of his credentials, let me cut and paste the following from the FreeBSD
GBDE manual page:

HISTORY
     This software was developed for the FreeBSD Project by Poul-Henning Kamp
     and NAI Labs, the Security Research Division of Network Associates, Inc.
     under DARPA/SPAWAR contract N66001-01-C-8035 (``CBOSS''), as part of the
     DARPA CHATS research program.

AUTHORS
     Poul-Henning Kamp <phk@FreeBSD.org>

So for the benefit of any xenophobes here, rest assured, he's cleared
for a NASA-powered Mars-walk when the time comes.

As for the issue at hand, I can only support phk, it's an open and shut
case. Of course, anybody and everybody is free to argue that if one
walks through Central Park (or whatever your local Mugger's Delight is
called), one is merely fighting a losing battle. At the end of the day,
however, it becomes a question about what behaviour we deem acceptable,
whether or not fighting back seems fruitful.

There's an old Scandinavian proverb (quite possibly of foreign origin)
which simply says ""Those who keep quiet, agree".

Do you agree?

  -- Per

15

The repeated suggestions that the best response to this sort of situation is to 'deal with it' are saddening. Did he "deserve it" because of the short skirt he was wearing?

Companies behaving irresponsibly and releasing (selling!) code that abuses a shared public resource should not be the norm.

Lots of people crap in the commons. But saying it is inevitable, and actually mocking people who care, is inexcusable.

Someone's accountable here, and in this case it is simple to determine who. The behavior is clearly antisocial and should be addressed.

Member-waving about spare gigabit ports and what traffic levels are inconsequential really is beside the point.

matto

PS: This puts D-Link on the same list I had Belkin and Netgear on.

--matt@snark.net------------------------------------------<darwin><
   Moral indignation is a technique to endow the idiot with dignity.
                                                 - Marshall McLuhan

16

And, if a service were available to the public *as a matter of courtesy* (or
even as a matter of accident) and not *advertised to the public*, then those
using the [unadvertised] service must cope if and when the service
disappears, or even starts misbehaving deliberately.

17

Jeff Shultz wrote:

By no means am I encouraging legally actionable activity,
however, and as noted, (b) just might be.

LOL! Did you read down to the end?...

/quote/
I can't afford to sue D-Link. It seems that they have managed to arrange
their corporate affairs so that there is no way I can sue them here in
Denmark, but will have to do it either in Taiwan or USA. Needless to say, I
can't afford that.
//

I would buy a ticket to witness the sublime poetic justice of D-Link trying
to appeal to the Danish legal system over Poul's efforts to fend off their
connections.

And in response to Kevin Day:

I think the lesson here is that any service you make available to the
public (NTP, DNS, IRC, SMTP, whatever) is going to be used in ways
that do not match with your desires.

I take it then that you are among those who believe that abuse of any
of these types of services should only be countered by technical
means then, and never through legal remedies?

18

Ok let me answer two at once here:

Did you read the posting? His ISP is charging him. He's also put in
a fair amount of time trying to get this resolved. As for transit --
NTP works much better with short RTTs, which is precisely why it's
good to have a server in Denmark.

Actually, no. Incase it wasn't clear, the IP (192.38.7.240) is out of an
IX subnet for the DIX. Even if you didn't know this particular block,
looking at the reverse DNS for nearby IPs makes it painfully obvious.

See: http://www.peeringdb.com/dns-scan/192-38-7-0-24.txt

The real issue here is that DIX used a /24 from an aggregate block which
is announced in BGP (198.38.0.0/17) for their IX space, thus making it
reachable from anywhere on the Internet. Incase anyone didn't know this
before, now you do, this is a Bad Idea (tm).

The prices phk mentions appear to be the cost of a DIX port. According to
their website:

A connection at the DIX with 10 or 100 Mbit/s ethernet has a yearly fee of
DKK 27.000.
A connection at the DIX with 1000 Mbit/s Ethernet costs a yearly fee of
DKK 38.700.

According to the service description, this NTP server was intended to be
used only by DIX connected networks. If the /24 had been pulled from a
direct /24 allocation or EP.net space, this would never be a problem,
because the /24 for the IX shouldn't be propagated globally. In this
particular case they could filter packets coming in via AS1835's border
links, but since the block is announced globally already this may create
further problems from people who don't know they need to carry the /24 and
propagate it to their customers.

Personally I'm not sure what to be more appalled by, that DIX would want
to charge him for something that is clearly a service which benefits only
them and which they should probably be paying HIM for (and which wouldn't
cost them a dime if not for their poorly implemented architecture), or
that a consultant charged $5000 to track this down. Both concepts are
actually more repulsive to me than dlink picking 25 publicly accessable
nameservers.

19

That will fix the problem. In 2012 or so.

I have a hostname that just now saw 500 NTP packets in 112 seconds. OK, so
it's only 5 packets per second.

Mind you, that hostname *was* at one time a stratum-2 server. But it moved to
a different host on April 7, 2000 - 6 *years* ago. One year after that, it
stopped answering NTP entirely at that IP address. And that IP address went
away entirely during a building renovation 4 years ago. There's still an ARP
every 2-3 seconds for it caused by people who hard-coded the IP address.

I'm not sure which is scarier - the fact that of those 500 queries, 367 were
*still* running NTPv1 - or that 89 were NTPv3 and and 44 were NTPv4, when the
host in question has never answered an NTPv4 query from off campus.

So somebody mentioned a stratum-1 is seeing 37 PPS - I'm still seeing 1/6 of that
level for something that went away *last century*.

20

I've been configuring up a few ciscos recently. In the config, I enter
"ntp server pool.ntp.org", at which point IOS resolves pool.ntp.org, and stores
the IP address it gets in the config. Not entirely what is expected, but an
explaination for why IPs get hardcoded...

Simon