At least in my neighborhood, Comcast appears to be running BIND 9.2.4rc6
--Prof. Steven M. Bellovin, http://www.cs.columbia.edu/~smb
Steve (and all),
At least in my neighborhood, Comcast appears to be running BIND 9.2.4rc6
Ah... Then there are to possible paths...
1) There was a real memory-leak bug and this was an unfortunate operations event. The CHANGES file for 9.3.1 and bind-9.2.5rc1 show various big fixes related to memory leak issues. I leave it to someone else to comment on the potential of being tickled within a Comcast environment.
-or- (And on a much more cynical note.)
2) Someone checked the latest CHANGES file for bind and realized that saying it was a memory leak was a good cover (see quick pseudo-grep of file below. Note that not all the bug's affect the running bind name server code).
Whichever it was, I wonder how it could affect so many name servers at only one provider and all at the same time. This is just plain strange. I would have thought that best practices for a DNS service would recommend staggered upgrades, heck, even forced different s/w releases. etc. etc.
Martin
Several of the servers that were down are not BIND, at least these:
prospero:~/Desktop/fpdns-0.9.1 dgold$ ./fpdns.pl 68.87.66.196
fingerprint (68.87.66.196, 68.87.66.196): Cisco CNR
I ran fpdns against them between outages. They now respond differently.
prospero:~/Desktop/fpdns-0.9.1 dgold$ ./fpdns.pl 68.87.66.196
fingerprint (68.87.66.196, 68.87.66.196):
q0r?1,IQUERY,0,0,1,1,0,0,REFUSED,0,0,0,0
These are the Comcast "national" DNS servers. (I am using plural, because
there are several reverse DNS entries for this IP address -
ns.cmc.co.denver.comcast.net and ns.inflow.pa.bo.comcast.net)
I wouldn't rush to blame BIND for this. For purposes of investigation, does
anyone have DNS servers from those periods of downtime other than the ones
above? Comcast is quite a patchwork, that's to the incomplete integrations
of MediaOne, AT&T Broadband, etc.
It would be interesting to see data on other DNS servers during the downtime
periods. Many folks on various forums were suggesting the use of ns1. And
ns2.level3. Of course, logic suggests that the vast majority of folks,
having no Internet access, could not have read the advice.
* Daniel Golding:
I wouldn't rush to blame BIND for this.
Maybe the leak wasn't in the DNS service, but some other software
component which company policy required on each server (think of
Tivoli, antivirus software, or CSA). Who knows? The possiblities are
endless.
There was, at one time, a fairly serious memory leak in Cisco CNR...
I believe I saw a post indicating that CNR was possibly in use?