All,
Just wanted to apologize for the attack over the weekend. The
posts came from a email address that was subscribed to the list, so
it was not subjected to moderation. While a filter was added
to block further posts (which were made in a short time window),
there were existing message queues that were not cleared in a
timely basis.
As Job Snijders (a fellow Communications Committee member) noted
in an earlier post, we will be implementing some additional protection
mechanisms to prevent this style of incident from happening again. We
will be more aggressively moderating posts from addresses who have
not posted recently, in addition to other filtering mechanisms.
Regards,
Larry Blunk
NANOG Communications Committee Admins@nanog.org
To add to that: several people reached out off-list, offering help and
recommendations. We'll be following those up in the next few days. Thank
you for your support!
Some people found the admins@nanog.org readership unresponsive, but I
assure you this is not the case under normal circumstances. The admins
mail distribution was clogged up for the same reasons as the main list.
We'll work on improving our reachability.
I'd made a post to the members list, in the vain hope that it was on a
different server, and perhaps might go through (and it certainly did,
bright and early this morning). There's a couple of things I'd said
that are worth noting here. For those who didn't visit the archives,
where it was at least possible to see that the deluge was noticed by
folks, I'd suggest a quick look.
In my very unscientific method of knowing approximately how many lines
were visible in my browser, I guesstimate that there were about 1750
messages, and they were issued in the span of perhaps twenty minutes
(perhaps less), before the alarm bells went off, and the problem was
addressed.
For those who quickly looked at the archives, it was clear that others
had noticed that there was a problem (I even had off list emails with
a couple of them). I might have been more draconian in the clean up
(i.e. purge the queues, including valid emails), but honestly, that
was a pretty tough assault, and it's a good object lesson on what
might happen. You *are* all updating your security approaches and
data recovery plans, right?
Thanks to both Job Snijders and Larry Blunk. The check is in the mail.
Thank you Larry and Job for the responses, mitigation steps taken, and work to further resolve these kind of events.
Food for thought for the rest of us out there. Had there been a network attack on Sunday (for example) and several of these lists (multiple received this spam "attack") were switched to require a moderator to filter all emails manually. How quickly would information have gotten out through the networking community? No NANOG and Outages are not the only places I check or subscribe to but I DO check them to see if anyone else is reporting anything. And they are some of the places I would report real network problems to.
For me this didn't kill my weekend or destroy my ability to check my emails. I know for many others it didn't either.
I use my android mail client to group emails with the same subject and after checking multiple of them I didn't worry about those threads anymore. Yes I received several hundred emails about it but I was still able to function and watch for anything that came in that would note a threat to the network as a whole.
Maybe if this event has caused such a stir and inconvenience we should look at what we are doing and how we are doing it. These lists are tools that can be valuable to get information out to a large group of people. Anything that would block that I would consider a threat to the purpose of the list as well. This event caused blockage as well and the NANOG staff are looking into mitigation for that.
For what it's worth, while I did see all of these that made it through the list itself, the larger portion that I saw did not come through the list but were sent directly to me, and the Received header trail shows that those did not come through the nanog mailman. So I applaud what you do with the list itself, but it wouldn't have made (and won't make, in the future) much difference, since e-mails were sent out bypassing the list server.