Best practices for sending network maintenance notifications

All,

We recently, at $dayjob, had one of our peers (at Symantec) send out a network maint notification, putting 70 addresses in the "To:" field, rather than using BCC or the exchange's mailing list.

Naturally, when you mail 30 addresses, of the forms peering@ and noc@ various organizations, you're likely to hit at least a few autoresponders and ticket systems...

And at least one or two of those autoresponders are of course brainded and configured to reply-all. (In this case, Verizon's ServiceNow setup was such a stupid responder). And that made things fun in our own ticket system, as our RT setup happily created a bunch of tickets.

My question for the group -- does anyone know if there's a "best practices" for sending maint notifications like this? An RFC sort of thing?

While it would define a social protocol, rather than a truly technical one, if there's not such a document, it seems like it could useful. And once such a thing exists, exchanges could of course helpfully point their members AT it (for both their humans, and ticket systems, to follow).

-Dan

I think there was a BCP being worked on. I seem to recall it was being discussed as a Facebook group. But there's no RFC, at least that I know of.

Regards,

Hal Ponton

Senior Network Engineer

Buzcom / FibreWiFi

Tel: 07429 979 217
Email: hal@buzcom.net

I think there was a BCP being worked on. I seem to recall it was being
discussed as a Facebook group. But there's no RFC, at least that I know
of.

And additionally, putting the recipients in the To: line sounds like a
really bad idea. Sharing PII without permission and stuff like that.

Make absolutely certain that all the SPF, DKIM and DMARC stuff is perfect.
Make sure that any links are to the corporate domain... always.

You want a neutral, paranoid 3rd party who is receiving the notice to be
absolutely convinced of its Bona Fides. Do not suppose that your abundance
of Sincerity excuses sloppiness. It won't. :frowning:

Regards,

Hal Ponton

Senior Network Engineer

Buzcom / FibreWiFi

Tel: 07429 979 217
Email: hal@buzcom.net

All,

We recently, at $dayjob, had one of our peers (at Symantec) send out a
network maint notification, putting 70 addresses in the "To:" field,
rather than using BCC or the exchange's mailing list.

Naturally, when you mail 30 addresses, of the forms peering@ and noc@
various organizations, you're likely to hit at least a few
autoresponders and ticket systems...

And at least one or two of those autoresponders are of course brainded
and configured to reply-all. (In this case, Verizon's ServiceNow setup
was such a stupid responder). And that made things fun in our own
ticket system, as our RT setup happily created a bunch of tickets.

My question for the group -- does anyone know if there's a "best
practices" for sending maint notifications like this? An RFC sort of
thing?

While it would define a social protocol, rather than a truly technical
one, if there's not such a document, it seems like it could useful. And
once such a thing exists, exchanges could of course helpfully point
their members AT it (for both their humans, and ticket systems, to
follow).

-Dan

--

Aloha mai Nai`a.

All,

We recently, at $dayjob, had one of our peers (at Symantec) send out a
network maint notification, putting 70 addresses in the "To:" field,
rather than using BCC or the exchange's mailing list.

Naturally, when you mail 30 addresses, of the forms peering@ and noc@
various organizations, you're likely to hit at least a few
autoresponders and ticket systems...

And at least one or two of those autoresponders are of course brainded
and configured to reply-all. (In this case, Verizon's ServiceNow setup
was such a stupid responder). And that made things fun in our own
ticket system, as our RT setup happily created a bunch of tickets.

My question for the group -- does anyone know if there's a "best
practices" for sending maint notifications like this? An RFC sort of
thing?

In general I'd push for a little automation for the sending of
notifications as reducing the likelihood of mishap.

Targeting bcc is nice, but so does simply generating a message for each
peer precludes this. we store contact information which bgp neighbor
parameters in our config generation.

It falls in the category of "Doctor, it hurts when I do this. Don't do that." Even the most dense CSR managers figure it out after a few attempts.

The other "don't do that" is never configure Music on Hold for any NOC/SOC
lines. Few things are more annoying than a eight hour trouble shooting conference bridge, and one of the dozen NOC/SOCs on the bridge hits
the Hold button.

"The other "don't do that" is never configure Music on Hold for any NOC/SOC
lines. Few things are more annoying than a eight hour trouble shooting
conference bridge, and one of the dozen NOC/SOCs on the bridge hits the Hold
button."

Now that you've said it it seems so obvious. But, honestly I'd never thought
it until right now. Thanks!

Regards,
Ray Orsini – CEO
Orsini IT, LLC – Technology Consultants
VOICE DATA  BANDWIDTH  SECURITY  SUPPORT
P: 305.967.6756 x1009 E: ray@orsiniit.com TF: 844.OIT.VOIP
7900 NW 155th Street, Suite 103, Miami Lakes, FL 33016
http://www.orsiniit.com | View My Calendar | View/Pay Your Invoices | View
Your Tickets

And some genius at an ISP's NOC has put
rick_asley_never_gonna_give_you_up.mp3 in the their hold queue music
rotation list.

I've been on hold a few times with some companies that had great 80's music. I almost asked them to put me back on hold when they finally took me off. Sometimes it's a party when one of the people on the call hits the hold button, it depends on how bad the outage is :slight_smile:

True.

https://www.facebook.com/groups/maintnote/

Currently under development, but fairly far along...

Cheers,
~Chris