`
Date: Tue, 06 May 2008 14:29:03 -0700
From: Nathan Anderson/FSR <nathana@fsr.com>
Subject: Re: [NANOG] Microsoft.com PMTUD black hole?
Now, although that makes sense, in order to avoid issues like the one we
are facing with Microsoft, would it not make _more_ sense for the stack
to look at the PMTU cache first, and then adjust its own MSS just for
connections to that one host?
This _is_ Microsoft we're talking about, remember. 'sense' and 'Microsoft'
are, at a =minimum= orthogonal to each other -- and may not even inhabit
the same address-space. <wry grin>
As for standards, it is official Microsoft policy to "embrace and extend",
not to implement in a way compatible with the rest of the world. *sigh*
I -don't- believe the rumor that "PMTUD/Vista Ultimate" sends incrementally
increasing-size packets, and uses the first one that -doesn't- get through
as the size limit. <giggle>
Interestingly, Windows XP, Sp3, released today, describes changes in
PMTUD behavior.
Black Hole Router detection is now on by default:
http://download.microsoft.com/download/6/8/7/687484ed-8174-496d-8db9-f02
b40c12982/Overview%20of%20Windows%20XP%20Service%20Pack%203.pdf
All,
A member of Microsoft's GNS network escalations team saw my postings on NANOG about this issue and took offense at my use of this forum to raise this issue with them, and criticized me as being unprofessional and lacking in business acumen.
Therefore, I would like to publicly apologize for my actions here. It was not my intention to "humiliate" Microsoft into compliance but rather to find a means of effective contact with them since none was to be found before today. However, I recognize that I did step over the line, especially with regards to one comment I made in an earlier post about "giving Microsoft too much credit." I apologize for this and retract this, and ask their forgiveness.
As I promised, I will not be posting any more to this list regarding this issue unless it is to report the final verdict that I receive from my now-open ticket with Microsoft (thanks to this list, I found an effective contact), or to discuss the mechanics of PMTUD in general.
Regards,
Tomas L. Byrnes wrote:
Interestingly, Windows XP, Sp3, released today, describes changes in
PMTUD behavior.
Black Hole Router detection is now on by default:
As I pointed out in my post earlier today timestamped at 2:29PM, I was using an XP SP3 host to perform my tests with, and it made no difference. I also used BBR's DrTCP application to make sure that black hole router detection was, in fact, enabled on my XP box before commencing my packet captures.
I cannot explain why it made no difference, but at the same time I don't know enough about how WinNT's black hole router detection works to begin speculating at this point. I do plan on looking into it, however.
A member of Microsoft's GNS network escalations team saw my postings on
NANOG about this issue and took offense at my use of this forum to raise
this issue with them, and criticized me as being unprofessional and
lacking in business acumen.
they try that intimidation every time a vulnerability or bug is
revealed. laugh and post their overly-aggressive message on a public
web site.
randy
Nathan Anderson/FSR wrote:
A member of Microsoft's GNS network escalations team saw my postings on NANOG about this issue and took offense at my use of this forum to raise this issue with them, and criticized me as being unprofessional and lacking in business acumen.
Hang on a tick. Aren't you one of their customers...<looking through
mail spool>...
> As I pointed out in my post earlier today timestamped at 2:29PM, I was
> using an XP SP3 host to perform my tests with...
...why yes, you are.
I can't think of any other supplier that would be so unprofessional and
so lacking in business acumen as to say that their customer was UALIBI.
Amazing. A fine case study of a person in customer contact undoing the
work of millions of dollars in PR. Whatever you say about Steve Ballmer
he's a great sales person at heart. He must despair at some of his staff.
I wouldn't worry too much about it, Glen. My observation is that the
millions of dollars in PR isn't working very well either 
- mark
This is a typical Microsoft reaction: blame the messenger for their
own incompetence, laziness, stupidity, and greed. I think you should
post their assinine message so that it can receive the public ridicule
it surely deserves.
---Rsk
Glen Turner wrote:
Amazing. A fine case study of a person in customer contact undoing the
work of millions of dollars in PR. Whatever you say about Steve Ballmer
he's a great sales person at heart. He must despair at some of his staff.
The rest of us however, despair at having to support their crap.
Patrick Giagnocavo
patrick@zill.net
Thus spake "Nathan Anderson/FSR" <nathana@fsr.com>
A member of Microsoft's GNS network escalations team saw my
postings on NANOG about this issue and took offense at my use
of this forum to raise this issue with them, and criticized me as
being unprofessional and lacking in business acumen.
First, it's "unprofessional and lacking in business acumen" for someone to criticize their customers to their face. As one manager taught me, "The customer may not always be right, but they're never wrong."
Second, it's their own damn fault for not maintaining their contact information properly in public databases. If the only option they leave you is to post to NANOG, because they don't respond to (or even accept) direct requests to the listed contacts, then that's what you have to do.
Many companies are guilty of the latter, and we all get the benefit of seeing the state of their customer service for reference when making future buying decisions. Very few are arrogant enough to do the former, though.
S
Stephen Sprunk "God does not play dice." --Albert Einstein
CCIE #3723 "God is an inveterate gambler, and He throws the
K5SSS dice at every possible opportunity." --Stephen Hawking
Here is a brief update on the situation:
I have been in contact with someone at Microsoft's service operations center, who has confirmed for me that MS does in fact block _all_ ICMP at the edge of their network, that they are aware that this will in fact break PMTUD, and that they have no current plans to change this practice which they have implemented in the interest of security.
Nevertheless, the person I have been in contact with is naturally not the final decision-maker on this issue and is going to continue to pass the issue on up the chain of command for me. So although this issue is not over and I do not have a final verdict from MS yet, I felt that, given that I don't know how much time to expect to pass between now and when that final verdict is rendered, it would be appropriate to let everybody here know what I have learned thus far. Hopefully public dissemination of this information factoid will prevent others in a position similar to mine from having to helplessly beat their heads into their keyboards.
I, naturally, voiced my strong objection over this security policy, and attempted to make a reasoned argument with the contact I have over there. We will see what comes of this.
Some have asked me to post copies of my private communication with my Microsoft contact here. I don't think it is appropriate for me to post copies of private communication without the other party's consent, so I will have to decline unless he first gives me said consent.
Others have asked for valid contact information for the Microsoft NOC, since the ARIN records for their 207.46.0.0/16 do not appear to be up to date. I eventually found a working e-mail address from somebody off-list who pointed to the WHOIS lookup from TUCOWS for microsoft.comosoft.com (which I'm still not clear on what exactly this is...). The e-mail address that was gleaned from this lookup was msnhst@microsoft.com, which goes to the Microsoft Corporate Domains Team. They, in turn, forwarded my message on to msnalerts@microsoft.com, which generated a ticket # for me and is, as I understand it, the e-mail address I was looking for in the first place (leads to their network/system people).
I hope this is helpful to others.
Regards,
Nathan Anderson/FSR wrote:
Here is a brief update on the situation:
I have been in contact with someone at Microsoft's service operations center, who has confirmed for me that MS does in fact block _all_ ICMP at the edge of their network, that they are aware that this will in fact break PMTUD, and that they have no current plans to change this practice which they have implemented in the interest of security.
Although the need for your previous apology has already been questioned in this forum, the confirmation that they block not only certain ICMP types, but all ICMP, further vacates the need for any apology for criticizing this behavior in a pubic forum. It is disheartening for those of us who use and support MSFT's products to learn that their understanding of security lacks even the basic nuance to know not to block an entire--critical--portion of the Internet Protocol. Perhaps they should also block _all_ TCP and UDP as well, and then we can move on.
I agree with Iljitsch that it happens frequently, but I think I am justified in expecting more than that from Microsoft. Anything less would be unprofessional.
*Speaking for myself only, of course!*
michael
Right.
Now Microsoft is also the company that built the OS that could be crashed by a maliciously crafted fragmented IP packet, so maybe there's something to this security policy. (One hopes that this bug and others like it are now fixed.)
However, in that case the only workable course of action would be TO DISABLE PATH MTU DISCOVERY!
You can't have your cake and eat it too.
The remedy you have below is NOT the only one, and is, in fact, a
non-sequitur in this case.
PMTUD uses the DF (for Don't_Fragment) bit, and works by getting an ICMP
Fragmentation needed response from the hop on the path where the packet
is too large, not a fragmentation and forward, so the union of PMTUD
packets and fragmented ones is 0.
The network-level solution to ping of death is to BLOCK fragmented
packets, and the way to ensure this doesn't self-deny-service is to
perform PMTUD and Black-Hole Router discovery.
Some Edumacation on the topic is here:
http://www.netheaven.com/pmtu.html
Tomas L. Byrnes wrote:
The remedy you have below is NOT the only one, and is, in fact, a
non-sequitur in this case.
How so? Iljitsch is suggesting that ICMP blockers originate packets without DF set if they are going to block the ICMP messages that PMTUD needs in order to work in the first place. That's what (I think) he means by "disabling path MTU discovery."
The network-level solution to ping of death is to BLOCK fragmented
packets, and the way to ensure this doesn't self-deny-service is to
perform PMTUD and Black-Hole Router discovery.
Which end are you talking about here, the servers or the client? If the servers, how do you expect them to do PMTUD if they _can't hear the ICMP messages_?
Also, for some reason, as I pointed out before, XP black hole router discovery doesn't seem to be working for me for whatever reason. Does anybody have any clue why that might be the case?
Tomas L. Byrnes wrote:
Some Edumacation on the topic is here:
You do know who it is that you are responding to, right?
http://www.oreillynet.com/pub/au/970
How so? Iljitsch is suggesting that ICMP blockers originate packets
without DF set if they are going to block the ICMP messages that PMTUD
needs in order to work in the first place. That's what (I think) he
means by "disabling path MTU discovery."
Yes.
Also, for some reason, as I pointed out before, XP black hole router
discovery doesn't seem to be working for me for whatever reason. Does
anybody have any clue why that might be the case?
The problem is in the direction from M$ to you, so you can't fix that from your end. I wonder if they've installed SP3 on their servers...
I was responding to his post that blocking or disabling PMTUD was the
way to avoid the ping of death, which is False, nothing more, nothing
less.
As far as who Iljitsch is, everyone misspeaks from time to time. Even
those of us who have been at this for nearly 3 decades.