Hi guys, several ISP's are experiencing a flood of calls from customers
who get failed installations of the recent IE 0day - VML - (vgx.dll).
If you are getting such floods too, this is why.
This is currently discussed on the botnets@ list, as raised by Cox, and I
figured I will float it out here.
No patch is currently available from Microsoft, workaround are available.
Gadi.
Ok I'll admit I've been reading less and less of this godforsaken list
with each passing day, but at what point did we change the name to North
American Network Tech Support Operators Group? Was the memo distributed
via HTML e-mail only or something? Maybe it was redacted from the archives
so I didn't see it...
Seriously Gadi, what *possible* relevence could this have to network
operations?
Gadi Evron wrote:
Hi guys, several ISP's are experiencing a flood of calls from customers
who get failed installations of the recent IE 0day - VML - (vgx.dll).If you are getting such floods too, this is why.
This is currently discussed on the botnets@ list, as raised by Cox, and I
figured I will float it out here.No patch is currently available from Microsoft, workaround are available.
Gadi.
And this has to do with Network Operations in what way?
-Bill
In my book, if very large ISPs abuse desks become saturated, this is a
problem ISPs face. Most ISPs would like to know how to respond to these
questions, as well as know what's going on.
Are you telling me tech support overflow at this immense scale does not
affect the ISP and its network staff as well?
It's not BGP, it's on-topic to others here.
Gadi.
define 'immense scale' ... no calls here... so 'immense scale' in this
case is 'nothing'.
No, one thing you might say is that increased (channelling Vijay here...)
calls from customers means increased 'Support Cost' and decreased profit
margin over time. I'd also say:
1) how is this different from a large scale network outage for a provider
2) how is this different from any other large worm outbreak thing
3) is this blackworm all over again? (all hype no bite... byte?)
-Chris
>
> Are you telling me tech support overflow at this immense scale does not
> affect the ISP and its network staff as well?define 'immense scale' ... no calls here... so 'immense scale' in this
case is 'nothing'.No, one thing you might say is that increased (channelling Vijay here...)
calls from customers means increased 'Support Cost' and decreased profit
Thank you for providing me with a correct explanation.
margin over time. I'd also say:
1) how is this different from a large scale network outage for a provider
Exactly the same, only seen at a few, so is likely to be seen with others.
2) how is this different from any other large worm outbreak thing
It's not.
3) is this blackworm all over again? (all hype no bite... byte?)
A lot of bite. Unfortunately. Every month on the third many still lose
their files. What was interesting to nanog then was the IMMENSE global
cooperation and coordination, encompassing too many and working, to
mitigate it. Unless some us us, others here try and keep nanog in the
loop.
I know this interested many here, and nanog is the best way to reach
them. Such occasional operational issues not interesting to you are
interesting to us. These emails cause more disturbance.
Is nanog to be BGP only? Please let me know and I won't email these
here. Simple enough. If not, we all take note of what is interesting to
us.
Gadi.
I'm seeing email saying my employer's (large broadband) call centers are taking extremely high call volumes due they believe to this exploit. I don't think this is a case of crying wolf, since there are apparently several broadband providers who are getting hit with this, based on Gadi's email.
I'll leave the flamewar as to whether this is on topic for NANOG or not to the experts.
Bob
Which makes it operational in which sense?
I'm starting to think that these "alerts" need to be filed along with the daily "OMG, evil people are taking over your computer if you don't send this to at least 10 people" IMs.
Paranoia has its place, but this ain't the place.
The report is NOT paranoia. Several LARGE user ISPs suffer immensely from
this. Use this information if it is useful to you and you encounter the
same problems.
Thanks,
Gadi.
Does it impact the network operation?
Eg, does it adversely affect the network? (say, like Beagle did.)
Adrian
> > Paranoia has its place, but this ain't the place.
>
> The report is NOT paranoia. Several LARGE user ISPs suffer immensely from
> this. Use this information if it is useful to you and you encounter the
> same problems.Does it impact the network operation?
Eg, does it adversely affect the network? (say, like Beagle did.)
Not like Bagle did, to my knowledge. That said, this is spreading at an
increasing rate that is unbelievable. That means worms, bots, and yes, ISP
support, network and system personnel time depending on ISP.
I was thinking sql-slammer, massive flood causing signifcant
amount of network infrastructure to go down. (people on low speed links
with large blocks of address space were DoS'ed off the network).
I don't think of drive-by browser/desktop infection as a networking
issue, more of an end-host issue.
- Jared
Gadi Evron wrote:
2) how is this different from any other large worm outbreak thing
It's not.
Which makes it operational in which sense?
I'm starting to think that these "alerts" need to be filed along with
the daily "OMG, evil people are taking over your computer if you
don't send this to at least 10 people" IMs.Paranoia has its place, but this ain't the place.
The report is NOT paranoia. Several LARGE user ISPs suffer immensely from
this. Use this information if it is useful to you and you encounter the
same problems.
Gadi, your initial query lacked the factual background that would have
been useful for someone to decide if it was relevant to them or not.
While I do believe that the intersection of host and applications issues
and networking has applicability here I will make two observations that
I hope are not wildly off the mark.
Many of the people on the operations side of networks do not spend a lot
of time on security mailing lists. They also don't spend a lot of time
looking into their own support organizations until until problems get
escalated to them, so your initial post could have used more background.
Even in an enterprise it's really hard to justify the expenditure that a
rapid response to a host security problem involves. For an isp which is
not likely to be in the position to recover the cost of being reactive
let alone pro-active I can't imagine how they would possibly support
desktop issues like this.
joelja
Gadi, your initial query lacked the factual background that would have
been useful for someone to decide if it was relevant to them or not.
While I do believe that the intersection of host and applications issues
and networking has applicability here I will make two observations that
I hope are not wildly off the mark.Many of the people on the operations side of networks do not spend a lot
of time on security mailing lists. They also don't spend a lot of time
looking into their own support organizations until until problems get
escalated to them, so your initial post could have used more background.Even in an enterprise it's really hard to justify the expenditure that a
rapid response to a host security problem involves. For an isp which is
not likely to be in the position to recover the cost of being reactive
let alone pro-active I can't imagine how they would possibly support
desktop issues like this.
Thank you, I will make sure and learn from this in the future!
Gadi.
> Does it impact the network operation?
> Eg, does it adversely affect the network? (say, like Beagle did.)I was thinking sql-slammer, massive flood causing signifcant
amount of network infrastructure to go down. (people on low speed links
with large blocks of address space were DoS'ed off the network).I don't think of drive-by browser/desktop infection as a networking
issue, more of an end-host issue.- Jared
so, how many netops folks use or are forced to use IE
in the mgmt of their particular sector of an IP network?
netops being deaf/blind; "... the MRTG/Cricket graphs are
not visable... does that mean nothing is happening?..."
might be considered operationaly significant. Or not..
YMMV...
--bill
jared@puck.nether.net (Jared Mauch) writes:
I was thinking sql-slammer, massive flood causing signifcant
amount of network infrastructure to go down. (people on low speed links
with large blocks of address space were DoS'ed off the network).
right.
I don't think of drive-by browser/desktop infection as a networking
issue, more of an end-host issue.
given that "network operations" now includes all kinds of non-bgp activities
like datacenter design, tcp syn flood protection, nonrandom initial tcp
sequence number prediction, and a googolplex or two of other issues, i've
assumed that the hardcore bgp engineering community now meets elsewhere.
(i wouldn't be needed or welcome "there" if so, so i'm just guessing.) so,
for lack of a better forum, "things that can beat the hell out of your abuse
desk" does indeed seem like safe fare for nanog@ in 2006, even though in 1996
maybe not so much so. (hell, in 1996 one could still send MIME attachments
to abuse desks, since they were generally running solaris on NCD terminals
rather than microsoft outlook, and attachments were "just opaque data", grrr.)
can we all agree to stop shooting the messenger? every time gadi speaks up
here, three or four folks bawl him out for being off-topic. time has proved
that (a) gadi's not going to STFU no matter whether he's flamed or isn't, (b)
those flaming arrows sticking out of his chest don't seem to injure him at all,
(c) the flames completely outweigh gadi's own original posts, and (d) some of
the folks lurking here actually tell me that they benefit from gadi's stuff.
henceforth if you see a post, a poster, or a thread that you aren't interested
in, "just hit delete". it'll save more bandwidth than flaming about it would.
joelja@uoregon.edu (Joel Jaeggli) writes:
Even in an enterprise it's really hard to justify the expenditure that a
rapid response to a host security problem involves. For an isp which is
not likely to be in the position to recover the cost of being reactive
let alone pro-active I can't imagine how they would possibly support
desktop issues like this.
and yet, when i consider my nontechnical friends with their DSL and cablemodem
connections, i know that if they get hit by an exploding DLL, their ISP is one
of the likely places they will place a call. and then they'll carefully nav
their way through what they call "voice mail hell" until they can talk to a
"live operator", no matter how complex that is, no matter how many steps, and
no matter how much musak-on-hold they'll have to listen to.
the perfect storm is a million extra customers calling over the course of a
week just to explain that they have "exploding DLL symptoms" and listen to a
"live operator" tell them that this isn't a network problem and they should
contact the dealer where they bought their computer, which is likely CostCo.
assuming that this takes less than 60 seconds per affected customer, it's
still a nasty unbudgeted expense and as a secondary burn it will make real
network problems harder to report.
Ok so:
1) Gadi sends his org email out stating bla bla bl abla
2) a dozen people reply back with to-all.. which causes further controversy
3) Gadi replys, trying to save him self
Can we please keep the flamewar offlist! .. if you got something to say..
say it to the person and not the entire list of people on nanog!
-ps, my apologies for contributing to this useless thread and mass listing
nanog.
-Payam
Paul Vixie wrote:
joelja@uoregon.edu (Joel Jaeggli) writes:
Even in an enterprise it's really hard to justify the expenditure that a
rapid response to a host security problem involves. For an isp which is
not likely to be in the position to recover the cost of being reactive
let alone pro-active I can't imagine how they would possibly support
desktop issues like this.
<snip>
the perfect storm is a million extra customers calling over the course of a
week just to explain that they have "exploding DLL symptoms" and listen to a
"live operator" tell them that this isn't a network problem and they should
contact the dealer where they bought their computer, which is likely CostCo.
assuming that this takes less than 60 seconds per affected customer, it's
still a nasty unbudgeted expense and as a secondary burn it will make real
network problems harder to report.
Indeed. I'm fairly certain that in the life-cycle of some network
maladies that decision has to be made as to whether you want to go out
of business sooner (no more customers) or later (costs). When given the
choice, I prefer the later.
Even more to the point, a lot of people with network infrastructure that
couldn't handle random destination traffic were affected. Such impact is
precisely the kind of thing that should be discussed on NANOG, both from
an operational "how do we deal with this" and a design "what you should
know about your gear when it doesn't have a prepopulated table in its fast
path" perspective.
A web browser crapping out has nothing to do with networks, or network
operations. I'm not aware of any network of any consequence where the
people who run, design, or build the infrastructure have any relationship
to end user tech support call centers. I'm sure there are many fines
places where this particular issue is great on-topic discussion, but since
as Gadi said it not only has nothing to do with BGP but nothing to do with
networks at all, this just isn't it.
To the people who say we throw in the towel and just say "Gadi will never
stop posting off-topic crap, so why bother trying to correct him?", I'd
suggest that this is a self-defeating attitude. Not only because Gadi
could actually be posting useful stuff if set on the right path as to what
is appropriate and what is not, but because 10,000 other people are going
to be reading that post and thinking that this is appropriate subject
matter. One off-topic post you can delete, but an entire list which has
been co-opted by off-topic material can not be fixed.
Unless we're ready to admit that NANOG is completely and totally worthless
as a forum for discussing network operations, people NEED to step up and
take responsibility for the "self policing" that we're all supposed to be
doing in srh's absence.