Any validatity to this and if so I am suprised that our team has got no calls on not be able to get to certain websites.
Elijah Savage wrote:
Any validatity to this and if so I am suprised that our team has got no calls on not be able to get to certain websites.
WOW trying to do to many things at once. What a horrible email LOL.
Any validity to this? Because I am suprised that we have not received any phone calls/tickets of customers complaining that they can't get to any of these domains.
LOL
Any validatity to this and if so I am suprised that our team has
got no calls on not be able to get to certain websites.
http://webhostingtalk.com/showthread.php?t=477562
I for one applaud godaddy's response. If more piddling "Hosting
Providers" with "Datacenters" got turned off when they started
spewing abusive traffic, the net would be a much nicer place.
Whoever the heck "nectartech" is, I guess they might act a little
more responsibly in the future. Or, more probably, they'll just
change to another DNS registrar who doesn't care as much about
abuse.
matto
--matt@snark.net------------------------------------------<darwin><
The only thing necessary for the triumph
of evil is for good men to do nothing. - Edmund Burke
I think the main thing I learned from that is that there are a surprising number of hosting companies and self-professed data centre operators who really don't know much about the DNS.
Joe
I think the main thing I learned from that is that there are a surprising number of hosting companies and self-professed data centre operators who really don't know much about the DNS.
Or even what the word "datacenter" means. Sounds to me like a rack of servers or a cage was suspended, not "an entire datacenter" which was claimed several times.
The recorded phone call was basically a lesson in how NOT to escalate a call, from both sides involved. From the customer's side if he'd not been so confrontational, he probably would have gotten his problem solved. From the operator's side, they should have a procedure for dealing with abuse and critical escalations 7/24.
Just my perception.
--chuck
I'm astonished GoDaddy pulled anyone for spamming. Isn't spamming the
whole point of GoDaddy, what with its content-free WHOIS records,
integrated no-name domain registry and hosting division? In fact, I
would go so far as to say taking out entire GoDaddy would probably be
a small increase in the amount of useful information on the Net..
FYI, Nectartech is a small hosting shop out of 55 S Market in San Jose. I
wouldn't describe them as a "datacenter", since I don't think they own or
operate any facilities.
Perhaps if they ever managed to find "the command to make two routers talk
to each other and be redundant" (a real quote from what has been loosely
described as their network admin, I'm not kidding, you can't make stuff
like this up :P), their next step might be to find the command to make dns
servers talk to each other and be redundant.
Reality check time, what we have here is a small hosting shop with a long
history of shady customers. I doubt GoDaddy nukes nameservers on a whim,
my money is that there was a lot of abuse which went on for a long time
without getting any response. Its amazing how quickly some people who
don't respond or address abuse issues at all when you're asking nicely
will appear and take care of things once you turn them off. The rest is
just some random blowhard web hosting customer who gets off on being an
ass and blaming everyone but himself and his choice in hosting companies.
Hardly an uncommon sight. 
Richard,
On the other hand �, I'm not comfortable with the idea that an organization
that provides network infrastructure services under the aegis of the US
Government could unilaterally revoke those services for something that is
not illegal.
By all means, the Justice Dept. and police should move against anyone
performing illegal acts such as phishing, I just don't think that it is
ICANN or ARIN and GoDaddy's job to police good net citizenship.
Joe
FYI, Nectartech is a small hosting shop out of 55 S Market in San Jose. I
wouldn't describe them as a "datacenter", since I don't think they own or
operate any facilities.
Heh, I used to work at a small hosting shop out of 55 S. Market- it was (then) called BBN Planet. I guess these schmoes rent a cage from Genuity (or whatever they are called now).
Perhaps if they ever managed to find "the command to make two routers talk
to each other and be redundant" (a real quote from what has been loosely
described as their network admin, I'm not kidding, you can't make stuff
like this up :P), their next step might be to find the command to make dns
servers talk to each other and be redundant.
Seriously. You need to be spewing a lot of cak onto the net for your _domain registrar_ to take notice.
The rest is just some random blowhard web hosting customer who
gets off on being an ass and blaming everyone but himself and his choice in hosting companies.
Hardly an uncommon sight.
The priceless part is that we probably never would have noticed, had he not had the hubris to record the conversations, and then publish the URL to them. I love it when the lusers are nice enough to clearly identify themselves.
matto
--matt@snark.net------------------------------------------<darwin><
The only thing necessary for the triumph
of evil is for good men to do nothing. - Edmund Burke
Richard,
On the other hand �, I'm not comfortable with the idea that an organization
that provides network infrastructure services under the aegis of the US
Government could unilaterally revoke those services for something that is
not illegal.
It does not have to be illegal. All that is necessary is that customer
who purchased the service beware and agree to the policies prior to making the purchase (of course, almost nobody fully reads that long
agreement you get presented on the website, but that's another story...)
Not being somebody who've ever used godaddy's services, I'm just speculating based on various reports, but I think their registration
service agreement is more extensive then domain registration agreement from most other registrars and prohibits use of the domain in connection with spamming as well as in connection with illegal activities.
If policies are violated then domain maybe suspended until problem is resolved. I suspect they don't suspend right away and have system of requiring domain owner be available for notification and conversation
in case such use (prohibited by their service agreement) is reported.
If they do not hear anything about it and reports continue then they
take action as allowed by domain registration agreement.
What we probably saw is such action after nectartech failed to respond
to several notifications and probably kept server running without
fully cleaning it up and possibly more then one of their servers was hacked too. This is similar enough situation to what may happen when
you run servers on the connection purchased from your ISP and that
ISP actually takes abuse reports seriously and has working abuse
department that follows up on what is sent them.
That this was spinned around as datacenter shutdown on WHT and even
got here is a result of both how nectartech wanted itself seen and
who they had for dealing with such vendor actions.
Casting blame may be a fun exercise. Listening to others cast blame gets old fast. The more useful question here is whether there are lessons the rest of us can learn from this incident.
The most important lesson is probably that your problems will almost always be more important to you than to somebody else. If you end up with a business killing problem, it doesn't matter if it's somebody else's fault -- you're the one who will be out of business. Likewise, you shouldn't go wandering out into heavy traffic just because the drivers are required by law to stop for you.
Choosing your vendors carefully is important. Having a backup plan for what to do if your vendors fail you is a good thing, but it's nice not to have to use the backup plan. Likewise, if something is really important to you, make sure your vendors know that. Nobody wants to suddenly find out in the middle of the night that they're responsible for something critical.
Knowing what's important to you in advance can help you figure out what arrangements need to be made. If your hosting operation won't run without power, Internet connectivity, and DNS, making sure your power, connectivity, and DNS are robust matters a lot. If your business can continue to operate for a few days without toner for your laser printer, choosing a less reliable toner supplier is probably ok.
If you do need to call your vendors, having a clear explanation of what's going on is often a good thing. "An entire datacenter" is an awfully vague term. If that were all of, say, Equinix Ashburn, it would be a big enough deal that government regulators would probably be concerned. But a room in the back of somebody's office with a rack of servers in it could also be justifiably called a "datacenter" (and a rack of servers in the back of somebody's office could also be important to somebody). It's probably better to be able to say, "x number of domains are down, representing y amount of revenue for our company and z critical service that the rest of the Internet relys on. This might put us out of business." This still may not get the desired response -- it's not your vendor who is going to be put out of business -- but it at least gives the person on the other end of the phone call some idea of what they're dealing with.
Protecting everything you've decided is important may be expensive. It may not be worth the cost. It's best to have made that calculation before the problem starts, when there's still time to spend money on protection if you do decide it's worth it.
Not having all your DNS servers in the same domain, or registered through the same registrar, isn't a "best practice" that has previously occurred to me, but it makes a lot of sense now that I think about it. Looking at the big TLDs, .com and .net have all their servers in the gtld-servers.net domain, but Verisign controls .net and can presumably fix gtld-servers.net if it breaks. UltraDNS has their TLD servers (for .org and others) in several different TLDs. Maybe that is to protect against this sort of thing.
And there's a PR lesson here, too. I'd never heard of Nectartech before this, and I'm guessing that's the case for a lot of NANOG readers. Having heard this story, I'd be hesitant to register a domain with GoDaddy, and that was presumably the goal. But I'd be hesitant to rely on a company with a name like GoDaddy anyway, just because of the name. Now that I've heard of Nectartech, I know them as the company that had the outage. That's not exactly a selling point.
I've certainly got sympathy for Mr. Perkel. I've learned a lot of the lessons above the hard way, some due to my own miscalculations and some due to working for companies that didn't value my time and stress levels as highly as I would have liked (choosing your employers carefully is important too...).
These lessons don't apply just to networking. The loss prevention department of a bank once locked my account for "suspicious activity" on a Friday afternoon and then left for the weekend. I had two dollars in my wallet, and didn't have much food. Escalating as far as I could through the ranks of people working the bank's customer service lines on Friday evening, I didn't manage to find anybody who didn't think I should just wait until Monday. Multiple accounts at different banks, neither of which is the bank that locked my account, now seem like a very good idea.
-Steve
I want to say, from an outsider's perspective, that I whole heartily applaud GoDaddy on the actions they took and the consistent professionalism exhibited by their tech support representative. Despite obvious (and heavily edited) calls to the same agent, the consumer was informed in a professional manner of his/her avenue for resolution. No doubt remains in my mind that the caller was not caught blind by this situation. Go Daddy has a privacy policy that no doubt prohibits them from releasing details of their side of this case, however to me the recording suggests that the caller knew this was the end result, not a sudden surprise move, and they just wanted to circumvent standard proceedure. The caller's prior thought to record, what appears as a standard call to tech-support, is insightful and should be an obvious sign of his motivation.
Let me explain my perspective. I am a long standing customer of data center services, and I fully appreciate network operators' efforts to stem the spread of spam and viruses. I run a few non-profit public mailing lists and the emails from my systems traverse your networks hourly. I work quikly and diligently with service providers to overcome issues where our paths cross. I have never been a Go Daddy customer, but I certainly appreciate their stand on this issue. I will probably never be a Nectartech customer after this episode.
-Jim P.
There seems to be a wide split on this topic. I was wondering if people would privately tell me yes or no on a few questions so I can understand the issue better.
1) Do you think it is acceptable to cause any collateral damage to innocent bystanders if it will stop network abuse?
2) If yes, do you still think it is acceptable to take down 100s of innocent bystanders because one customer of a provider is misbehaving?
3) If yes, do you still think it is acceptable if the "misbehaving" customer is not intentionally misbehaving - i.e. they've been hacked?
3) If yes, do you still think it is acceptable if the collateral damage (taking out 100s of innocent businesses) doesn't actually stop the spam run / DoS attack / etc.?
These are important question to me, and I'm surprised at the number of people who seem to feel so very differently than I thought they would feel - than I personally feel. Would people mind sending me private e-mails with yes/no answers? Longer answers are welcome, but yes/no will do.
Using the case under discussion as an example, I am wondering why anyone thinks taking down 100s of innocent domains is a good way to stop a single hacked machine from doing whatever it is doing? If you somehow think all that is worth it, take a close look at your cost / benefit analysis. At this rate, every business on the Internet will be out of business before we take out even a single moderately large botnet.
I am also wondering why anyone thinks the miscreant will stop just because the legitimate owner's domain no longer resolves? Not only is the machine likely to continue sending spam as if nothing happened, we aren't even "catching" the guy. I guess you could say "well, it put pressure on his hosting provider to clean the infected machine", which is true. I just think that's a bit silly. But maybe I'm the one who's silly.
Lastly, I wonder what "average" people - people who run businesses on hosting providers who really don't understand all this computer stuff - think about such actions. How many 100s of people have we just alienated for life to stop - er, NOT stop - a single zombie? And how many of their friends are going to hear over an over how the Internet is not a real business and no one should put any faith in it?
Is this really a good thing?
[jim, please wrap your text!]
I have never been a Go Daddy customer, but I certainly appreciate
their stand on this issue. I will probably never be a Nectartech customer after this episode.
Hear Hear.
After reading the GoDaddy domain registration legal agreement, available at:
https://www.godaddy.com/gdshop/legal_agreements/show_doc.asp?se=%2B&ci=1839&pageid=REG_SA
especially section 7, "Restriction of Services, Right of Refusal", I have to give them a big thumbs up.
It is good to see that wielding a Big Stick, and actively working for the Good Guys has not hindered GoDaddy from achieving quite a bit of success in the market.
matto
--matt@snark.net------------------------------------------<darwin><
The only thing necessary for the triumph
of evil is for good men to do nothing. - Edmund Burke
I don't think anyone (well ok, anyone sane, I know we have a few nutjobs
on this list :P) thinks that arbitrarily blocking service to hundreds or
thousands of users because someone is unknowingly hacked is an appropriate
way to address network abuse. I really have no idea how aggressive GoDaddy
is with enforcing their AUP, as I don't personally use their services, but
based on what I know about the affected customer and what I can read from
the affected whiner's website I'm certainly not going to jump to the
conclusion that GoDaddy is running around like a hopped up abuse desk
worker on a power trip, shutting off service to random innocent people
because they feel like it.
The question at hand is, at what point does a registrar providing services
have an ethical or moral obligation to step in and do something when they
do encounter an excessive level of abuse by someone using their services?
At what point does ARIN revoke the allocation of a blatant and persistant
spammer who is violating the law without being stopped? I think the answer
is that clearly this isn't something they want to be doing on a regular
basis, any more than an ISP wants to be responsible for filtering every
packet that goes through their routers looking for warez and kiddie porn,
yet I have seen them do it in certain rare and severe cases of unrelenting
abuse.
Maybe it is a judgement call, maybe it isn't. Bottom line, dealing with
abuse is an ass job, and I certainly wouldn't want it. Some days you're
doing a good thing because you shut down a spammer, some days you're doing
a bad thing because you shut down innocent services along with it (and
some days you're just fending off "stop hax0ring me on port 80 or I'll sue
you and call the CIA" e-mails).
I highly suspect that GoDaddy doesn't involve itself in these kinds of
issues lightly, which means that in all likelihood the level of abuse was
severe, with no communication from the person they suspended service to. I
for one have never heard of anyone I know having their GoDaddy service
suspended for this kind of thing. Unless someone has some actual facts
that GoDaddy is engaging in this kind of activity, I'm inclined to give
them the benefit of the doubt. This means, at least for now lumping them
in the "respecting them for taking a stand regarding the abuse of their
service" category, rather than the "wackjob conspiracy theorist
power-crazed zealot" category we all know and love.
I think the general consensus in the DNS field is that for security reasons it
is preferable to have as small a set of DNS servers (or perhaps as small as
set of differently configured servers! Hmm physical security....) in the
hierarchy above you as possible, since compromise of any of these could
affect the results obtained for your domain.
See also DJBs "Trusted Servers" note.
http://cr.yp.to/djbdns/notes.html
Here there is a clear conflict between security through redundancy against
accident, and resistant to compromise. Although it can be mitigated by
choosing well managed parents zones.
Incidently we have DNS servers in two domains, but that is historical, and
both top level domains are managed by Verisign, and delivered via the same
set of servers. Thus we are dependent on "root-servers.net",
"gltd-servers.net" and our own servers, only in the resolution of our own
domain names (and customer domains, where those domains are in .com/.net).
Of course arguably the effective working of some services (email?) are now
also dependent on reverse DNS working well, and the delegation of that is
different again.
That said I think the idea is sound against some issues (at which point one
should probably also use different providers for the DNS registration
services, since if their procedures are flawed....). However it does increase
the risk of certain types of malicious activity, as in general it is
sufficent to compromise one DNS server involved in serving a name to
compromise the majority of the traffic (at least in theory, I haven't had a
chance to prove this in anger yet).
Since we are moving a couple of our nameservers from their current domain, I
think I'll look at putting them under co.uk, as the UK seems to have tidied
up its DNS management quite nicely in recent years.
Also during recent event it has struck me that the hierarchy of servers
involved in providing DNS services is quite small, and has quite different
characteristics to the other records in the DNS. I'm beginning to wonder if
having the scaffolding in the protocol itself is the right way, but that is a
debate that has raged before, and is off topic here.
Theres a clear case of he said they said going on with this case. Nectartech is making claims that they fixed the issue. Also note that the caller is not a Nectartech employee at all. He's a customer who's also friends with the owner. Atleast that's what he says in WHT thread. In any event I don't think Nectartech handled this very well, and more likely than not still had a problem and were given ample time to properly correct it.
Patrick W. Gilmore wrote:
I want to say, from an outsider's perspective, that I whole heartily applaud GoDaddy on the actions they took [...]
There seems to be a wide split on this topic. I was wondering if people would privately tell me yes or no on a few questions so I can understand the issue better.
1) Do you think it is acceptable to cause any collateral damage to innocent bystanders if it will stop network abuse?
In some cases. Our policy is to minimize such. Example: Customer has a NATted network with multiple machines sharing one global address. One of the machines at customer's premise is causing abuse (virus, etc.) Null-routing one specific IP address will cause collateral damage to the non-infected machines at that customer, but I think most of here would agree that such is justified. Obviously, if the impact of the abuse is minimal, having the customer fix the problem before shutting anything down is preferred. Another example would be a customer's webserver which has many name-based virtual hosts, one of which is abusive, and you are providing IP connectivity. By null-routing one IP you are causing collateral damage to the non-abusive virtual host customers of your customer, but I think most would think that justified.
2) If yes, do you still think it is acceptable to take down 100s of innocent bystanders because one customer of a provider is misbehaving?
I assume here that you mean "Customer of a customer". Again, it depends. If the customer has continual problems controlling abuse from his customers, or you suspect that your customer is playing "whack-a-mole", or the abuse is ongoing and/or serious and you can't identify which of customer's customers is the cause (spoofed source addresses, etc.) in some cases yes.
3) If yes, do you still think it is acceptable if the "misbehaving" customer is not intentionally misbehaving - i.e. they've been hacked?
Again, it depends on the seriousness of the abuse and its affect on the network, as well as the frequency thereof and the seriousness of the customer in rectifying the problem. Also whether you can reasonably isolate the abuse and disconnect only the customer's abusive customer.
3) If yes, do you still think it is acceptable if the collateral damage (taking out 100s of innocent businesses) doesn't actually stop the spam run / DoS attack / etc.?
If it doesn't stop it but stops your network from being a part of it, yes. If it has no affect on it at all, then you're probably pulling the wrong plug.
These are important question to me, and I'm surprised at the number of people who seem to feel so very differently than I thought they would feel - than I personally feel. Would people mind sending me private e-mails with yes/no answers? Longer answers are welcome, but yes/no will do.
This is IMHO operational, so posting publicly. I don't think this is as black-and-white as to warrant simple yes-no answers. There are policies involved as well as your agreements with your peers/upstreams. If the issue is serious enough that you risk losing your own connectivity because you can't stem the abuse from a customer's customer, then you may need to do so, or the end result will be that you become part of greater collateral damage.
Using the case under discussion as an example, I am wondering why anyone thinks taking down 100s of innocent domains is a good way to stop a single hacked machine from doing whatever it is doing? If you somehow think all that is worth it, take a close look at your cost / benefit analysis. At this rate, every business on the Internet will be out of business before we take out even a single moderately large botnet.
The present example seems to be a combination of poor communication, bad attitude and sloppy network design from what I've seen here. It's unclear to me exactly what GoDaddy shut down, and the only data points we have to go on are admittedly edited conversations that took place after the plug was pulled. What went on beforehand? Did Nectar indeed make a good faith effort to correct the original problem? Was their attitude the same as shown on the phone calls? How long had the problem existed, had it happened before, and did Nectar keep an open dialogue as to the steps they were taking to fix it? Did GoDaddy have less intrusive options to shut down just the abuser?
I am also wondering why anyone thinks the miscreant will stop just because the legitimate owner's domain no longer resolves? Not only is the machine likely to continue sending spam as if nothing happened, we aren't even "catching" the guy. I guess you could say "well, it put pressure on his hosting provider to clean the infected machine", which is true. I just think that's a bit silly. But maybe I'm the one who's silly.
I think this was a case of a fake phishing website rather than outgoing spam spew. If the domain was the target of a phish, then causing it not to resolve would keep the phisher from reaping any benefit from the abuse although the spam run would likely continue, at least for a while until the phisher realizes it is in vain.
Lastly, I wonder what "average" people - people who run businesses on hosting providers who really don't understand all this computer stuff - think about such actions. How many 100s of people have we just alienated for life to stop - er, NOT stop - a single zombie? And how many of their friends are going to hear over an over how the Internet is not a real business and no one should put any faith in it?
Well, "average" people who run businesses on hosting providers" probably should hire someone who does understand all this computer stuff to do some due diligence on the providers they are considering. If their prospective providers netblocks are repeatedly mentioned in SPEWS, Spamhaus, Spamcop, and NANAE, they may want to look elsewhere.
Googling "Nectartech abuse" is interesting. As far back as July of last year they were battling GoDaddy over spam and abuse issues. It doesn't look like this should have been all that big of a surprise. In fact, Nectartech's predictions in post 23 of the following thread are eerily accurate.
http://www.webhostingtalk.com/showthread.php?s=&threadid=422612
Is this really a good thing?
If steps are taken to minimize collateral damage, yes. Allowing the abuse to continue causes collateral damage to the rest of the Internet for as long as it continues. The choice often boils down to severe collateral damage to a few or raising the noise level and collateral damage to the Internet as a whole. Is cutting off ten customers of an infected customer better than allowing this customer's virus to infect tens of thousands of random hosts on the net worth it? If you're one of the tens of thousands, yes. If you're one of the ten customers, no.
I think the issue here is not so much what happened, but how it
happened. The phishing problem was originally reported to godaddy and
then passed on to nectar on 1/9 (a Monday). It also appears the nectar
folks resolved the problem on the same day. After that point godaddy
continued to receive complains about the same problem and rather than
checking to see if the problem still existed, they just assumed it did.
Nectar appears to have even responded to godaddy stating that the
problem had already been resolved long before service was cut.
IMHO the big issue is that service was cut on a Friday night just as the
only folks empowered to resolve the situation have left for the weekend.
I can see cutting service during a weekday morning to get the client's
attention on the matter. Doing it at a time when you know you'll be
causing a long term outage is just plain nasty.
HTH,
Chris