Apparently it took authorize.net with it, too:
http://twitter.com/authorizenet~Seth
No technical explanation of course but it also took down
their 'backup facility' according to them on twitter;
I assume some bad routing/DNS if they do actually have
a backup facility. Lots of online stores are
offline right now because of this, and the holiday is
unfortunately keeping those store owners from knowing
they are not making sales right now. Life in ecommerce...
David
This begs the question of what basic parameters should be for a "carrier
hotel" or co-lo.
Given that we're getting designated "Critical Infrastructure", we'd
getter start coming up with some, or we'll have them defined for us.
The old NEBS standards were too much of a straightjacket, but the
current situation, where any buffoon who wants to can claim to be
something they aren't (redundant and reliable) undermines the business
of those who actually spend the money, and make the effort, to provide a
true "carrier grade" co-lo.
This is life in the current Internet: Overpromise, and Underdeliver.
Tomas L. Byrnes wrote:
This begs the question of what basic parameters should be for a "carrier
hotel" or co-lo.Given that we're getting designated "Critical Infrastructure", we'd
getter start coming up with some, or we'll have them defined for us.The old NEBS standards were too much of a straightjacket, but the
current situation, where any buffoon who wants to can claim to be
something they aren't (redundant and reliable) undermines the business
of those who actually spend the money, and make the effort, to provide a
true "carrier grade" co-lo.
Absolutely. Then your pricing is so far out of whack with the apparent
competition that it's hard to get customers when it appears one can get
the same/better for far less. Me, personally, I just don't say things
like "100% uptime" or claim to be a carrier-grade facility. But I think
that scares people off when my competitors (and I've seen the insides of
some of the horrid trash heaps they call a NOC) claim they do.
This is life in the current Internet: Overpromise, and Underdeliver.
"Our flywheel systems are so failure-proof and thinking outside the box
that we don't need a silly battery UPS that can cold-start!"
I know outages and related discussion end up attracting the off-topic
hammer here on NANOG, but I do find them interesting and worthwhile.
~Seth
This begs the question of what basic parameters should be for a "carrier
hotel" or co-lo.
Given that we're getting designated "Critical Infrastructure", we'd
getter start coming up with some, or we'll have them defined for us.
Tomas,
There is a useful standard: ANSI/TIA-942. It offers specifications for
four tiers of data centers ranging from tier 1 (a basic data center
with no redundancy) to tier 4 (fully fault tolerant).
http://www.tiaonline.org/standards/catalog/search.cfm?standards_criteria=TIA-942
(the 2005 one)
Judging from http://www.techlinks.net/community/articles/article/1-article-submission-forms/14833-a-quick-primer-on-data-center-tier-classifications
there's even research that projects what sort of annual downtime you
can expect for each of the tiers described by the standard.
When I walk into a data center, I make a habit of asking which tier
they achieve, at least for the HVAC and electrical systems. And then I
ask to see the components which the tier claim says they should have.
Regards,
Bill Herrin
Are you better off with a single "tier 4" data center, multiple
"tier 1" data centers, or something in between?
Distance and quantity versus complexity and scaling versus cost and risk. Sometimes no matter what you choose, you might be wrong.
Earth is a single point of failure, where is your backup site?
Earth is a single point of failure, where is your backup site?
[TLB:] Given that all my customers are on Earth, I don't need one if my
customers also are "down".
Wasn't Authorize.net affected by this? We received a support ticket
about why Authorize.net is down today (I don't know either, I don't
ask too many questions).
Jeff
Yes it was.
In a message written on Fri, Jul 03, 2009 at 03:22:14PM -0400, Sean Donelan wrote:
Are you better off with a single "tier 4" data center, multiple
"tier 1" data centers, or something in between?
It depends entirely on your dependency on connectivity.
One extreme is something like a Central Office. Lots of cables
from end-sites terminate in the building. Having a duplicate of
the head end termination equipment on the opposite coast is darn
near useless. If the building goes down, the users going through
it go down. "Tier 4" is probably a good idea.
The other extreme is a pure content play (YouTube, Google Search).
Users don't care which data center they hit (within reason), and
indeed often don't know. You're better off having data centers
spread out all over, both so you're more likely to only loose one
at a time, but also so that the loss of one is relatively unimportant.
Once you're already in this architecture, Tier 1 is generally
cheaper.
There are two problems though. First, most folks don't fit neatly
in one of these buckets. They have some ties to local infrastructure,
and some items which are not tied. Latency as a performance penality
is very subjective. A backup 1000 miles away is fine for many
things, and very bad for some things.
Second, most folks don't have choices. It would be nice if most
cities had three each Tier 1, 2, 3 and 4 data centers available so
there was choice and competition but that's rare.
Very few companies consider these choices rationally; often because
choices are made by different groups. I am amazed how many times
inside of an ISP the folks deploying the DNS and mail servers are
firewalled from the folks deploying the network, to the point where
you have to get to the President to reach common management. This
leads to them making choices in opposite directions that end up
costing extra money the company, and often resulting in a much lower
uptimes than expected. Having the network group deploy a single point
of failure to the "Tier 4" data center the server guys required is,
well, silly.
However, more important than all of this is testing your infrastructure.
Would you feel comfortable walking into your data center and ripping
the power cable out of some bit of equipment at random _right now_?
If not, you have no faith your equipment will work in an outage.
Power to some of the affected sections of the building has been restored via
existing onsite generators. The central power risers cannot be connected to
current generators in a timely manner due to excessive damage to the
electrical switching equipment (and those generators may still be in
standing water). These provide power to a number of colocated systems.
Temporary generators are on order to be connected to the central risers,
and the site expects that to be complete sometime late this evening. As
best I can tell, there is still no utility power connected to any of the
systems.
The AC systems (chiller and crac) are currently not working. It is not
clear to me whether these will be brought back on line when the temporary
generators are available, but I am assuming so.
It was pleasant to see the general positive attitude, sharing of information
and offers of assistance that were made by representatives of the various
tenants, customers and carriers that were on the scene. The usual suspects
(companies and individuals) stepped up and took care of things, as they
always seem to.
--D
Yes it was.
Wasn't Authorize.net affected by this? We received a support ticket
about why Authorize.net is down today (I don't know either, I don't
ask too many questions).
Authorize.net was for a while completely off the air, and companies that relied upon them
were not getting credit card authorizations (and, thus, no ecommerce). I think it is still only
partially functional.
Authorize.net has been communicating with customers mostly (entirely ?) with twitter - they are
@AuthorizeNet with a hash tab of #authorizenet
If you go there, you will see a lot of status messages like
#authorizenet (cont.) Do not manually submit ARB transactions b/c you run the risk of your merchants being double billed.
10 minutes ago from web
(i.e., 4:47 EDT).
You will also see a lot of posts from annoyed people if you search on #authorizenet
It's an interesting use of Web 2.0 for emergency communications.
Regards
Marshall
That's a great idea, use some lame Web 2.0 trend to communicate with
actual real life customers. </sarcasm>
Jeff
Bad Day !
That's a great idea, use some lame Web 2.0 trend to communicate with
actual real life customers. </sarcasm>
I would assume they figured it was better than just remaining silent.
Regards
Marshall
Jeff
Yes it was.
Wasn't Authorize.net affected by this? We received a support ticket
about why Authorize.net is down today (I don't know either, I don't
ask too many questions).Authorize.net was for a while completely off the air, and companies that
relied upon them
were not getting credit card authorizations (and, thus, no ecommerce). I
think it is still only
partially functional.Authorize.net has been communicating with customers mostly (entirely ?) with
twitter - they are@AuthorizeNet with a hash tab of #authorizenet
If you go there, you will see a lot of status messages like
#authorizenet (cont.) Do not manually submit ARB transactions b/c you run
the risk of your merchants being double billed.
10 minutes ago from web(i.e., 4:47 EDT).
You will also see a lot of posts from annoyed people if you search on
#authorizenetIt's an interesting use of Web 2.0 for emergency communications.
Regards
MarshallJeff
Earth is a single point of failure, where is your backup site?
[TLB:] Given that all my customers are on Earth, I don't need one if my
customers also are "down".--
Jeffrey Lyon, Leadership Team
jeffrey.lyon@blacklotus.net | http://www.blacklotus.net
Black Lotus Communications of The IRC Company, Inc.Look for us at HostingCon 2009 in Washington, DC on August 10th - 12th
at Booth #401.--
Jeffrey Lyon, Leadership Team
jeffrey.lyon@blacklotus.net | http://www.blacklotus.net
Black Lotus Communications of The IRC Company, Inc.Look for us at HostingCon 2009 in Washington, DC on August 10th - 12th
at Booth #401.
Regards
Marshall Eubanks
CEO / AmericaFree.TV
In article <786BA8C0-B534-40FF-9126-1E33BD11CB3C@americafree.tv>, Marshall Eubanks <tme@americafree.tv> writes
That's a great idea, use some lame Web 2.0 trend to communicate with
actual real life customers. </sarcasm>I would assume they figured it was better than just remaining silent.
I'm about to recommend to an organisation that it [a twitter account] is better than posting news of an outage on their low-volume website, which will get swamped when too many people poll it for news.
What does the team think?
Paying a lot more to host the website with higher "burst" capacity during an emergency, isn't an option.
The only other idea I've had is to sign all the customers up to receive an SMS via some sort of broadcast service (the news will fit easily in one SMS).
In article <786BA8C0-B534-40FF-9126-1E33BD11CB3C@americafree.tv>, Marshall Eubanks <tme@americafree.tv> writes
That's a great idea, use some lame Web 2.0 trend to communicate with
actual real life customers. </sarcasm>I would assume they figured it was better than just remaining silent.
I'm about to recommend to an organisation that it [a twitter account] is better than posting news of an outage on their low-volume website, which will get swamped when too many people poll it for news.
What if the outage takes out their website too ?
I don't think that their website was up, and I would guess that they didn't have email either. That
is a bad situation to be in.
Note, BTW, that twitter itself is subject to frequent planned and unplanned outages.
Marshall
What does the team think?
Paying a lot more to host the website with higher "burst" capacity during an emergency, isn't an option.
The only other idea I've had is to sign all the customers up to receive an SMS via some sort of broadcast service (the news will fit easily in one SMS).
--
Roland Perry
Regards
Marshall Eubanks
CEO / AmericaFree.TV
Roland Perry wrote:
In article <786BA8C0-B534-40FF-9126-1E33BD11CB3C@americafree.tv>, Marshall Eubanks <tme@americafree.tv> writes
That's a great idea, use some lame Web 2.0 trend to communicate with
actual real life customers. </sarcasm>I would assume they figured it was better than just remaining silent.
I'm about to recommend to an organisation that it [a twitter account] is better than posting news of an outage on their low-volume website, which will get swamped when too many people poll it for news.
What does the team think?
I don't understand why this is an either/or question. Why not post to both?
Twitter is a great method of communication, for those that use twitter. But some people don't use twitter. So use every avenue you have. If you have a customer mailing list for announcements, send email. If you have a blog, post to the blog.
I'm in the process of setting up posterous to post to a blog, myspace, facebook, and twitter all with one email. Can't get much simpler than that for getting the word out via all the channels (assuming that outbound email is working). (Obviously some businesses don't want/need myspace or facebook, but if you use them, post there too.)
Quoting:
How does autoposting work?
Just set up your other accounts here. The next time you post to posterous, we will instantly autopost everywhere else.
Facebook profile newsfeeds will be updated each time you post to notify your friends. You can also autopost photos to your photo album and embed your blog directly in your profile.
Twitter messages will use the title of your post up to 130 characters, and then append a shortened post.ly url.
Flickr photos will be put automatically in your photostream. If you attach multiple photos, we'll post them all in the order we receive them.
Blogs will be updated with the full content you send us. We'll host your images, music and files, so you don't have to lift a finger.
You control where we post.
Just email the right address and we'll do the right thing.
Post Everywhere? post@posterous.com as usual
Twitter? twitter@posterous.com
Flickr? flickr@posterous.com
Facebook? facebook@posterous.com
Tumblr? tumblr@posterous.com
Any other blog? blog@posterous.com
Posterous only? posterous@posterous.com
Combine them! flickr+twitter@posterous.com
You can also address an email to #{text}@posterous.com and it will post to any site where the url contains that text.
#apple@posterous.com will go to apple.wordpress.com and flickr.com/apple, but NOT banana.blogspot.com.
jc
In article <F832A12A-0AED-4A01-955D-E24DCA6181C8@americafree.tv>, Marshall Eubanks <tme@americafree.tv> writes
That's a great idea, use some lame Web 2.0 trend to communicate with
actual real life customers. </sarcasm>I would assume they figured it was better than just remaining silent.
I'm about to recommend to an organisation that it [a twitter account] is better than posting news of an outage on their low- volume website, which will get swamped when too many people poll it for news.
What if the outage takes out their website too ?
The website is hosted elsewhere, however the entire message can be delivered in one Tweet, so there's no need to confirm by looking at a website.
I don't think that their website was up, and I would guess that they didn't have email either. That is a bad situation to be in.
They don't plan to respond to email in real time.
Note, BTW, that twitter itself is subject to frequent planned and unplanned outages.
The question being, how often will they co-incide with the events I'm trying to track?
fwiw, I've been using twitter for about three months now, and have never encountered either kind of outage.