That specific example depends on how order-dependent your consistency
constraint is; you can have time-asynchronous locally ACID changes
across databases which are widely separate. If your consistency
requires order synchronicity across the geographic DB cluster then
this is a potential epic fail, of course.
The general point is valid.
Being able to tell if your application *really* does require strict
consistency or not, and if it requires strict ordering or not if it
requires strict consistency, is unfortunately beyond most line-level
system designers. A lot of people guess wrong in both directions, and
either cripple the app's performance unnecessarily or end up with
dangerous failure modes inherent in the architecture.
"Physics"?
Isn't that an entirely inadequate substitute for "desire"?
Cheers,
-- jra
Not really. For some applications, it is physics:
1) You need two or more locations separated by say 500km for
disaster protection (think Katrina, or Japan Tsunami).
2) Those two locations need to be 100% consistent, with in-order
"serializable" ACID semantics for a particular database entity. An
example would be some sort of financial account - the order of
transactions against that account must be such that an account cannot
go below a certain value, and debits to and from different accounts
must always happen together or not at all.
The above implies a two-phase commit protocol. This, in turn, implies
*at least* two network round-trips. Given a perfect dedicated fiber
network and no switch/router/CPU/disk latency, this means at least
10.8 ms per transaction, or at most 92 transactions per second per
affected database entity. The reality of real networks, disks,
databases, and servers makes this perfect scenario unachievable -
often by an order of magnitude.
I don't have inside knowledge, but I suspect this is why Wall Street
firms have DR sites across the river in New Jersey, rather than
somewhere "safer".
Amazon's EBS service is network-based block storage, with semantics
similar to the financial account scenario: data writes to the volume
must happen in-order at all replicas. Which is why EBS volumes cannot
have a replica a great distance away from the primary. So any
application which used the EBS abstraction for keeping consistent
state were screwed during this Amazon outage. The fact that Amazon's
availability zones were not, in fact, very isolated from each other
for this particular failure scenario compounded the problem.
From: "Ryan Malayter" <malayter@gmail.com>
> > From: "Ryan Malayter" <malay...@gmail.com>
> > I like to bag on my developers for not knowing anything about the
> > infrastructure, but sometimes you just can't do it right because
> > of
> > physics. Or you can't do it right without writing your own OS,
> > networking stacks, file systems, etc., which means it is
> > essentially
> > "impossible" in the real world.
>
> "Physics"?
>
> Isn't that an entirely inadequate substitute for "desire"?
Not really. For some applications, it is physics:
You misinterpreted me. I was making fun of people who think "I want
it, and therefore it WILL be so" trumps physics, of whom there are
altogether too many in positions of power these days to suit me.
I don't have inside knowledge, but I suspect this is why Wall Street
firms have DR sites across the river in New Jersey, rather than
somewhere "safer".
You don't need inside knowledge; that issue's the subject of much
general press lately; that's exactly why they do it.
And they think it's good enough.
I truly wish that their finding out it's not wouldn't be so massively
disrupting for the rest of us poor slobs...
Amazon's EBS service is network-based block storage, with semantics
similar to the financial account scenario: data writes to the volume
must happen in-order at all replicas. Which is why EBS volumes cannot
have a replica a great distance away from the primary. So any
application which used the EBS abstraction for keeping consistent
state were screwed during this Amazon outage. The fact that Amazon's
availability zones were not, in fact, very isolated from each other
for this particular failure scenario compounded the problem.
Oh, so maybe "letting someone else do the cloud for you"'s a bad idea?
Whod'a thunk *that*? 
Cheers,
-- jra
I am preparing a graduate level course for network managers. As part of this course I would like to use a series of case studies looking at problems such as described in the report from Amazon. If anyone has something similar or knows where I could find such things, I would appreciate a copy or a link to it.
Kenneth M. Chipps Ph.D.