Out-of-band paging (was: Web expert ...)

Mailman List Mirror (Read Only)
1

> As wonderful as the new communications paradigms are, do we also

> > have a situation now developing where it might eventually become
> > very difficult or even impossible to ensure out-of-band lines of
> > communications remain available?
> >
>That's already a problem for getting alert pages. Any actual *pager*
>companies left? They all seem to have gone to SMS systems.

Yes, several, although SMS is a better strategy today as far as I can tell. Skytel (now Velocita Wireless) has a fine 2-way network, which we used until early last year. We switched over to Metrotel, which had a smaller form-factor unit w/o 2-way which was better for us, for about a year.

However, we have completely cut over to SMS for alert pages now. Multitech makes a nice little GSM modem that sits on a serial port on your alerting systems. I threw AT&T SIMs in them, wrote a tiny bit of glue to convert from email alerts to SMS alerts, and now we get all of our alerts using SMS. There's lots of open source code to handle the modems.

It's completely out-of-band, even more so than our old touch-tone-phone-paging system was, so I'm actually happier with the total performance. Given that GSM coverage is increasing while pager coverage seems static or decreasing, SMS via out-of-band GSM looks like a great solution.

Even situations like major power outages which would eventually take down cell towers w/o generators (or with malfunctioning generation) aren't a real concern because you usually have plenty of notice before the power goes all the way out...

jms

2

Be wary, there is a fast growing trend amongst mobile operators to outsource backhaul from their towers to IP network operators. So far there are only a few that are using the same network as for other IP traffic, but the economy of scale motivations to combine onto a single IP network are strong and will not be resisted for long.

3

I would definitely consider the direction that cell and SMS is moving to
be at-risk and probably effectively in-band during a communications
crisis. As I pointed out to someone else last night in private e-mail:

: [...] but TDM as a backhaul
: technology for cellular will eventually give way to all-IP based
: backhaul. The pressures in the cellular space are particularly intense
: with the "advanced"(*) IP services that networks such as at&t wireless
: are selling to customers. In some areas, data traffic already exceeds
: voice loads, and maintaining both TDM and IP backhaul for wildly varying
: loads effectively means ensuring excess capacity available on two
: different networks. TDM in particular may be viewed as wasteful; it's
: possible to get better network efficiencies out of SIP/IMS based voice
: processing.
:
: And then consider landlines.
:
: TDM is an expensive and inefficient technology, when you look at it from
: the point of view of cost to implement and maintain. If you're at&t and
: you're selling Uverse, for example, you're already encoding the POTS
: line as data to haul it over the copper/fiber to the customer. Does it
: make a lot of sense to maintain a local central office switch that's
: essentially a dinosaur, converting TDM to VoIP at the CO, just to justify
: the continued existence of a switch at the CO?
:
: Point is, TDM's goose is cooked. Your cell phone's going to wind up on
: the same IP network that your landline's going to be on, and that's also
: likely to have overlap with consumer Internet connectivity. It may not
: be that way today, or tomorrow, or next year, but let's be realistic, as
: efforts to cut costs are made, telcos are not going to see value for
: their dollar in maintaining completely separate networks, and they're
: going to touch.
:
: (*) "advanced" == "Internet access", we NANOG'ers consider it basic.

Please remember before anyone tries to "correct" me that I'm making
forward-looking statements about where things are likely to go, and not
just looking at the current state of the technology. I see mobile data
as being strong growth, and mobile devices becoming plentiful, but the
demand for mobile voice is not going to grow in the same ways. Just as
the early days of the Internet were dialup and low bandwidth sites, but
we transitioned to broadband and bandwidth-hungry sites that were made
possible as a result, we'll see a lot of that happen with wireless data
too.

What that really implies is that voice demand is going to remain more
or less constant when compared to the explosive growth of data; data
demand is going to grow, and carriers will get to the point where
they're running gigE to a cell tower. Right now, maybe voice is of
sufficient importance and data is sufficiently new and problematic
that there is some segregation internally of that traffic within the
carrier's networks, but even in the most optimistic case for network
segregation, I see it getting to the point where someone looks at the
picture in a few years and says, "we've already got 1Gbps data pipes
to our cell sites, why are we running voice over a separate 45Mbps
pipe?"

And as far as I can tell, that's happening a lot more quickly than many
people have expected.

I strongly agree with your conclusions about economy of scale motivations.

... JG

4

I agree with you & Brandon in terms of the directions: yes, your local access (and your tower access for GSM) will likely all be backhauled over the same unexpectedly attenuated piece of fiber, causing your alerts to be as silent as your dial tone.

But... you can take this sort of 'single point of failure' argument almost as far as you want. In the security business (where I spend most of my time), I see people do this a lot--they get deep into the ultra-ultra-ultra marginal risk, which takes then an enormous amount of money to mitigate. It's an easy rat hole to explore, and often fun.

Obviously, using SMTP-to-SMS-over-the-Internet to tell yourself that your SMTP infrastructure is hosed is the wrong answer.

On the other hand, triply-redundantly engineering things to deal with the outage of the fiber that connects your building, POTS, GSM, and everything else may not be the right answer. To some extent, there's the practical question of "if my entire city is disconnected, do I really need to know about it since I probably can't do anything about it?" (Yes, I know your help desk would want to know, but realistically...)

I guess my point is: yeah, Brandon, Joe, you're right. But, I've built the alerting solution that minimizes the risk I will miss an alert I care about while also minimizing my overall cost and minimizing the complexity of the alerting system. I'm happy to make it better, cheaper, more robust, etc., but I think it's important to balance these things. (I should also note, if anyone had any doubts, that I'm also one of those mom-and-pop ISPs, not Time-Warner or Verizon, so my concept of alerting is a bit different from someone who is trying to keep tabs on 1300 POPs in 40 countries...)

jms

5

In a message written on Wed, Jul 28, 2010 at 04:38:25PM +0200, Joel M Snyder wrote:

But... you can take this sort of 'single point of failure' argument
almost as far as you want. In the security business (where I spend most
of my time), I see people do this a lot--they get deep into the
ultra-ultra-ultra marginal risk, which takes then an enormous amount of
money to mitigate. It's an easy rat hole to explore, and often fun.

I agree worring about the cell site is not the worry.

However I suspect many of the folks relying on SMS have no idea how
it works inside the carrier. There are in fact other points of
failure that may be much more "single point". For instance your
SMS likely passes through a database in the carrier network (in
case your phone is off). That's redundant, right? Fully RAID'ed
and a hot standby spare and all that, after all it probably handles
SMS's for a few million customers.

Not always.

6

(view from the UK where SMS is very very prevalent)

TXT's can take ages to deliver (hours days not uncommon).

GSM networks can get put to emergency access only so they don't get swamped
when a civil emergency occurs and emergency workers need priority access to
mobile network. eg 7 July 2005 in London

7

Once upon a time, Joel M Snyder <Joel.Snyder@Opus1.COM> said:

Obviously, using SMTP-to-SMS-over-the-Internet to tell yourself that
your SMTP infrastructure is hosed is the wrong answer.

We even ran into this with paging and direct submission via TAP. We had
a POTS line not provisioned over fiber (so not the same physical layer
as our regular connectivity), used a modem on a computer with a
dedicated UPS, etc.

Then we realized that our local paging provider was connected to us for
Internet access and sent pages to towers outside the immediate area over
the Internet. Oops.

Now we use SMS and a GSM modem. Since the cell carriers don't buy any
access from us, we're at least somewhat better off.

8

I guess my point is: yeah, Brandon, Joe, you're right. But, I've built
the alerting solution that minimizes the risk I will miss an alert I
care about while also minimizing my overall cost and minimizing the
complexity of the alerting system. I'm happy to make it better,
cheaper, more robust, etc., but I think it's important to balance these
things. (I should also note, if anyone had any doubts, that I'm also
one of those mom-and-pop ISPs, not Time-Warner or Verizon, so my concept
of alerting is a bit different from someone who is trying to keep tabs
on 1300 POPs in 40 countries...)

I think my point's more along the lines of: don't expect to be able to
magically hand off a message to a service provider and expect that it
will be delivered; they have the same sorts of problems that you do,
and the way things are going, they may even be using the same
infrastructure that you are. That last bit in particular is worth
thinking about.

From my point of view, my ideal alerting system is probably something

like a smartphone running an app that's connected to the network
monitoring system, and can tell me:

1) when it has lost that connection, and

2) whatever problems the network monitoring system chooses to let me
   know about.

The old-timers would recognize this as one form of supervised circuit.

I don't really care about the possibility of lost messages so long as
I'm aware that I may not be "in touch". I'm perfectly capable of
sorting that situation out myself.

... JG

9

Joe Greco wrote:

From my point of view, my ideal alerting system is probably something

like a smartphone running an app that's connected to the network
monitoring system, and can tell me:

1) when it has lost that connection, and

2) whatever problems the network monitoring system chooses to let me
   know about.

I use the triple approach myself. Old fashioned TAP line, helpdesk notifications (they have plenty of methods of contacting me), and an out of band hard relay alarm that goes to the telco operators.

Some methods use direct circuits to neighboring town's fiber node, some things use the local town's fiber node, both taking different paths. It's extremely hard to get fully isolated. Monitoring server even has it's own separate UPS, though I really need to just throw an offsite redundant monitoring server up.

The app solution is one I actually believe to be the best method, but I'm a poor country folk and smart isn't exactly what I'd call this little phone.

Jack

10

I think people are getting lost in the weeds here, and confusing technologies with paths.

My current employer has been upgrading its transit circuits, and spent time in the last few months worrying about diversity of the transit paths. But we didn't insist that one provider come in via metro ethernet, one via SONET, and one via a GRE tunnel. What we did was have them bring in network maps, and make them sell us circuits that weren't running down the same streets as our other providers.

The same goes for your paging network. If it's running over IP, that's not a huge problem. If anything, if you're an IP engineer, it probably makes it easier for you to audit the setup. Where you do have a problem is if it's running over YOUR IP network, but that's just a more accute version of the problem you'd have if your paging company were using fiber along the same path as somebody you were buying fiber from.

So, for paging, or out of band management, or redundant capacity, the rules seem pretty simple. Buy from somebody who's not your customer. Audit whatever information you can get about their network paths to verify that they're not sharing segments with you. And, for good measure, have some backup plans in case the notifications don't work.

You probably are better off if you have humans in a NOC, rather than a purely automated alerting system. Those people can notice if you're not responding, and be creative. Maybe they can figure out how to fix problems themselves. If all else fails, they may be able to dispatch somebody to your house. Remember, organizations have been tracking down critical personnel for far longer than there have been telephones.

Or are people here worried about a scenario in which the entire world is run off of one big interconnected IP network, and that when it fails it's not only not possible to make a phone call, but also not possible to get across town to alert the people who could fix it? It seems to me that if things really got that bad, it might be pretty hard for even the most oblivious on-call person to miss.

-Steve