Diversity - was: Fiber cut in SF area

Hopefully none of these customers had service and protect ckts that went down... I would be pissed as a ceo if that happen to my company. Hopefully level3's new service offering is 100@percent redundant as stated

The new service offerings include: - Protected Wavelengths: Level 3 now provides automatic protection-switching to a dedicated diversely routed wavelength in the event of a network failure. The protection switch, fully automated and managed by Level 3, happens at switching speeds approaching SONET restoration times. The single interface to the customer requires no additional capital cost for customer optical ports, and the diverse restoration path is fixed and fully known to the customer. These features allow customers to achieve fast restoration with predictable performance in their network without adding significant cost and routing complexity. -

I'm considering use of AT&T / Verizon / Sprint WWAN services and the
Cisco 3G router interface cards/integrated module in C880 routers for
primary or backup WAN network connectivity for routers.

I'm looking for information from users of these services on the
following:

- addressing - Do these WWAN services use dynamic, PPPoE or static IP
assignment typically? Any of the 3? All?
   - is static IP assignment available?

- do these service providers use NAT within their network?

- How is the service reliability? In most cases, is the service
available for use when you need to use it?
- How is the service coverage area? Do you have problems getting
sufficient coverage in the deplouyment location to support desired
speeds (say 512kbps up/down as a minimum)?
- is ESP / IKE / IPsec permitted through un-rate-limited and un-molested
by the providers?
- If you build a IPsec/GRE tunnel over these services, do you have
frequent issues with the tunnel dropping, or a dynamic routing protocol
running through the tunnel going down frequently?

Also interested in similar information on impressions of similar EMEA
WWAN service providers, particularly Vodaphone and T-Mobile, if anyone
has experiences with these.

Replies on-list or off-list are welcome.... Your choice.

Cisco 3G interface and provider information:

http://www.cisco.com/en/US/products/ps7272/index.html

http://www.cisco.com/en/US/prod/routers/networking_solutions_products_ge
nericcontent0900aecd80601f7e.html#~north-america

Regards,

Sam Crooks

I've seen it with "static" public IP pppoe assignment.

No NAT.

Reliability? Best effort at best.

Coverage area is ok.

Speed and reliability is completely dependant on your location. Test first. Always. And then do not set a decent expectation.

IPSec tunnels dropping? Could be. Again, depends on your locations.

My overall impression? Get a T1.

My overall recommendation? Complete site surveys and then have a back out plan.

tv

Crooks, Sam wrote:

I'm considering use of AT&T / Verizon / Sprint WWAN services and the
Cisco 3G router interface cards/integrated module in C880 routers for
primary or backup WAN network connectivity for routers.

My comments are only for Sprint EVDO/1xRTT since that's what I use.

I'm looking for information from users of these services on the
following:

- addressing - Do these WWAN services use dynamic, PPPoE or static IP
assignment typically? Any of the 3? All?

My IP changes every time the session establishes.

   - is static IP assignment available?

I've never asked about static because there was no benefit to me when
other workarounds were available, i.e. DMVPN.

- do these service providers use NAT within their network?

Sprint doesn't, you get a public IP and I can establish inbound
connections. They seem to filter incoming port 80 though. I regularly
SSH to the wireless IP without any problems, although if the radio is
sleeping sometimes it takes two attempts.

- How is the service reliability? In most cases, is the service
available for use when you need to use it?

I've been using it for years with no complaints.

- How is the service coverage area? Do you have problems getting
sufficient coverage in the deplouyment location to support desired
speeds (say 512kbps up/down as a minimum)?

I get full EVDO rates. It's as reliable as any other CDMA phone I've
used in my area. Standard bad and good coverage areas apply. They will
do site surveys for you though, plus you can get fancy antennas for the
cards. I picked EVDO because it has a better upstream rate.

- is ESP / IKE / IPsec permitted through un-rate-limited and un-molested
by the providers?

As far as I can tell.

- If you build a IPsec/GRE tunnel over these services, do you have
frequent issues with the tunnel dropping, or a dynamic routing protocol
running through the tunnel going down frequently?

Sometimes latency sucks and timers will expire. It always recovers on
its own though.

Also interested in similar information on impressions of similar EMEA
WWAN service providers, particularly Vodaphone and T-Mobile, if anyone
has experiences with these.

Replies on-list or off-list are welcome.... Your choice.

Cisco 3G interface and provider information:

Cisco 3G Wireless WAN High-Speed WAN Interface Card - Cisco

http://www.cisco.com/en/US/prod/routers/networking_solutions_products_ge
nericcontent0900aecd80601f7e.html#~north-america

If uplink rates matter, for AT&T, you'll have to wait for the
HWIC-3G-HSPA-A to come out. If you want better than 384 up right now, go
EVDO Rev. A and make sure they do a site survey for you first. In the
end, it's just a fancy cell phone in your router.

~Seth

I agree do not commit without POC or trial bases.

Mike Goldman

Crooks, Sam wrote:

I'm considering use of AT&T / Verizon / Sprint WWAN services and the
Cisco 3G router interface cards/integrated module in C880 routers for
primary or backup WAN network connectivity for routers.

My comments are only for Sprint EVDO/1xRTT since that's what I use.

I use Sprint EVD0 and really like it. SSH, tunnels, etc. all seem to work fine. I have never tried to host a mail server on it,
though.

About once per month I get the same IP address if the session dies and I immediately restart it, but generally not.
They are public IP addresses.

I have heard that there is now a 5 GB per month cap, but I never got a notice of this and have never been capped.

My biggest complaint is that Sprint internally regards this as a phone, and so the automated services are
typically useless. There is nothing like spending 25 minutes on the phone dealing with some issue, only to be
told "the information you requested has been texted to your phone," when, as far as I can tell, I have no
way to receive such texting.

Regards
Marshall

That service is probably very expensive.

There is no known way to provide cheap 10 wave protection. Not carrier grade. Protected 10 GigE service (LAN PHY 10 GigE) will tolerate a very high BER before switching. And the cost of switching STM64 is very high as well.

Bottom line is that it will cost more than two diversely routed 10 gig waves.

There is no real market for protected 10 gig waves. Occasionally a bank will request the service, but backoff as soon as they see the price tag.

"Hopefully none of these customers had service and protect ckts that went down... I would be pissed as a ceo if that happen to my company. Hopefully level3's new service offering is 100@percent redundant as stated

The new service offerings include: - Protected Wavelengths: Level 3 now provides automatic protection-switching to a dedicated diversely routed wavelength in the event of a network failure. The protection switch, fully automated and managed by Level 3, happens at switching speeds approaching SONET restoration times. The single interface to the customer requires no additional capital cost for customer optical ports, and the diverse restoration path is fixed and fully known to the customer. These features allow customers to achieve fast restoration with predictable performance in their network without adding significant cost and routing complexity. -"

Roderick S. Beck
Director of European Sales
Hibernia Atlantic
13-15, rue Sedaine, 75011 Paris
http://www.hiberniaatlantic.com

That's funny, because our company is a (very small) LEC and a member of a
(small) regional network, and we've been asked by a larger consortium to
give them protected 10-Gig waves between two cities. It's not been a
problem to find DWDM vendors that can do that.

Frank

Rod Beck wrote:

That service is probably very expensive.

There is no known way to provide cheap 10 wave protection. Not carrier grade. Protected 10 GigE service (LAN PHY 10 GigE) will tolerate a very high BER before switching. And the cost of switching STM64 is very high as well.

Bottom line is that it will cost more than two diversely routed 10 gig waves.

There is no real market for protected 10 gig waves. Occasionally a bank will request the service, but backoff as soon as they see the price tag.

"Hopefully none of these customers had service and protect ckts that went down... I would be pissed as a ceo if that happen to my company. Hopefully level3's new service offering is 100@percent redundant as stated

The new service offerings include: - Protected Wavelengths: Level 3 now provides automatic protection-switching to a dedicated diversely routed wavelength in the event of a network failure. The protection switch, fully automated and managed by Level 3, happens at switching speeds approaching SONET restoration times. The single interface to the customer requires no additional capital cost for customer optical ports, and the diverse restoration path is fixed and fully known to the customer. These features allow customers to achieve fast restoration with predictable performance in their network without adding significant cost and routing complexity. -"
  
Surely a simple wideband optomechanical switch, actuated by detected signal degradation on a pilot wavelength or wavelengths, would do the job with high reliability and relatively low cost, without any extra need for switching the STM64 signal at the bitstream level?

-- Neil

Adjacent cities is not what the long haul providers generally do.

My clients want Chicago Equinix to Frankfurt Interxion or Chicago Equinix to 60 Hudson. Not Pittsburgh to Cleveland.

The capex for those services is many hundreds of thousands of dollars.

Consider all cards required to a provide a protected 10 gig wave service when you have substantial DWDM infrastructure. Not only regen huts, but the POPs in between the desired end points. We have lots of regen huts and POPs in between Chicago and NYC.

You can't built protection with only four 10 gig wave cards on most routes.

To take the point further, if you are building a TransAtlantic circuit, you're going to need cards at every landing station. If you have two landing stations on both sides of the Atlantic, then you are talking eight cards. Hmmm ...

Every span has to be protected.

And it doesn't make sense usually to be put in separate platforms to reduce the capex involved in those rings.

Roderick S. Beck
Director of European Sales
Hibernia Atlantic
13-15, rue Sedaine, 75011 Paris
http://www.hiberniaatlantic.com
Wireless: 1-212-444-8829.
French Landline: 33+1+4355+8224
French Wireless: 33-6-14-33-48-97.
AOL Messenger: GlobalBandwidth
rod.beck@hiberniaatlantic.com
rodbeck@erols.com
``Unthinking respect for authority is the greatest enemy of truth.'' Albert Einstein.

And if the 10 gig wave is from 1 Wilshire to 60 Hudson with hundreds of regen huts and 30 POPs in between?

How that affect the capex cost?

Roderick S. Beck
Director of European Sales
Hibernia Atlantic
13-15, rue Sedaine, 75011 Paris
http://www.hiberniaatlantic.com
Wireless: 1-212-444-8829.
French Landline: 33+1+4355+8224
French Wireless: 33-6-14-33-48-97.
AOL Messenger: GlobalBandwidth
rod.beck@hiberniaatlantic.com
rodbeck@erols.com
``Unthinking respect for authority is the greatest enemy of truth.'' Albert Einstein.

Rod Beck wrote:

And if the 10 gig wave is from 1 Wilshire to 60 Hudson with hundreds of regen huts and 30 POPs in between?

How that affect the capex cost?

Sure, the capex cost of offering full diversity is substantial; my point was just that the cost of switching STM64 signals at the endpiints need not be a significant issue, since you only have to switch the optical path, which is cheap to do and highly reliable, and the kit to do that will only make up a tiny fraction of the rest of the capital and operations cost.

-- Neil

Agreed.

But bear in mind that DWDM infrastructure that does 80 to 120 waves per fiber pair is very expensive.

REgards,

Roderick S. Beck
Director of European Sales
Hibernia Atlantic
13-15, rue Sedaine, 75011 Paris
http://www.hiberniaatlantic.com
Wireless: 1-212-444-8829.
French Landline: 33+1+4355+8224
French Wireless: 33-6-14-33-48-97.
AOL Messenger: GlobalBandwidth
rod.beck@hiberniaatlantic.com
rodbeck@erols.com
``Unthinking respect for authority is the greatest enemy of truth.'' Albert Einstein.

There is no known way to provide cheap 10 wave protection. Not carrier
grade. Protected 10 GigE service (LAN PHY 10 GigE) will tolerate a
very high BER before switching. And the cost of switching STM64 is
very high as well.

Bottom line is that it will cost more than two diversely routed 10 gig
waves.

...

Every span has to be protected.

Hi Rod,

I don't think thats true. Most "carrier grade" DWDM platforms deployed
over the last few years have been capable of doing protected 10GE LAN
PHY service without a SONET/STM layer and without costing more than two
diversely routed waves.

Also, many of the modern systems in use by modern competetive carriers
are capable of providing > 2 degree (ring) protection. They essentially
act like an optical "switch", and can automatically seek out (and signal
via GMPLS) an available channel to restore or protect the overall path
on a dynamic basis, and in more than 2 directions.

There is no real market for protected 10 gig waves. Occasionally a
bank will request the service, but backoff as soon as they see the
price tag.

I think the pricing is the result of trying to charge what the market
will bear rather than an underlying technical cost to deliver service.
Think "if the customer wants a want stop solution where we're managing
everything for them they should be willing to pay more for the
convenience".

"Hopefully none of these customers had service and protect ckts that
went down... I would be pissed as a ceo if that happen to my company.
Hopefully level3's new service offering is 100@percent redundant as
stated

Protected vs 2x diverse unprotected circuits each have their advantages
and disadvantages. One thing a protected circuit is not good at is
providing higher availability than 2x diverse unprotected circuits.
That's because you're trading diversity at the endpoints for simplicity,
so you've still done nothing to protect yourself against endpoint
failures. Protected circuits may provide other advantages though, such
as > 2 degree protection, or better latency than may be reasonably
available to purchase independently. It depends on the carrier, the
network, and even the customer to figure out which is the better
solution.

The new service offerings include: - Protected Wavelengths: Level 3
now provides automatic protection-switching to a dedicated diversely
routed wavelength in the event of a network failure. The protection
switch, fully automated and managed by Level 3, happens at switching
speeds approaching SONET restoration times. The single interface to
the customer requires no additional capital cost for customer optical
ports, and the diverse restoration path is fixed and fully known to
the customer. These features allow customers to achieve fast
restoration with predictable performance in their network without
adding significant cost and routing complexity. -"

I believe this is what I was talking about above, on their Infinera
platform. This is much more powerful than traditional ring designs.

But bear in mind that DWDM infrastructure that does 80 to 120 waves
per fiber pair is very expensive.

I suppose expensive is in the eye of the beholder. Every modern
long-haul "carrier grade" DWDM platform I know of has done at least 80
channel 50GHz spacing at the same cost as a 40ch solution for quite a
few years now. Only in the metro space does the statement above hold
true.

My understanding is that AT&T uses an MPLS/VRF CE router facing the user
such that the resulting network connectivity is a private MPLS VPN. VZW
apparently requires the user to implement a GRE/IPSec configuration just
to reach their MPLS/VRF layer. The resulting user router config is thus
much simpler with AT&T. Haven't heard about Sprint though.

Hi Richard,

I never said that protected LAN PHY 10 GigE was more expensive than two diversely routed waves. However, Hibernia's engineers have advised that route protected LAN PHY 10 GigE will tolerate a relatively high BER before switching. I stand by that statement.

I said that protected STM64 service was more expensive and that is true. Not only do you need two diversely STM64 waves, but you need protection as well.

Finally, you're wrong about "trying to charge what the market will bear".

I have sold almost 30 ten gig waves (leases) and I have only received one request (global bank) for protected service. When I priced at the twice the price of an unprotected service plus a 10% premium, that request was downsized to a protected STM16.

Customers in general are simply not willing to pay for protection. Indeed, most of them prefer to load balance among diversely routed 10 gig waves or buy waves on several network or cable systems.

And there are incumbents and competitive carriers that want to protect the service themselves.

How many protected waves do you have? :slight_smile:

Roderick S. Beck
Director of European Sales
Hibernia Atlantic
13-15, rue Sedaine, 75011 Paris
http://www.hiberniaatlantic.com

Rod,

Unless you are lucky enough to be doing large,cost +, IRU deals all day,
supply and demand economics should prevail, right? The minimum a market
would bear is based on costs and then supply vs. demand.

Best,

Marty

Crooks, Sam wrote:

I'm considering use of AT&T / Verizon / Sprint WWAN services and the
Cisco 3G router interface cards/integrated module in C880 routers for
primary or backup WAN network connectivity for routers.

I haven't used the integrated cards with cisco gear. However I do have 300+ cards deployed throughout the United States (EVDO USB modems on Linux boxes).

I'm looking for information from users of these services on the
following:

- addressing - Do these WWAN services use dynamic, PPPoE or static IP
assignment typically? Any of the 3? All?
   - is static IP assignment available?

We have static IP assignment for our Verizon cards. Sprint cards aren't static.

- do these service providers use NAT within their network?

Verizon doesn't. Not sure about Sprint. T-mobile doesn't either.

- How is the service reliability? In most cases, is the service
available for use when you need to use it?

We have found it to be quite reliable, although a small subset (about 15 to 20 connections) have been giving us issues. I posted on this last week or so. No resolution from Verizon as of yet.

- How is the service coverage area? Do you have problems getting
sufficient coverage in the deplouyment location to support desired
speeds (say 512kbps up/down as a minimum)?

Frequently you will need to deploy an external antenna as a booster. Dunno if the Cisco cards have the option, but I would imagine they do. It's almost a necessity in the vast majority of indoor deployments.

- is ESP / IKE / IPsec permitted through un-rate-limited and un-molested
by the providers?
- If you build a IPsec/GRE tunnel over these services, do you have
frequent issues with the tunnel dropping, or a dynamic routing protocol
running through the tunnel going down frequently?

We use OpenVPN without incident. Dunno bout GRE/IPSEC.

Also interested in similar information on impressions of similar EMEA
WWAN service providers, particularly Vodaphone and T-Mobile, if anyone
has experiences with these.

I have used T-mobile EDGE via Linux with great success (even ran a skype conference call over it). See my blog post on the configuration at:

Speed tests I did gave me 126k. So you would most likely want HSDPA for sure. I have yet to try HSDPA but hear excellent things about it. They recently released a USB dongle which does wifi/hsdpa/edge. See i4u - Mejores Noticias Y Reseñas De Deportes Y Casinos USA for more.

I agree with the other posters about POC and site survey. All sorts of strange environmental issues can pop up and wreak havoc on signal.

This for branch office environments? Retail? Industrial? (My deployments are retail locations).

Hi Martin,

That statement is true in the long run. But not the short run.

No would argue that current TransAtlantic pricing could justify a new cable system. :slight_smile:

If you look at the last three TransAtlantic builds, they spanned from $600 million to $980 million. No backhaul included.

Current market pricing could never justify another system or for that matter doing a true terrestrial build (trenching and creating a conduit system).

Everything has been based on recycled assets to this point.

Regards,

Roderick S. Beck
Director of European Sales
Hibernia Atlantic
13-15, rue Sedaine, 75011 Paris
http://www.hiberniaatlantic.com
Wireless: 1-212-444-8829.
French Landline: 33+1+4355+8224
French Wireless: 33-6-14-33-48-97.
AOL Messenger: GlobalBandwidth
rod.beck@hiberniaatlantic.com
rodbeck@erols.com
``Unthinking respect for authority is the greatest enemy of truth.'' Albert Einstein.

Hi Richard,

I never said that protected LAN PHY 10 GigE was more expensive than
two diversely routed waves.

Strange, the e-mail from you that I quoted specifically said:

Bottom line is that it will cost more than two diversely routed 10 gig
waves.

But at any rate...

However, Hibernia's engineers have advised that route protected LAN
PHY 10 GigE will tolerate a relatively high BER before switching. I
stand by that statement.

I said that protected STM64 service was more expensive and that is
true. Not only do you need two diversely STM64 waves, but you need
protection as well.

Modern DWDM systems don't care about they content of the payload, they
use a system called OTN (optical transport networks) as a generic
digital wrapper around the payload, and then they deal entirely with the
OTN frame. This makes features like optical protection protocol
agnostic, and remove any kind of cost difference based on the type of
service.

I think you're confusing the old style system of implementing a
SONET/SDH based ring as a method of delivering protected services, with
the modern techniques of delivering 10G or other subrate services as LAN
PHY or SONET/SDH or some other protocol. These are completely different
things.

I have sold almost 30 ten gig waves (leases) and I have only received
one request (global bank) for protected service. When I priced at the
twice the price of an unprotected service plus a 10% premium, that
request was downsized to a protected STM16.

Well, I DID point out some compelling reasons why one might want to do
2x (or more) diversely routed unprotected wavelengths rather than a
protected service. There are many other reasons, such as statistically
multiplexed oversubscribtion on multiple unprotected circuits during the
normal non-failure state.

At any rate, I'm not in a position to explain the logic or motivations
of the people who buy waves from you, all I can tell you is how the
technology works and what it costs to deploy it. As such, my previous
explanation was correct. :slight_smile:

Customers in general are simply not willing to pay for protection.
Indeed, most of them prefer to load balance among diversely routed 10
gig waves or buy waves on several network or cable systems.

All perfectly legitimate reasons why one might want to do multiple
diverse unprotected wavelengths, but this is still orthogonal to the
assertions that protected wavelengths are not possible, not reliable, or
cost more to implement than 2x unprotected waves.

Also, keep in mind that the availability of > 2-degree protection on
modern DWDM platforms could *easily* result in optically protected
wavelengths which are much cheaper to deliver than diversely routed
unprotected paths. For example, lets consider the scenerio you
previously gave of a 10G wavelength from Chicago to Frankfurt. Using an
optical switching protection system, a provider could survive a fiber
cut between Chicago and Cleveland in Detroit by wrapping the wavelengths
via Chicago Indianapolis Cincinatti Cleveland, before continuing on its
way to Frankfurt. This eliminates the need to provision capacity on two
completely diverse paths, which may not even exist or which may have
extremely poor latency choices, and reduces the cost to deliver the
service. As always, the benefits of such a system depend on both the
carrier's and the customers' footprints. I suspect you'll start to see
more of this in the future, as Level3 seems to be adopting it.