Rollup: Small City Municipal Broadband

Ok, here's a rough plan assembled from everyone's helpful contributions
and arguing all week, based on the City with which, if I'm lucky, I
might get a job Sometime Soon. :slight_smile: (I'm sure some of you can speculate
which city it might be, but Please Don't.)

It's about 3 square miles, and has about 8000 passings, the majority of
which are single or double family residential; a sprinkling of multi-tenant,
about a dozen city facilities, and a bunch of retail multi-unit business.

Oh, and a college campus, commuter.

My goal is to fiber the entire city, with a 3-pr tail on each single-family
residence (or unit of a duplex/triplex), and N*1.5 on multi-tenant business
buildings, and probably about N*1.1 or so on large multi-unit residences.
Empty lots, if we have any, will also get a 3-pr tail, in a box.

My plan is for the city to contract out the design and build of the physical
plant, with each individual pair home-run to a Master Distributing Frame in
a city building. Since the diameter of the city is so small, this can be
a single building, and it need not be centrally located -- since we are a
coastal city, I want it at the other end. :slight_smile:

I propose to offer to clients, generally ISP, but also property owner/
renters, L1 connectivity, either between two buildings, or to a properly
equipped ISP, and also to equip for and offer L2 aggregated connectivity
to ISPs, where the city, instead of the ISP, will provide the necessary
CPE termination gear (ONT). The entire L0 fiber build, and all L2
aggregation equipment (except potential GPON splitters mentioned next)
will be the property of the City.

Assuming that the optical math pans out, we will hang GPON splitter frames
in the MDT, and cross connect subscriber ports to the front of them, and
the back of them to Provider equipment in an associated colo, in rooms
or cages; we'll also probably do this for our L2 subscribers, using our
own GPON splitters. Those will then be groomed into Ethernet handoffs
for whatever providers want to take it that way, at a higher MRC.
Splitters installed for Providers who take L1 handoffs will be their
property, though installed in our MDF room.

We will do all M-A-C work on the MDF, into which Provider employees will
generally not be admitted, at least unescorted, on a daily basis, except
in "emergencies", for which an extra NRC will be levied.

The cost we will charge the Providers, per subscriber, will be a fixed
MRC, similar to a 'tariffed' rate, which is published, and all Providers
pay the same rate, which is subject by contract to occasional adjustment
in either direction, and which is set to recover our costs to provide
the service, based on take rates and depreciation periods which I have not
yet determined. I'm assuming I can get 30 year depreciation out of the
fiber plant with no problems, probably 40... maybe 50 if it's built to
high enough standards -- I do not expect passive glass fiber to become
obsolete in 50 years.

Active equipment, a much shorter period, of course, probably between 4
and 7 years, depending on how far up the S-curve of terminal equipment
design it proves that we've already traveled. At the moment, my
comparison device is the Calix E7-20, with either 24-port AE or the
GPON cards; either 836GE interior ONTs, or their equivalent exterior
ones (since the power module has to be inside anyway, I'm not sure you
gain that much by putting the ONT outside, but...)

My motivation for not doing L3 is that it is said to greatly improve the
chances for competition at the ISP level, a fact not yet in evidence.

My motivation for not doing GPON in the field is that it's thoroughly
impractical to do that in an environment where an unknown number of
multiple providers will be competing for the subscribers, and anyway
it breaks point to point, which the city will need for itself, and which
I want to offer to residents as well.

My motivation for doing L2 is that it takes a lost of the front-end cost
burden off of potential smaller 'boutique' ISPs specializing in various
disciplines (very low cost/lifeline service, very high speed, 'has a big
local usenet spool', or what have you); such providers will have to pay
(and recover) a higher per-subscriber MRC, in exchange for not having to
themselves provision and install GPON splitters and something like a Calix
E7 -- such hardware will be installed by the City, and cost-shared; if/when
such a provider gets big enough, they can install their own, and we'll
cut them over.

I propose to take the project to the council for funding and approval
having in my pocket a letter of intent from a local 2nd tier ISP of
long standing to become our launch provider, with no incentives over
the published rates except the guarantee of additional subscribers.

My underlying motivation, which is intended to answer any tradeoff queries
which I haven't explicitly addresses before this point, is to increase
the City's position as being "full service" (as small as it is, it does
it's own fire, police, garbage and water already), and improve it's
chances of selection by people who are deciding where to move. The City
already has a relatively good image, within its target market, but as
time marches ever forwards, the maximum available broadband in its
footprint will become less and less acceptable, and I expect that there
are a significant number of people around the country for whom "I can
get Gigabit in my house? Bidirectional? I'm moving" is a valid viewpoint.

I know already that "what kind of broadband can I get" is a top-5, and
sometimes top-3 selection issue for people contemplating a move.

Things, therefore, which improve the city's image with potential immigrants,
be they residents or small businesses, are a Good Thing, whether because
those people actually want or need those services, or whether it's merely
because they like to bask in the reflected glow there of.

These things will likely reduce the city's vacancy rate, and thus increase
property tax revenue and hence the city's budget, in addition to slowly
improving the city's socioeconomic demographics, which will itself likely
have a salutary effect on the small businesses already here, and in the
decision processes of people thinking to move one here or start one.

That's my thinking so far. Now comes the hard part: assembling enough
other budgetary numbers to determine how much it will cost, how much we'll
have to charge, and whether people will *pay* that much.

I don't have any illusions that the wholesale charges will be a revenue
stream for the City, and I won't let the council get any such ideas either;
the benefits to the city (aside from dark fiber to all our own buildings)
are a bit deeper than that, and will require sufficient time to come to
fruition.

I wrote this as a summary for all the helpful NANOGers who chimed in this
week, and as a clarification for those who weren't quite sure where *I*
was trying to go -- all muni builds are sui generis, and this one moreso
than most.

If any of you see anything we've already said, but I left out, please
let me know...

And have a Whacky Weekend. If any of you pass through the west coast
enroute to ORL, let me know. :slight_smile:

Cheers,
-- jra

Why on earth would you do this with PON instead of active Ethernet? What
GPON vendor have you found where their technical staff will tell you this
is a good architecture for their PON offering?

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Out of curiosity, do you have plans for legal battles or anything? There
have been some other places attempting or running muni broadband that
have resulted in crap like the hilariously named "AN ACT TO PROTECT JOBS
AND INVESTMENT BY REGULATING LOCAL GOVERNMENT COMPETITION WITH PRIVATE
BUSINESS" bill.

Sorry, misclicked, delete the first incomplete email if possible.

Ok, here's a rough plan assembled from everyone's helpful contributions
and arguing all week, based on the City with which, if I'm lucky, I
might get a job Sometime Soon. :slight_smile: (I'm sure some of you can speculate
which city it might be, but Please Don't.)

It's about 3 square miles, and has about 8000 passings, the majority of
which are single or double family residential; a sprinkling of

multi-tenant,

Asked and answered, Scott; have you been ignoring the threads all week?

I'm pretty sure I even answered it in the posting, but just in case:

1) Line cards for the OLT frames appear to be 2 orders of magnitude denser
for GPON termination than AE (480 ports per 10U vs 10k ports per 10U in
Calix, unless I've badly misunderstood my sources), and

2) GPON is what potential L3 providers large enough to want an optical
handoff are generally used to.

If someone wants AE, they can certainly have it.

(C'mon; miss the *next* turn, too :slight_smile:

Cheers,
-- jra

Believe me, budgeting for both legal, and PR to make Verizo<um><ahem>any
large proprietary carrier look greedy and malign is in my plans, yes. :slight_smile:

TTBOMK, Florida is not a state where Verizon succeeded in making muni
ownership of the phy layer illegal. And since they're on record that

a) they were cherrypicking and
b) they're done now

it shouldn't be too hard to suggest their intentions.

Cheers,
-- jra

Jay,

I'm spotty on mailing lists since most of my time is spent building these
kinds of networks.

1) Talk to more vendors than just Calix, especially if they're quoting
their Ethernet density on the C7. Also, keep in mind that port density may
or may not be relevant to your situation since space for muni shelves isn't
usually a problem. Port density is much more important if you're deploying
in existing telco enclosures but muni networks tend (not universally of
course) to reuse existing city infrastructure building to house the nodes
of their network. Please note that I am not reccomending against Calix,
they're a good solution in many cases, but AE is not a strong point on the
C7. The E7 and the B series, which is the old Occam product, is much
better than the C7. For that matter I wouldn't consider doing a new build
on the C7 since that platform's EoL can't be too far in the future.

2) I have no idea who told you this, but this is completely and utterly
incorrect in nationwide terms. If you have a specific layer 3 provder in
mind that tells you they want a GPON hand off then that's fine, but ISPs in
general don't know what GPON is and have no gear to terminate that kind of
connection.

I hope I said "E7"; it's what I meant to say. Yes, I wasn't going to
stop at Calix; I'm just juggling budgetary type numbers at the moment;
I'll have 3 or 4 quotes before I go to press. It's a 36 month project
just to beginning of build, at this point, likely.

Assuming I get the gig at all.

The E7 is a good shelf, so that's a decent starting point. I'd also talk
with Zhone, Allied Telesys, Adtran, and Cisco if for no other reason but
get the best pricing you can. I'd also focus much more on your cost per
port than the density since your uptake rate will be driven by economics
long before port density and how much space your gear takes becomes an
issue.

> 2) I have no idea who told you this, but this is completely and utterly
> incorrect in nationwide terms. If you have a specific layer 3 provder
> in mind that tells you they want a GPON hand off then that's fine, but
> ISPs in general don't know what GPON is and have no gear to terminate
that
> kind of connection.

Other people here, said it. If nothing else, it's certainly what the
largest nationwide FTTH provider is provisioning, and I suspect it serves
more passings than anything else; possibly than everything else.

I'm not sure what you mean by this. The largest PON offering in the US is
Verizon's FIOS, but AFAIK they don't interconnect with anyone at layer 2
and their layer 3 fiber connections are either Packet Over SONET, Gig
E(most common), or very occasionally still ATM. I have heard of a few
instances where they'd buy existing GPON networks but I've never heard of
them cross connecting like this even with operators that they do
significant business with in other ways.

But it doesn't matter either way, except in cross-connects between my MDF
and my colo cages; except for GPONs apparent compatibility with RF CATV
delivery (which I gather, but have not researched) is just block-upconvert,
I don't care either way; there's no difference in the plant buildout.

This is not correct. DOCSIS is an MPEG stream over QAM or QPSK modulation
and there is nothing about it that is compatible to any flavor of PON. In
fact if you look at the various CableLabs standards you'll see DPoE (
http://www.cablelabs.com/dpoe/specifications/index.html) which lists how a
DOCSIS system can inter-operate and provision an PON system. If you look at
the two largest PON networks (FIOS and Uverse) you'll see the two different
approaches to doing video with a PON architecture. Verizon is simply
modulating a MPEG stream (this is block compatible to a cable plant, in
fact its the same way that a HFC network functions) on a different color on
the same fiber that they send their PON signalling. ATT takes another
approach where they simply run IPTV over their PON network. I've listened
to presentations from Verizon's VP of Engineering (at that time) for FIOS
and he said their choice was driven by the technology available when they
launched and they did modulated RF over their fiber instead of IPTV because
that technology wasn't as mature when they started. Verizon's approach may
be what someone was thinking of when they said that PON was compatible to
cable signaling but that's not how it works.

I was musing, off-list, as to why there isn't a Mad-Lib you can just plug
some numbers into and get within, say, 40% or so of your target costs for
a given design.

Well,

  http://www.ftthcommunitytoolkit.wikispaces.net

isn't it, but it does have a bunch of useful information for this project,
looks like.

A couple of clarifying points:

1) I had posited GPON as I assumed that was where most of the CATV over
FTTH hardware work was, vice FiOS. Turns out there's lots of hardware for
IPTV as well, and quite a number of smaller deployments, so apparently
that path is easier than I thought. The only difference is cross-connect
fiber counts, and possibly some link budget.

2) I was planning to provide an IX switch in my colo, so all my L3 providers
could short-circuit traffic to my *other* providers through it, unloading
my uplinks.

3) Given that, I suppose I could put Limelight and Akamai racks in there,
and couple them to the IX switch as well, policies permitting.

4) Given what a pisser it's going to be to get tags to me on the local
backbone loops (about 3 are with 5 miles of my city border), I'm also
considering having a 10G or 2 hauled in from each of 2 backbones, and
reselling those to my L3 providers (again at cost recovery pricing),
while not precluding any provider wanting to haul in their own uplink
from doing so.

5) There's a possiblity my college campus may be on I2 (or want to);
perhaps I can facilitate that as well -- and possible (again, policies
permitting) extend such connections to relevant staff members or students
who live in the city) (I'm not as familiar with I2 as I should be).

6) And pursuant to 3, perhaps I could even set up the IPTV service and
resell that to the L3 provider to bundle with their IP service, so
they don't have to do it themselves; while it's not a difficult as I
had gathered, it's still harder than them doing VoIP as part of their
own triple-play.

Cheers,
-- jra

I can't believe I'm going to beat Owen to this point, but considering you a building a brand new infrastructure, I'd hope you'd support your service provider's stakeholders if they want to do IPv6. To do so securely, you'll want your neutral layer 2 infrastrcuture to at least support RA-guard and DHCPv6 shield. You might also want/need DHCPv6 PD snooping, MLD snooping. We have found VERY disappointing support for these features in this type of gear.

So you are going to prohibit the operator of the fiber plant from running layer 3 services, but then turn around and let them offer IPTV? That seems quite inconsistent to me. And just because it's "hard"?

Running a decent layer 3 service is "hard" too. Isn't the whole point to let these service providers compete with each other on the quality and cost of their services?

From: "Jay Ashworth" <jra@baylink.com>

It's about 3 square miles, and has about 8000 passings, the majority
of which are single or double family residential; a sprinkling of
multi-tenant, about a dozen city facilities, and a bunch of retail
multi-unit business.

I was musing, off-list, as to why there isn't a Mad-Lib you can just plug
some numbers into and get within, say, 40% or so of your target costs for
a given design.

Well,

http://www.ftthcommunitytoolkit.wikispaces.net

isn't it, but it does have a bunch of useful information for this project,
looks like.

A couple of clarifying points:

1) I had posited GPON as I assumed that was where most of the CATV over
FTTH hardware work was, vice FiOS. Turns out there's lots of hardware for
IPTV as well, and quite a number of smaller deployments, so apparently
that path is easier than I thought. The only difference is cross-connect
fiber counts, and possibly some link budget.

2) I was planning to provide an IX switch in my colo, so all my L3 providers
could short-circuit traffic to my *other* providers through it, unloading
my uplinks.

3) Given that, I suppose I could put Limelight and Akamai racks in there,
and couple them to the IX switch as well, policies permitting.

4) Given what a pisser it's going to be to get tags to me on the local
backbone loops (about 3 are with 5 miles of my city border), I'm also
considering having a 10G or 2 hauled in from each of 2 backbones, and
reselling those to my L3 providers (again at cost recovery pricing),
while not precluding any provider wanting to haul in their own uplink
from doing so.

A better model, IMHO, is to encourage the backbones to come meet your
providers at your IX/Colo.

5) There's a possiblity my college campus may be on I2 (or want to);
perhaps I can facilitate that as well -- and possible (again, policies
permitting) extend such connections to relevant staff members or students
who live in the city) (I'm not as familiar with I2 as I should be).

Do some research before you pursue this too vocally or commit to it.

6) And pursuant to 3, perhaps I could even set up the IPTV service and
resell that to the L3 provider to bundle with their IP service, so
they don't have to do it themselves; while it's not a difficult as I
had gathered, it's still harder than them doing VoIP as part of their
own triple-play.

Pandora's can of worms.

Owen

That's one of the reasons to look at active ethernet over gpon. There is
much more of a chance to do v6 on that gear, especially cisco's Metro
ethernet switches.

From: "Brandon Ross" <bross@pobox.com>

> 6) And pursuant to 3, perhaps I could even set up the IPTV service and
> resell that to the L3 provider to bundle with their IP service, so
> they don't have to do it themselves; while it's not a difficult as I
> had gathered, it's still harder than them doing VoIP as part of
> their own triple-play.

So you are going to prohibit the operator of the fiber plant from
running layer 3 services, but then turn around and let them offer IPTV? That
seems quite inconsistent to me. And just because it's "hard"?

No; I wouldn't offer it retail; I'd offer it to all provider-comers
wholesale, at cost plus, just like everything else.

Running a decent layer 3 service is "hard" too. Isn't the whole point to
let these service providers compete with each other on the quality and
cost of their services?

You could say the same thing about the uplink, though; I note you didn't
throw a flag at that, or at Akamai; is the IPTV issue different to you?

Fair point.

Cheers,
-- jra

IPv6 would be on my ticklist, yes. :slight_smile:

Cheers,
-- jra

It's just been pointed out to me that I can farm that out. :slight_smile:

Cheers,
-- jra

From: "Brandon Ross" <bross@pobox.com>

6) And pursuant to 3, perhaps I could even set up the IPTV service and
resell that to the L3 provider to bundle with their IP service, so
they don't have to do it themselves; while it's not a difficult as I
had gathered, it's still harder than them doing VoIP as part of
their own triple-play.

So you are going to prohibit the operator of the fiber plant from
running layer 3 services, but then turn around and let them offer IPTV? That
seems quite inconsistent to me. And just because it's "hard"?

No; I wouldn't offer it retail; I'd offer it to all provider-comers
wholesale, at cost plus, just like everything else.

It sure seems like just pushing the competition (or lack thereof) up the stack.

Running a decent layer 3 service is "hard" too. Isn't the whole point to
let these service providers compete with each other on the quality and
cost of their services?

You could say the same thing about the uplink,

Which uplink is that? I'm a little confused.

though; I note you didn't throw a flag at that, or at Akamai; is the IPTV issue different to you?

If you were to open your colo to all comers that have similar models to Akamai, that seems fair. After all, it's not the city selling Akamai services to either the ISPs or end-users, the city is just providing a convenient way for the providers that are there to interconnect with content providers that care to show up.

Now if you were to encourage an IPTV services provider that WASN'T the city to co-locate at the facility, that seems reasonable as long as terms were even if another one wanted to show up. I could imagine that some might sell service direct retail, others might go wholesale with one of the other service providers. Maybe both?

This whole thing is the highway analogy to me. The fiber is the road. The city MIGHT build a rest stop (layer 2), but shouldn't be allowed to either be in the trucking business (layer 3), nor in the business of manufacturing the products that get shipped over the road (IPTV, VOIP, etc.), and the same should apply to the company that maintains the fiber, if it's outsourced.

In a message written on Sat, Feb 02, 2013 at 06:14:56PM -0500, Brandon Ross wrote:

This whole thing is the highway analogy to me. The fiber is the road.
The city MIGHT build a rest stop (layer 2), but shouldn't be allowed to
either be in the trucking business (layer 3), nor in the
business of manufacturing the products that get shipped over the road
(IPTV, VOIP, etc.), and the same should apply to the company that
maintains the fiber, if it's outsourced.

I think your analogy is largely correct (I'm not sure Rest Stop ==
Layer 2 is perfect, but close enough), but it is a very important
way of describing things to a non-technical audience.

FTTH should operate like roads in many respects. From ownership
and access, to how the network is expanded. For instance a new
neighborhood would see the developer build both the roads and fiber
to specifications, and then turn them over to the municipality.
Same model.

Having multiple people build the infrastructure would be just as
inefficeint as if every house had two roads built to it by two private
companies.

I was going to trot on the Manhattan 26-crossbuck telephone pole, and
multiple power wires and water pipes, but roads is *much* better.

Especially since my dad did this in the 70s, and I am a *big* fan of
<deep breath>

The Dwight D Eisenhower System of Interstate and Defense Highways.

Yes, the Interstate System has a home page:

  https://www.fhwa.dot.gov/interstate/homepage.cfm

Cheers,
-- jra

Jay,

While its certainly technically possible to offer linear video in a shared
network model the content owners have big objections of that. There really
is no way to do wholesale IPTV except for a very few organizations like the
cable coop (NCTC http://www.nctconline.org/).

......

This is not correct. DOCSIS is an MPEG stream over QAM or QPSK modulation
and there is nothing about it that is compatible to any flavor of PON. In
fact if you look at the various CableLabs standards you'll see DPoE (
http://www.cablelabs.com/dpoe/specifications/index.html) which lists how a
DOCSIS system can inter-operate and provision an PON system. If you look

at

Jay may be referring to something I alluded to earlier, what Calix refers
to as RF overlay. The RF signal from the traditional cable system is
converted to 1550nm and combined onto the PON before the splitter with a
CWDM module. Certain model ONT's split the 1550 back off and convert back
to an RF port.