Communications

Network Neutrality is the wrong fight!

Winning would mean giving up much more important rights – historical rights that were in place in the US as recently as 1995 and remain in place in most of Europe even today. 

We shouldn’t settle for network neutrality. It’s a poor substitute for what we had and much less than what we need. Let me explain. There are two topics to discuss. The first is “common carriage,” a centuries old legal concept that applied to the US telecom industry throughout most of the 20th century. The second involves communications protocols. Both topics are complex, so I will cover only what’s needed to understand why we shouldn’t accept network neutrality and why, at a minimum, we should fight for enforcement of existing common carriage rules. 

Network neutrality is about allowing any Internet application to run over an Internet connection, i.e. over a connection that uses Internet Protocol (IP). But under common carriage as it applied prior to the late 1990s, we had a more powerful right – the right to run any kind of network protocol, IP or otherwise, over a lower, simpler service which today we call a “bit stream*.” Why does this matter? Because real innovation is also possible at these lower layers and that innovation continues to be important. But today, such lower layer innovation is restricted to inside one building or one campus. Yes, we can tunnel some lower level innovations over IP, but not all of them and only at a cost. 

IP telephony (VoIP) is one place where problems arise. Most enterprises use IP PBXs internally, yet calls between enterprises use the PSTN. Many companies have attempted to address this gap, but progress is slow and expensive. Within an enterprise, IP telephony packets are given priority, but that priority is not supported on Internet access links and network neutrality doesn’t help. As a result, to interconnect VoIP calls, enterprises must lease separate dedicated access circuits – circuits usually based on bit stream access – to support “SIP trunks.” Up until the late 1990s, these circuits were regulated under common carriage. Today they are an unregulated monopoly, with prices derived from the cost of voice circuits 15-20 years ago, i.e. abnormally expensive for today. 

 Common carriage is the legal concept that, in exchange for government granted monopoly access to rights-of-way, the monopolist must carry anyone’s traffic over the resulting infrastructure, at regulated rates. For centuries this has applied, to canals, to roads, to railroads, to telegraph lines and, until nearly the end of the 20th century, to telecommunications lines. But during the legal battles after the Telecom Act of 1996, the FCC basically gave up on common carriage.

If we accept Network Neutrality instead of common carriage, we guarantee future innovations happen only above the IP layer. Innovation at lower layers will be restricted to enterprise or campus applications. That’s too bad as it was the existence of common carriage that allowed the Internet to develop in the first place. Do we want to eliminate that kind of innovation in the future? 

If anything, we should be fighting to extend the ideas of common carriage to lower layers, e.g. dark fiber. Installing dark fiber is expensive and requires access to rights-of-way that are limited. The installed fiber is capital expensive infrastructure that lasts for decades. Such conditions justify granting monopoly access, in exchange for common carriage and regulated rates of return. But when you light up a dark fiber, you use (relatively) low cost gear with a short life (even if it can survive for ten years, Moore’s Law renders it functionally obsolete within 2-3 years). What's more, there’s rapid innovation in opto-electronics gear. Just look at the order of magnitude difference in cost between enterprise and carrier fiber-optic gear. 

Today, the US is loosing leadership in all things Internet. Network Neutrality will just put a nail in our coffin. To stop our decline, fight for restoration of the common carriage principals that existed through most of the 20th century and still exist in law. To regain world leadership, fight to extent those principals to include access to dark fiber at regulated rates.

---------------------------------------------------------------------------------------

* Bit stream access.   In the 20th century two regulated services provided what the 21st century calls bit stream access. These were voice telephony and T1 circuits. T1 circuits directly carry a stream of digital bits. Modems allowed voice connections to carry digital bits, for example, for bulletin board services and other purposes long before the Internet became popular.

Ben West's coverage of ISCWN

While at the ISCWN in Vienna, I met Ben West of WasabiNet in St. Louis. He’s provided excellent coverage of the sessions he attended in his day-by-day posts here:

Day 1;  

Day 2;  

Day 3.

My notes from the Community Wireless Networks Summit in Vienna

As with most conferences I attend, I took most of my notes using Twitter.  Since I can't always depend on accessing old tweets, here is the entire tweet stream from the conference in one blog post.  I don't know if this is of use to anyone else, but at least I know I'll have a record of my notes that I can refer back to.

The full name of the conference was The International Summit for Community Wireless Networks (ISCWN) and it was held in Vienna on August 12-15, 2010.  See the conference website.

International Summit for Community Wireless Networks http://bit.ly/9Lp41p
Good conference Wi-Fi http://bit.ly/dkwUh2
OK, I'm finally set up to Tweet &/or blog the International Summit for Community Wireless Networks http://wirelesssummit.org
While there's plenty of broadband behind the Wi-Fi, there are very few power outlets, at least as yet.
Community Wireless conf finally getting underway
Sascha Meinwrath opening: 1/2 the participants are missing, even some who were here an hour ago, so expect stragglers, but we start.
Sascha: 80% of cash raised goes to travel allowances; 10% everything else; plus donations.
Aaron Kaplan, Funkfeuer, on TechGate facility science park estbsh'd 2001 by City; conf venue; also supports Funkfeuer
Aaron there is swimming in the old Danube - it's clean now! Also boat rental nearby!
Aaron: Funkfeuer hotspots since 2003, now reaching almost to Bratislava. Mixed mesh and P2P backbone
Sascha on previous feedback - social time is key
Jim Baller: America at the Crossroads: Greatness or Mediocrity. 15+ yrs fighting ILECs on behalf of muni's - mostly successful!
Baller: credits Broadband Coalition http://bb4us.net/  Also as key to Congress requiring FCC to do a BB Plan
Baller on the US national BB plan: We need big goals, the BB plan didn't deliver! It's goals for 2020 happening elsewhere today.
Baller: US BB Plan focuses on inches, not yards and miles.
Baller: US BB Stimulus: Awards coming quickly (9/2010 deadline): middle mile, rural first mile
Baller on Google gigabit initiative yielded 1100 muni applications and showed pent up demand
Baller: Ugly side - incumbents bigger, stronger & nastier than five yrs ago. Cable industry even nastier. Now Google is in question.
Baller: But btwn BB Plan (it's a plan), Google fiber init., $7B Stimulus goes to new parties, community BB efforts
Baller responding to Q about wireless: I'm a fiber guy, then disappointing comments on FCC being pro-wireless. Misses FCC focus :(
Baller responding to Q about Australia admits that most in US don't know what's happening in the rest of the world.  US always has excuses - "we're different."
Sascha introducing Ramon Roca, President, guifi.net Foundation, a fantastic alternative that's happening in Spain. 
Guifi runs a very large network across rural (and urban) Catalonia in Spain. http://guifi.net/
Ramon Roca: Vision for the Future. Will talk about: Why scale? and their sustainable economic model!
Ramon: Guifi has 10K+ nodes and 15 Km of networks, but still tiny compared to telco. Scale as a goal. Need growth to be sustainable.
Ramon: need sustainable model - critical mass; org issues; legal; partners. Non-profit core w/biz around.
Ramon: Demand is there; also incumbent not interested in low 40%.
Ramon on mgmt: horizontal, bi-directional & collaborative - examples from their website. Website appears to be key for communit.
Ramon on public/private: public requirements must be written: contracts, license, P2P agreements. Must be clear, and legal.
Ramon on public/private - like OS licenses, the network is considered private by govnmt. Need documents just as open src does.
Ramon facilitates integration and cooperation between diverse groups, e.g. need churches for their steeples! need public ROW access.
Ramon: Guifi license is required if you want to participate! Like GNU license, Guifi license even deals with revisions.
Ramon: Guifi license defines everything
Ramon: Guifi is also a ""public"" network in the legal sense in Spain because they had to in order to deploy some fiber.
Ramon: Guifi ROI for many: supply chain dlrs/shops, prof svcs and svc provdrs who mediate wholesale mkts.
Ramon: Catalonia is a bit ahead of Spain but w/o Guifi Osona (rural Catalonia) way behind. Now Osona ahead of rest of Spain & even the UK.
Ramon: Guifi network now 15% of BB links in Osona; and yet DSL subscription haven't dropped.
An English version of the Guifi Wireless commons license is here: http://guifi.net/WCL_EN
Ramon futures: working on fiber from farms (FFtF) cheap & easy & Gbps; always a combination.
Ramon has had fights with incumbent about ROW (even when on private property) and about access to poles. Ramon - do it! then fight.
Ramon - their main focus is to stay on private ROW, but as a public carrier they can now go after public ROW.
Ramon: stories about fights over public fiber & public ROW - everything you expect, except Guifi seems to win, espc. in rural area.
Ramon: Guifi participants getting access to public fibers laid for cameras after threaten authorities - tractor might hit camera...
Ramon concludes by emphasizing need to enlist service businesses and computer shops, e.g. 3rd party for profit businesses.
Lightning talks where individuals describe their networks.
Austin (TX) Wireless - hotspot service; wanted free Wi-Fi; customers incl coffee shops that compete with Starbucks.
Austin Wireless - other things to help venues, e.g. splash pages
Austin Wireless: splash pages useless if not totally relevant for the customer: Weather, sports, local info, Facebook links, venue.
Austin Wireless: Now making $ by providing marketing services to their venues, restaurants, coffee shops, etc.
Austin Wireless: Also does Chimpit which brings venue portal even if you're on another network.
Chambana.net joint project of IMC and Acorn IT collaborative - ~6 active people and 9 servers. Walk-in computers, media prod. lab.
Chambana.net host websites, many other IT services, but also some wireless! Small network, switching from NetBSD to OpenWRT.
Chambana.net: It sounds like their wireless net is secondary to what they do, and is currently down and being reworked.
Chambana.net: Wireless network is completely open, no splash page. Runs on donated hardware
Chambana.net runs on donations, but entire uplink is one Comcast business service. But part of a stimulus grant, so near future good
Tribal Digital Village: 19 tribes HQ in San Diego CA, only way to get Internet to reservations. sovereign nations within US, but...
Tribal Digital Village: 350 miles of P2P and P2MP links mostly license exempt; 18 bldgs
Tribal: Paid NW in parallel reaches 200 individual homes. 2/2 Mbps for $34.95 per month. Hope to reach 2000 of 2700 homes.
Tribal: 1K devices connected to net. Fiber at headend feeds an arc of P2P links. Now many gamers accessing via local centers.
Tribal: Too small and separate to think about running their own radio regulatory regimes. But not too worried about conforming :)
Djursland http://www.diirwb.net/ Rural area in Denmark; 95% can get DSL (fr 1600 exchgs). Last 5% not svc'd.
Djursland P2MP design
Djursland: NW built by volunteers
Djursland gets favorable Interent transit because the farmers drove (& control) the fiber backbone deployment.
Funkfeuer: mesh using OSLR; got help from jaap@scii.nl & friends in Berlin.
Funkfeuer: 240 roofs; financial sustainability based on hosting center revenues; Gbps uplink
Funkfeuer: slightly below mkt for hosting; TV streaming experiments; running Wi-Fi for events.
Funkfeuer: Fiber splicing is easy, if you have the expensive machine (~6000 Euros). Basically welding glass.
Funkfeuer is a closed user group. This is important for VoIP and other regulatory issues, including streaming TV to you members!
Funkfeuer Graz: like NW in Vienna; Technical Univ in Graz helped them grow fast; slower growth >2007 as 3G faster.
Funkfeuer Graz: now working to link with Bratislava (Slovokia) which could be the first international community network in EU.
Funkfeuer Graz: longest link today is 30 km which gets them 1/2 way to Bratislava (Ubiquiti radios)
Malcolm Matson knows of two networks connected (quietly) btwn Solovakia and Hungary.
Belgrade Wireless: up to 20 km links (> 50 Mbps) using corner antennas as shown in previous conf. NW now extends over 100 Km.
Belgrade: 3D corner antennas invented by Prof in Belgrade. Big focus on community events.
Belgrade: 3D Corner Antennas http://su.pr/AjXm1p - feeder for dish reflector - mixed polarization
More 3D corner antenna info here  http://su.pr/7t9990 Need to follow up on whether this remains relevant w/ MIMO using polarization.
Updates from the OLSR-NG Project - Henning Rogge (FKIE) & Aaron Kaplan (Funkfeuer.at) - history, today & futures.
OLSR-NG: 1 of 2 major mesh stds (other AODV). RFC 3626. Tonnesen PhD, Lopatic LQ & Fisheye extensions
OLSR-NG working on OLSR. Guys in Berlin starting over (BATMAN); HSLS hazy sighted link state (CUWin).
OLSR-NG session: CUWin HSLS didn't get beyond simulations
OLSR-NG: OLSR is link state - every node knows whole graph (100K entries = 4.8 MB); but MPR now off
OLSR-NG: MPRs only matter with really dense NWs, but with only 2-3 links per node, they don't pay.
OLSR-NG: ETX link quality metric used instead of basic hop count, i.e. sum of ETXs not sum of hops. But heavy compute load!
OLSR-NG bringing down compute load (now linear not exponential). 100-to-1 benefit with 400 nodes. Still Dijkstra, but optimize data.
OLSR-NG: malloc() thrashing fixed
OLSR-NG futures: soft refresh (CSN), better metrics (ETT, MIC), multipath routing (experimental), Q of layer 2 capabilities.
OLSR-NG: Henning comments on how few multi-path routing projects have made any progress... Problem: must choose whole path. very hrd
OLSR Henning: OLSRd 0.6.0 is current. Clean rewrite of routing code; smarter gateways to reduce thrashing btwn GWs.
Henning: 0.6.0 has very few (& only site-specific) bugs - very stable!
Henning on future plans: telnet/http server (done); config mgmt (stability, flexiblty.
Henning also thinking about better metrics but limited by packet format prior to OSLR v2. Many metrics in discussion in academia.
Henning responding to Q: negative about dual protocol mode to support migrations. Thinks it wold be very hard.
Henning - active developers= ~1.5 people
Henning complaining about academics who've made patches w/o consulting (thus doing stupid things) and without plans to give back.
Henning & Aaron on how plugin system makes it easy to try new things in a clean fashion.
Freifunk Berlin started 2002; then OSLR in 2003-04; 28 devices @ 2004 OS conf; PC to openWRT for embedded.
Freifunk took off in 2005 despite labeling website ""OSLR experiment"" but users wanted reliability
Freifunk can't even switch to B.A.T.M.A.N. because OSLR widely deployed
Freifunk net is now shrinking, as people who came only for bandwidth are getting DSL and 3G mobile.
Freifunk has issue of switching gateways which doesn't affect Frunkfeuer with their fixed gateways and public IPs.
guifi.net is a bunch of communities, not all interconnected. 10,300 nodes using same software, same tools and same license.
guifi.net is showing off an impressive set of tools for examining nodes, plus there's extra data that the node owner can access.
guifi.net Using MRTG http://oss.oetiker.ch/mrtg/
guifi.net is a large wireless LAN. Must search for & connect to services, like Internet access. Libraries & other offer Inet-GWs.
Île sans fil network in Montreal  http://www.ilesansfil.org/  Now 200 hotspots installed, free Internet thru portal page.
Île sans fil limits users to 7 GB/wk. 150K users registered. Each hotspot has own page. Projects: Authpuppy, WiFiDog
Wireless Toronto - captive Wi-Fi portals, started with Wifidog from ◊le sans fil.
Wireless Toronto has struggled compared to Montreal. All volunteers. Hotspots, but now adding mesh, e.g. in parks.
Wireless Toronto - no spt fr Government who started and sold a parallel NW. Hard to find location-based content for portal pages.
Wireless Toronto survives on annual fees from businesses with portals; organized as a club, not a nonprft
Wireless Toronto new mesh started with BATMAN but found OSLR more reliable. Using Open-mesh but moving to Authpuppy.
Wireless Toronto - the less you talk about Wi-Fi and the more you talk about mktg, the better you do selling business hotspots.
Wireless Toronto using AutoAP in part to just monitor what's going on, wirelessly, in their neighborhoods.
Open Wireless Networks http://consume.net in London
Consume fell out of use ~2003
OWN: nodes all clustered in London near Greenwich Park. Not 400 nodes
Good line-of-sight planning tool:  http://www.heywhatsthat.com/
Village Telco http://www.villagetelco.org/about/mobiles have brought telecom to Africa, but not Internet, yet...
Village Telco: it didn't take off
Village Telco: Built ""mesh potato"" - solar mesh device w/analog phone adapter; production units next month.
Village Telco: Mozilla is filming the project.
Village Telco: Retail cost $119
Wlan Ljubljana: http://wlan-lj.net Can't beat widespread fiber at $14/month, but each has excess capacity and willing to share.
Wlan Lj now up to 50 nodes & adding rural areas - now wlanslovenija! http://wlan-si.net entirely volunteers
Wlan Lj is clearly a group of hackers having fun, but it's not clear to me if they are really serving a need. Sustainability???
Wlan Lj has done solar nodes - 24 hr reliability but froze during the winter and wasn't restored until spring (cold on roof!).
Athen Wireless Metropolitan Network (AWMN) strtd 2002 because no DSL; open experimentat WLAN.
AWMN has some people who offer Internet access, but it's not the primary goal. Participants tend to be Univ types - young, educated.
AWMN mostly at 5.4 GHz with Linux and MikroTik routers. Islands of OSLR connected by BGP; 2505 nodes; 1100 backbone nodes.
AWMN speeds vary 11 Mbps to 150 Mbps. 730 access points. Organized by an association; events; community; no grants.
AWMN Recently, large deployments of 11n
AWMN As a local LAN, they have mirrors of many Internet services, also transliterated version of Google (Woogle), Yahoo, etc.
Richard MacKinnon of Austin Wireless follows me in the Freemium session. AW strt'd as all free, but wasn't sustainable.
MacKinnon: Austin Wireless used automation to replace 50 volunteers with 2 fulltime staff. Merchants pay to provide free access.
MacKinnon: Pull together local news for hotspot portal pages
MacKinnon: Used to charge $5/mo
MacKinnon: Integrate Facebook into portals, incent patron's to talk about the business they're visiting on their FB page. Biz value!
MacKinnon: installations were fun at 1st
MacKinnon: $55/mo buys support. Restaurants hate Wi-Fi but have to have it to be competitive. So support is key - ""power cycle box!
MacKinnon: deals with cable company; POS crdit crd installers.
MacKinnon: Ads didn't work but local ads may be coming back.
Nemanja Topovic, Belgrade Wireless, Serbia started as all volunteers. Had problems with Government (spectrum laws).
Topovic low cost svc didn't work (1 Euro/mo) as people expect full svc. Discusses many paths they've tried, unsuccessfully...
Topovic - network grew rapidly 2004-2006 but growth has stopped. Looking for a program that could restart their network.
Topovic: BGWireless not a mesh, uses high speed P2P and P2MP. Closed network.
MacKinnon suggests his biz mode for Topovic. Key is offering free svc to avoid spt issues and then find a premium svc to cover costs
MacKinnon - important to offer new paid service as something new, not as a price increase on old service.
MacKinnon: 200 customers today (and 1 & 2 yrs ago) but different group & paying more & more loyal. Free customers were least loyal.
MacKinnon user community divided: some just want Internet
Paul from NFP? talking about SW defined GNU radio work going on in the building. Available for discussions later...
Robin Chase, Meadow Networks (previously founder of Zipcar) will be evening keynote spkr - next up.
Robin Chase on tie-in of transportation and networks: financing (fuel tax moving to road tax eventually - per km!).
Chase Auto density in cities (congestion) but expense of rural rds. Moving to congestion pricing. Will need more technology...
Chase on transport costs not reflecting true costs (fuel/environment, etc.)
Chase Public-Private Partnership discussions are missing the individual. Transport tech has decided they need their own stuff.
Chase: transportation guy gets his spectrum; EMS/medical types need their own stuff- it's just comms
Chase: Transport problems: Lumpy density, congestion, financing, right pricing same problems as in comms infrastructure -> dist. NW
Chase: future for transport and comms is distributed networks
Chase focus on distributed and collaborative inputs as a path to innovation.
Chase: words to use when talking about Gov. spending: open data; open standards; open source
Chase talking about Comuto (ride sharing in France)  http://www.comuto.fr/ now has more unique users than ZipCar (& just in France).
Chase: talking about CouchSurfing: 7 yrs old; 200 countries, 71K cities; more ""beds"" than major hotel chains.
Chase on Chatroulette - built it in 3 days; but in 6 months it now gets 30M unique visitors - mindboggling!
Chase: Andrey could do ChatRoulette because platform (Pcs, Internet) was there and there was excess capacity available.
Chase on all the crazy ideas that are now iPhone apps and yet the excess capacity has fostered some incredible innovations.
Chase: People & Platforms -> Speed & Scale
Chase wants open mesh device in every car that will have to be paying congestion fees, getting traffic data
Chase wants an open platform in cars so we foster new apps we can't can't even envision now.
Chase: what if all nodes (smart grid, smart cars, smart infrastructure) were peers? and open platforms!
Chase: www.networkmusings.blogspot.com & @rmchase & rchase@alum.mit.edu
Vic Hayes of TU Delft University: Spectrum Assessment for Wi-Fi. History of FCC & license exempt spectrum and Wi-Fi market.
Hayes: FCC landmark decision 1985- license exempt; use more spectrum than required; spread spectrum tech.
Hayes: Standard CDMA history including Hedy Lamar's patent, but I learned something new - she was born in Vienna!
Hayes has some good slides to explain CDMA. I'll probably stick with my standard slides http://su.pr/1LMCYl
Hayes: FCC wanted to allow spread spectrum over wider bands but got objections so they settled on ISM because no one cared.
Hayes on attempts to get similar rules thru CEPT - succeeded in 1991 for 2.4 GHz band only and with slightly different rules.
Hayes: CEPT said only -10 dBW ERP (100 mw) and 10 mw per MHz
Hayes: because of cost of electronics, 900 MHz took off 1st (in 1989)
Hayes on how Wi-Fi 11 Mbps beat HomeRF even though the FCC permitted wideband hoppers (1998-2000). Wi-Fi faster
Hayes: 2002 FCC permits intelligent hoppers (reduce blutooth intf) & power spectral density rule (opens the way for OFDM, i.e. 11g).
Hayes: Now 5 GHz - FCC NII proceeding - Apple, Lucent, etc. release 1997. Adds 5 GHz spectrum.
Hayes: meanwhile CEPT yielded to Satellite industry and reduces pwr at 5 GHz, but add more spectrum for HIPERLANS (455 MHz).
Hayes: CEPT decides to go to WRC 2003 to make 5 GHz primary and global. US problem because NTIA refused. Took until Jan2003 to win.
Hayes: In June03 WRC 2003 allocates 455 MHz co-primary in 5 GHz band. Accepted in US & EU - still in flux in many countries.
Hayes: Once spectrum is allocated, it still must be defended (find reasons in style with current political agenda).
Hayes promoting his upcoming book: The Innovation Journey of Wi-Fi, Edited by Vic Hayes et al. to be published by Dec2010.
Clarification of 1st unlicensed spectrum was 1937 for baby monitors, etc.
Hayes: Questions about Ad Hoc mode
Rabi Karmacharya & Basanta Shrestha of OLE Nepal using wireless to deliver Internet connectivity to schools in Nepal
OLE Nepal: Initial focus (2006) was OLPC, but if you got PCs, there was still no Internet
OLE Nepal: Don't have to teach kids how to use computers - that comes naturally - but need to create content in local language.
OLE Nepal: Teachers have to be retrained, and need to realize the kids may know more about the PCs that they do.
OLE Nepal: Need network infrastructure, in schools and between schools and the Internet.
OLE Nepal: Open source, open content
OLE Nepal: 28 people on staff
OLE Nepal: First priority is a network to and btwn schools as Internet upstream is very expensive in Nepal.
OLE Nepal: One highway with fiber runs length of country in the southern lowlands (near India border). Northern areas to 8K meters!
OLE Nepal: fiber line is connected to India so Internet transit pricing has come down substantially, but still very expensive.
OLE Nepal: approaching 38 schools in six districts & 4K students connected. P2P WLAN w/ typ. 5-10 Km distances
OLE Nepal: ADSL where available, used with VLAN.
OLE Nepal: On the plains, need to find a hill to act as relay point. Sometimes >20 km links needed (example 26km).
OLE Nepal: In hilly areas, may need 3-4 relay points at remote sites (4 hr walk)
OLE Nepal: School NW have: radio; switch; & 2-3 Wi-Fi Aps; all w/UPS. Select equip. for low power! Linksys WRT54GL w/DDWRT.
OLE Nepal: MikroTik 433 AH & 411 CPE - have used this up to 26 km. Also EnGenius radios, but not reliable.
OLE Nepal: SW tools: Google Earth, Radio Mobile, mirror tst'g. Use trees as towers (cut away foliage).
Ben West of Wasabi Networks (St. Louis) has posted his Community Wireless Day 1 notes here:http://su.pr/3FG39a
Ben West's Day 2 notes include notes from the session I spoke in: http://su.pr/1DHUFi
There's a big EU network that's not here. It's Czech Freenet: http://su.pr/2qMyeE 
Whoops - and this Czech Freenet: http://su.pr/24yYU8
Future for Community Wireless Networks session lead by Aaron Kaplan and Vic Hayes. EU wired net getting better and CWN shrinking...
Future of CWN: ideas fr. grp: underserved rural areas; hacker grps want to stay that way.
CWN futures: my categories: hacker comm, social comm & svc provider. All NWs have some of each, but one tends to be primary.
CWN futures: several people object to categorizations (in general?)
CWN futures: Movement to integrate node databases fr. many diff. networks. Exchange knowledge on making node db's. Mtg in <6 mo.
CWN futures: A world node database would create a larger vision of the movement, espc. for members in individual networks.
CWN futures: some desire for turnkey node solution prompts objection from Berlin hacker community that people would stop learning.
CWN futures: Ramon (guifi) sees evolution in the discussions over past 4 yrs: more consensus, meta focus (e.g. world db).
Networking rural areas: Rantanen - Tribal NW mostly done with grant money.
Networking rural areas: Rantanen offerring his cast off equipment to Krusevac Open
Networking rural areas: Krusevac network is technically illegal under Serbian law (ISPs must be licensed w/many problems & cost).
Telecom for disaster relief - Mark Summer of Inveneo talking about wireless in Haiti after the earthquake -http://www.inveneo.org/
Mark Summer: Mountain range around Port au Prince was key
Mark Summer: so many Sat Phones came into Haiti that satellite capacity was saturated.
Mark Summer: got Ubiquiti to get stuff from distributors (in 3 days)
Summer: Kitted everything over a weekend; VSAT vendor committed 5 Mbps link if dish fixed.
Summer: Google got very high res imagery on-line within 48 hrs and updated it every few days. Open Street Maps for former streets.
Summer: got VSAT up in <24hrs
Summer: Got 3-4 radios in per day
Summer: Linked 18 NGOs at 23 locations in 3 wks; NGOs didn't want it turned off, as better than pre-disaster.
Closing keynotes just starting. Tomorrow: open spectrum alliance
Aaron Kaplan thanking many, many. Node DB SIG being set up for this coming winter.
Sascha Meinrath asking for ideas we should ponder over next yr. Answers: 1. OS physical layer (GNU radio?) - seeking FPGA designers.
Answers 2) Allison Powell (@postdocal) learned we're still not good at telling stories. She's seeking people to interview.
Answers - Rabi Karmacharya (OLE Nepal) very struck by the discussions of biz models and how NW can help in disaster recovery.
Audience comments: worried that several major networks are loosing nodes
Audience comments: Session on splicing fiber was exciting. Community radio + community fiber (+ community satellite?) !
Audience comments: Credit to Matt Rentenen for offerring to pass on older equipment. Sascha suggests an email to CWN list.
Audience comments reinforcing idea of using Wikipedia as the master list of Comm Wireless NWs + spectrum laws http://su.pr/2C3YFz
At OSA Mtg http://su.pr/2AtH5O 1st formal mtg took 1.5 hrs as bylaws are in German & English. Hopefully we'll talk spectrum soon.
OSA Mtg: @postdocal asks OSA view on net neut & suggests links with other advocacy grps. Open a rathole? Sent for email discussion.
Now in GNU radio discussion. Ah techies, > interesting than policy talk.. Getting into to USRP http://www.ettus.com/ which I already know of.
Paul Fuxjaeger on GNU radio: OFDM for 11a, 11g required substantial mods to GNU radio blocks. Also hard meet IEEE timing req.
Fuxjaeger, GNU: For 2x2 MIMO, externally sync 2 USRP2s (not w/Ettus cable!) so as to maintain 2 Gig data streams, vs 1 w/Ettus cable
Fuxjaeger: http://www.oz9aec.net has interesting GNU radio stuff, also an update on Gumstix & GNU radio! More shortly...
Fuxjaeger: http://su.pr/A899kx has the pointers for Gumstix and GNU radio
Fuxjaeger: http://su.pr/1TZZeQ is the PR for Gumstix's product Stagecoach which packages TI OMAP processors that may be used w/Ettus
Kaplan wraps up the GNU radio session w/demo pasting macro blks & receiving signal from a Ham radio in 70cm band, i.e. 420-450 MHz.

Good conference Wi-Fi

Thanks ISWCN!

 

That's especially nice as it let me upload my photos from yesterday afternoon's stroll around Vienna -- all completed in a couple of minutes while waiting for the conference registration desk to get set up.

 

International Summit for Community Wireless Networks

I'm off to Vienna to attend a conference.  Conference info is here.  On Friday afternoon, I'll be leading a session in which I hope to get useful feedback on the "freemium" business model we're using for netBlazr.

Europe has a number of community wireless networks that have been very successful.  The networks in Vienna, Berlin, Athens and Catalonia stand out in my mind.  Of course, in the US we've had active legislation that's made it hard for governments to create community network.  But the EU networks mentioned above are member-based.  They may have benefitted from government funds at some point, but they formed independent of any government effort (at least as I currently understand things).  The other issue I've observed in a number of US community networks is a reliance on grants that aren't renewed and/or on enthusiasts who eventually leave to go to grad school or otherwise move on.  Something is different in the EU networks mentioned above.  Hopefully I'll better understand this by Sunday.  :)

As is my habit, I'll be taking notes in real time using Twitter (@brough).  After the conference, I'll gather those notes into a single page for a blog post.  This is primarily for my own benefit, as it's been an extremely useful way for me to take and save notes (assuming it's an event like the ISCWN where I want to keep notes!).

AT&T Wireless still struggling with the basics

Has AT&T gotten better since I last wrote about their network performance at some length?   Maybe not ...

 

Comcast: Use your bandwidth and we'll throttle you

Recently a friend decided to use an on-line service to backup his 750G raid array. He selected CrashPlan which, among other things, generates a projected completion time. With his existing DSL service it was going to take months, so he signed up for Comcast Internet with a plan that offered "up to" 10 Mbps down and 2 Mbps up. As he put it:

... when I was signing up for Comcast I specifically asked about high-volume on-line backup, and in particular whether it was Ok for me to buy the 2x10 service and run a heavy duty upstream job 24x7 at 1 mbit, and I was reassured that this was OK.

To see what happened, look at the "wan-out" statistics in gray above. Comcast came on-line in late March and CrashPlan was configured to use 1 Mbps of the "2 Mbps upstream" bandwidth. This worked for less than two weeks, then Comcast throttled the upstream to less than 600 Kbps.  Over the next six weeks, he was further squeezed to ~400 Kbps. Nothing in his house changed during this period, so the data above reflects Comcast's performance. As my friend puts it:

I guess what they meant was "... it's OK to try, since we can rate limit you ... to anything we want, you won't bother us by trying.  You won't get your bandwidth either, but that's not our problem. "

Comcast:  "Up to X Mbps," but only for occasional use.

Commercial use of TV White Spaces in 2 years?

One of the interesting discussions at the Wireless ISP Association meeting in St. Louis last week was around TV White Spaces.  Alex Goldman has a good summary from the WISPA point of view.  We also heard from Julius Knapp of the FCC during lunch on Thursday.

What's clear:

The FCC is going to act, in the 3rd quarter (before October 1st), on the 17 petitions for reconsideration that are pending.  While the details of their decisions will have to wait until their announcement, it's likely the final rules will allow viable commercial markets to develop.

At the same 3Q meeting they are going to adopt policies that allow multiple independent database managers to compete.  (There will be a mandatory coordination function).

My conclusions:

Once the final rules are determined, we'll see both Wi-Fi and WiMAX equipment release - in both cases by re-banding existing equipment and adapting existing antenna technology.  Over time, other products should emerge, but rebanding existing standards will happen first.  We should see products that WISPs can deploy within 2 years.

My notes from the WISPA conference in St Louis

The following is based on my Tweet stream (@brough) during the conference plus some other notes I took off-line during the Tower session (when there was no Wi-Fi).  

Brief summary:  The conference was a great success, certainly for me, but seemingly for all.

Notes from the Wireless ISP Association (WISPA) meeting
St. Louis, July 21-22, 2010

WISPA meeting scheduled to start @8:45am (20 min) but room near empty. Sleeping late in St. Louis? Conf Agenda, see http://su.pr/24Zlii near bottom.

Twitter's location service keeps telling me I'm in Berkeley CA when I'm in St. Louis. Having to reset it makes the svc less than helpful.

The 40 Mbps wireless link set up for this meeting died in last night's lightening storm. Hotel has DSL? ugh. Hope radio get fixed!

Rick Harnish speaking: WISPA has signed up ~50 new members just because people wanted to come to this meeting.

Nice to finally meet Matt Larsen after many years of seeing him on email lists. Matt comments that WISPA membership now over 400.

Steve Coran, Doug Karl, Dewayne Hendricks, Patrick Leary visionaries panel is up, but 1st the ""Did you know, shift happens"" video...

Dewayne H started as a ham (age 12, 1961), then 56kbps packet radio in 1986; went commercial 1990 - wireless Internet, mesh, early!

Doug Karl started in IT as OSU, needed Internet for off-campus folks and started P2MP wireless net.

Patrick Leary started in fiber but focus is broadband Internet and that led him into wireless (initially in Atlanta ex-urbs.

Doug Karl became advocate for WaveLAN (from NCR) radios and their use for P2MP, then an advocate for early WISPs.

Dewayne Hendricks - the gating issue is regulatory. Original NOI discussed open spectrum, no power limits! Today rules are tightening.

Doug Karl commenting on having your town put extra requirements on cable company when up for renewal.

Dewayne's mission impossible plan for spectrum - demo alternate spectrum approaches in other countries (e.g. Tonga); existence proof!

Dewayne: reinvent, repurpose mass-market stuff - Wi-Fi silicon, DOCSIS modems (use transverters!).

Dewayne is pessimistic about starting a new WISPA today. I have to have an argument with him!

Doug Karl: remember you are in the communications biz - wireless may be only part of that (don't be wireless only).

Now it’s hard to pick which of three tracks... Whitespace policy; 4G; or VoIPoW. Love policy issues, know 4G & VoIP but wish I had all PPTs.

Raja Gopal - Alvarion - pushing WiMAX. $5K for a 16e basestation today but admits in 2-4 years LTE will win. But deploy WiMAX now.

VoIPoW discussion justified by how much money can be made with voice, but need QoS and high level of service - must be dependable.

TV White space discussion: politics! politics! chg sensing requirements; still large antenna issue.

Marlon Schafer asks about eventual pollution of TVWS by home LAN devices. Jack Unger on current interference mitigation discussions.

another 3 tracks decision: Mktg; FCC's 3rd way regulation; or Fiber deployment. But the only Wi-Fi access in the Mktg room. :(

Mktg Elizabeth Bowles loves on-line but (on behalf of Forbes Mercy) pushing radio (PR & Ads) for rural WISPAs; also yard signs!

Mktg: Ken Janc, Lorex Inc, on targeted direct mail and door-to-door - expensive but can be cost effective (cost per acquisition).

Mktg: Ken Janc - at least have your installers do door hangers on next door houses when doing an install.

Elizabeth Bowles of Aristotle.Net on SEO, pay for advice if you are not found via ""broadband in <your location>"", get prof advice.

Elizabeth Bowles big proponent of video on your website - dramatically boosts response rates.

Elizabeth Bowles: mobile phone enable your website. Remember smart phones with Wi-Fi access. Unlimited mobile data is dead, so people using WiFi.

Elizabeth Bowles: need a Facebook Fan page! Twitter account; LinkedIn; YouTube, Flickr, Yelp, etc. Track and respond.

Elizabeth Bowles: Optimized press releases (select key words to get coverage) is to get on-line coverage, not local press.

Martha Huizenga of DC Access a WISP in small DC neighborhoods: pick targets; then: postcards, signs <cars, yards>, local newspapers.

Martha Huizenga postcards are the only printed media; leaves them at dry cleaners & other local stores. Rest are PDFs by email/online.

Elizabeth Bowles responding to a Q: Market Wire costs $400-$700 per release; so search optimization your release first!

Elizabeth Bowles answering Q: need 3-4 touches to get a response. Link sign, postcard, website landing page to drive touches.

Michael Calabrese of the New America Foundation is luncheon keynote - talk is focused on open spectrum and especially TVWS issues.

Michael Calabrese: big problem - Genachowski's focus on auctioning large parts of the TVWS.

Calabrese: FCC, Congress & WhiteHouse intent on auctioning everything, including 20 TV channels (after paying off TV broadcasters).

Calabrese: If 20 TV channels went, after repacking broadcasters, there wouldn't be enough TVWS for an equipment market to emerge.

Calabrese: Spectrum sharing still sailing under DC radar; he's pushing that TVWS database extend to additional bands (e.g. military?).

Calabrese: we should use the same database to allow temporary access to ""warehoused spectrum"" i.e. until spectrum holder uses it.

Calabrese: Why shouldn't WISPs have temporary access to any spectrum band that is not yet in use in their geography?

Calabrese: FCC is not nearly as aware of the WISP community as it should be. Pls visit them; write to congress; file ltr to NOI.

Network Management: Matt Larsen, Vistabeam with Butch Evans; Brian Vargyas, Baltic; Alex Phillips, HSLink; Cameron Crum, Wispmon.

Larsen favors open source. Issues: integrate related items, respect business (may have to fire specific customers), leverage tech.

Larsen on processes: Billing, Provisioning, Dispatch, NW Mgmt (core & customer), Support, CRM, CustSat, Mktg, Employee accountability.

Larsen: Vistabeam has 1700 miles of wireless backbone over 600 miles. Documentation is critical before someone goes out in the field.

Larsen: Billing Freeside, Powercode, Platypus, ... He uses Freeside.

Larsen: NMS - What's Up Gold, Nagios, Xymon; proactively call people whose service is down. Also RT for trouble tickets.

Larsen on phone system - uses Asterisk but also familiar with Tribox and others. Msg when there's an outage cuts support calls!

Larsen on documentation - uses Network View, Visio, Dude and a wiki to capture other things (e.g. details of every tower).

Larsen reevaluate your performance metrics regularly. Ditto for biz processes.

Brian Vargyas on NM Mgmt: Be proactive, not reactive; BW usage; link history - Dude, Cacti, Nagios. New Dude 4.0 (beta) on SQL DB.

Vargyas - Dude works with any SNMP enabled device, not just MikroTik devices. Any MT board can be an agent to offload central box.

Vargyas - Cacti is open src PHP-based graphing system with web GUI but no notifications. Looks at anything with SNMP OID.

Cameron Crum of Wispmon - started a WISP, needed an IT infrastructure and a way to monitor his network. Visual kind of guy.

Crum is focused on demo'ing the Wispmon product.

Butch Evans speaking on QoS and prioritizing traffic to improve customers' performance - all on MikroTik. Others use Imagestream.

Butch Evans QoS script runs at an aggregation point. QoS won't fix massive overload problems; will help schedule during congestion.

Matt Larsen uses remote power control by Digital Loggers - auto power cycle if no ping on time.

Butch Evans QoS script for MikroTik is $175. Uses IP tables; categorizes most traffic and assigns it priorities; updated as needed.

Butch Evans was trying to emulate NetEqualizer when he started on his MikroTik QoS scripts.

Off to a Tower Technology session but there is no wireless in that room, so no Tweets.

WISPA Tower tech panel - Walked after they had started so I missed the introductions - not sure who is who…

Seems everyone is using 11b or 11g as they regard selecting polarization as critical.

Jack Unger (and at least one panelist) believes narrower channels (10 MHz or 5 MHz) cause the signal to go farther.   Statement that narrower channels have less background noise to contend with, thus the signal goes farther.  Given OFDM, this doesn't make sense to me.  Need to investigate what’s really happening.

One panelist argues for regulated power as the power supplies in the MikroTik's are cheap.

One approach to power and UPS is to use 24v PoE with a central 24v DC supply and two 12v deep cycle batteries.

Everyone advocates UPS.  One guy has 8 hours for every site.

At least one guy tells his customers to buy APCs for the client site.  Tornado zone, take laptop to basement and track tornado in real time.

Weatherproofing for coaxial connections:

1. Courtesy wrap (white tape) protects connector from butyl layer
2. Butyl layer (a mastic like putty)
3. 3M Super 33+ electrical tape (3 layers, shingled down on last layer) to extend at least an inch past the butyl layer.

The Andrew website has good tutorial on weatherproofing antenna connectors.

Use DC4 dielectric grease to protect Ethernet connectors.  Note this is not clear silicon! (even though it looks similar).

Inspect everything at least once per year.

Just posted some pictures from the conference here: http://su.pr/2XQuqp  More to come (will be added to this set!).

Conf has 253 registrations and 40 vendors. Mtg will breakeven or be slightly profitable. 43 new members joined in past one(?) month.

Board Mtg.: I should look into the Ambassador's program... They're looking for members to represent WISPA at regional events.

CALEA update - WISPA spec needs 3 updates to meet new rules; complex negotiation btwn FBI & WISPA. Next issue up is IPv6.

FCC Committee efforts: TVWS (not auctions); TDWR Database (stay 30 MHz away) - problems in Boston & PR); 3650 MHz (4 rule changes).

FCC Comm spent $5K in June, slightly lower than average; but advocacy and meetings are the bulk of what WISPA spends $ on.

Discussion of FCC-specific, or issue-specific, fund raising + volunteers to help engr'g (versus raising dues). Also partnering!

Steve Coran pitch at Marlon's request. (Rini-Coran is WISPA's law firm for DC politics).

Several members suggest that FCC Comm provide boilerplate that members can use to send letters to their representatives.

An aside in my twitter stream, not directly related to the WISPA meeting:
Where do telecom lobbyists come from?  There's a fantastic graph about half way down this page: http://su.pr/1tjZnZ    Hint: they used to work for congress!

Peter Stanforth, CTO, Spectrum Bridges up next on TVWS Trials. First focus on 2ndary mkts - time of day, location, freq. >10yr effort.

Stanforth has nice graphics on what TV white spaces are and how they arise. Hope to get his slides, eventually.

TVWS database administrator. Stanforth suggests we can have competition even here, and still accomplish desired results

Stanforth Summary of trials' field results in 04-186 FCC proceedings.  http://su.pr/2F9dOx

Stanfort: All TVWS trials have worked; getting 3x-5x coverage vs. 2.4 GHz systems; no interference complaints!

Stanforth: sensing unnecessary; drop antenna height limits; use fixed power limits & masks. Make DB higher fidelity than orig req'd.

Stanforth running TVWS BB antenna next to TV antenna in trials. Don't need separation! Must relax power limits and masks!

Stanforth notes that many TV stations are spraying out-of-band energy (a license violation!) but they haven't cared (so far).

Stanforth: if there are multiple database admins, they will coordinate btwn each other once each night, at least as proposed so far.

Rob Kubik of Motorola talking about FAA radars (TDWR) and DFS as basis for renewed outdoor use of 255 MHz of 5 GHz spectrum.

Kubik: experiments show 30 MHz away is ok, even in close (<35 km) to radar; combined w/DB approach should solve problem, but FCC..??

Jack Unger showing TDWR database which WISPA & Spectrum Bridge did jointly - going on-line next week. Meanwhile  http://su.pr/1Xk1SC

Stanforth showing prototype of TDWR database that will be up next week. Currently at http://su.pr/2iHAdO although that will change.

Panel on Buying and Selling WISPs. Prices seem to be from 1x to 2x revenues but many factors. Based on buyer estimate of cash flow expected after the transaction.

Across the board, panel is really cautious about any deal that involves stock or deferred cash. :)

Shiraz Moosajee's advice to buyers: plan on 20% reserve to fix problems and be sure you have 20% beyond that just in case.

Buy/Sell panel: 90-180 day clawback or true up clauses are common, with 20-25% in escrow.

Buy/Sell panel: Even if you don't plan to sell anytime soon, while growing your biz, keep revenue history, per tower.

Buy/Sell panel: Even though everyone prefers cash, very few deals are all cash (and they were fire sales).

Buy/Sell panel: Re buying small neighboring WISPS (100 subs) - frequently done with seller financing over 2-3 years. Estimate growth.

Buy/Sell panel: The smaller the network the less info you are liable to get!

Buy/Sell panel: Need transferability in all your lease agreements as buyers want assets (sellers prefer stock sale for Cap gains).

Buy/Sell panel: Sellers frequently have personal guarantees associated with corporate accts - got to know how to unravel these.

Julius Knapp  http://su.pr/9BXRJs  FCC, lunch keynote: Update on spectrum activities, but starts by hyping BB Plan.

Knapp on spectrum: ack's spectrum that's not yet built, but still pushing FCC/mobile operator's line: need more (licensed) spectrum.

Knapp promoting: spectrum dashboard; incentive auctions; more spectrum for mobile BB & for Backhaul - but this w.b. licensed. Ugh!

Knapp - only items that resonate with me: more unlicensed (Rec 5.11) and opportunistic use of spectrum NPRM (Rec 5.15) - Q3-2010.

Knapp & Ruth Milkman co-chair FCC Spectrum Task Force. Good, but he doesn't mention that FCC only controls part of the spectrum.

Knapp - Pres memo is forcing FCC & NTIA to work jointly on 500 MHz of new spectrum & Pres used words "licensed and unlicensed.

Knapp on TVWS 17 petitions for reconsideration. Expects multiple database mgs to be approved as part of this action.

Knapp acknowledges that 3650 MHz lite licensing has been a significant success, even as there is work afoot to improve things.

Knapp - TDWR protection w.b.: avoid freqs & 30 MHz guardband within 35 km. Make sure this works, so further licensing isn't required.

Knapp: the proceeding on backhaul scheduled for August 5th commissioners' meeting.

Knapp Q&A on Presidential memo suggests that ""licensed & unlicensed"" means some mix. How that's divided is to be argued...

Knapp says Aug 5th mtg will result in increased flexibility in use of spectrum for backhaul. But he wouldn't give details.

Knapp defends existing build-out reqmts. Charles Wu asks about Pt 101 licensing. A: spectrum transparency point is to find squatters.

Ray Savich of Moto describing software tools they have to help WISPs plan their businesses. These are biz tools, not just RF stuff.

Savich on ""Leased line payback calculator"" another tool to help sell WISP services.

Jay Lawrence of Gigabeam showing wireless carrier in Las Vegas who's built a wireless carrier Ethernet network using >50 1Gbps links.

xG Technology is here. Have dropped their original magical claims http://su.pr/1fBB9d for some completely conventional RF technology.

xG now has a proprietary mobile voice solution that runs on unlicensed spectrum. Not interoperable, but more a community cordless solution.

Cameron Crum of Wispmon demo'g their WISP-in-a-box IT solution including qualifying prospects against towers & terrain map. Very cute!

Crum emphasizes taking & preserving pictures of every install indoor incldg where power is connected (tell cust. how to power cycle).

Robert Olive of Wispmon is now pushing their new iPhone apps that work with Wispmon.

Nathan Stooke of WisperISP started in 2003; took a yr to breakeven; 3 yrs to make a salary; today, $2.4M revenue & 20 employees.

Stooke focus on best network and best customer service; has since taken over 15 competitors (many evolved from dial-up days).

Stooke: competitors were dial-up ISPs who entered wireless reluctantly. Diff was he wanted to build a wireless NW and he focused on svc.

Stooke: struggle to find good people; hirer slowly, carefully; some start as contractors, become employees later. He doesn’t provide benefits...

Stooke: employee training espc customer svc - one full day a month devoted to on-going training.

Stooke: run it as if you were setting up to sell it, i.e. keep it clean, fix problems, don't let them linger. Make it a showcase.

Stooke: Has a fire truck with 100' extension ladder that he got for $1K (but spent $26K getting fixed & certified) = temp tower.

Jim Murphy of NetSapiens describing their VoIP solution for WISPs - scales up, also down to system for just 10 simultaneous calls.

Murphy positioning against Asterisk (lower opex, expertise) and hosting (facilities based, more control; potentially better QoS).

Murphy also offering local virtual switch where realtime elements are local but NetSapiens handles everything else.

Now I have to leave to catch a plane, so I'll miss the final prize drawings. :( But this has definitely been worth it!! 

Thanks WISPA!!

WISPA - Wireless ISP Association meeting in St. Louis

I've just landed in St. Louis for a WISPA regional meeting that sure looks national to me.  Of course, I'm from Boston but the first person I talked to was Jim Murphy, COO of NetSapiens who had just flown in from La Jolla California. 

I'm new to the wireless ISP business but it was nice to see several familiar faces in addition to Jim (who I know from my VoIP days in the late 90s), for example, Alex Goldman, formerly of ISPCon and now writing for WISPA and Donny Smith of Jaguar Communications who I had met at FiberFete in April.

During the conference I'm be taking notes via twitter:  @brough

Word use as a key to US Broadband Plan

The inestimatable Frank Coluccio has finally started to read the US National Broadband Plan but, before starting, he did a very interesting document analysis.  He counted the number of times various words and phrases are used.  Here's a suggestive subset I picked out from Frank's more detailed document.

Commons - 1
Symmetric (not "symmetrical") - 1
Backbone - 1
Line of sight - 1
Gigabit Ethernet - 4
Dark Fiber - 9
Wi-Fi  - 10
FTTH/B/C/P (combined) - 20
Cellular - 31
Backhaul - 39
DSL (all types) - 40

TV - 122
Video - 201
Rural - 266
Wireless - 377
Mobile - 397
Internet (all contexts) - 453
Government - 669
Spectrum - 715
FCC - 1439

Stealth no more! Also, I need your vote

A few of you have commented on my various profiles that say I'm associated with Ashtonbrooke Corporation or with BigBroadband.Net.  Of course neither website provides much information...

Well our service launch is still weeks away and we haven't found a name we like yet, but we have entered the MassChallenge and that means we've revealed what we're up to. MassChallenge is running a competition for startups and they're giving out grants.  Since we are entirely self-funded so far, a grant would be sweet.

To see our entry you must register at the MassChallenge website, http://www.masschallenge.org/user/register?role=Supporter and then go to Competition, Team Pitches, search for "BigBroadband.Net" and click Details.

Once you are there, you can not only get an idea of what we're up to, but you can vote for us!  Ideally before midnight tonight.  :)

Please Vote for BigBroadband.Net

As a registered supporter of MassChallenge you can rank any team's entry.  Just click on the right most of the five stars under "Ranking" on the right of our details page to give us a 5 Star/Awesome rating, ideally by midnight tonight.

Thank you.

Longer term..., we're seeking a new name

Once we roll out our pilot network (in Boston later this summer), I'll have a lot more to say about how we expect to pull a wireless end run around the duopoly, at least in dense urban areas in the US.

Meanwhile, we are seeking a better name. WirelessEndRun (our internal name) is either brazen or obtuse, or both.  BigBroadband.Net says something about what we're doing but has a lot of negatives.

All suggestions gratefully welcomed.

Audiocast tomorrow: Satellite phones, spectrum games and more

Tomorrow, Tuesday the 15th at 11am eastern time, I'll be live on an audiocast organized by Carl Ford of 4GWE and TMCNet.  

The title is "4G is in the Stars: Satellite to Terrestrial" and speakers include Barlow Keener, Lyman Chapin and yours truly.

The prize is 68 MHz of spectrum at 1.4 GHz.  In the US and Canada, this spectrum has been allocated for Mobile Satellite systems but it can also be used for Ancillary Terrestrial Services - that is services that are ancillary to the satellite services! This spectrum would be good cellular spectrum if you could get it, being above the 800 MHz original cellular bands but below the 1.9 GHz PCS bands. Technically, the satellite signals go up and down while the ancillary signals go mostly horizontally, so they are relatively independent. The big question is: Can anyone afford to launch satellites and run satellite services just so they can also use this spectrum for LTE services at ground level?

Gigaom has been skeptical, but when you stack up the estimated $5B to build a network against the possible value of this spectrum at auction, i.e. 68 MHz at US$1.2 per MHz-POP (the price paid in the US 700 MHz auctions) and 335 million POPs in the US & Canada you get roughly $27B. That math could work...

Join us tomorrow at 11am eastern by registering here.

Top cities ranked by Broadband Speedtest results

 

For those who are US-centric, note that MD is not Maryland, it's Moldova!

These results are aggregates of measurements made at Speedtest.net.  These and other results have been published by Ookla (the company that created Speedtest.net and Pingtest.net).  See Net Index for more results.

It's interesting that many of these cities are in eastern Europe, e.g. Latvia, Moldova, Romania, Bulgaria and the Ukraine.  Indeed, three of the top 15 cities are in Latvia.  I don't have current data on Latvia but I wonder if it benefits from the same relatively free access to rights-of-way that is present in Bucharest and several other parts of the world.

Africa moves from satellite to fiber links

2009-2010 marks an exciting transition for the Internet in Africa. 

Historically, most countries had only satellite connections or a connection to a single submarine fiber optic cable (SAT-3) controlled by an incumbent monopolist. Not surprisingly, prices for SAT-3 bandwidth in the African countries it serves were high (US$4,500–12,000 per Mbit/s per month, vastly more than than bandwidth prices in the US or EU).

But finally, things are changing, radically.  Beginning in 2009, the Seacom cable connected eastern and southern Africa with Asia and the TEAMs cable reached Kenya. Now four more cables (Glo One, Main One, EASSy and LION) are coming on line in 2010, with WACS and ACE expected in 2011-12. Steve Song has an excellent map and more details here.

Impact on Satellite services

Balancing Act Africa has a new market study out which projects African bandwidth markets through 2014. Among other conclusions summarized in recent letter from Russell Southwood:

The overall balance of satellite vs fibre use in Africa’s top Sub-Saharan markets 20 markets goes from 45.6% vs 54.4% in 2008 to 11.9% to 88.1% in 2014. Of these top 20 markets, the top 5 markets (South Africa, Nigeria, Kenya, Angola and Sudan) make up the majority of bandwidth demand across the continent and are the countries that will experience the fastest bandwidth growth over the period. So satellite operators and resellers will not only be selling into a smaller share of the overall bandwidth market but will become niche players in many countries. There will be growth but most of it will be on fibre.

After three and a half years since December 2005 in which only ten satellites were launched with coverage over Africa (four of which failed), at least 36 new satellites will be launched by the end of 2013, costing between US$4.4 billion and bringing the equivalent of 26,325 MHz of additional capacity over the region. By the time the second constellation of O3b satellites is launched, there will be almost as much Ka-band capacity over Africa as the total stock of C- and Ku-band capacity.

The satellite operators have invested huge amounts in new and replacement satellites. US$ 4.395 billion will be invested in new and replacement satellites against US£2.15 billion in seven international submarine cables, excluding ACE where the budget has not yet been announced. The satellite operators are investing at a time when the tide is turning against satellite use.

What are satellite operators thinking?

Broadband speeds - Boston and Bucharest

Recently I was conversing with a friend from Romania and we got to discussing consumer access to the Internet. Not surprisingly, connectivity in Bucharest is substantially faster than what I have here in the Boston area (even though I have Verizon FiOS service).  Anyway, rather than go by two isolated samples, I thought to look at the international records at speedtest.net to see averages of many, many measurements.

Download measurements

   

Notice that the first three measurements on the Boston side are business services, not things a consumer could afford. On the Bucharest side, Voxility is a wholesaler but #2 and #3 are selling to anyone.

Upload measurements 

   
 Yes, it's one rather specific set of measurements, but still...

City comparisons

The greater Boston metropolitan area has a population of ~4.5 million making it the largest city in New England and the tenth largest metropolitan area in the US.  Bucharest is the capital of Romania and the Bucharest metropolitan area has a population between 2.15 and 3 million. Bucharest is a bit older than Boston and today has a bit higher density, while Boston has somewhat more high rise buildings.  Still the cities are not so dissimilar that it would explain the wide divergence in speeds of Internet connectivity.

Structural Separation: Open for discussion, at last?

On April 1, 2003, I gave a keynote address at the Spring VON conference in San Jose.  I opened with some misdirection about a fiber project, implying it was near my home in the US.  The project, which delivered 100/100 Mbps fiber to the home for $24/month, in 1999-2000(!), was real enough.  But it was in Ulmea Sweden.  Even at that time, I had been talking for several years about the need for either: user control of first mile fiber or multiple competitors gaining open access to dark fiber in the first mile.

But whenever I used the term "structural separation" in the presence of US policy wonks, I was put down with comments like "Be serious. That could never happen in the US."  Indeed, while the idea is openly discussed in international circles, in the US, only fringe people mention structural separation.  The friendliest comment was from a communications lawyer acquaintance who advised me to avoid the words 'structural separation' if I wanted anyone to take me seriously.

So I downplayed those specific words but continued to advocate open access to dark fiber.

Good News!  The words 'structural separation' may no longer be verboten in the US.

While the US Broadband Plan avoids mention of any kind of structural reform, the topic is finally showing up in mainstream US discussions.  Harold Feld (Public Knowledge) directly mentions structural separation in his comments on the plan.  Esme Vos has slightly different views but is also "appalled that structural separation is not part of the Plan."  And when Paul Budde (admittedly an Australian) opened the topic, he prompted a vigorous discussion from Richard Bennett in his comments section.

A Long Way to Go

The US has a long way to go and the US Broadband Plan is not bold enough to get us there, but at least it is now possible to discuss a major issue in public in the US.  Let's be thankful for any step forward.

eComm Panel Discussion on the US Broadband Plan

At the Emerging Communications Conference in San Francisco next month, I'm organizing a panel discussion on the US Broadband Plan. Independent of my personal views, I've tried to get panelists with as diverse a set of views as possible.  Controversy always makes for a more interesting, and more informative, session.

Growing interest in TDD versions of LTE

A few weeks ago I put the provocative title "LTE and spectrum stupidity" on a blog post, prompting a flurry of objections in the comments.  (Thanks guys. I enjoy controversy).  My complaint was that, despite defining both FDD and TDD versions of LTE, the industry had only implemented the FDD version even though TDD can better match available capacity to data traffic demands that can vary widely between 10 ms LTE frames.

It's obvious from the comments that I didn't win over all objectors and I haven't had time to write an article on short term (10 ms) Internet traffic variations, but I did notice this article at dailywireless.org and, in particular, this article at Light Reading which lists a large group of companies, both operators and equipment providers, who have supported a request by Clearwire that the 3GPP specify how TD-LTE should work across the 2.6 GHz spectrum.  In addition to Clearwire, 

... there was broad industry support for the proposal from other companies, including: Sprint Nextel Corp. (NYSE: S), NII Holdings Inc. (Nasdaq: NIHD), China Mobile Communications Corp. , UK Broadband Ltd. , Motorola Inc. (NYSE: MOT), Huawei Technologies Co. Ltd. , TD Tech Ltd. , WiChorus Inc. , ZTE Corp. (Shenzhen: 000063; Hong Kong: 0763), Chinese Academy of Telecommunications Technology , Nokia Siemens Networks , Cisco Systems Inc. (Nasdaq: CSCO), Sequans Communications , Alcatel-Lucent (NYSE: ALU), Alcatel Shanghai Bell Co. Ltd., and Rohde & Schwarz GmbH & Co. KG.

Part of the 2.6GHz band is already specified for TDD, namely the 2570MHz-to-2620MHz band. The new work at the 3GPP will ensure that all of Clearwire's spectrum will have a definition for TD-LTE operation and will get the TD-LTE specs in line with Federal Communications Commission (FCC) requirements as well.

It will take a few years to play out, but we should get real data on real traffic using TD-LTE within five years.  Before then, I'll try and find time to gather the data and write the definitive article on millisecond-to-millisecond Internet traffic variation (but not in the next few weeks - sorry).

As the dailywireless.org article says, "In a data-driven world, symmetrical pipes can be a waste of space." And even when long term data averages are symmetric, short term variations in IP traffic are extreme (see the 3rd diagram here: http://su.pr/1Vmge3 ).

Community fiber ripoff - Exclusive marketing & bulk service agreements

Four years ago I read an article in Broadband Properties magazine with frightening implications.  This caused me to write a blog post, Community Fiber - Read the fine Print. Well, through to the wonder of Google, I've just been connected to a group of people who were caught in exactly the kind of trap I was worried about.

It works as follows:  The developer of a planned community or condominium enters into an exclusive marketing agreement or a bulk services agreement for communications services (Internet, TV, phone, etc.) for the community.  The contract is placed with a firm run by the developer's brother-in-law, or otherwise set up to benefit the developer.  The initial rates look good and prospects are sold on the idea that their new residence comes with state of the art communications services.

Later, after the developer has sold most units and the community is up and running, it turns out the communications contract is not such a great deal, but it's a solid contract and it runs for 25 years!

That's right, 25 years.

As chronicled by BanBulkBilling.com, this is now a problem for at least many dozen communities developed by many different developers.  Of course there are consumer protection laws in most states, but that's a long hard slog - good for the attorney's but not so great for home owners associations.

The Wi-Fi solution for mobile data offload

A recent Google alert pointed me to this article which I wrote in December 2009 for my monthly column in Internet Telephony magazine.  It was published in the February issue and has only now become available on line. None-the-less, it's points are timely. It opens with:

LTE versus WiMAX is a standard topic in the press and at conferences, as if something disruptive was happening or might happen. Wrong!  WiMAX and LTE are technical variations on the same business model providing similar services. If we’re looking for disruption, we need to catch up on what’s happening with Wi-Fi.

and concludes:

Expect to see both fixed and mobile carriers including free Wi-Fi access in their subscription bundles as Wi-Fi trumps femtocells. Conventional operators are not going away but, over the next decade, it’s Wi-Fi that will shake up business models and drive disruption.

Check it out.

LTE and spectrum stupidity

Mobile operators are counting on Long Term Evolution (LTE) technology to handle surging demand for mobile data access.  But LTE developers made some poor choices, cutting spectral efficiency and thus driving up operator costs.

LTE was envisioned as an all IP system, but the RF allocations follow the voice-centric approach of earlier generations.  While LTE standards allow for either Frequency Division Duplexing (FDD) or Time Division Duplexing (TDD), all initial LTE equipment uses FDD.  FDD requires two separate blocks of spectrum - one for each direction.  FDD makes perfect sense for bi-directional voice traffic. It makes no sense for data. With the exception of peer-to-peer file sharing (which most mobile operators block), data traffic is very asymmetric. Sending data via FDD means one block of spectrum is fully utilized and the other, equal sized block, is dramatically under utilized.  Result: the operator pays for almost twice the spectrum they actually use.

Verizon is deploying LTE in the 700 MHz C block which means they are using 746 MHz to 756 MHz (a 10 MHz channel) for their downlink (to the mobile device) and wasting most of 777 MHz to 787 MHz (another 10 MHz channel) for the uplink. If Verizon could deploy TDD (as used by WiMAX and as defined for LTE but not implemented), they could fully utilize both 10 MHz blocks for data tranfers, almost doubling their data capacity.

I don't know the actual capacity Verizon will realize on average with their first generation LTE infrastructure. But suppose Peter Rysavy is correct (as implied by Gigaom) that Verizon will initially average 15 Mbps per 10 MHz channel. That's 15/15 Mbps, symmetric, even though average traffic is likely to be 15/2 Mbps. No single user is likely to see 15 Mbps; rather that 15 Mbps is shared among all users in that sector. With TDD (the default for WiMAX and an unimplemented option for LTE), the Verizon spectrum could support two channels of perhaps 13/2 Mbps each in that same sector. Again, no single user will see 13 Mbps, but all the users in the cell will be sharing 30 Mbps of capacity that can be dynamically divided between up and down - mostly like averaging 26/4 Mbps but able to allocate 15/15 or 28/2 as the traffic mix changes.

It's ironic the LTE implementors got this wrong when you consider their decision to use only IP in the rest of the LTE design, thereby dropping support for traditional voice or SMS services.  That's right, initial LTE deployments won't support voice telephony or SMS messages, only data services, and yet LTE spectrum assignments were made as if voice comes first.

That's ironic.

Broadband Wireless — the future is 5 GHz

and eventually, spectrum from 4 GHz to 10 GHz.  

Unfortunately, the mobile industry doesn't understand the implications of MIMO or beamforming, so they are asking for the wrong thing.  They are asking for more spectrum near their current bands (below 2.1 GHz) or lower, e.g. in former TV bands below 700 MHz.  This is all wrong.

Until recently you could say: lower frequencies "work better" meaning they go farther.  But this was a technology limitation, not something in the physics. From a physical point of view, 5 GHz photons pass through the atmosphere just as well as 700 MHz photons. Today, MIMO and beamforming are correcting some of these historic technology limitations and changing everything.

• MIMO not only delivers significant performance improvement, it makes 5 GHz spectrum as useful as lower frequency spectrum for all sorts of applications.
• Beamforming delivers the next performance increment (20x and up), and it's actually easier to implement at higher frequencies.

Addressing mobile data demand

There is no doubt data consumption is growing more rapidly than mobile operators' data capacity (see slides 10 and 11 of my presentation at 4GWE). Unfortunately, US cellular operators are pressing for more low frequency radio spectrum and they are being heard in Washington. In his keynote at this year's Consumer Electronics Show, FCC Chairman Julius Genachowski said an impending shortage of wireless spectrum in the U.S. will dampen future economic growth unless action is taken to fix the problem. 

However, the short term issue is investment dollars not spectrum and the long term solution is new technology at much higher frequencies (above 4 GHz) rather than more spectrum for existing technology near existing bands.

Why 5 GHz spectrum is more useful than TV frequencies

Everyone know TV signals go long distances but remember than TV broadcasters use from 100 Kilowatts to 5 Megawatts ERP.  That's a million to a hundred million times more signal than your mobile handset puts out.  No wonder it covers a lot of distance.

There's also an equation most wireless engineers use in one form or another to calculate free space path loss which says higher frequencies have more loss. But this equation encapsulates two factors:  the true path loss and the size of the antenna.  It assumes a 1/2 wavelength antenna. Higher frequencies have shorter wavelengths, so it assumes the antenna gets smaller as the frequency goes up. Smaller antenna, less signal. With comparable antenna apertures, path loss in the atmosphere is the same from below 50 MHz to nearly 10 GHz. Thus in open air, 5 GHz photons go just as far as 500 MHz (US channel 19) photons or indeed photons for Channel 2 or Channel 50.

Multi-path and MIMO

The reason people have had trouble with higher frequencies for the past 100+ years is "multi-path" propagation.  As a signal radiates from a source, some of it goes directly to the receiving antenna but some of it goes in other directions where it may be reflected or refracted by objects it encounters. When reflected signals also reach the receiving antenna, they are slightly delayed because they traveled a slightly longer distance.  In the days of over-the-air analog TV, we saw these delayed signals as "ghosts" or shadows around images on our TV screens.  For digital data transmission, multi-path contributes to the "noise" in the signal-to-noise ratio. The historic problem with higher frequencies is their shorter wavelengths made it easier for them to be reflected and refracted by man-made objects like buildings, window frames and even closely spaced double pane glass surfaces.

 

But with MIMO, all this changes. With MIMO's multiple antennas and multiple radio front ends, it's possible to separately decode and make use of the multi-path signals.  Now "multi-path" is not only removed as a source of "noise," it adds additional signal and helps carry more data.

Beamforming

A beamformer uses signal processing to control the phase and relative amplitude of the signal at each of a group of independent antenna elements.  Radiation from multiple antenna elements causes a pattern of constructive and destructive interference in the resulting wavefront.  This can produce a tight beam just like one from a highly directional antenna.  But with beamforming, the antenna beam can be steered in software on a packet-by-packet basis.

With eight antenna elements spaced 1/2 wavelength apart (total 3.5 wavelengths), you can create a beam like this:

 

Highly directional beams significantly extend the usable range of a wireless system. And since the beam is computed with digital signal processing software, it can be steered to different directions in microseconds. What's more, the benefits of beamforming apply both while transmitting and while receiving. Either way, the beamformer accentuates the signal in the desired direction while surpressing signals to/from other directions.

But what about wavelength?  To obtain this narrow beam, the outer antenna elements are 3.5 wavelengths apart.  At 5.8 GHz, that's less than 7.5" so the whole antenna array easily fits in a ceiling mounted access point just 8" x 3" by 2".  At 700 MHz, that degree of beamforming still requires 3.5 wavelengths, but wavelengths are longer so now we need 5 feet of separation — something that may fit on a cell tower, but is difficult for a microcell and impossible for a femtocell.  For beamforming, higher frequencies are an advantage.

More spectrum at higher frequencies

Finally, there's a lot more spectrum potentially available at higher frequencies and, today, in the 5GHz band, there is over 555 MHz of license-exempt spectrum already available for applications like Wi-Fi. That's more spectrum than Verizon Wireless, AT&T Wireless, Sprint PCS and T-Mobile USA have, combined!

Wi-Fi blazes the trail

The Wi-Fi market place is vastly more diverse than the 3G/4G mobile operator market place, which means many new technologies show up in Wi-Fi years before they are deployed in mobile networks.  That is certainly true of so-called 4G modulation (OFDM) which was deployed for Wi-Fi with 802.11a (1999) and 802.11g (2003), years ahead of WiMAX (2005) or LTE (2010).

The 802.11n specification includes MIMO and optional beamforming and silicon technology is appearing to support 4x4 MIMO with beamforming.  MIMO products (2x2) have been shipping since 2007 and 4x4 MIMO in consumer products expected in the next six months. Meanwhile, many players are scrambling to deliver 11n options, including beamforming. Early systems are already deployed.

Mobile operators, please pay attention

The technology benefits of MIMO and beamforming apply to the mobile phone industry, it will just take a few years for the deliberate pace of the industry to catch up.  Meanwhile, mobile phone operators should be tracking real world measurements of Wi-Fi performance to understand what spectrum they will really need five to eight years hence.

Network Neutrality, Common Carriage and Layers - an international perspective

I'm attending an FCC workshop on 'The Open Internet" which means Network Neutrality. I am very much in favor of open access to the Internet. However, by settling for network neutrality rules, we won't solve the real problem, which is a lack of competing ISPs.  If we don't address this, we'll insure the US falls further and further behind other countries. Consider:

 

In the US, we claim to be supporting facilities-based competition. This means the rights-of-way are open to multiple competitors and you expect to see competing networks built from the ground up.  I call this open access at layer zero.  Unfortunately, this approach hasn't worked in the US. Most locations have only one or two access networks (the phone company-cable company duopoly).  

The reason is fairly easy to understand:

 

Building a new set of facilities represented a large fixed cost.  This investment must be paid back and typically only larger players have enough revenue to be able to make the interest payments. If you only capture 10% market share, you can't payback the investment.  As a result, facilities-based competition usually results in 2-3 competitors, at most.

Note that in Korea, facilities-based competition is part of a fairly complex history of government and private network evolution.  Hong Kong also has facilities-based competition, most notably by Hong Kong Broadband Networks Ltd.  But that's one fiber over-builder in a market that has open access to copper (i.e. layer 1 access) and many, many DSL providers. Facilities-based competition is a great idea, but by itself it doesn't work.

The ideal solution, at least for metropolitan population densities should be clear from this diagram:

The horizontal axis shows years of useful life. Notice that dark fiber is expensive to install but has a useful life of decades. But the equipment you use to light up the fiber will be functionally obsolete in 30 months just due to Moore's Law.

So what happened in Stockholm?  In 1994, Stockholm chartered a company, Stokab AB, to lay dark fiber and offer it to all comers.  Throughout the 90s, Stokab only offered dark fiber.  The result is many, many ISPs and corporate customers who lease fiber for diverse purposes.  A bigger result, is one of the lowest costs for some of the best Internet access of any city in the world.

What the US really needs

We need open access to bare copper and dark fiber from a utility that is regulated and is structurally separate from companies who provide telephony, TV or ISP services.

Rather than talk about network neutrality, we should be talking about structural separation, or at a minimum, functional separation with common carriage requirements. That would be a big change and is probably politically impossible, but that's the only way to get ISP competition and ISP competition is the only approach that will get us world class broadband.

FCC workshop on Open Internet at MIT this Wed January 13th

I haven't seen much publicity for this, but an FCC staff "workshop" on "Innovation, Investment and the Open Internet" will be held in the Bartos Theater at MIT"s Media Labs on Wedensday January 13, 2010.  "Open Internet" is usually political speak for Network Neutrality, although presumably some of speakers will differentiate between the advantages of open access versus the specifics of network neutrality, common carriage, structural separation and other approaches to achieving open access.

Details were provided in a press release which was picked up by a few people, but not many that I follow so, if you are in the Boston area, here's the scoop:

JANUARY 13 WORKSHOP ON INNOVATION, INVESTMENT, AND THE OPEN INTERNET

The Federal Communications Commission will hold a staff workshop on innovation and investment as part of the Commission’s Open Internet proceeding. This workshop will examine how the Internet’s openness affects the ability of network operators, Internet content and application providers, and other Internet technology developers to innovate and to drive investment, job creation, and economic growth throughout the Internet ecosystem. These issues will be explored from the diverse perspectives of innovators and entrepreneurs, investors, network operators and equipment vendors, and experts in Internet innovation and investment. 

WHAT: Innovation, Investment, and the Open Internet
WHEN: Wednesday, January 13, 4:30 p.m.
WHERE: MIT Media Lab, Bartos Theater Cambridge, Massachusetts
ONLINE: www.openinternet.gov/workshops/ 

PANELISTS:
Ajay Agarwal, Bain Capital Ventures
Tim Berners-Lee, World Wide Web Consortium (W3C)
David Clark, MIT Computer Science & Artificial Intelligence Laboratory
Jeffrey Glueck, Skyfire
Shane Greenstein, Kellogg School of Management, Northwestern University
Nabeel Hyatt, Conduit Labs
Susie Kim Riley, Camiant, Inc.
Paul Sagan, Akamai
Lynn St. Amour, Internet Society
Amy Tykeson, BendBroadband
Barbara van Schewick, Stanford Law School
Marcus Weldon, Alcatel-Lucent
Christopher S. Yoo, University of Pennsylvania Law School 
Sharon Gillett, Wireline Competition Bureau (Moderator)
Paul de Sa, Office of Strategic Planning and Policy Analysis (Moderator) 

The workshop will be open to the public; however, admittance will be limited to the seating available. Audio/video coverage of the workshop will be streamed live with open captioning over the Web at www.openinternet.gov/workshops/.

Broadband capacity - sizing buffers to handle traffic bursts

This is part 4 in a sequence of posts examining how broadband services actually work.  Part 1 looked at ISP concentration ratios.  Part 2 examined the effect of  averaging traffic from many subscribers.  Part 3 considered the impact of congestion on user experience. Here we'll take a finer grained look at traffic in the access network.

How do we achieve "zero" packet loss? Most websites are "hosted" and connect to the backbone by Fast Ethernet or Gigabit Ethernet links. Individual websites may be mismanaged, but there is no excuse for packet loss here.

Internet backbone links and backbone-to-backbone hand-offs are extremely high capacity multiplexing the traffic of thousands to millions of users.  Traffic statistics are predictable and operators compete on the basis of service levels, so packet loss is typically zero.

If there are performance problems, they're usually in the access network. Of course, one bottleneck is the service I've signed up for, for example a DSL link at 6 Mbps /1 Mbps. By definition my traffic is shaped to the service level I've signed up for.  This shaping typically happens at the first aggregation point (CMTS, DSLAM or WISP P2MP radio) or, on an IP or MAC address basis, at a bandwidth management device further upstream.  If they arise, problems are in the aggregation points and/or links between me and the backbone.

Since packet loss significantly degrades user experience, we'd like to understand how much over-provisioning a broadband service provider needs in order to absolutely minimize packet loss in the access network.  To understand this we need to look at smaller levels of aggregation (dozens and hundreds of users) and shorter intervals (sub-second rather than 5 minute averages, as few routers buffer more than a few tens or hundreds of milliseconds of traffic).

But first lets look at some five minute averages from a WISP with ~300 customers, each with either 1 Mbps or 3 Mbps service.

If one took these graphs at face value, it seems 20 Mbps of backhaul would be more than adequate, as the traffic peaks at 17 Mbps.  But 5 minute averages don't speak to what's really happening.

I don't have sub-second averages for the specific WISP above, but Fergal Toomey, Chief Scientist at Corvil Ltd., has some very detailed measurements on a similar Internet access network providing services to users at 512 Kbps, 1 Mbps and 2 Mbps. These are available in a whitepaper you can request from Corvil's website. Fergal's measurements were made at the Internet POP looking at traffic headed towards individual users.  Also, his measurements were made in 2004 when most hosted servers had 100 Mbps connections to the backbone and the majority of computers had TCP stacks configured for a maximum TCP receive window of 32KB.  In this graph, green shows 5 minutes averages, while light blue shows 500 ms averages. Dark blue shows 5 ms averages and they are striking!

What's Happening?

Web browsers typically use persistent TCP connections to minimize connection overhead and to avoid repeating TCP's slow start.  As a result, servers will frequently respond to a new request with a full window's worth of data.  In 2004 that was 32 KBs of data or ~23 packets of up to 1500 bytes each. Today it could be much more. Absorbing one such burst should be no problem, but to what extent are such bursts coincident?

Beyond web browsing, detailed traffic analysis uncovers a variety of odd behavior and some less savory behavior. For example, one burst was a blast of many 41-byte packets sent by a web application that apparently does a series of 1-byte writes to a TCP socket with the NO_DELAY flag set.  Several thousand packets arrive in quick succession causing packet loss.  Worse yet, follow-on losses occur as retransmissions cause a decaying series of echo bursts.

On a more malicious note, the data includes bursts caused by port scans.  These consist of many, many minimum size probe packets that attempt to scan all of a subscriber's ports in one enormous burst.  Luckily these lost packets are not retransmitted.

Beyond these, Fergal Toomey's analysis uncovered a variety of application anomalies that I found unexpected, but which crop up in the real world.

What headroom is really required?

Clearly we could just increase the size of the buffers in the router and smooth out any of these bursts, but that would also increase the delay while the burst lasts and thus introduce significant jitter.  A better question is how much headroom is required to guarantee zero packet loss and no more than 20 ms of incremental delay?

Conveniently, Fergal Toomey has run that calculation on the data set above and the result looks like this:

In this graph, the average traffic (the "mean" shown in light green) peaks just shy of 5 Mbps.  To guarantee zero packet loss and no more than 20 ms incremental delay 100 % of the time requires 19 Mbps of capacity or nearly 4x.  However, to guarantee zero packet loss and no more than 20 ms of delay 99% of the time, requires only 8 Mbps, i.e. a little less than 2x.

Interestingly, while ISP practices tend to be trade secrets, the rule of thumb that 2x headroom provides good service crops up repeatedly.

Conclusion

As a practical matter, real Internet traffic is fairly complex and includes a wide variety of anomalous behaviors. Rules of thumb based on averaging the traffic of a few hundred or even a few thousand subscribers are likely to be very expensive (in terms of excess capacity).  A better approach is to continuously monitor packet loss and delay variation (jitter), adding capacity as needed to keep those measures below desired levels.  

Zero packet loss and less than 20 ms incremental jitter, more than 99 percent of the time sounds plausible objective to me.  

Now if only my broadband service provider was willing to share any information about their service level objectives or performance.  Ooops, I must be dreaming.

Broadband capacity - the impact of congestion

This is part 3 in a sequence of posts examining how broadband services actually work.  Part 1 looked at ISP concentration ratios and part 2 examined the impact of averaging many subscribers. Future posts will consider how to size backhaul links and how to configure buffers.

Two major reasons people purchase broadband (or sign up for more capacity) are to improve interactivity and to get access to new applications, for example streaming movies.  New apps is clear, but it's worth looking congestion and how congestion affects interactivity.

Congestion has two impacts. First you get increased delays, as packets are buffered in queues waiting for free capacity on the next link. Usually this adds tens or hundreds of milliseconds to round trip times. Then, if the average traffic exceeds link capacity, you get packet loss. If you do have congestion, it's these packet losses that cause TCP traffic flows to back off, thus avoiding congestion collapse (and even less throughput).

Here is a graph of packet loss on a service that most of us would consider unacceptable.  These measurements were taken by Tom Dunigan on an early cable modem service (February 2001) that was substantially upgraded about a year later.

 

The next graph (also from Tom Donigan) shows how this congestion impacts interactivity.  The graph shows the echo delay for typing into a remote program using telnet in character-at-a-time mode.  One test (red line) was run in the early morning and one test (green line) was run during the evening when packet loss was peaking.  Each test measured the echo response time for 100 successive individual characters. Notice the typical echo delay goes up from ~130 ms in the morning to roughly 200-300 ms in the evening, but with occasional delays of more than one second.

Relatively few programs operate character-by-character, so the extra 100 ms or so might not matter.  On the other hand, a once second delay is noticeable!

A more common activity is viewing content on websites.  Here, the dominant time in any interaction is the time spent waiting for web content to download to your browser.  Since web content is downloaded using TCP (and more generally, TCP is the dominant protocol in use today), it's worth looking at the impact of packet loss on TCP throughput.  

The TCP protocol includes a congestion avoidance algorithm which is triggered by packet loss. When a packet is lost, the TCP sender slows down. As a result, the data rate for a single TCP flow looks like this (thanks to Guido Appenzeller):

Of course this assumes the TCP flow lasts long enough to saturate the bottleneck link and that this is the only flow in the network!  

What happens in real networks is complex, but RFC 3155 has an approximate formula for how a single flow is affected by packet loss in real network.  Even better, Bill Gibson at Niwot Networks has produced a nice graphic based on that formula for his paper on TCP limitations on file transfer performance.  It looks like this:

 

This is dramatic!  The different lines reflect different end-to-end round trip times (RTT) - times that will vary depending on the site you are connecting with.  Also, they represent the maximum throughput you could achieve with long duration TCP flows and no other bottlenecks.  What's notable is the logarithmic throughput scale on the left and the fact that, at 1% packet loss (0.01 on the bottom scale), potential throughput drops by a factor of 100!

Again, there are many caveats.  A real network has a mix of short- and long-lived flows. The local operating system may not be optimized to take full advantage of broadband speeds (although MS Windows actually got better at this with Vista). None-the-less, even 1% or 2% packet loss is correlated with poor user experience.

In short, to actually obtain the instantaneous throughput you thought you were purchasing, you don't want packet loss in upstream portions of the network.

I'll discuss what this actually implies for routers and backhaul links in a subsequent post.

Broadband capacity - the impact of averaging many subscribers

The fundamental idea of packet networks like the Internet is to transport variable rate data flows over a shared infrastructure.  Individual end points may generate little or no traffic for minutes or hours on end and then suddenly want to send or receive a large data set as quickly as possible.  Packet networks efficiently share common transport facilities among multiple users but how does this sharing actually work?  Yesterday I posted some actual, relatively current data from three small ISPs.  Today, let me show you how averaging traffic from many individual end points produces a completely stable core network flow.

Each of the graphs below shows successive five minutes measurements during a 24 hour period, but the first graph reflects a few hundred subscribers while the final graph represents the average of many, many millions of subscribers.

In the first graph, the blue line reflects outbound traffic from a few hundred subscribers.  While the average is just over 1 Mbps, the peak (at ~1615 hrs) is 6.9 Mbps for a peak-to-average of nearly 7-to-1.  The green bars represent inbound traffic with an average of 5.6 Mbps and a peak of nearly 16.8 Mbps for a peak-to-average of 3-to-1.  Some of this is time-of-day dependency but a lot of it, particularly the 6.91 Mbps spike at 1615 hrs, is the result of averaging only a few hundred subscribers.

 

Here's a traffic measurement at AMS-IX, the largest Internet Exchange in the world.  With an average of nearly 1 Gbps, the peak-to-average is a modest 1.43-to-1 and it's almost entirely due to time of day.  The largest change from one 5 minute measurement to the next appears to be about 140 Mbps so the short term peak-to-average is perhaps 1.1-to-1.

 

Finally, here a measurement of all the traffic flowing through AMS-IX – over 822 Gbps peak.  Here the time-of-day variation remains but there are virtually no statistical fluctuations.  The short term peak-to-average is almost 1-to-1.

 

To put this in economic terms, for an Internet backbone link that runs at many Gbps your daily traffic profile is completely stable and you can guarantee zero packet loss merely by providing 10%-20% extra capacity above your daily peak.  

But if you are running a small ISP, both the capacity you need per subscriber and the extra "headroom" for unanticipated peaks must be substantially larger.  To get a handle on what is required we also need to look at shorter intervals (shorter than five minutes).  But more on that in a subsequent post.

ISP Contention Ratios and average traffic per user – some data points

Recently there’s been a flurry of discussions about bandwidth shortages, not to mention bandwidth hogs, bandwidth caps, network neutrality and what constitutes fair practices by ISPs. But there is very little public data and no transparency! 

So let me offer a (very) few actual data points, one from a few years ago and three fairly current. These come from friends who are operators and from emails shared on trade association mail lists. Since some of it was not intended for publication, I have withheld specific names. 

Notice contention ratios vary widely and yet each of these operators has enough extra capacity that they claim to avoid link saturation and its impacts, like packet loss and excessive ping times. It’s true these are all small operators (280-1200 customers) so this data lacks the statistical smoothness one would expect when aggregating traffic from thousands or millions of subscribers. 

I also have some data on traffic averages by time and by number of subscribers that I’ll share in a subsequent blog post. 

Mar 2006 – Rural ISP (Dialup, DSL & Wireless)
130 Motorola Canopy subscribers (with 512 Kbps service)
150 DSL subscribers (110 with 512 Kbps service; 40 at 1 Mbps)
100 dialup lines serving 800 customers (average 40 Kbps)
Daily traffic was saturating 4.5 Mbps (3 T1s) for an hour per day but not with 6 Mbps.
Contention ratio: 164Mbps / 6 Mbps -> 27:1
Traffic per BB user: 6 Mbps / 280 subscribers -> 21 Kbps 

Sep 2009 – Rural ISP (DSL & Wireless)
1200 customers, DSL subs at 1.5, 3 or 6 Mbps; wireless subs at 384 Kbps, 768 Kbps, 1 Mbps or 3 Mbps. I don’t have the mix, so I'm guessing the contention ratio.
Daily traffic peaks at around 35-40 Mbps; up to 150 Mbps Internet transit available.
Contention ratio: 2400 Mbps(?) / 150 Mbps -> 16:1
Traffic per BB user: 40 Mbps / 1200 subscribers -> 33 Kbps 

Sep 2009 – Community-run WISP
340 customers, mostly on 3-4 Mbps service
Daily traffic peaks around 17 Mbps; 45 Mbps of Internet transit available.
Contention ratio: 1190 Mbps / 45 Mbps -> 26:1
Traffic per BB user: 17 Mbps / 340 members -> 50 Kbps 

Sep 2009 – Rural ISP
370 customers: 300 Customers on wireless (1-3 Mbps) and 70 customers with FTTH (10/100 & up, but most have 10 Mbps peak). Monthly fees based on a 10 or 20 GB cap with extra charges for heavier use.
Daily traffic peaks at 13-14 Mbps; 20 Mbps of Internet transit available.
Contention ratio: 1200 Mbps (?) / 20 Mbps -> 60:1
Traffic per BB user: 14 Mbps / 370 subscribers -> 37 Kbps

White spaces could be the broadcasters best hope

For years, the National Association of Broadcasters (NAB) fought the White Spaces Coalition and others interested in making US "TV white spaces" available for broadband, Wi-Fi or indeed, any new purpose. When the FCC voted 5-0 to permit license exempt use of TV White Spaces, the industry brought suit in Federal court. And they did this, despite rules in the FCC's decision that are so restrictive that, for now, white spaces devices are doomed to commercial failure. The NAB are savvy in the ways of Washington

But fighting the White Space Coalition is short sighted. The NAB faces a much bigger and more powerful enemy — mobile operators.

The White Spaces Coalition merely seeks permission to use spectrum where NAB members are not using it, i.e. on a non-interference basis as "secondary users" with purely secondary rights.  

The mobile industry wants it all.  They'd prefer that broadcast spectrum be taken back and auctioned off for mobile use. Discussions on recapturing broadcast spectrum ramped up after an October comment by FCC broadband czar Blair Levin. For example see the transcript of this December 1st panel discussion. Or consider last week's appointment of Duke Law Professor Stuart Benjamin as the FCC's first Distinguished Scholar in Residence. Benjamin is a vocal proponent of reclaiming the TV broadcast spectrum!

Broadcasters beware!  

Unlike the White Spaces Coalition, the mobile operators are political experts. They are part of a 100+ year telecom lobbying heritage. The Bell System was lobbying government agencies before the broadcast industry existed. Now Congress is considering spectrum policy. The FCC is considering spectrum policy.  

Broadcasters may eventually extort large sums of money out of the public, but over the next decade they will lose more and more of their spectrum.  I am no fan of the broadcast industry. Even after converting to more modern "digital" broadcasting, they are sitting on spectrum they don't need in order to deliver a limited number of channels of broadcast TV to the 14% of households who don't subscribe to cable. I'm one of those 14% and I don't even watch TV, so I have no interest in broadcasters' survival. But I can't help noticing there is one thing broadcasters could do that would block mobile operators from taking over broadcast spectrum.

White spaces can save broadcasters' spectrum rights

If license exempt white space devices are commercially successful, it will become increasingly difficult and then politically impossible for Congress or the FCC to recapture TV spectrum for exclusive use by the mobile industry. If Wi-Fi, WiMAX and other consumer devices appear using TV frequencies, it will become harder and harder to displace these consumer uses and recapture the exclusive use the mobile industry requires.  

So, if the broadcast industry really wants to hold onto their current spectrum rights, they should get as many non-interfering "secondary users" into their band as possible.  Otherwise, they will eventually lose their primary rights to the quest for more mobile broadband.

User owned fiber in Utah

Here's some wonderful news via Geoff Daily's excellent Broadband blog, Apprising.  Brigham City Utah is offering its citizens the chance to own their own fiber connection from their premises to the Utopia network where they can pick and choose among a growing number of services and service providers.

Some highlights from Geoff's post:

Today in Brigham City, for $3,000 you can buy your own fiber. And in fact more than 1,600 local residents have already bought in to this new opportunity.

The biggest problem with the economics of deploying fiber is that you have to carry a massive debt load and begin paying it off before much revenue starts coming in. Plus you have to invest a lot of money into neighborhoods without any real idea of how many people are going to sign up for service.

The user-owned model totally changes these dynamics. First off, by having users pay for their own pipes you disaggregate most of the debt. Just look at Brigham City. They're building a $5.5 million network and the city's putting up less than $700,000. So no massive debt load for the city (or a private provider for that matter) to carry.

Of course the other issue with is fiber builds is take rate - what percentage of the prospective customers actually sign up?  For Verizon FiOS it's 21%.

With user owned fiber the take rate is 100% !  and the churn is zero.

Best of all, the owner controls their destiny while multiple service providers compete for the business.

Leaving for eComm in Amsterdam - Will tweet conference talks

I'll be leaving for Amsterdam this afternoon to attend Fall eComm 2009. 

I plan to record my notes as I did at the HD Communications summit in NYC in September, i.e. I'll write my notes on Twitter in real time and then, after the conference, collate the results into one giant blog post (for my own sake as much as for yours, esteemed reader).

If you're interested in live coverage, follow my tweets at: http://twitter.com/brough

Whoever and whatever you follow, I assume the common tag people will be using (as before) will be "eComm" so that's another approach...

Shameless self-promotion:  My part in the conference:

On Wednesday morning, I'm speaking on Open Spectrum issues.  I will point out some laws of physics and emerging technology changes that, together, suggest a new, better focus for open spectrum efforts.

On Wednesday afternoon, I'm a panelist on Andy Abramson's panel, "Are current wireline and wireless eco-systems still relevant?."

And then on Friday afternoon, I'm moderating a panel that I've organized on "Spectrum 2.0 - What's really happening?"

Has AT&T Wireless data congestion been self-inflicted?

We've all heard or read stories about how iPhone usage has overloaded the AT&T Wireless network but it's likely at least some of their problems are the result of configuration errors ― specifically, congestion collapse induced by misconfigured buffers in their mobile core network.

In early September, David Reed sent this interesting message to the IRTF's "end-to-end" email list.  List members include some world experts on Internet protocols.  During the next couple of days, there were over 40 messages in related threads.  While some of these experts were over-thinking the problem, if you are patient enough to read through the many messages, what emerges is clear.  At least in the case David measured (from a hotel room in Chicago, while he had 5 bars of signal strength, using an AT&T Mercury 3G data modem in his laptop), the terrible throughput and extreme delays he experienced appear to result from overly large buffers in the routers &/or switches in AT&T's core network.  Note:  if you don't want to read all the list messages the short summary is:  >8 second pings times!  What's more the effect was bymodal:  either ping times under 200 ms, or over 5 seconds.

Recently I was talking with a friend whose company continuously operates (and monitors) multiple 3G data links on the Verizon Wireless, AT&T Wireless and Sprint  PCS networks.  His data shows periods when the round trip time for http requests goes over 8 seconds, on the AT&T Wireless network only!  I don't have a copy of his data that I can examine in detail, but when combined with David Reeds report, it certainly appears AT&T Wireless has configuration problems.  If you read on you'll see this may not be the result of gross stupidity, but someone at AT&T Wireless should be a little embarrassed.

My (techie outsider's) Analysis of What's Happening

Buffers in Packet Networks

Routers (& switches) in a packet network have to include buffering in order to absorb transient traffic bursts.  Unfortunately, despite decades of research and operational experience, there is no simple formula for how much buffering is optimal at any given location in a network.  If you're interested in more detail, Ravi Prasad has a good review of the literature on pages 10 & 11 of his (April 2008) PhD thesis.  But decades of operational experience have yielded some basic precepts and it appears AT&T Wireless is violating at least one basic precept.

The buffer in front of a congested link must induce some packet loss.  TCP (the dominant Internet protocol) continuously increases it's transmit rate until it experiences packet loss, then it cuts its rate in half and enters a congestion avoidance mode.  If the network becomes full but there is no packet loss, each TCP sender will keep increasing it's rate causing the network to suffer a congestion collapse.

In the case of a mobile network, the limited resource is over-the-air capacity.  Backhaul may also be expensive, but it's relatively easy to over provision anything else in the operator's network.  So the issue is, how big should the buffer be in the last router between the high capacity core network and the actual over-the-air data path to a subscriber?  Ideally we'd like enough buffer to absorb momentary packet bursts that, averaged out, don't exceed the available over-the-air capacity.  But as soon as the offered traffic exceeds the available over-the-air capacity, we want some packet loss.  The complicating factor is the way 3G wireless networks schedule over-the-air traffic.

Jitter in 3G radio networks

One cellular base station serves multiple users and the quality of the connection to any specific user depends upon instantaneous wireless propagation characteristics.  These can vary second by second and even millisecond by millisecond when a user is moving.  To deal with over-the-air losses, the basestation (the "Node-B" in a 3G network) keeps copies of each packet until a positive acknowledgement (ACK) is received, retransmitting the packet if the ACK is not received in time. Of course retransmissions introduce delay and jitter.  Furthermore, at any given instant, some users wireless links are better than others.  In order to maximize the total traffic in a cell, the 3G MAC layer schedules transmissions to individual users based on who has the best instantaneous throughput.  This is an efficient solution but it also introduces different amounts of jitter into each user's data path.  Luckily these effects are well understood and not that severe.  With HSDPA, the basic transmission time interval is 2ms so total delay variation is relatively small.  This graph from Jang et al (1) is typical of measured values in an HSDPA network.

Most jitter is below 15 ms.  Measurements of ping latency between 3G wireless devices and the first IP server at the edge of the mobile core network (typically the GGSN) can extend out to over a second as this graph from Mun Choon Chan and R. Ramjee (2) shows:

but most of the time, total IP latency is a few hundred milliseconds. As mentioned earlier, David Reed reported bimodal operation on the AT&T Wireless network with normal behavior yielding ping times under 200 ms.

Likely cause of AT&T's problems

So what is happening in the AT&T Wireless network when ping times go over 8 seconds?  We know how a customer's IP packets are passed through an operator's mobile core network.  They are tunneled all the way from the handset to the Gateway GPRS Signaling Node (GGSN), i.e. to the router where the mobile core network connects to other networks.  The protocol stacks for this tunneling look like this:

More recent versions replace the ATM and AAL5 with Ethernet and IP, but the user never sees this as user IP data is tunneled across the top of the diagram (carried by PDCP and GTP-U).  As a result, user traceroutes can't reveal the detail of what's happening in the core network.  The first thing the user can see is the GGSN (the gateway to the next network).  So we can't make conclusive measurements from outside the network, but we do know a few more things.

The bottleneck link is the over-the-air link, i.e. the connection from radio access network or UTRAN to the Mobile Statation (MS) in the above diagram, therefore the critical buffers are those at the UTRAN.  In practice the UTRAN includes both the basestations (called Node-Bs) and the Radio Network Controllers (RNCs) which coordinate handovers between basestations (among other things).  Because of hand-overs, the amount of data buffered at the Node-B is relatively small. It's the buffers at the RNC that must be large enough to deal with the delay variations in the radio network and yet small enough to induce packet loss when the network gets congested.

While I don't personally have experience managing a 3G HSDPA network, my impression is UTRAN buffers are normally less than 200 ms.  Recently Yerima and Al-Begain presented an interesting paper (3) on buffer management in 3.5G wireless networks in which they concluded that 120 ms buffers were ideal for downlink traffic in a specific UMTS-HSDPA configuration.

Zero Packet Loss

It appears AT&T Wireless has configured their RNC buffers so there is no packet loss, i.e. with buffers capable of holding more than ten seconds of data.  Zero packet loss may sound impressive to a telephone guy, but it causes TCP congestion collapse and thus doesn't work for the mobile Internet!

=================================================

References

(1) 3G and 3.5G Wireless Network Performance Measured from Moving Cars and High-Speed Trains, by Keon Jang† (keonjang@an.kaist.ac.kr), Mongnam Han† (hgma11@gmail.com), Soohyun Cho∗ (shcho1@kt.com), Hyung-Keun Ryu∗ (hkryu@kt.com), Jaehwa Lee∗ (jhlee@koren21.net), Youngseok Lee‡ (lee@cnu.ac.kr), Sue Moon† (sbmoon@kaist.edu)

  † Computer Science Department, KAIST

  ‡ School of Computer Science & Engineering, Chungnam National University

  ∗ KT Central R&D Laboratory
http://an.kaist.ac.kr/~sbmoon/paper/intl-conf/2009-micnet-3g.pdf

(2) TCP/IP performance over 3G wireless links with rate and delay variation,

by Mun Choon Chan and R. Ramjee, Bell Labs  (presented by Jinqiang Yang)
http://wwwcsif.cs.ucdavis.edu/~ghosal/Teaching/ECS289I/Lectures/jinqiang-yang-wirelessTCP.pdf

(3) Dynamic Buffer Management for Multimedia Services in 3.5G Wireless Networks,

by Suleiman Y. Yerima and Khalid Al-Begain

Proceedings of the World Congress on Engineering 2009 Vol I

WCE 2009, July 1 - 3, 2009, London, U.K.
http://www.iaeng.org/publication/WCE2009/WCE2009_pp857-862.pdf

Download speeds of GSM & 3GSM networks

ARCchart is selling a new report entitled Mobile Broadband Performance of Carrier Networks.  I can't personally justify the purchase, but I notice this wonderful graph in their sample.

ARCchart gave mobile users free speed test applications, first for iPhone beginning in August 2008, then for Android and BlackBerry.  By June 2009, they had over 2 million tests across 268 wireless operators in 103 different countries.  For this graph, the data was filtered to focus on major urban areas.  That left 648K individual test results which were used to form the graph above.

What does this tell us?

• First, 2G GSM data rates are usually 50-60 Kbps.
• While EDGE data rates might go as high as 250 Kbps, a good bet is 140-200 Kbps.
• And finally, deployed 3GSM networks are good for 600 Kbps to perhaps 1.4 Mbps.
• Anything faster is extremely unlikely (at least as of 2Q09).

In a world of endless hype, it's nice to have some real data.

Structural Bypass - my talk from eComm last spring

At Spring eComm 2009 in California in March, I gave a presentation entitled:  Structural Bypass - A Simple, Proven Path to "Real Broadband" that dealt with policy approaches for first mile fiber deployments.  The video has just been posted here.  Unfortunately, this is my face and the audio with my words.  The slides are not shown in the video, instead you have to look at the slides here.

My arguments focused on urban areas only, as the economics are different between urban, suburban and rural deployments.  You can't just lump together all areas in a country and draw reasonable conclusions.  Also, the only data I have is for urban areas.  And, as a simplification for this talk (I only had 15 minutes including Q&A), my metric of success was the consumer price for 100 Mbps symmetric Internet connectivity.

My points were:

• Structural separation produces better results than other approaches.
• Separation between dark fiber and everything else is more effective than separations at higher layers.
• There is a political path, which I call Structural Bypass, which has been used at least once (in Stockholm), to effectively achieve structural separation without explicitly fighting that battle.

If you are interested, I suggest you open the video in one browser tab merely for the purpose of listening to the audio.  Then view the slides in another browser tab.  The audio is of some interest, but my facial expressions are of questionable value.

eComm agenda is on-line ―Check out the speakers!

There's a fascinating agenda building for eComm in Amsterdam in October.  Here's the latest list of talks, but what's best is the set of speakers who are giving these talks.  I personally know quite a few and know of many more (who I look forward to meeting).  The group includes a preponderance of innovators -- new views on what's happening, new ideas for how to drive change.  If you don't read through this list of talks, look at the speakers list here.  Then come to Amsterdam and meet these people.

21st Century Economics: Lessons for Telcos - Umair Haque (Havas Media Lab)

Advances In Spectrum Transparency, Software Defined Radio/Cognitive Radio - Darrin M Mylet (Spectru-Station)

Almost all Marketing & Product Management of Telco Services is Wrong - Rudolf van der Berg (Logica)

Current Trends in Community Wireless Networks and Beyond - Aaron Kaplan (FunkFeuer)

Death of the Handset: Evolving from Mobile Devices to the Mobile Digital Life - Mark Rolston (frog Design)

Edge As Value - Value As Edge - Graham Brierton (Voicesage)

Enslaving Humans using Communications Technology for Fun and Profit - Thomas McCarthy-Howe (Jaduka)

Entrepreneurial Advantages with New Open-Source Technologies - Jay Phillips (Voxeo)

European Telecoms 2015: Silent Death or Generative Bazaar? - Julien Salanave (IDATE Telecoms)

Finding Disruption - Michael Jackson (Mangrove Capital)

Goodbye Minutes, Hello Moments - Martin Geddes (BT)

Google Wave - David Wang (Google)

How the "Internet of Things" will Change the Way we Connect - Alexandra Deschamps-Sonsino (Tinker.it)

How to get More Value out of Customer Interactions by Blending Online with Voice - Stefan Hopmann (Swisscom)

Humans as a Service: Abstracting Communications to Reach New Applications - Todd Landry (NEC Sphere)

Is LTE being Held Hostage by Ordinary Voice Telephony? - Dean Bubley (Disruptive Analysis)

Lifestyle Segmentation from Carrier Location and Call Data - Greg Skibiski (Sense Networks)

Lifestyle Segmentation from Carrier Location and Call Data - Tony Jebara (Sense Networks)

LTE - Long Term Employment or Less Than Expected? - Moray Rumney (Agilent)

Open Access Makes Economic Sense - Benoît Felten (Yankee Group)

Open Screen Project: Next Generation Contextual Applications - Andrew Shorten (Adobe)

Opening of the Terahertz Region - Robert Horvitz (Open Spectrum Foundation)

Opportunities in Post-Telecom Connectivity - Bob Frankston (Frankston Innovating)

Post Financial Trauma - How is the Telecom Value Chain Now Positioned? - James Enck (mCapital)

Redefining Gifts in the Digital Space - Katie Lips (Little World Gifts)

Slowing Down Communication: Designs Inspired by Quality, Intimacy, and Humanity - Stefan Agamanolis (Distance Lab)

Spectrum 2.0 – What's Really Happening? - Brough Turner (Ashtonbrooke)

Stealth Approaches to Legislating Open Spectrum - Brough Turner (Ashtonbrooke)

Technology and Biological Evolution: What This Means for Media and Communications Technologies - Tomas Rawlings (University of the West of England)

Telemedia Futures - Gerd Leonhard (MediaFuturist.com)

The Emerging Telecology of Social Networks and the Status Update - Stuart Henshall (Phweet)

The Future of Interconnection - Rudolf van der Berg (Logica)

The Global Battle for Communications Justice: An Open Spectrum Manifesto - Sascha Meinrath (New America Foundation)

The Next Wave of Communications Applications - Cullen Jennings (Cisco)

The Practical Edge of Speech Technology - Moshe Yudkowsky (Disaggregate)

Ubiquitous Voice over Broadband - is There a Future Role for the Smart Pipe? - Martin Taylor (MetaSwitch)

Unlicensed Spectrum: Future Regulation - Prof. William Webb (Ofcom)

Video Killed the Telephone Czar? - Jan Linden (Global IP Solutions)

When Will HD Voice Become a Reality? - Martyn Davies (Dialogic)

What Would Telephony be Like if we Designed it Today? - Matt Ranney (RebelVox)

If you've read this far, there's a 10% discount via this registration link - register here.  And for more information, visit http://eComm.ec.  This one conference is at the top of my list for:  interesting, productive and an excellent set of contacts.

eComm in Amsterdam ― shameless self promotion

I've just wrapped up a focused effort that delayed blogging and many other things.  As a result, I finally submitted the detailed description for my plenary slot at eComm Fall 2009 which will be happening in Amsterdam October 28th-30th.  My abstract is not up on the website yet, but hopefully the next few weeks will bring details on my talk and many others.  For now, let me just say my title is
   Stealth Approaches to Legislating Open Spectrum
in which I propose what I hope is a novel approach to dramatically expanding the capabilities and commercial success of license-exempt consumer devices.

This will be the first time eComm has been held in Europe but, based on the first two eComm conferences (2008 and Spring 2009), this is the meeting for the future of communications.  It's not a trade show and it's not a mass event.  Instead, it's three days of rapid paced information ― high level, insightful and non-commercial.  Even more important, the people are very, very interesting.  It's not cheap, but it costs less if you sign up now (especially if you sign up before July 21st).  What's more, because I'm such an enthusiast, and I'm on the conference's advisory board, I've been given a discount code.  For an additional 20% off type in "BroughTurner" as the eComm discount code, i.e. where the registration form says "Click here to enter a promotional code."

I hope to see you in Amsterdam in October.

Opportunity Doesn't Always Knock. Sometimes It Calls.

The mammoth telecom industry ― fixed and cellular ― is in the process of being re-written. You can stand on the side and be written into history or join with the growing community that's writing the future. Opportunities have never been so great ― to influence how humanity connects, communicates and collaborates and to profit from radical restructuring.

The Dumb Pipe Paradox ― How open access networks build shareholder value

On more than one occasion, "natural monopoly" arguments have been used to obtain, and/or hold onto, a government granted monopoly,  This is definitely true of access networks where the physical right-of-way to any specific real estate parcel is a limited resource, but the economics of what you put in it have changed over time.  At one time it made sense to consider telephone service a natural monopoly.  And for most of the twentieth century, telephone service was operated that way (think Bell System in the US and government run PTTs in most of the rest of the world).  In the latter 20th century, cable TV became a second such monopoly.  Today, it's increasingly clear that telephony and television are higher layer services, not inherently tied to the access network.  Yet our laws and regulation have barely evolved ― each access network is still regulated as a different vertically integrated monopoly.  And, managers in each business focus on preserving their historic monopoly even as market forces or government regulations force them to also offer Internet access.

So it's interesting to see solid economic analysis showing that access network shareholders would make more money if management was willing to open up their access networks, i.e. become "dumb pipes."

The first such argument I encountered was by Craig Moffett and Amelia Wong of Bernstein Research who wrote an interesting paper The Dumb Pipe Paradox, early in 2006.  The original paper is not on line but I have some quotes here and there are some other comments here.  Craig and Amelia were looking at Cable TV's hybrid fiber-coax networks and concluded that cable companies could make more return on investment if they were in the pure dumb pipes business.

More recently, I reported on a speech given by Benoît Felten of Yankee Group and Fiberevolution in which he argued that new fiber to the home (FTTH) investments could be paid back more rapidly if the FTTH network were open, i.e. offered to competitors at attractive wholesale rates.

Now Benoît has written a detailed report describing his findings.  Although the report is for Yankee Group subscribers only, Benoît is also giving a webinar on the subject next Tuesday and that's open to all.  Register here.

I particularly liked the polite suggestion near the end of Benoît's report:

Recommendations for Incumbents
• Re-examine your economic fundamentals in light of the FTTH business model. It’s irrational to cling to antiquated business practices if new approaches, no matter how disruptive, deliver better shareholder value.

Highlights from eComm 2009

This year's Emerging Communications conference, eComm 2009, was the best telecom conference I've been to in ages (ever?).  Now presentations and videos from the conference are becoming available on the web.  The presentations are on SlideShare; search by speaker name or for the tag "eComm."  Here's my presentation, Structural Bypass - A simple proven path to "Real Broadband."

Brough Turner's Presentation at eComm 2009

View more presentations from eCommConf.

Videos and transcripts are also coming, although not as rapidly as I'd like (a matter of resources - one person editting and releasing 2-4 videos per week).  Here's a transcript of the Spectrum 2.0 panel that I moderated.

Videos will show up on Fora.tv, for example, here is the really cool keynote address by Ge Wang entitled "New Expressive Social Mediums on the iPhone."

For more information and pointers, subscribe to the eComm blog.

More bandwidth ― less delay, less latency

I was a little sloppy yesterday and several people have questioned my comment about latency.  I was reacting to slide by Herman Wagter of Amsterdam's Citynet in his presentation at F2C 2009.  His slide said:

Latency is the cause, bandwidth is the effect.

From his discussion it was clear he meant "Latency is the cause, bandwidth is the cure."  At the time he was talking about real-time person-to-person communications and illustrating it with a housebound person in Amsterdam who wanted to play cards with friends in other places (not nearby).  Verbally he mentioned the issue of sending large files.  In short he was addressing the real reason for high capacity Internet access links.

Why people want more "bandwidth"

It's not because they need or want to send and receive 100 Mbps of data all the time or even a significant part of the time.  The issue is delay, specifically serialization delay.  If I have a 1 Mbps upstream Internet connection and I want to send an email with a 5 MB Powerpoint file attached, it will take more than 40 seconds (5 MB ~= 40 Mbits).  On a 100 Mbps link, the same email is delayed only a fraction of a second.

Serialization delay also effects media streams, although much less.  If I want to send 500 Kbps of continuous video over that 1 Mbps uplink, serialization delay will cause added latency.  IP is a packet protocol and the 500 Kbps video stream will be broken into a stream of packets, typically ~1500 bytes (12 Kbits) each. While the serialization delay is only 12 ms on the first link, there is serialization delay on every link.  If there is another 1 mbps link at the other end, that's another 12 ms of delay.  And here, 12 ms is significant.  For a natural interaction between two people, you'd like to keep the round trip delay below 200 ms.  Nothing goes faster than the speed of light so transoceanic links introduce many 10s of ms of delay, each way.  It is very easy to eat up a 200 ms budget, so saving 12 ms at each end is significant.

Burst rate versus continuous

For me and for most people, the issue that drives demand for high speed access links is delay, not the amount of information that is to be sent or received.  Indeed, I'd love a service offered only a few Mbps average over a month, if I could be guaranteed 1 Gbps on a burst rate basis whenever I wanted.

David Weinberger is blogging F2C rather completely

And David is a expert blogger!  i.e., really good coverage.
  http://www.hyperorg.com/blogger/

The first Freedom to Connect conference entry is here. 

Latency is the cause; bandwidth is the solution

Actually, what Herman Wagter of Amsterdam's Citynet said at F2C 2009 was:  latency is the cause, bandwidth is the effect.  But his explanation matched my title above.

If you are attempting to interact with other people, whether by VoIP or just playing cards together (with video) you need less than 200 milliseconds of end-to-end delay.  If it's playing cards together, with video, and you need to exchange 500 Kbps in less than 200 ms, you need a 100 Mbps pipe!

It's latency that drives the need for high bandwidth.  Most people won't fill that pipe most of the time, but they need the pipe to guarentee that what they do send gets through rapidly.

Highlights from Day 1 of F2C

Shamelessly ranked by my areas of interest...

Tim Nulty of East Central Vermont Community Fiber.  Tim is a veteran network builder and a forceful speaker, so he's happy to tell it like is.  He's also got that Yankee mix of liberal politics with extreme fiscal conservatism.  He's building fiber networks, in rural Vermont, which pay for themselves.

Ken Biba of Novarum has been measuring actual wireless networks in buildings and in cities for years.  While the detail is in a report available for purchase, the summary is that WiFi-based Muni WiFi yields significantly better performance than 3G cellular.  Interestingly coverage and reliability is right up there in selected cities, as well.  The take-away - 802.11n really rocks. I.e., the next cycle of WiFi is going to be vastly better than what he's been measuring over the past 3 years.

Ellen Miller of Sunlight Foundation was low key by comparison with Tim or Ken, but her stories were compelling - multiple instances of Internet community feedback creating the kind of information that the "open government" initiatives aspire to.

Finally, Dewayne Hendricks is always interesting.  This year he seemed more optimistic than last, presumably the result of the recent election.  In any event, here's another speaker with deep experience in building networks.

Freedom to Connect 2009 and Municipal Fiber

I'm attending F2C 2009 in Silver Spring Maryland. If you are here, please say hello.

Things are just getting started with a panel on Municipal Networks, led by Joanne Hovis, President of Columbia Telecommunications Corporation (CTC).  Panelists are:

Tim Nulty, formerly of Burlington Telecom and now running East Central Vermont Community Fiber, has tons of experience building fiber networks in low density areas (Vermont).

Dirk van der Woude, from Amsterdam's CitiNet, to talk about the Amsterdam's municipal fiber to the home project..

Lev Gonick, CIO Case Western Reserve, who was a key player in creating a 4000 mile fiber network for Cleveland and northeast Ohio under a community organization called OneCommunity.

Bill Schrier, CTO for Seattle, which is starting a fiber project, but already has it's own electric power utility.  (Although Bill implies they have had to drag their utility brethren into this).

What's interesting is the discussion on the chat backchannel is not about muni vs. commercial, but wireless versus fiber.

Tim Nulty has a strong argument that wireless is excellent for mobility, but not economical for fixed access.  In rural Vermont, a WiMAX network would cost $35M if you could get access to the spectrum (which is being horded by others).  Fiber would cost $70M but has 50 times the capacity and several times the revenue potential versus the wireless approach.  Further, if you deploy wireless as an addon after you have the fiber network (and the customer support infrastructure), the incremental cost is dramitcally less (perhaps $10M) and you get enough incremental revenue to get a good return on investment.  IN other words, you make more money if you do fiber first.  Tim's key to success is to get as near complete deployment as possible - something that is possible in areas where the incumbents are going.  Second, he goes for community ventures as a way to qualify for muncipal bonds.

Shareholders should demand phone companies open their fiber networks

A few weeks ago, Benoît Felten of Yankee Group and Fiberevolution gave a speech at a New Zealand Commerce Commission event which included a fascinating argument.

He points out that receiving a return on the substantial capital investments that a fiber to the home (FTTH) project requires is much more dependent on takeup rates than on the average revenue per user.  

Single player large scale deployments usually achive only 20%-25% initial adoption after which growth is exceedly slow.  On the other hand, systems which are open to competitors, i.e. with viable wholesale services, attract many more players who market, sell and deliver new services thus dramatically increasing adoption and accelerating the wholesaler's return on investment.

In short, obtaining a regulatory holiday so you can run your new fiber network as a monopoly is actually bad for your shareholders!

Benoît's speech was filmed.  For the full presentation (in four segments of < 10 minutes each on YouTube) see Benoît's post.

If you want just the essence, listen to the first three minutes of this:

Here is an enlarged version of the chart that Benoît is using while he speaks:

I'm off to the Emerging Communications Conference in San Francisco

It's snowing in Boston and my American flight has been cancelled but Virgin America claims their 8:35am flight is going to leave on time.  So here I am in the Virgin gate area.  Wish me luck.

At this point there are a ton of people I'm hoping to hook up with at eComm 2009.  The agenda looks really good.  And, of course I'm looking forward to good discussions around two favorite policy topics: broadband access and wireless spectrum.

My talk on Wednesday is: Structural Bypass: A simple, proven path to “Real Broadband.”

While the US struggles to define "broadband," high speed Internet access (100 Mbps & above) is widely available at modest cost in several countries and quite a few more cities. So far, US political discussion has largely neglected these successes. Brough will point out what's common among diverse international success stories and propose a path for the US that has proven to work elsewhere, despite established monopolies and political processes dominated by vested interests.

On Thursday, I've organized a panel entitled:  Spectrum 2.0 - What's really happening?

WiFi, UltraWideBand and now TV White Spaces represent new commons-based approaches to radio spectrum regulation. While some advocate commons-based approaches for all wireless spectrum, that's hardly acceptable to broadcasters or the mobile phone industry. By questioning a diverse panel of industry experts, we will expose the roots of today's controversy - technical, commercial and political - and see what's likely to occur over the next two to five years and in the long term.

The panelists are top notch:  Richard Bennett, Maura Corbett, Peter Ecclesine, Darrin Mylet and Richard Whitt.

If you're attending, please say hello.

Rate speakers - Interesting new website SpeakerRate

If you are attending eComm 2009 this week, please consider rating my presentation on SpeakerRate.  It's entitled "Structural Bypass" and you can rate me here.  

 

The SpeakerRate website just came out of stealth mode a few weeks ago.  It looks like a service I'd use and refer to, so I wish them luck.

Wireless Tutorial slides posted

The slides we used for our four part Wireless Tutorial at the 4G Wireless Evolution conference in Miami last week are now up on the web.

History and Evolution of Mobile Radio

Part One covers the history of mobile wireless from the earliest days

to the latest 4G technology.

Part one is also available as a webinar recorded in 3 sections last fall.

IEEE Wireless Ethernet Keeps Going and Growing

Part two covers the IEEE wireless systems:  WiFi, WiMAX and more...

Mobile Broadband: New Applications and New Business Models

Part three covers emerging world of mobile broadband access and some of the applications it enables.

White Spaces and Open spectrum Issues

Finally, part four focuses on Open spectrum and the recent decision by the FCC to permit unlicensed devices to operate on unoccupied TV channels - the so called TV White Spaces.  In the end, there's alot more that will be possible eventually...

Grey market Broadband access in Bangladesh

Here's another great picture from Tariq, taken when he was in Dhaka, Bangladesh to speak at the January 2008 meeting of SANOG.

At the top of the pole are power lines, but the rest of the stuff in the middle is an assortment of cable TV and fiber infrastructure, almost all run without the benefit of formal government regulation.

The result is people in at least some neighborhoods of Dhaka have access similar to what's on offer in Karchi, Pakistan.

Community Networks — the Robin Hood Approach

What kind of broadband Internet access would be possible if there were no government regulation?  Today every government in the world has laws or regulations on who can run cables in public rights-of-way but, in some places, these rules are not effective.  This has produced some very interesting case studies on what markets do in the absence government granted monopolies.

Last fall, I wrote two blog posts about the situation in Bucharest where essentially anyone can string cables in public rights-of-way.  The result are widespread fiber and Cat 5 cable networks and bundles of CATV + FTTB (100 Mbps FastEthernet) + telephony are routinely available for as little as 9 Euros (~US $12) per month.   Here is another case:

Desi Cable Networks in Lahore

Recently, I had a very interesting email exchange with Tariq Mustafa in Lahore about informal broadband access arrangements in Southeast Asia. (Tariq blogs as Tee Emm on Pakistan NextGen Issues and is also active on Telecom Grid Pakistan).  Tariq also has direct knowledge of local networks in Lahore and Karachi and indirect information about Dhaka (Bangladesh).  With Tariq's permission, I'll quote extensively from his emails, but first a quick summary of the situation as I understand it.

Access to neighborhood rights-of-way is governed by local relationships rather than by official government license.  As a result, there are extensive local broadband networks (fiber and Cat 5 cable) providing excellent local connectivity (typically up to 100 Mbps symmetric), including high speed access to local content caches.  Actual connections to the Internet are still limited so, while one can download movies from a file server at 30 - 50 Mbps, connections to the real Internet tend to be limited to 512 Kbps each.  On the other hand, the whole package costs roughly Rs 600 or about US $8, per month.  And, while exact figures are not available, it appears the number of grey market subscribers far exceeds those officially counted by the Pakistan Telecommunications Authority.

In short, where there is no government and no government empowered monopolist (in this case, the first 5-15 km), both fiber and Fast Ethernet are widely deployed, and prices are extremely low.  Actual connections to the Internet have to go through licensed ISPs and need access to the country's limited International cables.  These remain expensive.

Tariq's Explanation of Desi Cable Networks

Everything that follows is quoted directly from Tariq's emails to me:

DCN - Desi Cable Nets as I want to call them. (Desi is the word that means 'local' in our South Asian Languages. The word was necessary to differentiate these networks from the original hybrid fiber coaxial network that are generally called Cable Networks in the developed countries).

To answer your questions, here are some more facts. Feel free to get back to me if you have more questions. I consider myself a vocalist when it comes to these DCNs and the impact that these have created (in the absence of the actual formal industry rising up to the occasion for the digital needs of the masses). I talked about this at Sanog XI in Dhaka (Bangladesh) earlier this January <2008> and the title of the talk was 'Community Networks - Crowning the Robinhood'. This summaries what I mean.

1. DCN is an eco-system of operators, consumers and social realities.

2. A DCN eco-system is characterized by high desires for middle class community to get online, low affordability, lack of regulation, low concerns of digital communication security and poor infrastructure of electricity and underground cables.

3. A DCN successfully delivers (often partial) Internet services in the wake of the above obstacles to large parts of population.

4. In a DCN eco-system, there is one part that can be referred to as the 'junknet'. The junknet is an instance of anywhere from 50 to 150 houses connected to empirically <located> Ethernet switches mounted generally on top of utility poles and interconnected to each other over Cat5 cables. A junknet is characterized by lack of any intelligence. There are no AAA servers, no bandwidth managers, no routers etc. It is essentially a collection of Cat5 cables coming out of houses and connecting to Ethernet switches on the utility poles. These Ethernet switches are cascaded to allow the junknet to connect together from 50 to 150 houses typically.

5. A junknet operator essentially performs the following in this ecosystem:

1. Maintenance of the Junknet cable system.
2. Making cable connections for end users.
3. Helping end users with the set up of the connection at the PC.
4. Securing rights (mostly informal, driven by local influence) of taking the cable in the vicinity.
5. Provisioning and upkeep of the Ethernet switches mounted atop the utility poles in the area.
6. Securing UPS based power for the switches where possible by offering discounted or free DCN service to participating teens in the area.
7. Fee collection from the end users and depositing to the aggregator. Bills or pre-paid balances are managed and are generated by another entity.

6. Junknets terminate at router points that are placed in the locality by another player within the DCN eco-system. Let us call them aggregators. To save costs and to remain at the lowest price points, the aggregator uses small Linux based PCs with up to 6 Ethernet interfaces running Zebra for routing. Aggregators are intelligent in that they have elements that do routing and silence the broadcast storms that are possible (and often quite common) in the individual Junknets they power.

7. Aggregator networks make extensive use of optical fiber, once again, running on utility poles. Again, to keep costs down, these operators do not use optical Ethernet switches as one would expect but employ low cost media converters to connect the router PCs together over great distances - generally 5 to 15 km.

8. These router PCs are aggregated on proper Cisco Ethernet switches (for the first time in the ecosystem).

9. Typical DCNs operate over 1 Gig of network capacity between the switches with recent news of the trunks being upgraded to 10 G on experimental basis. The DCNs in Pakistan run on a rich diet of fiber which gives them the ability to support hundreds of end points in densely populated areas.

10. A network wide DHCP hands out one set of IP addresses to any device that connects to this huge network of 10 to 15 thousand end points. However, these IPs are not routed and are of no direct use to the average end customer.

11. A home grown VPN client than connects a VPN concentrator at the DCN HQ to get a functional IP address which is either a real globally routed IP address or an IP address that is recognized by one of the many proxy servers in the DCN HQ and lets the user access the Internet.

12. Non-paying customers are denied access to the VPN box until the payment is received traversing the chain mentioned above (Junknet - Aggregator - HQ).

13. It is common for the end uses to download pirated movies and software from the file servers managed by the DCN HQs. Users expect these downloads to take place at around 4 to 5 MB per second and when this goes down below 2 MB per second, the users actually complain against the service to their operator.

14. As a DCN approaches higher number of users and buy bulk bandwidth (most that I know are running STM1 or multiple STM1s for transit capacity), the statistical multiplexing effects gets stronger and end users' experience gets better.

15. For Rs 600 ($8) per month, users of these DCN generally get a service upwards of 512 kbps in middle class localities.  The DCNs often do not explicitly cap the throughput. At times, when the load is less, users get to enjoy multi-megabit Internet access. However, of late, with better traffic shaping techniques being cheaply available to the DCN operators, some permanent traffic policing is being witnessed. This is managed at the DCN HQ.

16. The cable systems running these DCNs - the fiber or cat5 cables atop utility poles - are not governed by regulations of either PEMRA (Pakistan Electronic Media Regulatory Authority) or PTA (Pakistan Telecommunication Authority) as both bodies find the operations lying in the middle gray area. PTA deals with the formal, tax paying and well organized ISP and Telecommunications industry while PEMRA deals with the TV cable systems that, at the edge, depend on the coaxial cable running atop utility poles to reach their end users. The fiber that takes the DCNs from one locality to another locality (with the PC routers as bridges between them and the junknets) runs along with these TV channel cables.

And from a subsequent email answering some additional questions:

2. There is a tremendous potential for DCNs to peer, at least in the cities. The trump card here, again, is the near universal availability of fibers-on-the-poles which can allow DCN HQs to peer at multi gigabit speeds. However, the biggest challenge is not technological unfortunately. The biggest challenge is that the DCN top-guys are yesteryears street roamers and getting them to digest the idea of putting a dedicated switch (vs a PC based multi Ethernet port aggregator) is tough beyond expectations. Telling them why peering is important and the subsequent marvels this will bring in is like explaining a fine color shade to <someone> born blind. I've had personal experience in this respect.

3. Pakistan is paying through its nose for the Internet Transit it buys for its population. As a country, it is estimated that Pakistan subscribes to around 10 GB of Internet transit.  For operators like my employer, STM1 costs in the vicinity of USD 23,000 per month and STM-4 transit capacities in the vicinity of USD 65,000 per month. In Pakistan, where broadband penetration is just 170,000 broadband connections (BB is defined as anything above 128 kbps), there is a huge potential <demand for> domestic bandwidth which is just not being taken care of. The entire country is depending on what I insist to call 'imported Internet' and there is no way this model can scale if we are to touch a few million broadband users by the next 18 months.

4. Asymmetry sometimes comes into play because some DSN have HFC cablenet and/or DSL connection bundles as their upstreams or backups. When these are in play, the upstream performances get bad.

Real Broadband - Will the US ever catch up?

Here's a good item from Give Up Internet!

Tragically, the "North American" comparison here is with Canada, a country that comes out substantially ahead of the US in most broadband Internet access rankings.