Communications

Wireless ISP Conference(s) - WISPAPalooza 2011

I'm at the Wireless ISP Association's conference in Las Vegas.  The conference name: WISPAPALOOZA with these sessions. Following the main conference, there are several vendor conferences including the Ubiquiti World Conference on Thursday and a MikroTik User Meeting (MUM).  Since netBlazr is using both MikroTik and Ubiquiti gear, this is a good conference to attend.

I'm also speaking tomorrow and moderating a session on Antenna Propagation: Principals and Possibilities.

I'll be taking notes on Twitter, but on the netBlazr twitter account.  If you interested in, follow me at:

http://twitter.com/netblazr

 

Limitations of Carrier Grade NAT, and some workarounds

Qtel, the largest carrier in Qatar (and nearly the only Internet provider) appears to connect all their users (~600K) to the Internet through just one or a very few public IPv4 addresses.  82.148.97.69 was their single public address in 2006-2007.  How can network address translation (NAT) put all those users through just one IP address?

Well, both UDP and TCP include 16-bit port numbers, so for each IP address there are 2**16 = 65,536 unique IP address/ TCP port number combinations which can support network address translation (NAT) for more than 65K simultaneous connections.  Of course, individual browser sessions may open multiple connections so each on-line user can tie up several such connections, even several dozen.

But there are more unique fields that come into play.  On the Internet side of the NAT, each active IP session has five fields of interest:  source IP address, source port, protocol, destination address; destination port.  So you can use the same source address and port number for multiple independent sessions if the sessions are going to different destinations or using different protocols. With this approach, one public IP address can clearly support millions and millions of connections.

But what if most of your users are trying to reach just a few destination addresses?  That’s a problem Qtel had (or at least their users had) when accessing Wikipedia in 2006.

One obvious solution is to devote more than one public IP address for the whole country!  Indeed, if a few hundred people are attempting to access the same address simultaneously, then having a few hundred IP addresses (for the whole country!) would do it. Of course, if the destination IP address is that of Google’s search bar, you might need more than a few hundred IP addresses.

There is another path, one which Geoff Huston alludes to in a recent article on the IPv6 transition.

… we may expect to see a push to re-architect content into Content Distribution Networks that have points of presence in major access networks.

Today, major content distribution networks (CDNs) like Akamai, Amazon Cloudfront, Limelight and Google each maintain thousands of servers close to the edges of the Internet.  These servers cache popular content close to where the users are.

When many, many users in Qatar simultaneously try to access the same content, the load on Qtel’s NAT boxes could be dramatically reduced by putting the appropriate CDN’s caching servers within Qtel’s private address space.  Then requests for popular content wouldn’t even go through the NAT, reducing total traffic and dramatically reducing the occasions when two users are going through the NAT to the same public IP address.

It’s not in line with the Internet’s original end-to-end design, but as IPv4 addresses get more expensive, we can expect to see more and more use of NAT and CDN servers moving into carriers' private address domains.

 

YouTube video flow control actually causes packet loss (and retransmission)

There's an interesting article in the April 2011 issue of ACM SIGCOM Computer Communications Review. The study by Shane Alcock and Richard Nelson (also accessible here) looks at the way application level algorithms on YouTube's video servers interact with TCP and the actual buffers in a retail DSL delivery system.  I found this particularly interesting as I had already looked into the burstiness of other kinds of traffic in a set of posts in late 2009 (this post in particular).

Here's the way YouTube servers deliver data (at least as of January 2010) to high capacity clients:

By high capacity client I mean the client device has enough buffer space to absorb whatever data the video server wants to send and the client's broadband Internet access service has enough capacity to avoid congestion during delivery of these video bursts.  Unfortunately, the delivery rate YouTube was using in January 2010 was enough to overwhelm the data buffers in typical DSL infrastructure as this graph illustrates.

It appears the YouTube servers send data in blocks of 64 KBytes or 128 KBytes (or multiples thereof) and these blocks can exceed the available buffer space in the DSL delivery infrastructure - mostly likely the buffers in Broadband Remote Access Server (BRAS).  At least one BRAS I was familiar with in 2004-2005 had 200 ms of buffering, obviously not enough for one burst per second.

The good news is the net effect was only a 1% increase in the total data (due to retransmission) so this isn't a big problem and the authors discussed the issue with Google where engineers were working on a solution (in 2010).  What's interesting to me is the extent to which the sizing of buffers for IP infrastructure is still a complex issue and the ease with which a simple application level decision can go wrong.

 

Paid Peering: Issues and misunderstandings

Recently I was asked my opinion of Google paying France Telecom (FT) to deliver traffic to into FT's network, i.e. Google paying to peer with FT. I wasn't aware Google pays FT. I don't even know if it's true. But I do know this is a topic fraught with misunderstandings. Also, if there is a "problem" here, the problem is one of competition (or lack thereof) in portions of the French broadband access market. It is not a problem that can be or should be fixed by "network neutrality" regulations or legislation.

To understand what's happening, you need to understand the Internet's business model and you need to know something about peering as it works in the real world.

The most suscinct definition of the Internet I'm aware of is: 
Internet (n.) A voluntary agreement among network operators to exchange traffic for their mutual benefit.

This is very different from the telephone networks (fixed or mobile) where relations between operators are governed by laws and regulations, change at a regulatory pace, i.e. decades, and are continuously gamed by political processes.

Each Internet operator needs to be able to reach all possible Internet addresses, so interconnection is essential. If you are a small operator, like netBlazr, you purchase "Internet transit" services from a larger operator. Rates are volume dependent and arrangements are similar to those of any other business or individual purchasing an Internet access service.  If you are a large operator, a large content provider or a very large business, then interconnection costs and arrangements become negotiable. Two operators in Holland will likely agree to exchange that portion of their respective traffic that is destined for each other's networks at the AMS-IX which is local, rather than pay for Internet transit services that would likely carry that traffic to London or the US where a higher tier operator would handle the exchange.  Thus the two Dutch operators engage in peering for that portion of their traffic for which it makes sense.

This pattern is repeated throughout the Internet as each operator evaluates where, how, and with whom it makes sense to exchange portions of their traffic so as to minimize costs, increase performance or achieve other goals important to that operator. Thus there are vastly more interconnection contracts, formed under more diverse terms, for the Internet than ever was the case for the telephone network. Further, this network of agreements is in constant flux as individual operators find better deals.

It's instructive to follow the growth of YouTube from their founding through their acquisition by Google. When YouTube began, they had to pay standard rates for Internet access. As they grew to be a significant content source, their cost of Internet access fell as more and more operators asked to peer with them, as I discussed two years ago.

So between Google and France Telecom, who has negotiating leverage? and why?

France Telecom already has peering agreements with multiple other Internet backbones, so their customers have several paths to get to Google content.  Why would either party want to connect directly?

First, it might cut costs. Whatever pre-agreement path the traffic takes involves some cost. Perhaps that's money paid to other operators or it may just be the cost of maintaining the facilities needed for the pre-agreement traffic (e.g. to go via London).  I can't guess the specifics for Google or for France Telecom, but each has a possible interest here.

Second, it will likely improve performance. Even shaving 50 milliseconds off of round trip times (RTT) can improve user experience as accessing a single web page can require the exchange of dozens of messages, thus 50 ms less RTT can shave more than a second off the page load time. In a competitive market, France Telecom might want to advertise their network as the "fastest," but local access competition is not that great in France (or in many markets, like the US).  France Telecom has nearly 50% market share.  They are losing ground to Free, but apparently it's not enough to justify a marketing campaign based on the "fastest" network.  Google on the other hand, has a strong reason to want better user experiences. In Google experiments with search latency, longer latencies reduce the number of searches a user does per day. Given the number of Google search users on the France Telecom network, Google can directly compute their monetary return from cutting search latency by X ms. So Google's negotiators have hard numbers in their back pocket when they go to talk with France Telecom.

Is this good or bad? Well, if France Telecom doesn't have to compete on the performance of their service, that suggests a failure of competition policy. But then a solution should address competition in the French Internet access market, not government regulation of peering or the imposition of "network neutrality" regulations on operators that don't have a monopoly position.

For myself, I look forward to continued growth of new providers in France, like Free, as I discussed in the January issue of NGN Magazine.

 

Paul Baran and open spectrum

Paul Baran, an engineer who helped create the technical underpinnings for the Arpanet (the precursor to today’s Internet), died Saturday in Palo Alto, California.  While everyone else celebrates his early contributions to packet networking, I will remember him for his open spectrum advocacy.

I never had the privilege of hearing Paul Baran speak live, but as I became interested in wireless spectrum, one of the earliest references I encountered was this transcript of Paul Baran's Keynote Address at the 8th Annual Conference on Next Generation Networks Washington, DC, November 9, 1994.

His title says it all:

Visions of the 21st Century Communications: Is the Shortage of Radio Spectrum for Broadband Networks of the Future a Self Made Problem?

And here's an excerpt:

   The key point at issue that we will question is the widespread belief
   that we don't have enough radio spectrum to go around. This is a
   common, fundamental belief. Since we live in a world of scarcity or
   natural resources it is almost automatic that we believe that there is
   a shortage of frequencies. This particular resource is somewhat
   different.
   
   This morning, let's start by reviewing this presumption of a permanent
   shortage. Let's consider how, with an application of already known
   technology, we could create even a surplus of frequencies. What may be
   going on is an inadvertent shortage created by a regulatory structure
   which has yet to appreciate the potential capabilities of the new
   digital signal processing technology as applied to communications.

Paul Baran is hardly the originator of open spectrum ideas, but the complete transcript shows he once again led the industry.

 

Basic questions about NGNs

I get questions on telecom, mobile and Internet topics from students in different parts of the world and I try to reply to them all as best I can.  One kind of question that comes up repeatedly has to do with "Next Generation Networks" or NGNs - what are they?  why are they based on packet technology?  how do they support multi-services? etc.  

Recently I answered a whole string of such questions for a Masters student in the UK.  Here are my answers:

Hi  ... ,

For most kinds of communication, packet networks are more flexible and more efficient.  And, as a practical matter, the Internet (a packet network) has become a globally pervasive platform, which indeed can be used to support almost any kind of communication.  Most everything is migrating to the Internet because the Internet is pervasive and Internet technology is by far the highest volume and lowest cost of any communications technology.  So it's natural that any envisioned NGN would be built on packet technology.

The term NGN is a bit loaded, as it normally refers to a telco-centric view of how the Internet should evolve.  My views on that are here:

   http://www.tmcnet.com/ngnmag/columns/articles/126646-ngn-itu-misses-boat.htm

For the latest on how many AS numbers have been assigned see:

   http://www.cidr-report.org/as2.0/autnums.html

For the latest on how many AS numbers are currently advertising routes on the Internet backbone, see:

   http://www.cidr-report.org/as2.0/

The only issue with carrying a variety of services over a common platform is whether you need different relative levels of service (QoS) to support different higher level services like telephony, video, web browsing, etc.  The telco view is that a sophisticated range of priorities are essential (and that the telco should be able to differentially bill based on the higher level service that is being carried).  However, no one has figured out a commercially viable way to deliver QoS over the public Internet.  Also, if there is any requirement for QoS at all, it is for very simple differentiation at just those few points where there are bandwidth shortages.

The richest set of details on what an NGN is envisioned to be is presented in the standards of the 3GPP (the body that provides standards for GSM, HSPA, LTE and other wireless networks).  See: 

   http://en.wikipedia.org/wiki/IP_Multimedia_Subsystem

   http://en.wikipedia.org/wiki/System_Architecture_Evolution

and if you want to read the actual 3GPP specs, you might start here:

   http://www.3gpp.org/ftp/Specs/archive/23_series/23.402/23402-a21.zip

You'll have to dig through documentation on the 3GPP's IMS and SAE if you want to understand their plans for their NGN.  I don't expect any NGN to gain much traction compared to the open Internet and I personally have moved from a business where I supplied products to telco equipment vendors into a new business, netBlazr, where we are revolutionizing the delivery of fixed Internet connectivity in urban areas. So I'm not so interested in NGN any more.  :)

MPLS is an IP-centric protocol layer just below the IP layer which allows Internet Service Providers with multiple routes to define "virtual circuits" between points in the core network.  It's a way to hide (i.e. simplify) an otherwise complex network of IP routers and to channel certain kinds of traffic over specific virtual circuits.  This can be used to guarantee enough headroom for certain kinds of traffic and for other network management purposes.  For example, if you want to give priority to voice traffic, you could route all voice-related IP sessions over a different path (different at the MPLS level) and you could guarantee enough bandwidth on that path so voice traffic was never over-subscribed (i.e. never ran out of bandwidth).  When there was little or no voice traffic, other MPLS virtual circuits running over the same links could use the idle bandwidth, but when voice traffic increases, the voice circuit would get it's predetermined bandwidth.

Indeed, MPLS works by assigning labels (which are inserted between the link layer protocol header and the IP header).  Here's a view of how MPLS & QoS are envisioned to work:

   http://www.huawei.com/products/datacomm/catalog.do?id=2515

This is simpler than the ITU/3GPP view of NGN, but still likely more complex than what most network operators are deploying.  On the other hand, I don't have direct field experience operating a backbone network, so i can't give you the real details.  My impression is that Internet backbone operators use MPLS but only for very simple pre-defined bandwidth allocations between major backbone routers.

I hope this helps,

Brough Turner

 

My Column has moved to NGN Magazine

"Next Wave Redux," which has appeared in Internet Telephony magazine for many years has moved.  I'm now writing for NGN Magazine.

With my attention focused on netBlazr, I wasn't in a position to do justice to Internet telephony topics.  I thought of abandonning the column entirely, but the folks at TMCnet urged me to move to a different magazine, more in line with my current focus.  Of course, the term NGN carries a lot of baggage which I address in my first article, NGN: ITU Misses the Boat, written last fall for the November 2010 issue.

If you followed by columns in Internet Telephony, you may want to sign up for electronic delivery of NGN magazine (i.e. via email), as the issues typically don't appear on the TMC website until a couple of months later.

US Broadband - a personal rant

With netBlazr up and running, it's time for me to write some posts on the netBlazr blog.  I've started off with a personal rant about broadband in the US and why I'm putting all my time into our attempt to do an end run around the whole mess — the duopoly (phone company & cable company), the regulatorium (FCC, etc.) and Congress.

Of course this mention on my personal blog is not only to induce you to look at the netBlazr blog, but to attract some links pointing to the netBlazr website.  Right now, when Google sees "netBlazr," it thinks you've mistyped "netblazer." Please help us change that!  Write something that points to somewhere on the netBlazr website.  Thanks!

Network Neutrality is the wrong fight!

Winning would mean giving up much more important rights – historical rights that were in place in the US as recently as 1995 and remain in place in most of Europe even today. 

We shouldn’t settle for network neutrality. It’s a poor substitute for what we had and much less than what we need. Let me explain. There are two topics to discuss. The first is “common carriage,” a centuries old legal concept that applied to the US telecom industry throughout most of the 20th century. The second involves communications protocols. Both topics are complex, so I will cover only what’s needed to understand why we shouldn’t accept network neutrality and why, at a minimum, we should fight for enforcement of existing common carriage rules. 

Network neutrality is about allowing any Internet application to run over an Internet connection, i.e. over a connection that uses Internet Protocol (IP). But under common carriage as it applied prior to the late 1990s, we had a more powerful right – the right to run any kind of network protocol, IP or otherwise, over a lower, simpler service which today we call a “bit stream*.” Why does this matter? Because real innovation is also possible at these lower layers and that innovation continues to be important. But today, such lower layer innovation is restricted to inside one building or one campus. Yes, we can tunnel some lower level innovations over IP, but not all of them and only at a cost. 

IP telephony (VoIP) is one place where problems arise. Most enterprises use IP PBXs internally, yet calls between enterprises use the PSTN. Many companies have attempted to address this gap, but progress is slow and expensive. Within an enterprise, IP telephony packets are given priority, but that priority is not supported on Internet access links and network neutrality doesn’t help. As a result, to interconnect VoIP calls, enterprises must lease separate dedicated access circuits – circuits usually based on bit stream access – to support “SIP trunks.” Up until the late 1990s, these circuits were regulated under common carriage. Today they are an unregulated monopoly, with prices derived from the cost of voice circuits 15-20 years ago, i.e. abnormally expensive for today. 

 Common carriage is the legal concept that, in exchange for government granted monopoly access to rights-of-way, the monopolist must carry anyone’s traffic over the resulting infrastructure, at regulated rates. For centuries this has applied, to canals, to roads, to railroads, to telegraph lines and, until nearly the end of the 20th century, to telecommunications lines. But during the legal battles after the Telecom Act of 1996, the FCC basically gave up on common carriage.

If we accept Network Neutrality instead of common carriage, we guarantee future innovations happen only above the IP layer. Innovation at lower layers will be restricted to enterprise or campus applications. That’s too bad as it was the existence of common carriage that allowed the Internet to develop in the first place. Do we want to eliminate that kind of innovation in the future? 

If anything, we should be fighting to extend the ideas of common carriage to lower layers, e.g. dark fiber. Installing dark fiber is expensive and requires access to rights-of-way that are limited. The installed fiber is capital expensive infrastructure that lasts for decades. Such conditions justify granting monopoly access, in exchange for common carriage and regulated rates of return. But when you light up a dark fiber, you use (relatively) low cost gear with a short life (even if it can survive for ten years, Moore’s Law renders it functionally obsolete within 2-3 years). What's more, there’s rapid innovation in opto-electronics gear. Just look at the order of magnitude difference in cost between enterprise and carrier fiber-optic gear. 

Today, the US is loosing leadership in all things Internet. Network Neutrality will just put a nail in our coffin. To stop our decline, fight for restoration of the common carriage principals that existed through most of the 20th century and still exist in law. To regain world leadership, fight to extent those principals to include access to dark fiber at regulated rates.

---------------------------------------------------------------------------------------

* Bit stream access.   In the 20th century two regulated services provided what the 21st century calls bit stream access. These were voice telephony and T1 circuits. T1 circuits directly carry a stream of digital bits. Modems allowed voice connections to carry digital bits, for example, for bulletin board services and other purposes long before the Internet became popular.

My notes from the Community Wireless Networks Summit in Vienna

As with most conferences I attend, I took most of my notes using Twitter.  Since I can't always depend on accessing old tweets, here is the entire tweet stream from the conference in one blog post.  I don't know if this is of use to anyone else, but at least I know I'll have a record of my notes that I can refer back to.

The full name of the conference was The International Summit for Community Wireless Networks (ISCWN) and it was held in Vienna on August 12-15, 2010.  See the conference website.

International Summit for Community Wireless Networks http://bit.ly/9Lp41p
Good conference Wi-Fi http://bit.ly/dkwUh2
OK, I'm finally set up to Tweet &/or blog the International Summit for Community Wireless Networks http://wirelesssummit.org
While there's plenty of broadband behind the Wi-Fi, there are very few power outlets, at least as yet.
Community Wireless conf finally getting underway
Sascha Meinwrath opening: 1/2 the participants are missing, even some who were here an hour ago, so expect stragglers, but we start.
Sascha: 80% of cash raised goes to travel allowances; 10% everything else; plus donations.
Aaron Kaplan, Funkfeuer, on TechGate facility science park estbsh'd 2001 by City; conf venue; also supports Funkfeuer
Aaron there is swimming in the old Danube - it's clean now! Also boat rental nearby!
Aaron: Funkfeuer hotspots since 2003, now reaching almost to Bratislava. Mixed mesh and P2P backbone
Sascha on previous feedback - social time is key
Jim Baller: America at the Crossroads: Greatness or Mediocrity. 15+ yrs fighting ILECs on behalf of muni's - mostly successful!
Baller: credits Broadband Coalition http://bb4us.net/  Also as key to Congress requiring FCC to do a BB Plan
Baller on the US national BB plan: We need big goals, the BB plan didn't deliver! It's goals for 2020 happening elsewhere today.
Baller: US BB Plan focuses on inches, not yards and miles.
Baller: US BB Stimulus: Awards coming quickly (9/2010 deadline): middle mile, rural first mile
Baller on Google gigabit initiative yielded 1100 muni applications and showed pent up demand
Baller: Ugly side - incumbents bigger, stronger & nastier than five yrs ago. Cable industry even nastier. Now Google is in question.
Baller: But btwn BB Plan (it's a plan), Google fiber init., $7B Stimulus goes to new parties, community BB efforts
Baller responding to Q about wireless: I'm a fiber guy, then disappointing comments on FCC being pro-wireless. Misses FCC focus :(
Baller responding to Q about Australia admits that most in US don't know what's happening in the rest of the world.  US always has excuses - "we're different."
Sascha introducing Ramon Roca, President, guifi.net Foundation, a fantastic alternative that's happening in Spain. 
Guifi runs a very large network across rural (and urban) Catalonia in Spain. http://guifi.net/
Ramon Roca: Vision for the Future. Will talk about: Why scale? and their sustainable economic model!
Ramon: Guifi has 10K+ nodes and 15 Km of networks, but still tiny compared to telco. Scale as a goal. Need growth to be sustainable.
Ramon: need sustainable model - critical mass; org issues; legal; partners. Non-profit core w/biz around.
Ramon: Demand is there; also incumbent not interested in low 40%.
Ramon on mgmt: horizontal, bi-directional & collaborative - examples from their website. Website appears to be key for communit.
Ramon on public/private: public requirements must be written: contracts, license, P2P agreements. Must be clear, and legal.
Ramon on public/private - like OS licenses, the network is considered private by govnmt. Need documents just as open src does.
Ramon facilitates integration and cooperation between diverse groups, e.g. need churches for their steeples! need public ROW access.
Ramon: Guifi license is required if you want to participate! Like GNU license, Guifi license even deals with revisions.
Ramon: Guifi license defines everything
Ramon: Guifi is also a ""public"" network in the legal sense in Spain because they had to in order to deploy some fiber.
Ramon: Guifi ROI for many: supply chain dlrs/shops, prof svcs and svc provdrs who mediate wholesale mkts.
Ramon: Catalonia is a bit ahead of Spain but w/o Guifi Osona (rural Catalonia) way behind. Now Osona ahead of rest of Spain & even the UK.
Ramon: Guifi network now 15% of BB links in Osona; and yet DSL subscription haven't dropped.
An English version of the Guifi Wireless commons license is here: http://guifi.net/WCL_EN
Ramon futures: working on fiber from farms (FFtF) cheap & easy & Gbps; always a combination.
Ramon has had fights with incumbent about ROW (even when on private property) and about access to poles. Ramon - do it! then fight.
Ramon - their main focus is to stay on private ROW, but as a public carrier they can now go after public ROW.
Ramon: stories about fights over public fiber & public ROW - everything you expect, except Guifi seems to win, espc. in rural area.
Ramon: Guifi participants getting access to public fibers laid for cameras after threaten authorities - tractor might hit camera...
Ramon concludes by emphasizing need to enlist service businesses and computer shops, e.g. 3rd party for profit businesses.
Lightning talks where individuals describe their networks.
Austin (TX) Wireless - hotspot service; wanted free Wi-Fi; customers incl coffee shops that compete with Starbucks.
Austin Wireless - other things to help venues, e.g. splash pages
Austin Wireless: splash pages useless if not totally relevant for the customer: Weather, sports, local info, Facebook links, venue.
Austin Wireless: Now making $ by providing marketing services to their venues, restaurants, coffee shops, etc.
Austin Wireless: Also does Chimpit which brings venue portal even if you're on another network.
Chambana.net joint project of IMC and Acorn IT collaborative - ~6 active people and 9 servers. Walk-in computers, media prod. lab.
Chambana.net host websites, many other IT services, but also some wireless! Small network, switching from NetBSD to OpenWRT.
Chambana.net: It sounds like their wireless net is secondary to what they do, and is currently down and being reworked.
Chambana.net: Wireless network is completely open, no splash page. Runs on donated hardware
Chambana.net runs on donations, but entire uplink is one Comcast business service. But part of a stimulus grant, so near future good
Tribal Digital Village: 19 tribes HQ in San Diego CA, only way to get Internet to reservations. sovereign nations within US, but...
Tribal Digital Village: 350 miles of P2P and P2MP links mostly license exempt; 18 bldgs
Tribal: Paid NW in parallel reaches 200 individual homes. 2/2 Mbps for $34.95 per month. Hope to reach 2000 of 2700 homes.
Tribal: 1K devices connected to net. Fiber at headend feeds an arc of P2P links. Now many gamers accessing via local centers.
Tribal: Too small and separate to think about running their own radio regulatory regimes. But not too worried about conforming :)
Djursland http://www.diirwb.net/ Rural area in Denmark; 95% can get DSL (fr 1600 exchgs). Last 5% not svc'd.
Djursland P2MP design
Djursland: NW built by volunteers
Djursland gets favorable Interent transit because the farmers drove (& control) the fiber backbone deployment.
Funkfeuer: mesh using OSLR; got help from jaap@scii.nl & friends in Berlin.
Funkfeuer: 240 roofs; financial sustainability based on hosting center revenues; Gbps uplink
Funkfeuer: slightly below mkt for hosting; TV streaming experiments; running Wi-Fi for events.
Funkfeuer: Fiber splicing is easy, if you have the expensive machine (~6000 Euros). Basically welding glass.
Funkfeuer is a closed user group. This is important for VoIP and other regulatory issues, including streaming TV to you members!
Funkfeuer Graz: like NW in Vienna; Technical Univ in Graz helped them grow fast; slower growth >2007 as 3G faster.
Funkfeuer Graz: now working to link with Bratislava (Slovokia) which could be the first international community network in EU.
Funkfeuer Graz: longest link today is 30 km which gets them 1/2 way to Bratislava (Ubiquiti radios)
Malcolm Matson knows of two networks connected (quietly) btwn Solovakia and Hungary.
Belgrade Wireless: up to 20 km links (> 50 Mbps) using corner antennas as shown in previous conf. NW now extends over 100 Km.
Belgrade: 3D corner antennas invented by Prof in Belgrade. Big focus on community events.
Belgrade: 3D Corner Antennas http://su.pr/AjXm1p - feeder for dish reflector - mixed polarization
More 3D corner antenna info here  http://su.pr/7t9990 Need to follow up on whether this remains relevant w/ MIMO using polarization.
Updates from the OLSR-NG Project - Henning Rogge (FKIE) & Aaron Kaplan (Funkfeuer.at) - history, today & futures.
OLSR-NG: 1 of 2 major mesh stds (other AODV). RFC 3626. Tonnesen PhD, Lopatic LQ & Fisheye extensions
OLSR-NG working on OLSR. Guys in Berlin starting over (BATMAN); HSLS hazy sighted link state (CUWin).
OLSR-NG session: CUWin HSLS didn't get beyond simulations
OLSR-NG: OLSR is link state - every node knows whole graph (100K entries = 4.8 MB); but MPR now off
OLSR-NG: MPRs only matter with really dense NWs, but with only 2-3 links per node, they don't pay.
OLSR-NG: ETX link quality metric used instead of basic hop count, i.e. sum of ETXs not sum of hops. But heavy compute load!
OLSR-NG bringing down compute load (now linear not exponential). 100-to-1 benefit with 400 nodes. Still Dijkstra, but optimize data.
OLSR-NG: malloc() thrashing fixed
OLSR-NG futures: soft refresh (CSN), better metrics (ETT, MIC), multipath routing (experimental), Q of layer 2 capabilities.
OLSR-NG: Henning comments on how few multi-path routing projects have made any progress... Problem: must choose whole path. very hrd
OLSR Henning: OLSRd 0.6.0 is current. Clean rewrite of routing code; smarter gateways to reduce thrashing btwn GWs.
Henning: 0.6.0 has very few (& only site-specific) bugs - very stable!
Henning on future plans: telnet/http server (done); config mgmt (stability, flexiblty.
Henning also thinking about better metrics but limited by packet format prior to OSLR v2. Many metrics in discussion in academia.
Henning responding to Q: negative about dual protocol mode to support migrations. Thinks it wold be very hard.
Henning - active developers= ~1.5 people
Henning complaining about academics who've made patches w/o consulting (thus doing stupid things) and without plans to give back.
Henning & Aaron on how plugin system makes it easy to try new things in a clean fashion.
Freifunk Berlin started 2002; then OSLR in 2003-04; 28 devices @ 2004 OS conf; PC to openWRT for embedded.
Freifunk took off in 2005 despite labeling website ""OSLR experiment"" but users wanted reliability
Freifunk can't even switch to B.A.T.M.A.N. because OSLR widely deployed
Freifunk net is now shrinking, as people who came only for bandwidth are getting DSL and 3G mobile.
Freifunk has issue of switching gateways which doesn't affect Frunkfeuer with their fixed gateways and public IPs.
guifi.net is a bunch of communities, not all interconnected. 10,300 nodes using same software, same tools and same license.
guifi.net is showing off an impressive set of tools for examining nodes, plus there's extra data that the node owner can access.
guifi.net Using MRTG http://oss.oetiker.ch/mrtg/
guifi.net is a large wireless LAN. Must search for & connect to services, like Internet access. Libraries & other offer Inet-GWs.
Île sans fil network in Montreal  http://www.ilesansfil.org/  Now 200 hotspots installed, free Internet thru portal page.
Île sans fil limits users to 7 GB/wk. 150K users registered. Each hotspot has own page. Projects: Authpuppy, WiFiDog
Wireless Toronto - captive Wi-Fi portals, started with Wifidog from ◊le sans fil.
Wireless Toronto has struggled compared to Montreal. All volunteers. Hotspots, but now adding mesh, e.g. in parks.
Wireless Toronto - no spt fr Government who started and sold a parallel NW. Hard to find location-based content for portal pages.
Wireless Toronto survives on annual fees from businesses with portals; organized as a club, not a nonprft
Wireless Toronto new mesh started with BATMAN but found OSLR more reliable. Using Open-mesh but moving to Authpuppy.
Wireless Toronto - the less you talk about Wi-Fi and the more you talk about mktg, the better you do selling business hotspots.
Wireless Toronto using AutoAP in part to just monitor what's going on, wirelessly, in their neighborhoods.
Open Wireless Networks http://consume.net in London
Consume fell out of use ~2003
OWN: nodes all clustered in London near Greenwich Park. Not 400 nodes
Good line-of-sight planning tool:  http://www.heywhatsthat.com/
Village Telco http://www.villagetelco.org/about/mobiles have brought telecom to Africa, but not Internet, yet...
Village Telco: it didn't take off
Village Telco: Built ""mesh potato"" - solar mesh device w/analog phone adapter; production units next month.
Village Telco: Mozilla is filming the project.
Village Telco: Retail cost $119
Wlan Ljubljana: http://wlan-lj.net Can't beat widespread fiber at $14/month, but each has excess capacity and willing to share.
Wlan Lj now up to 50 nodes & adding rural areas - now wlanslovenija! http://wlan-si.net entirely volunteers
Wlan Lj is clearly a group of hackers having fun, but it's not clear to me if they are really serving a need. Sustainability???
Wlan Lj has done solar nodes - 24 hr reliability but froze during the winter and wasn't restored until spring (cold on roof!).
Athen Wireless Metropolitan Network (AWMN) strtd 2002 because no DSL; open experimentat WLAN.
AWMN has some people who offer Internet access, but it's not the primary goal. Participants tend to be Univ types - young, educated.
AWMN mostly at 5.4 GHz with Linux and MikroTik routers. Islands of OSLR connected by BGP; 2505 nodes; 1100 backbone nodes.
AWMN speeds vary 11 Mbps to 150 Mbps. 730 access points. Organized by an association; events; community; no grants.
AWMN Recently, large deployments of 11n
AWMN As a local LAN, they have mirrors of many Internet services, also transliterated version of Google (Woogle), Yahoo, etc.
Richard MacKinnon of Austin Wireless follows me in the Freemium session. AW strt'd as all free, but wasn't sustainable.
MacKinnon: Austin Wireless used automation to replace 50 volunteers with 2 fulltime staff. Merchants pay to provide free access.
MacKinnon: Pull together local news for hotspot portal pages
MacKinnon: Used to charge $5/mo
MacKinnon: Integrate Facebook into portals, incent patron's to talk about the business they're visiting on their FB page. Biz value!
MacKinnon: installations were fun at 1st
MacKinnon: $55/mo buys support. Restaurants hate Wi-Fi but have to have it to be competitive. So support is key - ""power cycle box!
MacKinnon: deals with cable company; POS crdit crd installers.
MacKinnon: Ads didn't work but local ads may be coming back.
Nemanja Topovic, Belgrade Wireless, Serbia started as all volunteers. Had problems with Government (spectrum laws).
Topovic low cost svc didn't work (1 Euro/mo) as people expect full svc. Discusses many paths they've tried, unsuccessfully...
Topovic - network grew rapidly 2004-2006 but growth has stopped. Looking for a program that could restart their network.
Topovic: BGWireless not a mesh, uses high speed P2P and P2MP. Closed network.
MacKinnon suggests his biz mode for Topovic. Key is offering free svc to avoid spt issues and then find a premium svc to cover costs
MacKinnon - important to offer new paid service as something new, not as a price increase on old service.
MacKinnon: 200 customers today (and 1 & 2 yrs ago) but different group & paying more & more loyal. Free customers were least loyal.
MacKinnon user community divided: some just want Internet
Paul from NFP? talking about SW defined GNU radio work going on in the building. Available for discussions later...
Robin Chase, Meadow Networks (previously founder of Zipcar) will be evening keynote spkr - next up.
Robin Chase on tie-in of transportation and networks: financing (fuel tax moving to road tax eventually - per km!).
Chase Auto density in cities (congestion) but expense of rural rds. Moving to congestion pricing. Will need more technology...
Chase on transport costs not reflecting true costs (fuel/environment, etc.)
Chase Public-Private Partnership discussions are missing the individual. Transport tech has decided they need their own stuff.
Chase: transportation guy gets his spectrum; EMS/medical types need their own stuff- it's just comms
Chase: Transport problems: Lumpy density, congestion, financing, right pricing same problems as in comms infrastructure -> dist. NW
Chase: future for transport and comms is distributed networks
Chase focus on distributed and collaborative inputs as a path to innovation.
Chase: words to use when talking about Gov. spending: open data; open standards; open source
Chase talking about Comuto (ride sharing in France)  http://www.comuto.fr/ now has more unique users than ZipCar (& just in France).
Chase: talking about CouchSurfing: 7 yrs old; 200 countries, 71K cities; more ""beds"" than major hotel chains.
Chase on Chatroulette - built it in 3 days; but in 6 months it now gets 30M unique visitors - mindboggling!
Chase: Andrey could do ChatRoulette because platform (Pcs, Internet) was there and there was excess capacity available.
Chase on all the crazy ideas that are now iPhone apps and yet the excess capacity has fostered some incredible innovations.
Chase: People & Platforms -> Speed & Scale
Chase wants open mesh device in every car that will have to be paying congestion fees, getting traffic data
Chase wants an open platform in cars so we foster new apps we can't can't even envision now.
Chase: what if all nodes (smart grid, smart cars, smart infrastructure) were peers? and open platforms!
Chase: www.networkmusings.blogspot.com & @rmchase & rchase@alum.mit.edu
Vic Hayes of TU Delft University: Spectrum Assessment for Wi-Fi. History of FCC & license exempt spectrum and Wi-Fi market.
Hayes: FCC landmark decision 1985- license exempt; use more spectrum than required; spread spectrum tech.
Hayes: Standard CDMA history including Hedy Lamar's patent, but I learned something new - she was born in Vienna!
Hayes has some good slides to explain CDMA. I'll probably stick with my standard slides http://su.pr/1LMCYl
Hayes: FCC wanted to allow spread spectrum over wider bands but got objections so they settled on ISM because no one cared.
Hayes on attempts to get similar rules thru CEPT - succeeded in 1991 for 2.4 GHz band only and with slightly different rules.
Hayes: CEPT said only -10 dBW ERP (100 mw) and 10 mw per MHz
Hayes: because of cost of electronics, 900 MHz took off 1st (in 1989)
Hayes on how Wi-Fi 11 Mbps beat HomeRF even though the FCC permitted wideband hoppers (1998-2000). Wi-Fi faster
Hayes: 2002 FCC permits intelligent hoppers (reduce blutooth intf) & power spectral density rule (opens the way for OFDM, i.e. 11g).
Hayes: Now 5 GHz - FCC NII proceeding - Apple, Lucent, etc. release 1997. Adds 5 GHz spectrum.
Hayes: meanwhile CEPT yielded to Satellite industry and reduces pwr at 5 GHz, but add more spectrum for HIPERLANS (455 MHz).
Hayes: CEPT decides to go to WRC 2003 to make 5 GHz primary and global. US problem because NTIA refused. Took until Jan2003 to win.
Hayes: In June03 WRC 2003 allocates 455 MHz co-primary in 5 GHz band. Accepted in US & EU - still in flux in many countries.
Hayes: Once spectrum is allocated, it still must be defended (find reasons in style with current political agenda).
Hayes promoting his upcoming book: The Innovation Journey of Wi-Fi, Edited by Vic Hayes et al. to be published by Dec2010.
Clarification of 1st unlicensed spectrum was 1937 for baby monitors, etc.
Hayes: Questions about Ad Hoc mode
Rabi Karmacharya & Basanta Shrestha of OLE Nepal using wireless to deliver Internet connectivity to schools in Nepal
OLE Nepal: Initial focus (2006) was OLPC, but if you got PCs, there was still no Internet
OLE Nepal: Don't have to teach kids how to use computers - that comes naturally - but need to create content in local language.
OLE Nepal: Teachers have to be retrained, and need to realize the kids may know more about the PCs that they do.
OLE Nepal: Need network infrastructure, in schools and between schools and the Internet.
OLE Nepal: Open source, open content
OLE Nepal: 28 people on staff
OLE Nepal: First priority is a network to and btwn schools as Internet upstream is very expensive in Nepal.
OLE Nepal: One highway with fiber runs length of country in the southern lowlands (near India border). Northern areas to 8K meters!
OLE Nepal: fiber line is connected to India so Internet transit pricing has come down substantially, but still very expensive.
OLE Nepal: approaching 38 schools in six districts & 4K students connected. P2P WLAN w/ typ. 5-10 Km distances
OLE Nepal: ADSL where available, used with VLAN.
OLE Nepal: On the plains, need to find a hill to act as relay point. Sometimes >20 km links needed (example 26km).
OLE Nepal: In hilly areas, may need 3-4 relay points at remote sites (4 hr walk)
OLE Nepal: School NW have: radio; switch; & 2-3 Wi-Fi Aps; all w/UPS. Select equip. for low power! Linksys WRT54GL w/DDWRT.
OLE Nepal: MikroTik 433 AH & 411 CPE - have used this up to 26 km. Also EnGenius radios, but not reliable.
OLE Nepal: SW tools: Google Earth, Radio Mobile, mirror tst'g. Use trees as towers (cut away foliage).
Ben West of Wasabi Networks (St. Louis) has posted his Community Wireless Day 1 notes here:http://su.pr/3FG39a
Ben West's Day 2 notes include notes from the session I spoke in: http://su.pr/1DHUFi
There's a big EU network that's not here. It's Czech Freenet: http://su.pr/2qMyeE 
Whoops - and this Czech Freenet: http://su.pr/24yYU8
Future for Community Wireless Networks session lead by Aaron Kaplan and Vic Hayes. EU wired net getting better and CWN shrinking...
Future of CWN: ideas fr. grp: underserved rural areas; hacker grps want to stay that way.
CWN futures: my categories: hacker comm, social comm & svc provider. All NWs have some of each, but one tends to be primary.
CWN futures: several people object to categorizations (in general?)
CWN futures: Movement to integrate node databases fr. many diff. networks. Exchange knowledge on making node db's. Mtg in <6 mo.
CWN futures: A world node database would create a larger vision of the movement, espc. for members in individual networks.
CWN futures: some desire for turnkey node solution prompts objection from Berlin hacker community that people would stop learning.
CWN futures: Ramon (guifi) sees evolution in the discussions over past 4 yrs: more consensus, meta focus (e.g. world db).
Networking rural areas: Rantanen - Tribal NW mostly done with grant money.
Networking rural areas: Rantanen offerring his cast off equipment to Krusevac Open
Networking rural areas: Krusevac network is technically illegal under Serbian law (ISPs must be licensed w/many problems & cost).
Telecom for disaster relief - Mark Summer of Inveneo talking about wireless in Haiti after the earthquake -http://www.inveneo.org/
Mark Summer: Mountain range around Port au Prince was key
Mark Summer: so many Sat Phones came into Haiti that satellite capacity was saturated.
Mark Summer: got Ubiquiti to get stuff from distributors (in 3 days)
Summer: Kitted everything over a weekend; VSAT vendor committed 5 Mbps link if dish fixed.
Summer: Google got very high res imagery on-line within 48 hrs and updated it every few days. Open Street Maps for former streets.
Summer: got VSAT up in <24hrs
Summer: Got 3-4 radios in per day
Summer: Linked 18 NGOs at 23 locations in 3 wks; NGOs didn't want it turned off, as better than pre-disaster.
Closing keynotes just starting. Tomorrow: open spectrum alliance
Aaron Kaplan thanking many, many. Node DB SIG being set up for this coming winter.
Sascha Meinrath asking for ideas we should ponder over next yr. Answers: 1. OS physical layer (GNU radio?) - seeking FPGA designers.
Answers 2) Allison Powell (@postdocal) learned we're still not good at telling stories. She's seeking people to interview.
Answers - Rabi Karmacharya (OLE Nepal) very struck by the discussions of biz models and how NW can help in disaster recovery.
Audience comments: worried that several major networks are loosing nodes
Audience comments: Session on splicing fiber was exciting. Community radio + community fiber (+ community satellite?) !
Audience comments: Credit to Matt Rentenen for offerring to pass on older equipment. Sascha suggests an email to CWN list.
Audience comments reinforcing idea of using Wikipedia as the master list of Comm Wireless NWs + spectrum laws http://su.pr/2C3YFz
At OSA Mtg http://su.pr/2AtH5O 1st formal mtg took 1.5 hrs as bylaws are in German & English. Hopefully we'll talk spectrum soon.
OSA Mtg: @postdocal asks OSA view on net neut & suggests links with other advocacy grps. Open a rathole? Sent for email discussion.
Now in GNU radio discussion. Ah techies, > interesting than policy talk.. Getting into to USRP http://www.ettus.com/ which I already know of.
Paul Fuxjaeger on GNU radio: OFDM for 11a, 11g required substantial mods to GNU radio blocks. Also hard meet IEEE timing req.
Fuxjaeger, GNU: For 2x2 MIMO, externally sync 2 USRP2s (not w/Ettus cable!) so as to maintain 2 Gig data streams, vs 1 w/Ettus cable
Fuxjaeger: http://www.oz9aec.net has interesting GNU radio stuff, also an update on Gumstix & GNU radio! More shortly...
Fuxjaeger: http://su.pr/A899kx has the pointers for Gumstix and GNU radio
Fuxjaeger: http://su.pr/1TZZeQ is the PR for Gumstix's product Stagecoach which packages TI OMAP processors that may be used w/Ettus
Kaplan wraps up the GNU radio session w/demo pasting macro blks & receiving signal from a Ham radio in 70cm band, i.e. 420-450 MHz.

My notes from the WISPA conference in St Louis

The following is based on my Tweet stream (@brough) during the conference plus some other notes I took off-line during the Tower session (when there was no Wi-Fi).  

Brief summary:  The conference was a great success, certainly for me, but seemingly for all.

Notes from the Wireless ISP Association (WISPA) meeting
St. Louis, July 21-22, 2010

WISPA meeting scheduled to start @8:45am (20 min) but room near empty. Sleeping late in St. Louis? Conf Agenda, see http://su.pr/24Zlii near bottom.

Twitter's location service keeps telling me I'm in Berkeley CA when I'm in St. Louis. Having to reset it makes the svc less than helpful.

The 40 Mbps wireless link set up for this meeting died in last night's lightening storm. Hotel has DSL? ugh. Hope radio get fixed!

Rick Harnish speaking: WISPA has signed up ~50 new members just because people wanted to come to this meeting.

Nice to finally meet Matt Larsen after many years of seeing him on email lists. Matt comments that WISPA membership now over 400.

Steve Coran, Doug Karl, Dewayne Hendricks, Patrick Leary visionaries panel is up, but 1st the ""Did you know, shift happens"" video...

Dewayne H started as a ham (age 12, 1961), then 56kbps packet radio in 1986; went commercial 1990 - wireless Internet, mesh, early!

Doug Karl started in IT as OSU, needed Internet for off-campus folks and started P2MP wireless net.

Patrick Leary started in fiber but focus is broadband Internet and that led him into wireless (initially in Atlanta ex-urbs.

Doug Karl became advocate for WaveLAN (from NCR) radios and their use for P2MP, then an advocate for early WISPs.

Dewayne Hendricks - the gating issue is regulatory. Original NOI discussed open spectrum, no power limits! Today rules are tightening.

Doug Karl commenting on having your town put extra requirements on cable company when up for renewal.

Dewayne's mission impossible plan for spectrum - demo alternate spectrum approaches in other countries (e.g. Tonga); existence proof!

Dewayne: reinvent, repurpose mass-market stuff - Wi-Fi silicon, DOCSIS modems (use transverters!).

Dewayne is pessimistic about starting a new WISPA today. I have to have an argument with him!

Doug Karl: remember you are in the communications biz - wireless may be only part of that (don't be wireless only).

Now it’s hard to pick which of three tracks... Whitespace policy; 4G; or VoIPoW. Love policy issues, know 4G & VoIP but wish I had all PPTs.

Raja Gopal - Alvarion - pushing WiMAX. $5K for a 16e basestation today but admits in 2-4 years LTE will win. But deploy WiMAX now.

VoIPoW discussion justified by how much money can be made with voice, but need QoS and high level of service - must be dependable.

TV White space discussion: politics! politics! chg sensing requirements; still large antenna issue.

Marlon Schafer asks about eventual pollution of TVWS by home LAN devices. Jack Unger on current interference mitigation discussions.

another 3 tracks decision: Mktg; FCC's 3rd way regulation; or Fiber deployment. But the only Wi-Fi access in the Mktg room. :(

Mktg Elizabeth Bowles loves on-line but (on behalf of Forbes Mercy) pushing radio (PR & Ads) for rural WISPAs; also yard signs!

Mktg: Ken Janc, Lorex Inc, on targeted direct mail and door-to-door - expensive but can be cost effective (cost per acquisition).

Mktg: Ken Janc - at least have your installers do door hangers on next door houses when doing an install.

Elizabeth Bowles of Aristotle.Net on SEO, pay for advice if you are not found via ""broadband in <your location>"", get prof advice.

Elizabeth Bowles big proponent of video on your website - dramatically boosts response rates.

Elizabeth Bowles: mobile phone enable your website. Remember smart phones with Wi-Fi access. Unlimited mobile data is dead, so people using WiFi.

Elizabeth Bowles: need a Facebook Fan page! Twitter account; LinkedIn; YouTube, Flickr, Yelp, etc. Track and respond.

Elizabeth Bowles: Optimized press releases (select key words to get coverage) is to get on-line coverage, not local press.

Martha Huizenga of DC Access a WISP in small DC neighborhoods: pick targets; then: postcards, signs <cars, yards>, local newspapers.

Martha Huizenga postcards are the only printed media; leaves them at dry cleaners & other local stores. Rest are PDFs by email/online.

Elizabeth Bowles responding to a Q: Market Wire costs $400-$700 per release; so search optimization your release first!

Elizabeth Bowles answering Q: need 3-4 touches to get a response. Link sign, postcard, website landing page to drive touches.

Michael Calabrese of the New America Foundation is luncheon keynote - talk is focused on open spectrum and especially TVWS issues.

Michael Calabrese: big problem - Genachowski's focus on auctioning large parts of the TVWS.

Calabrese: FCC, Congress & WhiteHouse intent on auctioning everything, including 20 TV channels (after paying off TV broadcasters).

Calabrese: If 20 TV channels went, after repacking broadcasters, there wouldn't be enough TVWS for an equipment market to emerge.

Calabrese: Spectrum sharing still sailing under DC radar; he's pushing that TVWS database extend to additional bands (e.g. military?).

Calabrese: we should use the same database to allow temporary access to ""warehoused spectrum"" i.e. until spectrum holder uses it.

Calabrese: Why shouldn't WISPs have temporary access to any spectrum band that is not yet in use in their geography?

Calabrese: FCC is not nearly as aware of the WISP community as it should be. Pls visit them; write to congress; file ltr to NOI.

Network Management: Matt Larsen, Vistabeam with Butch Evans; Brian Vargyas, Baltic; Alex Phillips, HSLink; Cameron Crum, Wispmon.

Larsen favors open source. Issues: integrate related items, respect business (may have to fire specific customers), leverage tech.

Larsen on processes: Billing, Provisioning, Dispatch, NW Mgmt (core & customer), Support, CRM, CustSat, Mktg, Employee accountability.

Larsen: Vistabeam has 1700 miles of wireless backbone over 600 miles. Documentation is critical before someone goes out in the field.

Larsen: Billing Freeside, Powercode, Platypus, ... He uses Freeside.

Larsen: NMS - What's Up Gold, Nagios, Xymon; proactively call people whose service is down. Also RT for trouble tickets.

Larsen on phone system - uses Asterisk but also familiar with Tribox and others. Msg when there's an outage cuts support calls!

Larsen on documentation - uses Network View, Visio, Dude and a wiki to capture other things (e.g. details of every tower).

Larsen reevaluate your performance metrics regularly. Ditto for biz processes.

Brian Vargyas on NM Mgmt: Be proactive, not reactive; BW usage; link history - Dude, Cacti, Nagios. New Dude 4.0 (beta) on SQL DB.

Vargyas - Dude works with any SNMP enabled device, not just MikroTik devices. Any MT board can be an agent to offload central box.

Vargyas - Cacti is open src PHP-based graphing system with web GUI but no notifications. Looks at anything with SNMP OID.

Cameron Crum of Wispmon - started a WISP, needed an IT infrastructure and a way to monitor his network. Visual kind of guy.

Crum is focused on demo'ing the Wispmon product.

Butch Evans speaking on QoS and prioritizing traffic to improve customers' performance - all on MikroTik. Others use Imagestream.

Butch Evans QoS script runs at an aggregation point. QoS won't fix massive overload problems; will help schedule during congestion.

Matt Larsen uses remote power control by Digital Loggers - auto power cycle if no ping on time.

Butch Evans QoS script for MikroTik is $175. Uses IP tables; categorizes most traffic and assigns it priorities; updated as needed.

Butch Evans was trying to emulate NetEqualizer when he started on his MikroTik QoS scripts.

Off to a Tower Technology session but there is no wireless in that room, so no Tweets.

WISPA Tower tech panel - Walked after they had started so I missed the introductions - not sure who is who…

Seems everyone is using 11b or 11g as they regard selecting polarization as critical.

Jack Unger (and at least one panelist) believes narrower channels (10 MHz or 5 MHz) cause the signal to go farther.   Statement that narrower channels have less background noise to contend with, thus the signal goes farther.  Given OFDM, this doesn't make sense to me.  Need to investigate what’s really happening.

One panelist argues for regulated power as the power supplies in the MikroTik's are cheap.

One approach to power and UPS is to use 24v PoE with a central 24v DC supply and two 12v deep cycle batteries.

Everyone advocates UPS.  One guy has 8 hours for every site.

At least one guy tells his customers to buy APCs for the client site.  Tornado zone, take laptop to basement and track tornado in real time.

Weatherproofing for coaxial connections:

1. Courtesy wrap (white tape) protects connector from butyl layer
2. Butyl layer (a mastic like putty)
3. 3M Super 33+ electrical tape (3 layers, shingled down on last layer) to extend at least an inch past the butyl layer.

The Andrew website has good tutorial on weatherproofing antenna connectors.

Use DC4 dielectric grease to protect Ethernet connectors.  Note this is not clear silicon! (even though it looks similar).

Inspect everything at least once per year.

Just posted some pictures from the conference here: http://su.pr/2XQuqp  More to come (will be added to this set!).

Conf has 253 registrations and 40 vendors. Mtg will breakeven or be slightly profitable. 43 new members joined in past one(?) month.

Board Mtg.: I should look into the Ambassador's program... They're looking for members to represent WISPA at regional events.

CALEA update - WISPA spec needs 3 updates to meet new rules; complex negotiation btwn FBI & WISPA. Next issue up is IPv6.

FCC Committee efforts: TVWS (not auctions); TDWR Database (stay 30 MHz away) - problems in Boston & PR); 3650 MHz (4 rule changes).

FCC Comm spent $5K in June, slightly lower than average; but advocacy and meetings are the bulk of what WISPA spends $ on.

Discussion of FCC-specific, or issue-specific, fund raising + volunteers to help engr'g (versus raising dues). Also partnering!

Steve Coran pitch at Marlon's request. (Rini-Coran is WISPA's law firm for DC politics).

Several members suggest that FCC Comm provide boilerplate that members can use to send letters to their representatives.

An aside in my twitter stream, not directly related to the WISPA meeting:
Where do telecom lobbyists come from?  There's a fantastic graph about half way down this page: http://su.pr/1tjZnZ    Hint: they used to work for congress!

Peter Stanforth, CTO, Spectrum Bridges up next on TVWS Trials. First focus on 2ndary mkts - time of day, location, freq. >10yr effort.

Stanforth has nice graphics on what TV white spaces are and how they arise. Hope to get his slides, eventually.

TVWS database administrator. Stanforth suggests we can have competition even here, and still accomplish desired results

Stanforth Summary of trials' field results in 04-186 FCC proceedings.  http://su.pr/2F9dOx

Stanfort: All TVWS trials have worked; getting 3x-5x coverage vs. 2.4 GHz systems; no interference complaints!

Stanforth: sensing unnecessary; drop antenna height limits; use fixed power limits & masks. Make DB higher fidelity than orig req'd.

Stanforth running TVWS BB antenna next to TV antenna in trials. Don't need separation! Must relax power limits and masks!

Stanforth notes that many TV stations are spraying out-of-band energy (a license violation!) but they haven't cared (so far).

Stanforth: if there are multiple database admins, they will coordinate btwn each other once each night, at least as proposed so far.

Rob Kubik of Motorola talking about FAA radars (TDWR) and DFS as basis for renewed outdoor use of 255 MHz of 5 GHz spectrum.

Kubik: experiments show 30 MHz away is ok, even in close (<35 km) to radar; combined w/DB approach should solve problem, but FCC..??

Jack Unger showing TDWR database which WISPA & Spectrum Bridge did jointly - going on-line next week. Meanwhile  http://su.pr/1Xk1SC

Stanforth showing prototype of TDWR database that will be up next week. Currently at http://su.pr/2iHAdO although that will change.

Panel on Buying and Selling WISPs. Prices seem to be from 1x to 2x revenues but many factors. Based on buyer estimate of cash flow expected after the transaction.

Across the board, panel is really cautious about any deal that involves stock or deferred cash. :)

Shiraz Moosajee's advice to buyers: plan on 20% reserve to fix problems and be sure you have 20% beyond that just in case.

Buy/Sell panel: 90-180 day clawback or true up clauses are common, with 20-25% in escrow.

Buy/Sell panel: Even if you don't plan to sell anytime soon, while growing your biz, keep revenue history, per tower.

Buy/Sell panel: Even though everyone prefers cash, very few deals are all cash (and they were fire sales).

Buy/Sell panel: Re buying small neighboring WISPS (100 subs) - frequently done with seller financing over 2-3 years. Estimate growth.

Buy/Sell panel: The smaller the network the less info you are liable to get!

Buy/Sell panel: Need transferability in all your lease agreements as buyers want assets (sellers prefer stock sale for Cap gains).

Buy/Sell panel: Sellers frequently have personal guarantees associated with corporate accts - got to know how to unravel these.

Julius Knapp  http://su.pr/9BXRJs  FCC, lunch keynote: Update on spectrum activities, but starts by hyping BB Plan.

Knapp on spectrum: ack's spectrum that's not yet built, but still pushing FCC/mobile operator's line: need more (licensed) spectrum.

Knapp promoting: spectrum dashboard; incentive auctions; more spectrum for mobile BB & for Backhaul - but this w.b. licensed. Ugh!

Knapp - only items that resonate with me: more unlicensed (Rec 5.11) and opportunistic use of spectrum NPRM (Rec 5.15) - Q3-2010.

Knapp & Ruth Milkman co-chair FCC Spectrum Task Force. Good, but he doesn't mention that FCC only controls part of the spectrum.

Knapp - Pres memo is forcing FCC & NTIA to work jointly on 500 MHz of new spectrum & Pres used words "licensed and unlicensed.

Knapp on TVWS 17 petitions for reconsideration. Expects multiple database mgs to be approved as part of this action.

Knapp acknowledges that 3650 MHz lite licensing has been a significant success, even as there is work afoot to improve things.

Knapp - TDWR protection w.b.: avoid freqs & 30 MHz guardband within 35 km. Make sure this works, so further licensing isn't required.

Knapp: the proceeding on backhaul scheduled for August 5th commissioners' meeting.

Knapp Q&A on Presidential memo suggests that ""licensed & unlicensed"" means some mix. How that's divided is to be argued...

Knapp says Aug 5th mtg will result in increased flexibility in use of spectrum for backhaul. But he wouldn't give details.

Knapp defends existing build-out reqmts. Charles Wu asks about Pt 101 licensing. A: spectrum transparency point is to find squatters.

Ray Savich of Moto describing software tools they have to help WISPs plan their businesses. These are biz tools, not just RF stuff.

Savich on ""Leased line payback calculator"" another tool to help sell WISP services.

Jay Lawrence of Gigabeam showing wireless carrier in Las Vegas who's built a wireless carrier Ethernet network using >50 1Gbps links.

xG Technology is here. Have dropped their original magical claims http://su.pr/1fBB9d for some completely conventional RF technology.

xG now has a proprietary mobile voice solution that runs on unlicensed spectrum. Not interoperable, but more a community cordless solution.

Cameron Crum of Wispmon demo'g their WISP-in-a-box IT solution including qualifying prospects against towers & terrain map. Very cute!

Crum emphasizes taking & preserving pictures of every install indoor incldg where power is connected (tell cust. how to power cycle).

Robert Olive of Wispmon is now pushing their new iPhone apps that work with Wispmon.

Nathan Stooke of WisperISP started in 2003; took a yr to breakeven; 3 yrs to make a salary; today, $2.4M revenue & 20 employees.

Stooke focus on best network and best customer service; has since taken over 15 competitors (many evolved from dial-up days).

Stooke: competitors were dial-up ISPs who entered wireless reluctantly. Diff was he wanted to build a wireless NW and he focused on svc.

Stooke: struggle to find good people; hirer slowly, carefully; some start as contractors, become employees later. He doesn’t provide benefits...

Stooke: employee training espc customer svc - one full day a month devoted to on-going training.

Stooke: run it as if you were setting up to sell it, i.e. keep it clean, fix problems, don't let them linger. Make it a showcase.

Stooke: Has a fire truck with 100' extension ladder that he got for $1K (but spent $26K getting fixed & certified) = temp tower.

Jim Murphy of NetSapiens describing their VoIP solution for WISPs - scales up, also down to system for just 10 simultaneous calls.

Murphy positioning against Asterisk (lower opex, expertise) and hosting (facilities based, more control; potentially better QoS).

Murphy also offering local virtual switch where realtime elements are local but NetSapiens handles everything else.

Now I have to leave to catch a plane, so I'll miss the final prize drawings. :( But this has definitely been worth it!! 

Thanks WISPA!!

Stealth no more! Also, I need your vote

A few of you have commented on my various profiles that say I'm associated with Ashtonbrooke Corporation or with BigBroadband.Net.  Of course neither website provides much information...

Well our service launch is still weeks away and we haven't found a name we like yet, but we have entered the MassChallenge and that means we've revealed what we're up to. MassChallenge is running a competition for startups and they're giving out grants.  Since we are entirely self-funded so far, a grant would be sweet.

To see our entry you must register at the MassChallenge website, http://www.masschallenge.org/user/register?role=Supporter and then go to Competition, Team Pitches, search for "BigBroadband.Net" and click Details.

Once you are there, you can not only get an idea of what we're up to, but you can vote for us!  Ideally before midnight tonight.  :)

Please Vote for BigBroadband.Net

As a registered supporter of MassChallenge you can rank any team's entry.  Just click on the right most of the five stars under "Ranking" on the right of our details page to give us a 5 Star/Awesome rating, ideally by midnight tonight.

Thank you.

Longer term..., we're seeking a new name

Once we roll out our pilot network (in Boston later this summer), I'll have a lot more to say about how we expect to pull a wireless end run around the duopoly, at least in dense urban areas in the US.

Meanwhile, we are seeking a better name. WirelessEndRun (our internal name) is either brazen or obtuse, or both.  BigBroadband.Net says something about what we're doing but has a lot of negatives.

All suggestions gratefully welcomed.

Overestimating mobile data growth

Cisco has recently updated their forecast of mobile data growth.  The good news is their forecast compound annual growth rate (CAGR) is down, from last year's forecast of 131% to their current forecast of 108% growth per year.

 
 

Unfortunately, when you look over the past 15 years of Internet traffic growth, the idea that any specific sector could exceed 100% growth for more than 1-2 years just isn't credible. 

In the early days of the Internet (prior to 1994) traffic did approximately double each year. And, as Andrew Odlyzko points out in his classic study, there was a period of perhaps 18 months in 1995-1996 when US traffic grew much more rapidly, however growth reverted to 100% per year in 1997.  And since then, Internet traffic growth has been slowing.

The most comprehensive data I am aware of is from Minnesota Internet Traffic Studies (MINTS) which shows the rate of Internet backbone traffic growth slowing so that in recent years (2008-2009) it has been in the range 40%-50%.  For a good discussion of long term trends see Internet Growth Trends & Moore's Law.

Mobile network traffic growth

It's interesting to look back on the growth of mobile networks so far.  Here are some numbers on cell sites, subscribers and voice minutes of use, extracted from CTIA data by Andrew Odlyzko.

                             1991              2006

subscribers            6.38 M          219.7 M

cell sites                6,685           197,576

MOUs                    5.2 B           858 B
 

Over 15 years, voice traffic grew 165x which is 41% CAGR.  

As an aside, it's interesting is to separate the growth in capacity due to more cell sites from the growth due to better wireless technology.  Over the 15 year period 1991-2006, we got a 30x growth in the number of cell sites, or 25% CAGR.  That suggests improvements in wireless technology (the transition from 1G AMPS to 2G and 3G systems) provided for only 5.58x of the increased traffic, or 12% CAGR.  This surprised me.  It can't be due to uncounted data traffic as data traffic was a small percentage of the mix in 2006. If true, it's rather embarrassing for the 2G and 3G equipment providers.  Any suggestions?

There's a different view in this study of available mobile data download speeds done by Novarum Inc.

This shows available download data rates doubling in a little less than 24 months, or ~45% CAGR. However, this is measuring available bandwidth, not actual traffic.  None-the-less, it's suggestive of capacity as these measurements were made on live networks during business hours.

A More Reasonable Projection

Given both Internet and wireless history, I suggest mobile Internet traffic growth over the next 5 years is more likely to be in the range 50% to 100% CAGR.  There may be a 12-18 month period when it grows more than 100% per year, but only for 12-18 months and only if some alternate technology enters the mix (for example, widespread use of Wi-Fi hotspots as I suggested a few weeks ago).

Broadband capacity - sizing buffers to handle traffic bursts

This is part 4 in a sequence of posts examining how broadband services actually work.  Part 1 looked at ISP concentration ratios.  Part 2 examined the effect of  averaging traffic from many subscribers.  Part 3 considered the impact of congestion on user experience. Here we'll take a finer grained look at traffic in the access network.

How do we achieve "zero" packet loss? Most websites are "hosted" and connect to the backbone by Fast Ethernet or Gigabit Ethernet links. Individual websites may be mismanaged, but there is no excuse for packet loss here.

Internet backbone links and backbone-to-backbone hand-offs are extremely high capacity multiplexing the traffic of thousands to millions of users.  Traffic statistics are predictable and operators compete on the basis of service levels, so packet loss is typically zero.

If there are performance problems, they're usually in the access network. Of course, one bottleneck is the service I've signed up for, for example a DSL link at 6 Mbps /1 Mbps. By definition my traffic is shaped to the service level I've signed up for.  This shaping typically happens at the first aggregation point (CMTS, DSLAM or WISP P2MP radio) or, on an IP or MAC address basis, at a bandwidth management device further upstream.  If they arise, problems are in the aggregation points and/or links between me and the backbone.

Since packet loss significantly degrades user experience, we'd like to understand how much over-provisioning a broadband service provider needs in order to absolutely minimize packet loss in the access network.  To understand this we need to look at smaller levels of aggregation (dozens and hundreds of users) and shorter intervals (sub-second rather than 5 minute averages, as few routers buffer more than a few tens or hundreds of milliseconds of traffic).

But first lets look at some five minute averages from a WISP with ~300 customers, each with either 1 Mbps or 3 Mbps service.

If one took these graphs at face value, it seems 20 Mbps of backhaul would be more than adequate, as the traffic peaks at 17 Mbps.  But 5 minute averages don't speak to what's really happening.

I don't have sub-second averages for the specific WISP above, but Fergal Toomey, Chief Scientist at Corvil Ltd., has some very detailed measurements on a similar Internet access network providing services to users at 512 Kbps, 1 Mbps and 2 Mbps. These are available in a whitepaper you can request from Corvil's website. Fergal's measurements were made at the Internet POP looking at traffic headed towards individual users.  Also, his measurements were made in 2004 when most hosted servers had 100 Mbps connections to the backbone and the majority of computers had TCP stacks configured for a maximum TCP receive window of 32KB.  In this graph, green shows 5 minutes averages, while light blue shows 500 ms averages. Dark blue shows 5 ms averages and they are striking!

What's Happening?

Web browsers typically use persistent TCP connections to minimize connection overhead and to avoid repeating TCP's slow start.  As a result, servers will frequently respond to a new request with a full window's worth of data.  In 2004 that was 32 KBs of data or ~23 packets of up to 1500 bytes each. Today it could be much more. Absorbing one such burst should be no problem, but to what extent are such bursts coincident?

Beyond web browsing, detailed traffic analysis uncovers a variety of odd behavior and some less savory behavior. For example, one burst was a blast of many 41-byte packets sent by a web application that apparently does a series of 1-byte writes to a TCP socket with the NO_DELAY flag set.  Several thousand packets arrive in quick succession causing packet loss.  Worse yet, follow-on losses occur as retransmissions cause a decaying series of echo bursts.

On a more malicious note, the data includes bursts caused by port scans.  These consist of many, many minimum size probe packets that attempt to scan all of a subscriber's ports in one enormous burst.  Luckily these lost packets are not retransmitted.

Beyond these, Fergal Toomey's analysis uncovered a variety of application anomalies that I found unexpected, but which crop up in the real world.

What headroom is really required?

Clearly we could just increase the size of the buffers in the router and smooth out any of these bursts, but that would also increase the delay while the burst lasts and thus introduce significant jitter.  A better question is how much headroom is required to guarantee zero packet loss and no more than 20 ms of incremental delay?

Conveniently, Fergal Toomey has run that calculation on the data set above and the result looks like this:

In this graph, the average traffic (the "mean" shown in light green) peaks just shy of 5 Mbps.  To guarantee zero packet loss and no more than 20 ms incremental delay 100 % of the time requires 19 Mbps of capacity or nearly 4x.  However, to guarantee zero packet loss and no more than 20 ms of delay 99% of the time, requires only 8 Mbps, i.e. a little less than 2x.

Interestingly, while ISP practices tend to be trade secrets, the rule of thumb that 2x headroom provides good service crops up repeatedly.

Conclusion

As a practical matter, real Internet traffic is fairly complex and includes a wide variety of anomalous behaviors. Rules of thumb based on averaging the traffic of a few hundred or even a few thousand subscribers are likely to be very expensive (in terms of excess capacity).  A better approach is to continuously monitor packet loss and delay variation (jitter), adding capacity as needed to keep those measures below desired levels.  

Zero packet loss and less than 20 ms incremental jitter, more than 99 percent of the time sounds plausible objective to me.  

Now if only my broadband service provider was willing to share any information about their service level objectives or performance.  Ooops, I must be dreaming.

Broadband capacity - the impact of congestion

This is part 3 in a sequence of posts examining how broadband services actually work.  Part 1 looked at ISP concentration ratios and part 2 examined the impact of averaging many subscribers. Future posts will consider how to size backhaul links and how to configure buffers.

Two major reasons people purchase broadband (or sign up for more capacity) are to improve interactivity and to get access to new applications, for example streaming movies.  New apps is clear, but it's worth looking congestion and how congestion affects interactivity.

Congestion has two impacts. First you get increased delays, as packets are buffered in queues waiting for free capacity on the next link. Usually this adds tens or hundreds of milliseconds to round trip times. Then, if the average traffic exceeds link capacity, you get packet loss. If you do have congestion, it's these packet losses that cause TCP traffic flows to back off, thus avoiding congestion collapse (and even less throughput).

Here is a graph of packet loss on a service that most of us would consider unacceptable.  These measurements were taken by Tom Dunigan on an early cable modem service (February 2001) that was substantially upgraded about a year later.

 

The next graph (also from Tom Donigan) shows how this congestion impacts interactivity.  The graph shows the echo delay for typing into a remote program using telnet in character-at-a-time mode.  One test (red line) was run in the early morning and one test (green line) was run during the evening when packet loss was peaking.  Each test measured the echo response time for 100 successive individual characters. Notice the typical echo delay goes up from ~130 ms in the morning to roughly 200-300 ms in the evening, but with occasional delays of more than one second.

Relatively few programs operate character-by-character, so the extra 100 ms or so might not matter.  On the other hand, a once second delay is noticeable!

A more common activity is viewing content on websites.  Here, the dominant time in any interaction is the time spent waiting for web content to download to your browser.  Since web content is downloaded using TCP (and more generally, TCP is the dominant protocol in use today), it's worth looking at the impact of packet loss on TCP throughput.  

The TCP protocol includes a congestion avoidance algorithm which is triggered by packet loss. When a packet is lost, the TCP sender slows down. As a result, the data rate for a single TCP flow looks like this (thanks to Guido Appenzeller):

Of course this assumes the TCP flow lasts long enough to saturate the bottleneck link and that this is the only flow in the network!  

What happens in real networks is complex, but RFC 3155 has an approximate formula for how a single flow is affected by packet loss in real network.  Even better, Bill Gibson at Niwot Networks has produced a nice graphic based on that formula for his paper on TCP limitations on file transfer performance.  It looks like this:

 

This is dramatic!  The different lines reflect different end-to-end round trip times (RTT) - times that will vary depending on the site you are connecting with.  Also, they represent the maximum throughput you could achieve with long duration TCP flows and no other bottlenecks.  What's notable is the logarithmic throughput scale on the left and the fact that, at 1% packet loss (0.01 on the bottom scale), potential throughput drops by a factor of 100!

Again, there are many caveats.  A real network has a mix of short- and long-lived flows. The local operating system may not be optimized to take full advantage of broadband speeds (although MS Windows actually got better at this with Vista). None-the-less, even 1% or 2% packet loss is correlated with poor user experience.

In short, to actually obtain the instantaneous throughput you thought you were purchasing, you don't want packet loss in upstream portions of the network.

I'll discuss what this actually implies for routers and backhaul links in a subsequent post.

Broadband capacity - the impact of averaging many subscribers

The fundamental idea of packet networks like the Internet is to transport variable rate data flows over a shared infrastructure.  Individual end points may generate little or no traffic for minutes or hours on end and then suddenly want to send or receive a large data set as quickly as possible.  Packet networks efficiently share common transport facilities among multiple users but how does this sharing actually work?  Yesterday I posted some actual, relatively current data from three small ISPs.  Today, let me show you how averaging traffic from many individual end points produces a completely stable core network flow.

Each of the graphs below shows successive five minutes measurements during a 24 hour period, but the first graph reflects a few hundred subscribers while the final graph represents the average of many, many millions of subscribers.

In the first graph, the blue line reflects outbound traffic from a few hundred subscribers.  While the average is just over 1 Mbps, the peak (at ~1615 hrs) is 6.9 Mbps for a peak-to-average of nearly 7-to-1.  The green bars represent inbound traffic with an average of 5.6 Mbps and a peak of nearly 16.8 Mbps for a peak-to-average of 3-to-1.  Some of this is time-of-day dependency but a lot of it, particularly the 6.91 Mbps spike at 1615 hrs, is the result of averaging only a few hundred subscribers.

 

Here's a traffic measurement at AMS-IX, the largest Internet Exchange in the world.  With an average of nearly 1 Gbps, the peak-to-average is a modest 1.43-to-1 and it's almost entirely due to time of day.  The largest change from one 5 minute measurement to the next appears to be about 140 Mbps so the short term peak-to-average is perhaps 1.1-to-1.

 

Finally, here a measurement of all the traffic flowing through AMS-IX – over 822 Gbps peak.  Here the time-of-day variation remains but there are virtually no statistical fluctuations.  The short term peak-to-average is almost 1-to-1.

 

To put this in economic terms, for an Internet backbone link that runs at many Gbps your daily traffic profile is completely stable and you can guarantee zero packet loss merely by providing 10%-20% extra capacity above your daily peak.  

But if you are running a small ISP, both the capacity you need per subscriber and the extra "headroom" for unanticipated peaks must be substantially larger.  To get a handle on what is required we also need to look at shorter intervals (shorter than five minutes).  But more on that in a subsequent post.

ISP Contention Ratios and average traffic per user – some data points

Recently there’s been a flurry of discussions about bandwidth shortages, not to mention bandwidth hogs, bandwidth caps, network neutrality and what constitutes fair practices by ISPs. But there is very little public data and no transparency! 

So let me offer a (very) few actual data points, one from a few years ago and three fairly current. These come from friends who are operators and from emails shared on trade association mail lists. Since some of it was not intended for publication, I have withheld specific names. 

Notice contention ratios vary widely and yet each of these operators has enough extra capacity that they claim to avoid link saturation and its impacts, like packet loss and excessive ping times. It’s true these are all small operators (280-1200 customers) so this data lacks the statistical smoothness one would expect when aggregating traffic from thousands or millions of subscribers. 

I also have some data on traffic averages by time and by number of subscribers that I’ll share in a subsequent blog post. 

Mar 2006 – Rural ISP (Dialup, DSL & Wireless)
130 Motorola Canopy subscribers (with 512 Kbps service)
150 DSL subscribers (110 with 512 Kbps service; 40 at 1 Mbps)
100 dialup lines serving 800 customers (average 40 Kbps)
Daily traffic was saturating 4.5 Mbps (3 T1s) for an hour per day but not with 6 Mbps.
Contention ratio: 164Mbps / 6 Mbps -> 27:1
Traffic per BB user: 6 Mbps / 280 subscribers -> 21 Kbps 

Sep 2009 – Rural ISP (DSL & Wireless)
1200 customers, DSL subs at 1.5, 3 or 6 Mbps; wireless subs at 384 Kbps, 768 Kbps, 1 Mbps or 3 Mbps. I don’t have the mix, so I'm guessing the contention ratio.
Daily traffic peaks at around 35-40 Mbps; up to 150 Mbps Internet transit available.
Contention ratio: 2400 Mbps(?) / 150 Mbps -> 16:1
Traffic per BB user: 40 Mbps / 1200 subscribers -> 33 Kbps 

Sep 2009 – Community-run WISP
340 customers, mostly on 3-4 Mbps service
Daily traffic peaks around 17 Mbps; 45 Mbps of Internet transit available.
Contention ratio: 1190 Mbps / 45 Mbps -> 26:1
Traffic per BB user: 17 Mbps / 340 members -> 50 Kbps 

Sep 2009 – Rural ISP
370 customers: 300 Customers on wireless (1-3 Mbps) and 70 customers with FTTH (10/100 & up, but most have 10 Mbps peak). Monthly fees based on a 10 or 20 GB cap with extra charges for heavier use.
Daily traffic peaks at 13-14 Mbps; 20 Mbps of Internet transit available.
Contention ratio: 1200 Mbps (?) / 20 Mbps -> 60:1
Traffic per BB user: 14 Mbps / 370 subscribers -> 37 Kbps

Has AT&T Wireless data congestion been self-inflicted?

We've all heard or read stories about how iPhone usage has overloaded the AT&T Wireless network but it's likely at least some of their problems are the result of configuration errors ― specifically, congestion collapse induced by misconfigured buffers in their mobile core network.

In early September, David Reed sent this interesting message to the IRTF's "end-to-end" email list.  List members include some world experts on Internet protocols.  During the next couple of days, there were over 40 messages in related threads.  While some of these experts were over-thinking the problem, if you are patient enough to read through the many messages, what emerges is clear.  At least in the case David measured (from a hotel room in Chicago, while he had 5 bars of signal strength, using an AT&T Mercury 3G data modem in his laptop), the terrible throughput and extreme delays he experienced appear to result from overly large buffers in the routers &/or switches in AT&T's core network.  Note:  if you don't want to read all the list messages the short summary is:  >8 second pings times!  What's more the effect was bymodal:  either ping times under 200 ms, or over 5 seconds.

Recently I was talking with a friend whose company continuously operates (and monitors) multiple 3G data links on the Verizon Wireless, AT&T Wireless and Sprint  PCS networks.  His data shows periods when the round trip time for http requests goes over 8 seconds, on the AT&T Wireless network only!  I don't have a copy of his data that I can examine in detail, but when combined with David Reeds report, it certainly appears AT&T Wireless has configuration problems.  If you read on you'll see this may not be the result of gross stupidity, but someone at AT&T Wireless should be a little embarrassed.

My (techie outsider's) Analysis of What's Happening

Buffers in Packet Networks

Routers (& switches) in a packet network have to include buffering in order to absorb transient traffic bursts.  Unfortunately, despite decades of research and operational experience, there is no simple formula for how much buffering is optimal at any given location in a network.  If you're interested in more detail, Ravi Prasad has a good review of the literature on pages 10 & 11 of his (April 2008) PhD thesis.  But decades of operational experience have yielded some basic precepts and it appears AT&T Wireless is violating at least one basic precept.

The buffer in front of a congested link must induce some packet loss.  TCP (the dominant Internet protocol) continuously increases it's transmit rate until it experiences packet loss, then it cuts its rate in half and enters a congestion avoidance mode.  If the network becomes full but there is no packet loss, each TCP sender will keep increasing it's rate causing the network to suffer a congestion collapse.

In the case of a mobile network, the limited resource is over-the-air capacity.  Backhaul may also be expensive, but it's relatively easy to over provision anything else in the operator's network.  So the issue is, how big should the buffer be in the last router between the high capacity core network and the actual over-the-air data path to a subscriber?  Ideally we'd like enough buffer to absorb momentary packet bursts that, averaged out, don't exceed the available over-the-air capacity.  But as soon as the offered traffic exceeds the available over-the-air capacity, we want some packet loss.  The complicating factor is the way 3G wireless networks schedule over-the-air traffic.

Jitter in 3G radio networks

One cellular base station serves multiple users and the quality of the connection to any specific user depends upon instantaneous wireless propagation characteristics.  These can vary second by second and even millisecond by millisecond when a user is moving.  To deal with over-the-air losses, the basestation (the "Node-B" in a 3G network) keeps copies of each packet until a positive acknowledgement (ACK) is received, retransmitting the packet if the ACK is not received in time. Of course retransmissions introduce delay and jitter.  Furthermore, at any given instant, some users wireless links are better than others.  In order to maximize the total traffic in a cell, the 3G MAC layer schedules transmissions to individual users based on who has the best instantaneous throughput.  This is an efficient solution but it also introduces different amounts of jitter into each user's data path.  Luckily these effects are well understood and not that severe.  With HSDPA, the basic transmission time interval is 2ms so total delay variation is relatively small.  This graph from Jang et al (1) is typical of measured values in an HSDPA network.

Most jitter is below 15 ms.  Measurements of ping latency between 3G wireless devices and the first IP server at the edge of the mobile core network (typically the GGSN) can extend out to over a second as this graph from Mun Choon Chan and R. Ramjee (2) shows:

but most of the time, total IP latency is a few hundred milliseconds. As mentioned earlier, David Reed reported bimodal operation on the AT&T Wireless network with normal behavior yielding ping times under 200 ms.

Likely cause of AT&T's problems

So what is happening in the AT&T Wireless network when ping times go over 8 seconds?  We know how a customer's IP packets are passed through an operator's mobile core network.  They are tunneled all the way from the handset to the Gateway GPRS Signaling Node (GGSN), i.e. to the router where the mobile core network connects to other networks.  The protocol stacks for this tunneling look like this:

More recent versions replace the ATM and AAL5 with Ethernet and IP, but the user never sees this as user IP data is tunneled across the top of the diagram (carried by PDCP and GTP-U).  As a result, user traceroutes can't reveal the detail of what's happening in the core network.  The first thing the user can see is the GGSN (the gateway to the next network).  So we can't make conclusive measurements from outside the network, but we do know a few more things.

The bottleneck link is the over-the-air link, i.e. the connection from radio access network or UTRAN to the Mobile Statation (MS) in the above diagram, therefore the critical buffers are those at the UTRAN.  In practice the UTRAN includes both the basestations (called Node-Bs) and the Radio Network Controllers (RNCs) which coordinate handovers between basestations (among other things).  Because of hand-overs, the amount of data buffered at the Node-B is relatively small. It's the buffers at the RNC that must be large enough to deal with the delay variations in the radio network and yet small enough to induce packet loss when the network gets congested.

While I don't personally have experience managing a 3G HSDPA network, my impression is UTRAN buffers are normally less than 200 ms.  Recently Yerima and Al-Begain presented an interesting paper (3) on buffer management in 3.5G wireless networks in which they concluded that 120 ms buffers were ideal for downlink traffic in a specific UMTS-HSDPA configuration.

Zero Packet Loss

It appears AT&T Wireless has configured their RNC buffers so there is no packet loss, i.e. with buffers capable of holding more than ten seconds of data.  Zero packet loss may sound impressive to a telephone guy, but it causes TCP congestion collapse and thus doesn't work for the mobile Internet!

=================================================

References

(1) 3G and 3.5G Wireless Network Performance Measured from Moving Cars and High-Speed Trains, by Keon Jang† (keonjang@an.kaist.ac.kr), Mongnam Han† (hgma11@gmail.com), Soohyun Cho∗ (shcho1@kt.com), Hyung-Keun Ryu∗ (hkryu@kt.com), Jaehwa Lee∗ (jhlee@koren21.net), Youngseok Lee‡ (lee@cnu.ac.kr), Sue Moon† (sbmoon@kaist.edu)

  † Computer Science Department, KAIST

  ‡ School of Computer Science & Engineering, Chungnam National University

  ∗ KT Central R&D Laboratory
http://an.kaist.ac.kr/~sbmoon/paper/intl-conf/2009-micnet-3g.pdf

(2) TCP/IP performance over 3G wireless links with rate and delay variation,

by Mun Choon Chan and R. Ramjee, Bell Labs  (presented by Jinqiang Yang)
http://wwwcsif.cs.ucdavis.edu/~ghosal/Teaching/ECS289I/Lectures/jinqiang-yang-wirelessTCP.pdf

(3) Dynamic Buffer Management for Multimedia Services in 3.5G Wireless Networks,

by Suleiman Y. Yerima and Khalid Al-Begain

Proceedings of the World Congress on Engineering 2009 Vol I

WCE 2009, July 1 - 3, 2009, London, U.K.
http://www.iaeng.org/publication/WCE2009/WCE2009_pp857-862.pdf

Structural Bypass - my talk from eComm last spring

At Spring eComm 2009 in California in March, I gave a presentation entitled:  Structural Bypass - A Simple, Proven Path to "Real Broadband" that dealt with policy approaches for first mile fiber deployments.  The video has just been posted here.  Unfortunately, this is my face and the audio with my words.  The slides are not shown in the video, instead you have to look at the slides here.

My arguments focused on urban areas only, as the economics are different between urban, suburban and rural deployments.  You can't just lump together all areas in a country and draw reasonable conclusions.  Also, the only data I have is for urban areas.  And, as a simplification for this talk (I only had 15 minutes including Q&A), my metric of success was the consumer price for 100 Mbps symmetric Internet connectivity.

My points were:

• Structural separation produces better results than other approaches.
• Separation between dark fiber and everything else is more effective than separations at higher layers.
• There is a political path, which I call Structural Bypass, which has been used at least once (in Stockholm), to effectively achieve structural separation without explicitly fighting that battle.

If you are interested, I suggest you open the video in one browser tab merely for the purpose of listening to the audio.  Then view the slides in another browser tab.  The audio is of some interest, but my facial expressions are of questionable value.

eComm agenda is on-line ―Check out the speakers!

There's a fascinating agenda building for eComm in Amsterdam in October.  Here's the latest list of talks, but what's best is the set of speakers who are giving these talks.  I personally know quite a few and know of many more (who I look forward to meeting).  The group includes a preponderance of innovators -- new views on what's happening, new ideas for how to drive change.  If you don't read through this list of talks, look at the speakers list here.  Then come to Amsterdam and meet these people.

21st Century Economics: Lessons for Telcos - Umair Haque (Havas Media Lab)

Advances In Spectrum Transparency, Software Defined Radio/Cognitive Radio - Darrin M Mylet (Spectru-Station)

Almost all Marketing & Product Management of Telco Services is Wrong - Rudolf van der Berg (Logica)

Current Trends in Community Wireless Networks and Beyond - Aaron Kaplan (FunkFeuer)

Death of the Handset: Evolving from Mobile Devices to the Mobile Digital Life - Mark Rolston (frog Design)

Edge As Value - Value As Edge - Graham Brierton (Voicesage)

Enslaving Humans using Communications Technology for Fun and Profit - Thomas McCarthy-Howe (Jaduka)

Entrepreneurial Advantages with New Open-Source Technologies - Jay Phillips (Voxeo)

European Telecoms 2015: Silent Death or Generative Bazaar? - Julien Salanave (IDATE Telecoms)

Finding Disruption - Michael Jackson (Mangrove Capital)

Goodbye Minutes, Hello Moments - Martin Geddes (BT)

Google Wave - David Wang (Google)

How the "Internet of Things" will Change the Way we Connect - Alexandra Deschamps-Sonsino (Tinker.it)

How to get More Value out of Customer Interactions by Blending Online with Voice - Stefan Hopmann (Swisscom)

Humans as a Service: Abstracting Communications to Reach New Applications - Todd Landry (NEC Sphere)

Is LTE being Held Hostage by Ordinary Voice Telephony? - Dean Bubley (Disruptive Analysis)

Lifestyle Segmentation from Carrier Location and Call Data - Greg Skibiski (Sense Networks)

Lifestyle Segmentation from Carrier Location and Call Data - Tony Jebara (Sense Networks)

LTE - Long Term Employment or Less Than Expected? - Moray Rumney (Agilent)

Open Access Makes Economic Sense - Benoît Felten (Yankee Group)

Open Screen Project: Next Generation Contextual Applications - Andrew Shorten (Adobe)

Opening of the Terahertz Region - Robert Horvitz (Open Spectrum Foundation)

Opportunities in Post-Telecom Connectivity - Bob Frankston (Frankston Innovating)

Post Financial Trauma - How is the Telecom Value Chain Now Positioned? - James Enck (mCapital)

Redefining Gifts in the Digital Space - Katie Lips (Little World Gifts)

Slowing Down Communication: Designs Inspired by Quality, Intimacy, and Humanity - Stefan Agamanolis (Distance Lab)

Spectrum 2.0 – What's Really Happening? - Brough Turner (Ashtonbrooke)

Stealth Approaches to Legislating Open Spectrum - Brough Turner (Ashtonbrooke)

Technology and Biological Evolution: What This Means for Media and Communications Technologies - Tomas Rawlings (University of the West of England)

Telemedia Futures - Gerd Leonhard (MediaFuturist.com)

The Emerging Telecology of Social Networks and the Status Update - Stuart Henshall (Phweet)

The Future of Interconnection - Rudolf van der Berg (Logica)

The Global Battle for Communications Justice: An Open Spectrum Manifesto - Sascha Meinrath (New America Foundation)

The Next Wave of Communications Applications - Cullen Jennings (Cisco)

The Practical Edge of Speech Technology - Moshe Yudkowsky (Disaggregate)

Ubiquitous Voice over Broadband - is There a Future Role for the Smart Pipe? - Martin Taylor (MetaSwitch)

Unlicensed Spectrum: Future Regulation - Prof. William Webb (Ofcom)

Video Killed the Telephone Czar? - Jan Linden (Global IP Solutions)

When Will HD Voice Become a Reality? - Martyn Davies (Dialogic)

What Would Telephony be Like if we Designed it Today? - Matt Ranney (RebelVox)

If you've read this far, there's a 10% discount via this registration link - register here.  And for more information, visit http://eComm.ec.  This one conference is at the top of my list for:  interesting, productive and an excellent set of contacts.

Pouzin Society - organizational meeting, today

The most interesting data networking ideas I've come across in the past N years are in the recent book by John Day, Patterns in Network Architecture: A Return to Fundamentals.  John Day's fundamental insight is that data networking is a distributed version of inter-process communications (IPC).  Those of us who come at this from the computer and computer operating system side of the world are quite familiar with IPC within one computer running one operating system.

The Pouzin Society (named after datagram inventor Louis Pouzin) is a group interested in investigating the implications of John Day's architecture and how it might be used in developing viable solutions to the current Internet architecture crisis.

Here's a top level picture (from a talk by John Day).  If you don't buy the book, there is an interesting summary in a presentation here.

Viewing data networking as a distributed IPC function (DIF) does a number of things.

• It clears up names versus addresses in a fashion that makes sense to me.  The application asks for connection to another named application while node addressing remains internal to the DIF.  Indeed, the point of attachment address for one DIF is a "name" presented to a lower level DIF.
• It makes management explicit.  To even establish an IPC, we must ask the DIF for the other party by name and then get their permission to exchange information (i.e. to establish the IPC session).
• It simplifies layers.  There is only one kind of layer, the DIF, that is used recursively.  Successive levels in the recursion differ in scope, and likely in policies, but architecturally they are the same.

I'm still getting up to speed, but this appears to be a fundamental theoretical breakthrough.  There are a mix of academics and practical networkers in the room for the Pouzin Society kickoff, so hopefully some practical implications for the evolution of the Internet will emerge in time.

Google's Peering and Caching Strategy

This is a followup on yesterday's post about how little Google/YouTube pays for bandwidth. Google wants to promote peering with ISPs, so they give presentations at ISP meetings.  After reading yesterday's post, Alex Benik of Battery Ventures sent me a link to this presentation given by Google at a 2008 meeting of the Latin America and Carribean Internet Addresses Registry (LACNIC).

As expected, Google peers with as many relevant ISPs as possible.  For the ISP, peering with Google eliminates their upstream costs for traffic to Google.  Since Google represents a substantial volume of traffic for most ISPs, this is a big saving.   As of May 2008, Google was present in 33 public Internet exchanges around the globe, so major ISPs already have connections in places where they can peer with Google.  The minimum qualifications are 5 Mbps of Google traffic and the ability to interconnect using Gigabit Ethernet at one of these 33 major Internet exchange points.

Google Global Cache

What's interesting is Google's caching strategy.  Just as Akamai puts servers in ISP's local facilities, Google is providing a distributed cache for their content.  This available to larger ISPs and allows them to serve Google content directly at the edge of their networks, thus reducing traffic on the ISP's backbone network.  Here's a representative rack that Google provides to the ISP.

YouTube's fine - Analysts don't understand Internet peering

As widely reported, Credit Suisse analysts have estimated Google's YouTube may lose $470M in 2009 and more in the future.  However, their estimates say Google will pay $360M for bandwidth in 2009.  I don't know how Google figures their cost of bandwidth, but anyone who understands anything about Internet transit/peering knows Credit is way off base.

Google does not pay for Internet Transit the way most tier 2/3 ISPs or most content providers must.  The economics are simple.  If you are a Tier 2 ISP, you have to purchase Internet Transit services from a Tier 1 network to handle that customer traffic which goes off your network and for which you cannot make other arrangements.  The most notable 'other arrangement' is peering.  If you have significant traffic to/from another specific network, you and the other network can both save Internet Transit costs by exchanging traffic locally, i.e. peering.  Of course an enormous amount of your traffic is directed to Google.  If you have a presence in any data center where Google has a presence, you would love to peer with Google, as that saves an enormous amount on your payments for upstream Internet Transit.

A similar effect plays out among Tier 1 providers.  If one tier 1 network cuts a special deal with Google, Google routes all their traffic through this provider and suddenly the other tier 1 networks have large asymmetries in their tier 1 peering arrangements.  Either they also cut deals with Google or they have to renegotiate their tier 1 peering arrangements to pay for the traffic asymmetry (something that's highly unlikely!).  Google is the one with leverage here!

I don't know what, if anything, Google pays for bandwidth, but it's not paying $360M for Internet transit.  Sorry Credit Suisse, you better go back to analyzing derivatives, credit swaps and other purely financial plays.

Google does have costs.  They have data centers in many parts of the world and they have a private fiber backbone that interconnects their sites and connects their private network to many, many potential peering points.  Operating their private backbone is a real cost to them and I haven't examined their financial reports to see if there is any way (from public data) of estimating their costs for this private network.

But until someone does this analysis, forget what you've read from Credit Suisse.

40th anniversary of RFC 1

April 7, 1969 was the day a graduate student at UCLA, Steve Crocker, published RFC 1 to the somewhat informal Network Working Group associated with the ARPA Network project.  I have heard Steve Crocker speaking informally about those days and my understanding is he created what became the "Request for Comments" series because anything more formal would have brought down several layers of governement bureaucracy.

It's interesting that the Internet owes its origins to a government project, but a lot of what happened was done by individuals and groups working around and/or in spite of government processes.

More bandwidth ― less delay, less latency

I was a little sloppy yesterday and several people have questioned my comment about latency.  I was reacting to slide by Herman Wagter of Amsterdam's Citynet in his presentation at F2C 2009.  His slide said:

Latency is the cause, bandwidth is the effect.

From his discussion it was clear he meant "Latency is the cause, bandwidth is the cure."  At the time he was talking about real-time person-to-person communications and illustrating it with a housebound person in Amsterdam who wanted to play cards with friends in other places (not nearby).  Verbally he mentioned the issue of sending large files.  In short he was addressing the real reason for high capacity Internet access links.

Why people want more "bandwidth"

It's not because they need or want to send and receive 100 Mbps of data all the time or even a significant part of the time.  The issue is delay, specifically serialization delay.  If I have a 1 Mbps upstream Internet connection and I want to send an email with a 5 MB Powerpoint file attached, it will take more than 40 seconds (5 MB ~= 40 Mbits).  On a 100 Mbps link, the same email is delayed only a fraction of a second.

Serialization delay also effects media streams, although much less.  If I want to send 500 Kbps of continuous video over that 1 Mbps uplink, serialization delay will cause added latency.  IP is a packet protocol and the 500 Kbps video stream will be broken into a stream of packets, typically ~1500 bytes (12 Kbits) each. While the serialization delay is only 12 ms on the first link, there is serialization delay on every link.  If there is another 1 mbps link at the other end, that's another 12 ms of delay.  And here, 12 ms is significant.  For a natural interaction between two people, you'd like to keep the round trip delay below 200 ms.  Nothing goes faster than the speed of light so transoceanic links introduce many 10s of ms of delay, each way.  It is very easy to eat up a 200 ms budget, so saving 12 ms at each end is significant.

Burst rate versus continuous

For me and for most people, the issue that drives demand for high speed access links is delay, not the amount of information that is to be sent or received.  Indeed, I'd love a service offered only a few Mbps average over a month, if I could be guaranteed 1 Gbps on a burst rate basis whenever I wanted.

David Weinberger is blogging F2C rather completely

And David is a expert blogger!  i.e., really good coverage.
  http://www.hyperorg.com/blogger/

The first Freedom to Connect conference entry is here. 

Latency is the cause; bandwidth is the solution

Actually, what Herman Wagter of Amsterdam's Citynet said at F2C 2009 was:  latency is the cause, bandwidth is the effect.  But his explanation matched my title above.

If you are attempting to interact with other people, whether by VoIP or just playing cards together (with video) you need less than 200 milliseconds of end-to-end delay.  If it's playing cards together, with video, and you need to exchange 500 Kbps in less than 200 ms, you need a 100 Mbps pipe!

It's latency that drives the need for high bandwidth.  Most people won't fill that pipe most of the time, but they need the pipe to guarentee that what they do send gets through rapidly.

Highlights from Day 1 of F2C

Shamelessly ranked by my areas of interest...

Tim Nulty of East Central Vermont Community Fiber.  Tim is a veteran network builder and a forceful speaker, so he's happy to tell it like is.  He's also got that Yankee mix of liberal politics with extreme fiscal conservatism.  He's building fiber networks, in rural Vermont, which pay for themselves.

Ken Biba of Novarum has been measuring actual wireless networks in buildings and in cities for years.  While the detail is in a report available for purchase, the summary is that WiFi-based Muni WiFi yields significantly better performance than 3G cellular.  Interestingly coverage and reliability is right up there in selected cities, as well.  The take-away - 802.11n really rocks. I.e., the next cycle of WiFi is going to be vastly better than what he's been measuring over the past 3 years.

Ellen Miller of Sunlight Foundation was low key by comparison with Tim or Ken, but her stories were compelling - multiple instances of Internet community feedback creating the kind of information that the "open government" initiatives aspire to.

Finally, Dewayne Hendricks is always interesting.  This year he seemed more optimistic than last, presumably the result of the recent election.  In any event, here's another speaker with deep experience in building networks.

Freedom to Connect 2009 and Municipal Fiber

I'm attending F2C 2009 in Silver Spring Maryland. If you are here, please say hello.

Things are just getting started with a panel on Municipal Networks, led by Joanne Hovis, President of Columbia Telecommunications Corporation (CTC).  Panelists are:

Tim Nulty, formerly of Burlington Telecom and now running East Central Vermont Community Fiber, has tons of experience building fiber networks in low density areas (Vermont).

Dirk van der Woude, from Amsterdam's CitiNet, to talk about the Amsterdam's municipal fiber to the home project..

Lev Gonick, CIO Case Western Reserve, who was a key player in creating a 4000 mile fiber network for Cleveland and northeast Ohio under a community organization called OneCommunity.

Bill Schrier, CTO for Seattle, which is starting a fiber project, but already has it's own electric power utility.  (Although Bill implies they have had to drag their utility brethren into this).

What's interesting is the discussion on the chat backchannel is not about muni vs. commercial, but wireless versus fiber.

Tim Nulty has a strong argument that wireless is excellent for mobility, but not economical for fixed access.  In rural Vermont, a WiMAX network would cost $35M if you could get access to the spectrum (which is being horded by others).  Fiber would cost $70M but has 50 times the capacity and several times the revenue potential versus the wireless approach.  Further, if you deploy wireless as an addon after you have the fiber network (and the customer support infrastructure), the incremental cost is dramitcally less (perhaps $10M) and you get enough incremental revenue to get a good return on investment.  IN other words, you make more money if you do fiber first.  Tim's key to success is to get as near complete deployment as possible - something that is possible in areas where the incumbents are going.  Second, he goes for community ventures as a way to qualify for muncipal bonds.

Shareholders should demand phone companies open their fiber networks

A few weeks ago, Benoît Felten of Yankee Group and Fiberevolution gave a speech at a New Zealand Commerce Commission event which included a fascinating argument.

He points out that receiving a return on the substantial capital investments that a fiber to the home (FTTH) project requires is much more dependent on takeup rates than on the average revenue per user.  

Single player large scale deployments usually achive only 20%-25% initial adoption after which growth is exceedly slow.  On the other hand, systems which are open to competitors, i.e. with viable wholesale services, attract many more players who market, sell and deliver new services thus dramatically increasing adoption and accelerating the wholesaler's return on investment.

In short, obtaining a regulatory holiday so you can run your new fiber network as a monopoly is actually bad for your shareholders!

Benoît's speech was filmed.  For the full presentation (in four segments of < 10 minutes each on YouTube) see Benoît's post.

If you want just the essence, listen to the first three minutes of this:

Here is an enlarged version of the chart that Benoît is using while he speaks:

Gold bits and brown bits ― When you need QoS, two classes will do it!

Quality of Service is a contentious subject.  Every now and again, I look at Wikipedia's QoS page.  (Here are my comments 18 months ago).  While the Wikipedia content continues to evolve, it always downplays anything that questions the value of sophisticated QoS.  As of today, the section titled "QoS Problems" has been cut back to a single sentence (which does reference a classic anti-QoS paper):  "Internet2's QoS Working Group concluded that increasing bandwidth is probably more practical than implementing QoS."

Rather than directly debunk QoS yet again, let me investigate where QoS is actually useful by looking real life situations and the deployed solutions that allow for robust services despite congestion.  Identifying common elements in successful deployed solutions is much more realistic than theorizing about NGNs that still await deployment.

The first thing to notice: the only problem is on access links.  As I've written before:

Once you get beyond the access network, every link in the Internet — local, regional, national or international — is carrying multiplexed traffic from many users. Multiplexing many, many, bursty flows results in relatively predictable volume. Traffic volumes vary by time of day, but these links don’t saturate, except as a result of poor engineering or forecasting on the part of an ISP, or failures in other parts of the network causing rerouted traffic. Either case generates a rapid response from any ISP that expects to remain in business. So “best efforts” in the Internet core means sub-millisecond delay variations and near zero lost packets.

Net, net:  you never need QoS priority schemes in the core, all traffic gets excellent service. 

Access links are another story for two reasons.  First it can be very expensive to add more bandwidth.  This may be a byproduct of political/ regulatory issues, but we won't go there, at least in this post. :-)  Second, an access link may saturate due to an outgoing email with large Powerpoint attachment (the home case) or an equivalent event in the corporate environment, for example, transfer of a large CAD file.

But these are solved problems with multiple commercial products on the market for residential and corporate use.  It's worth looking at what these products do in order to understand exactly what kind of QoS is really required, when you can't just provision more bandwidth.

In the corporate market, there are two approaches.  The brute force approach splits voice and video telephony traffic from data traffic using two separate Internet connections with the VoIP connection sized to preclude any potential congestions.  When a single access link is shared, the typical solution employs a QoS router at the customer’s end. The QoS router gives absolute priority to outbound VoIP packets and protects inbound VoIP packets by active traffic shaping; i.e., by signaling remote TCP hosts to throttle inbound TCP data flows so there’s enough capacity for inbound VoIP packets.

Elsewhere within corporate LANs, there is either no QoS or simple priorities. Since VoIP traffic is a tiny percentage of all traffic, it can be given absolute priority without noticeably impacting other applications.  Thus some enterprise VoIP installations use QoS based on DiffServ or Ethernet 802.11p/q.  Typically there are just two classes assigned, with absolute priority for VoIP and video telephony.

I've written about equivalent solutions for residential situations that support VoIP and interactive gaming.  The solution?  Consumer VoIP phone adapters incorporate simple priority and have the ability to fragment large packets (so as to reduce serialization delays).  Because it's so useful for VoIP and also for interactive gaming, this functionality is showing up in popular residential routers from Linksys, Netgear, etc.

Two classes and simple priority is what it takes ― Gold bits for voice (and video and interactive gaming); Brown bits for everything else.

And that's only when you can't get more bandwidth.  On your own premises, it's likely cheaper and easier to upgrade to Fast Ethernet or Gigabit Ethernet.

Customer owned fiber in Ottawa

Arstechnica had a nice article yesterday by Timothy Lee entitled The Long Tail following up on Derek Slater's article last week on the Google Public Policy Blog entitled What if you could own your Internet connection?  Both articles are about a pilot project in Ottawa.

The "tail" in Timothy's article is the "last mile" (or as I prefer, "first mile") fiber connection from individual homes to a network peering point or other aggregation point where individuals can then choose from among multiple competing ISPs.  The importance is, as Timothy Lee puts it,

Without the expense of rolling out last mile infrastructure to every home, many more ISPs could afford to serve a given neighborhood by running wiring to the peering point, leading to more competition and lower prices. Perhaps best of all, the growth of customer-owned fiber could make debates over "open access" and network neutrality moot, as robust telecom competition should prevent the worst of the monopolistic behavior exhibited by telco and cable incumbents.

Typical costs run below $3000 per household depending on density and the number of households that agree to participate.  Considering the cost of a home, this is one investment that should easily increase the value of the home, most likely by much more than the cost!

Mentioned by Derek although unmentioned by Timothy is the most important person here, Bill St. Arnaud, Chief Research Officer at CANARIE, a Canadian nonprofit that focuses on network infrastructure.

Bill is driving the Ottawa trial and is the leading proponent of condominium fiber as described in his presentations here.  His work has also had a major impact on my thinking about "layer zero" competition.

Either way, articles on the Google blog and at Arstechnica will help promote this excellent concept and, hopefully, improve the quality of discussion in policy circles.

3G's biggest success is as a dumb pipe

Thanks to Dean Bubley for pointing out this presentation by Antero Kivi on the use of 3G mobile networks in Finland.

92% of all data traffic is from PCs

Furthermore, of the 4% of traffic from Symbian devices, 79% of that is for web access and 10% is for email.  So users are paying for mobile data for the sole purpose of connecting to the Internet.

Put another way, mobile operators have failed to provide any operator-specific data application of any relevance.

Why invest another penny in IMS?

Voice telephony needs priority, but after that all people want is a dumb pipe. In a competitive market that what they'll get. So IMS is vastly more complex than appropriate.

The cost effective solution is simple (i.e. two) priorities on the radio link, softswitch-controlled circuits-over-packets for voice telephony and straight off-load to the public Internet for everything else. An IP/MPLS core network is more than sufficient.  And existing GSM voice telephony protocols are both adequate and appropriate for voice telephony.  Existing GSM signaling is already being carried over Sigtran over IP, so the pieces are falling in place.

What I've just described matches what I've seen of China Mobile's new 3G backbone.  If true, that means the Chinese network equipment providers (Huawei, ZTE, etc.) are well positioned to win 3G business in high growth emerging markets where costs are key.

It certainly sounds bad for IMS.

NGN ≠ the Internet, and never will

I see and hear a lot of confusion about next generation networks (NGN).  In most cases people are using the term roughly as the ITU-T defines it:

A Next Generation Network (NGN) is a packet-based network able to provide services including Telecommunication Services and able to make use of multiple broadband, QoS-enabled transport technologies and in which service-related functions are independent from underlying transport-related technologies.

but many people don't realize how little this has to do with the Internet.

The Internet is a "network of networks" that includes millions of smaller domestic, academic, business, and government networks interconnected using IP.  It is a hierarchy because there is a backbone of ~28,000 autonomous systems (ASs) which exchange IP packets using routes established by Border Gateway Protocol (BGP).  The remaining millions of networks connect to that backbone via hundreds of thousands of ISPs and other intermediaries who are ASs or connect to an AS.

All of the NGN proposals (Wikipedia has a good summary) involve sophisticated QoS.  But it is well established that there is no technical or commercial requirement for QoS on the Internet backbone (references discussed here and here).

The thousands of organizations that are ASs exist in hundreds of different jurisdictions.  While some ASs are heavily controlled by governments (there is basically one AS for all of China), AS interconnection is independent of any single government.  Interconnections occur based on tradeoffs between the cost of doing business locally and the cost of routes to other locations.

Indeed, to the extent Tier 1 ISPs have attempted to limit free peering, Tier 2 ISPs have established peering agreements that form a donut around the Tier 1s, thus cutting Tier 2s' transit costs to an absolute minimum.  So the effectively unregulated Internet backbone is working remarkably well based on commercial arrangements between thousands of parties, just as it has for 15+ years.

With no technical need for QoS on backbone routes (as discussed here) and no commercial reason that anyone has articulated, it's hard to see how the thousands of parties who make up the core of the Internet would agree to do anything with QoS, ever.

Established telephone companies will deploy NGNs for telephone service.  To the extent they have a monopoly on Internet access, they will be able to use their NGNs to block access to the Internet, but the existence of NGNs won't change the way the Internet core works or the way anyone else's network works.

So NGN's are an evolution path for existing telephony networks, not the Internet, and they will last as long as the existing telephone service model lasts and no longer.

Why there is no Internet QoS

In 2006, I wrote a short blog post on why there is no Internet QoS and likely never will be.  That post is a continuing source click throughs and email inquiries so, when Jon Arnold asked me to write a guest article for IP Convergence TV, I thought I'd tackle the definitive story on Internet QoS.

Here is the new article: Why there is no Internet QoS.  It may or may not be definitive, but I have filled in the engineering and economic background, and I investigate where QoS has been deployed in networks in order to identify where there may be QoS related opportunities for ISPs.

My conclusions:

• There is no Internet QoS today, and it’s unlikely any complex QoS scheme will ever be added to the Internet as a whole.
  
• To the extent next generation networks with rich QoS are deployed, this will only happen within walled gardens, not as part of the public Internet.
  
• But there may be opportunities for ISPs to create a simple premium service that could generate incremental revenue.

Tracing Internet connections by inter-packet timing correlation

Susan Crawford's post The Auction, the Cops, and Comcast, highlights a request from the US Department of Justice to the FCC that is likely to result in an FCC Notice of Proposed Rule Making (NPRM).  As I read her post, I recalled an interesting paper on surveillance.  Here's the relevant part of Susan's post (bold italic highlighting is mine):

The NPRM, if it follows the DOJ’s request, will suggest (among other things) that all of these providers should build their routers and network hardware to provide “packet activity reporting” for all packets crossing their networks, and physical location information for all of their customers at all times. It will also suggest that very fine-grained timing information is needed - something that the internet and its applications don’t provide at the moment. “Packet activity reporting” means that the broadband provider will need to know the destination IP address and port number for everything happening on its network.

The idea is that these designs will help law enforcement when they want to carry out a request for call-identifying information.

A few months ago I happen to read a 2002 paper by Xinyuan Wang, Douglas S. Reeves and S. Felix Wu entitled Inter-Packet Delay Based Correlation for Tracing Encrypted Connections Through Stepping Stones.

I should comment that, when I travel in China, I use TOR (The Onion Router)
to bypass the Great China Firewall.  TOR works by passing your traffic through a series of intermediate routers, with intermediate connections encrypted, so a third party observer can't tell whose traffic is going where.  This is mildly useful to me when I'm in China, but it can be life saving for dissident writers living in totalitarian states. 

So it was somewhat distressing when I read the paper by Wang et al.

From their abstract:

... we address the problem of tracing encrypted connections through stepping stones. The incoming and outgoing connections through a stepping stone must be correlated to accomplish this. We propose a novel correlation scheme based on inter-packet timing characteristics of both encrypted and unencrypted connections. We show that (after some filtering) inter-packet delays (IPDs) of both encrypted and unencrypted, interactive connections are preserved across many router hops and stepping stones.

and from their conclusion:

Our correlation metric does not require clock synchronization, and allows correlation of measurements taken at widely scattered points. Our method also requires only small packet sequences (on the order of a few dozen packets) for correlation. We have found that after some filtering, IPDs (Inter-Packet Delay) of both encrypted and unencrypted, interactive connections are largely preserved across many hops stepping-stones. We have demonstrated that both encrypted and unencrypted, interactive connections can be effectively correlated and differentiated based on IPD characteristics.

So it's clear what the Department of Justice has in mind.  Of course, if the Department of Justice gets this through, it will just be an added expense on all ISPs (and thus on their customers, i.e. you and me).  It won't actually work against the bad guys (or the good guys) as it's fairly simple to imagine an outbound packet scheduler that introduces jitter into each flow at each onion router.

Singapore's Next Generation National Broadband Network

My post on Tuesday was in response to a Singapore government press release that used the phrases "structural separation" and "passive network." 

Subsequently I've been pointed to this website where the Singapore government has posted the qualification documents, mainly document IDA(PQ)-010V2, dated 11 December 2007.  It appears the actual RFP is only available to qualified bidders, but the qualification document has good information on the project structure.

Here is their view of structural separation:

Here is the supporting text:

3.3.2  There will be three distinct types of functional entities operating on the NGNBN.  At the lowest layer, the NetCo would be responsible for the design, build and operation of the NGNBN's passive infrastructure, which includes ducts and wirelines.  The NetCo would offer this passive infrastructure at a wholesale, non-discriminatory and cost-competitive basis (Layer 1 Open Access) to the OpCo(s).  The NetCo's passive infrastructure would be rolled out extensively throughout Singapore.  The NetCo infrastructure is expected to be capital-intensive with massive implementation effort.

To me that implies multiple OpCos can get access to dark fiber.  Hopefully private companies, who want their own dark fiber from one building to another, would also be able to lease dark fiber.  However, they don't make this clear so we'll have to wait and see. 

Only the NetCo portion is covered by the current RFP.  The Singapore government expects the NetCo piece to be built and operated by a private company or consortium, and they are prepared to provide up to S$750M (US$521M) in grants to make this happen.  Responses are due 25 March 2008 and will be evaluated based on:

• Attractiveness of business plan to industry
• Quality of network infrastructure
• Level of Government grant
• Financial proposition and strength of bidder

This is one to watch!

Rate of growth of traffic at the Amsterdam Internet Exchange

Here's a great historical graph, from page 28 here, showing the growth in traffic at the Amsterdam Internet Exchange.

By my take off from this graph, there was 1000x growth in nine years.  That means traffic doubles every 11 months.  While this is only one data point, it's new to me and fairly up-to-date.  It's also roughly consistent with the 12 month doubling rate mentioned in my earlier post.

Managed Storage Futures

Recently I wrote about differences in the exponential growth rates of computing and networking and promised to say more about how these differences cause substantial shifts in the technology landscape.  Managed storage is one example.  The relevant doubling rates (from that earlier post) are:

Doubling Rates

Technology Measure	Months
Computing performance	18
Storage capacity	12
Networking performance	15
Access connectivity	20-26

The increase in storage capacity per dollar has been phenomenal and is one of the reasons that Google can offer Gbytes of free storage for email and that Amazon can offer their Simple Storage Service (Amazon S3) at extremely low rates.

But it's also caused headaches for IT directors, as installed equipment becomes obsolete long before it's fully depreciated, and employees and department heads grip about inflated internal billing rates for storage.  Pity the IT staffer who sends a broadcast message justifying corporate email storage limits because "it costs the company X cents per megabyte per month."  I've seen such messages, and the employee ridicule they engender.

This sounds like a perfect opportunity for a managed service — provide an interface that looks a storage area network or network attached storage, using multiple (for reliability and arbitrage) Internet-based storage services to provide the actual storage.  But now differential growth rates become a factor.

Storage costs decline a bit more rapidly than the cost of Internet transit.  So, already it's the case that network-based storage is extremely low cost for backup but less affordable for transactions.

But the real problem is the cost of access connectivity.  If you're selling managed services to IT departments, you need to provide services at their premises.  Local connectivity is not fast, cheap or reliable, and the pace at which it improves is glacial in comparison with storage or Internet transit.

Has this prevented the emergence of managed storage solutions?  Of course not.  But most existing solutions focus on remotely managing equipment that's physically on the enterprise premises.

Is there opportunity for network-based managed services.  Also, yes.  But you will need considerable focus on local connectivity, both for the numbers you use in your business plan and for the specifics of how you implement the service.  Some thoughts:  interface your managed service via a remotely managed on-premise box that includes caching?  use a dedicated access link to guarantee QoS?  ???

In any event, three years from now, you can count of disk storage being ~8X more affordable, Internet transit being perhaps 4x more affordable, but local connectivity only 2x or 2.5x.  Don't give up your great  business idea, but plan accordingly.

Communications' Exponential Growth Rates

Everyone in high tech is familiar with Moore's Law and most are aware that similar exponential growth occurs in many other measures of digital technology, for example disk storage.  What about communications?

Gordon Moore referred to the number of transistors that could be inexpensively placed on an integrated circuit.  The Wikipedia entry for Moore's Law mentions other measures that are increasing at exponential rates as alternate "formulations" of Moore's Law, e.g., computing power, hard disk capacity and pixels per dollar.  Unfortunately, Wikipedia (at least so far) doesn't have accurate doubling rates or supporting data for these other measures and they present little on the area that most interests me -— communications.

Why care?  Different rates of exponential growth can cause substantial shifts in the technology landscape, some of which I'll discuss in subsequent posts.

Here are the doubling rates for some measures I'm interested in:

Note 1:  Density at minimum cost per transistor, see the Wikipedia article on Moore's Law.

Note 2:  12-24 months depending on how computation is being measured.  See, for example, the arguments here.

Note 3:  Since the mid-1990s, disk capacity has doubled  every 12 months.  In earlier decades, it was more like 15 months (based on calculations from this data), but since 1995 (through today) the 12 month rule has held.

Note 4:  Based on these data points:

Note 5:  The classic reference is Odlyzko & Coffman (2001) (also 1998).

Note 6:  My personal Internet connectivity is documented here (for data to me).  For data rates from me to the Internet, the Feb 2007 value is 2.7 Mbps, which implies a 26 month doubling period (versus 20 months for data rates to me).

Off to Madrid for Connect 2007 Europe

The third and final Connect conference of 2007 is taking place in Madrid on Wednesday and Thursday, November 7th and 8th and I'll be there.  My blog comments on earlier conferences are here (& 1, 2, 3, 4, 5, 6, 7, 8).

Day One has a heavy focus on mobile industry issues and mobile applications. And, it's conducted as panel discussions with few or no slides.  Perhaps this only works because there are good speakers, chosen to promote controversy and discussion, but it really works!  In both 2006 and so far in 2007, the nature of the discussion has been much, much better than at a typical industry show.  The session descriptions for November 7th are here and the speaker bios are here.  If you can be in Madrid on Wednesday, you should attend.

Day Two is a more traditional developers conference focusing on NMS technology and products that are used to create many of the applications discussed by the Day One executives.  Check out the Day Two program.

The day the routers died (to music)

The RIPE 55 meeting in Amsterdam has ended and Raindeer just uploaded this summary to his blog Lunatic Thought:

"The RIPE 55 meeting has just concluded. There was much debate on what to do on the imminent depletion of the unallocated IPv4 pool in 2010. We could do nothing or we could create a market place and facilitate transfer of IP-adresses, but it's all a train wreck waiting to happen. This is best shown however by a beautiful song "The day the routers died" also available on Youtube written and performed by Gary Feldman. So please all upgrade to IPv6 soon, or else you will not get 40Gbit/s to your mother."

and points to this wonderfully funny video:

Internet Connectivity in Guilin, China — Great China Firewall, plus engineering & capacity issues

I'm in Guilin China for the Connect 2007 Asia conference.  I've got the Tor bundle (Tor & Privoxy & Vidalia) installed on my laptop so I can get around the Great China Firewall (GCF) and that works well.  Tor starts up automatically when my PC starts, but I only use it for browsing (by clicking the Tor button for Firefox) if I can't reach a site directly.  Yes, it's noticeably slower going through Tor, but it always works.  For example, just now, seeking the URLs to paste in about the GCF, I could get Rebecca MacKinnon's post directly, but timed out on this from Gigaom.  I clicked the Tor button, hit retry and the Gigaom post loaded.  So, the Tor bundle is a good setup which I highly recommend if you are traveling to China.

The bigger problem for me at the moment is basic throughput.  The Sheraton Hotel Guilin (where the conference is being held) has wired Internet connectivity in the rooms for 50 Yuan ($6.65) per day and free WiFi in the lobby, but only WiFi from China Mobile (CMCC) in the conference area (100 Yuan per day through the hotel business center for those of us without a China Mobile phone number).

Performance is roughly equivalent for all three connections.  Early in the morning or late at night (perhaps during the night, but I do have to sleep sometime), everything works well.  It's not the blazing speed I sometimes see in Beijing, but it works. Here are the results in my room at 7:37am against a server in Shenzhen (300 miles away):

and here's a test against a server in Los Angeles:

Unfortunately, by 10am things begin to slow down and by noon yesterday, I was seeing greater than 50% packet loss and it became impossible to blog the conference live.  I was using the China Mobile WiFi but going up to my room didn't help.

My guess:  There's a significant capacity problem between Guilin and the rest of China.  Not surprising I suppose, considering Guilin is a tourist city, not a business city.

IMS is about transport, not services

OK, that title is designed to grab your attention, but it's also the reasonable takeaway of people in the audience at the last session yesterday, i.e. day one at Connect 2007 in Boston.

The session was entitled "Increasing Service Velocity" and the session description focused on IMS and service delivery platforms.  I wasn't involved in setting up the session or signing up the speakers, but a few weeks ago I was recruited to moderate the session.  The panelists from left to right were:

• Kjell M. Johansson, Director of Solutions Management, Multimedia, Market Unit North America, Ericsson
• Susan Norris, Communications Industry Advisor, Norport Technology Management Consulting (but note that Susan spent many years at operators, most recently at Sprint, and was the most articulate about the operator point of view).
• Douglas Tucker, CTO NA, Ubiquity Software Corporation
• Jouni Welander, Head of New Solutions US, Nokia Siemens Networks

So there's a slight problem.  Everyone here is smart and knowledgeable, but everyone (with the possible exception of Susan) is involved in creating or selling IMS or IMS products.  Panel discussions are always better if you have divergent points of view, i.e. controversy on stage!  Since I was moderating, I obviously couldn't blog it live or even take notes.  Luckily, George Kontopidis did take notes (and the picture above), so that helps me reconstruct events.  George's complete notes are here: Brough1.jpg, Brough2.jpg and Brough3.jpg.

Some of the specific comments on service deployment platforms (SDPs), their relationship to the rest of IMS and to deployment of actual applications included:

Kjell - SDPs are essential to the development of actual services, but the problem here is too many standards in what's exposed to the developer.  Kjell alluded to pressures from operators that may result in the major equipment vendors converging their service creation environments, but he couldn't give specifics or a date (beyond soon, within the next year).

Susan said operators think of IMS as the solution for new services.  They are generally very conservative (particularly on the operations side) and wouldn't dream of opening up their networks for something like web services.

Doug offered that IMS and higher level development environments, including web services, are not conflicting.  IMS is the platform, but it's standardization stopped below the application layer.  IMS needs web services and/or other development layers to actually realize new services.

And it was Jouni who offered "IMS is not about killer apps, it's about a killer environment" i.e. IMS is the platform but it needs (the non-standard) SDP layer above to support service creation.

On the subject of what is actually real, there was some consistency.  Kjell and Jouni both said there were a few commercial deployments for specific applications like video sharing.  Susan said she couldn't identify any IMS deployments with full service, but knew of several with partial services, i.e. IMS lite.  And Doug commented that he knew of pieces of IMS deployed in many operators, but nothing that's pretty or matches the IMS vision as yet.  On the other hand, everyone on the panel was confident that things would continue to improve.  There were joking remarks that 2008 would be the year of IMS.  In response to the question of when I would be able to hand off a video sharing session across operators, there was some agreement that the GSMA was working on this, so it might be solved in the next 1-2 years (although it could take longer to propagate to AT&T !).

There's a lot more in George's notes...

Dean Bubley did the best job shaking up the panelists (via questions from the audience).

And the sense I was left with (as were several members of the audience with whom I talked later) is roughly as summarized in my title above.  IMS is plumbing that helps operators manage their networks and is a great platform to support a variety of new services, but it will take higher layers (not part of the standards) to actually facilitate new applications,  In addition, there are many, many other issues to resolve, both technical (like integration with billing systems and other operator IT infrastructure, simplification of handset diversity issues, and so on) and business model related, i.e the extent to which will operators open up to new applications.

In any event, an interesting session!

Architecturally induced denial of service susceptability

Fundamental design decisions have a big impact on how a network behaves under stress.  I just noticed two otherwise unrelated posts both touching on network design issues that result in (or protect against) network collapse under stress.

Yesterday, I wrote about August's Skype outage where the system took over 36 hours to recover from a collapse.  The issue was too many clients attempting to reconnect at once.  Instead of clients “backing off” when supernodes weren’t immediately available, they kept hammering away, trying to log in.

Last week I wrote about the ways the Internet and the Telecoms networks respond to congestion in the backbone.  In December 2006 an earthquake off the coast of Taiwan broke multiple fiber cables causing a congestion collapse on the Internet backbone in significant parts of Asia.  The issue was too many computers attempting to reestablish TCP sessions with the result there was no capacity left for any session to actually send data.

In both cases, there were externally induced problems but, rather than recovering, there was the equivalent of a denial of service attack, self inflicted!

Both of these failures have a fairly simple solution, at least architecturally.  Under conditions of severe overload, the system must be able to restrict new attempts (new TCP sessions, new Skype logins, etc.) to some small percentage of the available capacity.  This allows the rest of the capacity to serve the logins, sessions or calls that do get through, with the result that what capacity remains is put to good use.

As I commented last week, this is one place where the telecoms industry has the correct architecture.  When disaster strikes subjecting some part of the network to overload, it's easy to restrict new call attempts on trunks into the congested area, for example by call gapping. This limits the amount of new traffic to that which the network can handle.  Thus, if only 30% capacity is available, at least the network handles 30% of the calls, not 3% or zero.

Here's one place where network architects can learn from established telecoms practice.

Skype outage: post mortem evolves

Since my last summary, some new information has emerged.  Gerry Blackwell interviewed Skype's Director of Operations, Michael Jackson, got some comment from Martin Geddes, and wrote an interesting article.

I've also had an email exchange with Julian Cain, which I reprint below, and a brief exchange with Philippe Biondi of SecDev.org who referred my to an interesting blog post (in French) by his colleague Cédric Blancher at  EADS France, Innovation Works (Suresnes).

New information from Michael Jackson (in Gerry Blackwell's article) includes the idea of five supernodes per cell (of 300 users):

Each supernode handles about 300 nearby users. Skype configures five in each cell for redundancy. So with upwards of nine million users online, it takes something like 150,000 supernodes to make Skype work.

The triggering event is still attributed to massive computer reboots after Microsoft's Patch Tuesday.  Everything else in the article is consistent with the best earlier explanations, including Julian's which I summarized here, i.e. as Blackwell puts it:

... the real culprit, Skype now says—was a resource allocation algorithm in the client software that could not adapt to such a set of circumstances. Instead of clients “backing off” on their attempts to validate on the network when supernodes weren’t immediately available and waiting for the ship to right itself, they kept hammering away, trying to log in.

And the solution, having clients back off when supernodes aren't immediately responsive, is obvious.  What's left to understand?

1.  I still haven't seen a plausible explanation of why Microsoft's "Patch Tuesday" resulted in problems on Thursday morning, only a lot of questions.  If the problem was induced by massive reboots, why didn't it happen on Wednesday morning?

2.  I still haven't seen a reasonable discussion of scaling.  As I wrote back in August,

I wonder, apart from the login server cluster as a single point of failure, is there also a scaling issue?  FastTrack's breakthrough was the use of supernodes to make the system more scalable.  But was that just one layer of scalability?  If so, what happens when there are 300 million on-line users and one million supernodes?  Perhaps Julian (or another P2P expert) could comment...

Indeed I emailed Julian about scaling and also about Joost.  This was his reply.
On 09/05/2007 05:06 PM, Julian Cain wrote:

Brough,

Skype and Joost are utterly different however Skype is more like Fasttrack(Kazaa). Joosts' Network architecture is mainly "Centralized", they have their own server farm of Supernodes as well as Authentication and Jabber servers. The nature of Joost is less dependent on peer to peer routing as it's basis is tuned towards QoS. Joost peers route traffic and relay UDP based payload as media data streams as well as keep a small cache of what they have recently viewed, however currently every Joost peer is directly connected back to the Joost home servers unlike Skype and Kazaa where once authentication occurs it's "out of our hands".

I agree on the extent of Skype scalability being very limited because of the nature of the Supernodes. At any one time the Supernodes hold ~300-500 child nodes and maintain an "Overlay" network which consists of another several hundred Supernode to Supernode connections. Ie* The Supernode network is very dense in order to provide for best means routing of least cost, however the flaw in this architecture is where the "Overlay" network reaches a capacity and is unable to reliably route traffic. I do not currently have any statistics on how the "Overlay" layer is not scalable but as more Supernodes arise the management of the "Decentralized Data Store" becomes a very hard task as well as keeping this "Overlay" in one single "in sync" network. This was proven with the Skype outage as the network was "trying to heal" it had to start from many 10s of thousands of "Overlay" networks which very slowly were able to sync again as a "single" network however is still an issue today with presence.

For the current Skype "Overlay" network to scale indefinitely while maintaining  a "Single" network infrastructure it needs in place an organizational hierarchy of Supernodes and a level of Service for each of these Supernodes. *Ie. If Skype Supernodes worked in a way such as in Fasttrack then when the network reached 100 million users it would began to crawl. This is due to the dense nature of the upper "Overlay". I can only assume that Skype has thought of this and that when the Supernode ratio is beginning to "bottle neck" then there would be some ordered Hierarchy as to what role each Supernode was playing, otherwise the more Supernodes the more dense the "Overlay" the more the data is relayed back and forth before considering the "Supernode Overlay" into it's own Denial of Service attack.

I hope this helps to some degree, let me know if you have any other questions.

~Julian

So to the extent I have time to look into P2P technology further, I plan to explore what's been written about hierarchy in P2P networks.  Here are some references (which I've found but have not read as yet):

RFC 4981 on Survey of Research towards Robust Peer-to-Peer Networks: Search Methods

Hierarchical Peer-to-peer Systems by L. Garces-Erice, E.W. Biersack, P.A. Felber, K.W. Ross, and G. Urvoy-Keller.

An efficient peer-to-peer file sharing exploiting hierarchy and asymmetry, by G. Kwon and K. D. Ryu in the Proceedings of the 2003 Symposium on Applications and the Internet, 27-31 Jan. 2003 Page(s): 226 - 233.
< unfortunately only available on an IEEE pay-for site >

Congestion in the backbone: Telecom and Internet solutions

When a network is subject to a rapid increase in traffic perhaps combined with a rapid decrease in capacity (for example due to a fire or a natural disaster), there is a risk of congestion collapse.  In a congestion collapse, the remaining capacity is so overloaded with access attempts that virtually no traffic gets through.  In the case of telephony, everyone attempts to call their family and friends in a disaster area.  The long standing telephony approach is to restrict new call attempts upstream of the congested area, for example by call gapping.  This limits the amount of new traffic to that which the network can handle.  Thus, if only 30% capacity is available, at least the network handles 30% of the calls, not 3% or zero.

There are comparable issues for the Internet backbone, but they are not so completely solved.  As Wikipedia puts it:

Congestion collapse was identified as a possible problem as far back as 1984 (RFC 896). It was first observed on the early internet in October 1986, when the NSFnet phase-I backbone dropped three orders of magnitude from its capacity of 32 kbit/s to 40 bit/s, and continued until end nodes started implementing Van Jacobson's congestion control between 1987 and 1988.

TCP congestion control solved day-to-day congestion collapse, but it didn't deal with disasters such as the Taiwan earthquake of December 2006.

As a result of problems in the Internet backbone in Asia caused by that earthquake, there's new discussion of Internet congestion.  I stumbled on one interesting discussion on the NANOG (North American Network Operators Group) mail list.

In this thread, the NANOG group seems to be arriving at conclusions similar to those of the telecom industry, i.e. throttle new traffic attempts upstream of the congested area.  From Fred Baker:

So plan B would be to in some way rate limit the passage of TCP SYN/SYN-ACK and SCTP INIT in such a way that the hosed links remain fully utilized but sessions that have become established get acceptable service (maybe not great service, but they eventually complete without failing).

And from Sean Donelan:

This would be a useful plan B (or plan F - when things are really FUBARed), but I still think you need a way to signal it upstream 1 or 2 ASNs from the Extreme Congestion to be effective.

...what should the alternate queue plan B be?

Probably not fixed capacity numbers, but a distributed percentage across different upstreams. ...

  Session protocol start packets (TCP SYN/SYN-ACK, SCTP INIT, etc) 1% queue
  Everything else (UDP, ICMP, GRE, TCP ACK/FIN, etc) normal queue

And finally why only do this during extreme congestion?  Why not always do it?

The thread contains over 50 messages and doesn't reach a specific call-to-action (that I could detect), but it does show similar problems and similar potential solutions.

I would have thought there'd be a few old telecom folks on the NANOG list, but I guess not...

Best Skype outage post mortem so far

It's been more than ten days since the global Skype outage – time to reconsider what actually happened.  The most credible analysis is not from Skype, but from Julian Cain in a series of comments (here, here and here) that he made to a Gigom article about the outage (or see the single file in "References" below).  Julian is lead architect at Pando and, earlier, was head of Mac development for Kazaa at Sharmen Networks.  So he knows a lot about peer-to-peer networks and his work at Sharmen put him in a position to know quite a bit about the P2P technology that's also used by Skype (and likely by Joost).

Background

Skype's P2P technology was evolved from FastTrack, originally developed for Kazaa.  Their P2P network consists of clients and supernodes.  Skype distributes client software which includes all necessary supernode software, so any client that has appropriate capacity and connectivity can be promoted to become a supernode.  Supernodes dynamically link to other supernodes to support a distributed database and distributed index (called the distributed hash table or DHT).  For Skype, the DHT layer is responsible for maintaining client presence info, contacts and icons/avatars, and handling call routing.

Root Cause

But as I pointed out in several posts during the outage, there's also a centralized component to the Skype network.  That's the login servers.  Julian refers to them at the "authentication servers" and/or "login/connectivity servers."  They are implemented as one cluster of about 50 machines.  As for the root cause of the outage, he asserts:

Skype employees introduced code into the "login/connectivity" server farm that was not compatible with current Skype clients.

Other Issues

While that was the root cause, it was helped along by other network characteristics, notably that each client connects to only one supernode at a time.  According to Julian, there are 300+ clients per supernode and if a supernode goes off line, the 300 or so clients connected to it must reenter their "connecting" sequence, i.e., find and connect to another supernode.

A network with 8 million on-line users implies ~27K supernodes, a figure that's consistent with the ~20K supernodes estimated by Desclaux and Kortchinsky in 2005-2006 (see their June 2006 Recon presentation, PDF here).  The other point from measurements by Desclaux and Kortchinsky is that each supernode attempts to maintain a list of all other supernodes which means there is a substantial amount of traffic between supernodes.  This clearly contributed to the slow recovery, during which Julian commented:

Right now there are approximately 10,000 Skype networks instead of one single "in sync" network.

Scaling?

So I wonder, apart from the login server cluster as a single point of failure, is there also a scaling issue?  FastTrack's breakthrough was the use of supernodes to make the system more scalable.  But was that just one layer of scalability?  If so, what happens when there are 300 million on-line users and one million supernodes?  Perhaps Julian (or another P2P expert) could comment...

References:

I've extracted and assembled a complete copy of Julian's relevant comments.

Vanilla Skype part 1 and part 2, by Fabrice Desclaux and Kostya Kortchinsky, Recon, June 17th, 2006.

Skype traffic during the week of the outage, captured by Phil Wolff of Skype Journal.

Skype's centralized control of P2P network parameters

The blogsphere is abuzz with reaction to Skype's second attempt to explain what caused the recent crash of their entire "peer-to-peer" network, but I haven't seen any comment on the one thing that struck me (in their 4th paragraph):

Once we found the algorithmic fix to ensure continued operation in the face of high numbers of client reboots, the efforts focused squarely on stabilising the P2P core. The fix means that we’ve tuned Skype’s P2P core so that it can cope with simultaneous P2P network load and core size changes similar to those that occurred on August 16.

As I commented earlier, we know from presentations by Desclaux & Kortchinsky at Blackhat Europe (PDF) in March 2006 and at Recon in June 2006 (PDF in 2 files: one and two), that there is substantial traffic between the (3rd-party-owned, distributed, P2P) supernodes that form the core of the Skype P2P network and Skype's (centralized) login servers.

If Skype's explanation is correct, it's clear Skype also has a way of distributing parameters to supernodes that tune their behavior.  I'm not surprised.  It's a logical to design in both measurement and tuning capabilities. 

But such centralized capabilities also represent a potential venerability.  What would happen if a black hat got access to those tuning capabilities...

Understanding the Skype Outage

Skype's official explanation.  Phil Wolff has a good set of interpolated comments on the official explanation.  There are two things to add.

1.  As the Register points out, last Tuesday was Microsoft’s monthly patch day and those patches required a re-boot.  If we believe Skype that their problem started with excessive login attempts, this is the only plausible explanation on the table.

2.  There was no patch for the Skype client (i.e. this was routine and hasn't been widely adopted) so either:

• we face another problem next time Microsoft issues patches requiring a reboot, or
• they fixed something on their servers. 

I suggest the latter.  As I pointed out during the outage, Skype generates a lot of traffic between the login servers and supernodes (see slide 16 in  DESCLAUX and KORTCHINSKY's presentation.  I suggest Skype has patched something on the login servers.  It's well known (e.g. Desclaux & Kortchinsky) that Skype login is a centralized function.

Meanwhile, it will be interesting to see if any additional comments or new client releases appear from Skype in the coming days.  I suspect not, as their approach to security has always in included both encryption and obfuscation.