Communications

Broadband capacity - sizing buffers to handle traffic bursts

This is part 4 in a sequence of posts examining how broadband services actually work.  Part 1 looked at ISP concentration ratios.  Part 2 examined the effect of  averaging traffic from many subscribers.  Part 3 considered the impact of congestion on user experience. Here we'll take a finer grained look at traffic in the access network.

How do we achieve "zero" packet loss? Most websites are "hosted" and connect to the backbone by Fast Ethernet or Gigabit Ethernet links. Individual websites may be mismanaged, but there is no excuse for packet loss here.

Internet backbone links and backbone-to-backbone hand-offs are extremely high capacity multiplexing the traffic of thousands to millions of users.  Traffic statistics are predictable and operators compete on the basis of service levels, so packet loss is typically zero.

If there are performance problems, they're usually in the access network. Of course, one bottleneck is the service I've signed up for, for example a DSL link at 6 Mbps /1 Mbps. By definition my traffic is shaped to the service level I've signed up for.  This shaping typically happens at the first aggregation point (CMTS, DSLAM or WISP P2MP radio) or, on an IP or MAC address basis, at a bandwidth management device further upstream.  If they arise, problems are in the aggregation points and/or links between me and the backbone.

Since packet loss significantly degrades user experience, we'd like to understand how much over-provisioning a broadband service provider needs in order to absolutely minimize packet loss in the access network.  To understand this we need to look at smaller levels of aggregation (dozens and hundreds of users) and shorter intervals (sub-second rather than 5 minute averages, as few routers buffer more than a few tens or hundreds of milliseconds of traffic).

But first lets look at some five minute averages from a WISP with ~300 customers, each with either 1 Mbps or 3 Mbps service.

If one took these graphs at face value, it seems 20 Mbps of backhaul would be more than adequate, as the traffic peaks at 17 Mbps.  But 5 minute averages don't speak to what's really happening.

I don't have sub-second averages for the specific WISP above, but Fergal Toomey, Chief Scientist at Corvil Ltd., has some very detailed measurements on a similar Internet access network providing services to users at 512 Kbps, 1 Mbps and 2 Mbps. These are available in a whitepaper you can request from Corvil's website. Fergal's measurements were made at the Internet POP looking at traffic headed towards individual users.  Also, his measurements were made in 2004 when most hosted servers had 100 Mbps connections to the backbone and the majority of computers had TCP stacks configured for a maximum TCP receive window of 32KB.  In this graph, green shows 5 minutes averages, while light blue shows 500 ms averages. Dark blue shows 5 ms averages and they are striking!

What's Happening?

Web browsers typically use persistent TCP connections to minimize connection overhead and to avoid repeating TCP's slow start.  As a result, servers will frequently respond to a new request with a full window's worth of data.  In 2004 that was 32 KBs of data or ~23 packets of up to 1500 bytes each. Today it could be much more. Absorbing one such burst should be no problem, but to what extent are such bursts coincident?

Beyond web browsing, detailed traffic analysis uncovers a variety of odd behavior and some less savory behavior. For example, one burst was a blast of many 41-byte packets sent by a web application that apparently does a series of 1-byte writes to a TCP socket with the NO_DELAY flag set.  Several thousand packets arrive in quick succession causing packet loss.  Worse yet, follow-on losses occur as retransmissions cause a decaying series of echo bursts.

On a more malicious note, the data includes bursts caused by port scans.  These consist of many, many minimum size probe packets that attempt to scan all of a subscriber's ports in one enormous burst.  Luckily these lost packets are not retransmitted.

Beyond these, Fergal Toomey's analysis uncovered a variety of application anomalies that I found unexpected, but which crop up in the real world.

What headroom is really required?

Clearly we could just increase the size of the buffers in the router and smooth out any of these bursts, but that would also increase the delay while the burst lasts and thus introduce significant jitter.  A better question is how much headroom is required to guarantee zero packet loss and no more than 20 ms of incremental delay?

Conveniently, Fergal Toomey has run that calculation on the data set above and the result looks like this:

In this graph, the average traffic (the "mean" shown in light green) peaks just shy of 5 Mbps.  To guarantee zero packet loss and no more than 20 ms incremental delay 100 % of the time requires 19 Mbps of capacity or nearly 4x.  However, to guarantee zero packet loss and no more than 20 ms of delay 99% of the time, requires only 8 Mbps, i.e. a little less than 2x.

Interestingly, while ISP practices tend to be trade secrets, the rule of thumb that 2x headroom provides good service crops up repeatedly.

Conclusion

As a practical matter, real Internet traffic is fairly complex and includes a wide variety of anomalous behaviors. Rules of thumb based on averaging the traffic of a few hundred or even a few thousand subscribers are likely to be very expensive (in terms of excess capacity).  A better approach is to continuously monitor packet loss and delay variation (jitter), adding capacity as needed to keep those measures below desired levels.  

Zero packet loss and less than 20 ms incremental jitter, more than 99 percent of the time sounds plausible objective to me.  

Now if only my broadband service provider was willing to share any information about their service level objectives or performance.  Ooops, I must be dreaming.

Broadband capacity - the impact of congestion

This is part 3 in a sequence of posts examining how broadband services actually work.  Part 1 looked at ISP concentration ratios and part 2 examined the impact of averaging many subscribers. Future posts will consider how to size backhaul links and how to configure buffers.

Two major reasons people purchase broadband (or sign up for more capacity) are to improve interactivity and to get access to new applications, for example streaming movies.  New apps is clear, but it's worth looking congestion and how congestion affects interactivity.

Congestion has two impacts. First you get increased delays, as packets are buffered in queues waiting for free capacity on the next link. Usually this adds tens or hundreds of milliseconds to round trip times. Then, if the average traffic exceeds link capacity, you get packet loss. If you do have congestion, it's these packet losses that cause TCP traffic flows to back off, thus avoiding congestion collapse (and even less throughput).

Here is a graph of packet loss on a service that most of us would consider unacceptable.  These measurements were taken by Tom Dunigan on an early cable modem service (February 2001) that was substantially upgraded about a year later.

 

The next graph (also from Tom Donigan) shows how this congestion impacts interactivity.  The graph shows the echo delay for typing into a remote program using telnet in character-at-a-time mode.  One test (red line) was run in the early morning and one test (green line) was run during the evening when packet loss was peaking.  Each test measured the echo response time for 100 successive individual characters. Notice the typical echo delay goes up from ~130 ms in the morning to roughly 200-300 ms in the evening, but with occasional delays of more than one second.

Relatively few programs operate character-by-character, so the extra 100 ms or so might not matter.  On the other hand, a once second delay is noticeable!

A more common activity is viewing content on websites.  Here, the dominant time in any interaction is the time spent waiting for web content to download to your browser.  Since web content is downloaded using TCP (and more generally, TCP is the dominant protocol in use today), it's worth looking at the impact of packet loss on TCP throughput.  

The TCP protocol includes a congestion avoidance algorithm which is triggered by packet loss. When a packet is lost, the TCP sender slows down. As a result, the data rate for a single TCP flow looks like this (thanks to Guido Appenzeller):

Of course this assumes the TCP flow lasts long enough to saturate the bottleneck link and that this is the only flow in the network!  

What happens in real networks is complex, but RFC 3155 has an approximate formula for how a single flow is affected by packet loss in real network.  Even better, Bill Gibson at Niwot Networks has produced a nice graphic based on that formula for his paper on TCP limitations on file transfer performance.  It looks like this:

 

This is dramatic!  The different lines reflect different end-to-end round trip times (RTT) - times that will vary depending on the site you are connecting with.  Also, they represent the maximum throughput you could achieve with long duration TCP flows and no other bottlenecks.  What's notable is the logarithmic throughput scale on the left and the fact that, at 1% packet loss (0.01 on the bottom scale), potential throughput drops by a factor of 100!

Again, there are many caveats.  A real network has a mix of short- and long-lived flows. The local operating system may not be optimized to take full advantage of broadband speeds (although MS Windows actually got better at this with Vista). None-the-less, even 1% or 2% packet loss is correlated with poor user experience.

In short, to actually obtain the instantaneous throughput you thought you were purchasing, you don't want packet loss in upstream portions of the network.

I'll discuss what this actually implies for routers and backhaul links in a subsequent post.

Walled gardens have collapsed

Yesterday, Mary Meeker of Morgan Stanley gave her annual state of the economy and the Internet presentation at O'Reilly's Web 2.0 conference.  I wasn't there but her presentation is on line and it has some interesting insights.  Many others are commenting.  The one thing I'd like to draw to your attention is slide 46.

In one year (2007 -> 2008), UK users abandoned their wireless carriers' portals and went directly to numerous open mobile internet services.  I'd love to see comparable data for the US.

Happy One Web Day

Thanks to Susan Crawford for creating One Web Day and directing attention to just what the Web has done for mankind, and towards some of the issues in sustaining and growing the benefits of the World Wide Web.

The blog has moved

A little over two hours ago, my Communications blog moved to what I hope is it's final home.  With the help of Dialogic's web and IT staff, we've got "301 Redirects" established so that traffic that previously went to any of:

blogs.dialogic.com
blogs.nmss.com
blogs.nmss.com/communications/
blogs.nmscommunication.com
blogs.nmscommunications.com/communications/

should now be coming here, i.e. to

http://blogs.broughturner.com

Search engine optimization and "301 redirects"

When I started consulting to Dialogic on strategy, we established a second set of links to my blog.  As a result, both Dialogic-based and NMS-based URLs worked.  The problem is, with multiple URLs pointing to each post, incoming links are diluted and search engine rank suffers.  The correct solution to multiple historic URLs is to redirect them to the new URL.  The http protocol defines "301" for this purpose.  A "301 redirect" tells the inquiring user (or search engine) that the post has moved, permanently.

I've known this for some time, but I've been buried in other work (thus the paucity of blog posts!).  This past week we finally straightened things out, culminating in this morning's changeover.

To establish a redirect, you have to get DNS to point to a server you control and then install the "301 redirect" on that server.  Getting DNS set up was further complicated by the multiple parties involved - that's multiple parties each, for Dialogic and for me.  For broughturner.com, the domain registrar is Network Solutions.  But I host my personal website at pair.com, so the Network Solutions DNS registry points to DNS name servers at pair.com.  Meanwhile my blog is hosted on TypePad, so I established a "custom DNS" record, otherwise know as a CNAME record, on the pair.com name servers so the subdomain, blogs.broughturner.com, points to brough.typepad.com, i.e. to the internal URL of my blog at Typepad. 

One final issue:   I was aware of how you establish "301 redirects" on an Apache server (via a .htaccess file) but I wasn't aware of how it worked on Microsoft IIS.  Luckily, it was easy there as well and, better yet, Dialogic's web team handled it for me.  :)

Thanks Fernando, Sean, Kevin and Louis!

Google's Peering and Caching Strategy

This is a followup on yesterday's post about how little Google/YouTube pays for bandwidth. Google wants to promote peering with ISPs, so they give presentations at ISP meetings.  After reading yesterday's post, Alex Benik of Battery Ventures sent me a link to this presentation given by Google at a 2008 meeting of the Latin America and Carribean Internet Addresses Registry (LACNIC).

As expected, Google peers with as many relevant ISPs as possible.  For the ISP, peering with Google eliminates their upstream costs for traffic to Google.  Since Google represents a substantial volume of traffic for most ISPs, this is a big saving.   As of May 2008, Google was present in 33 public Internet exchanges around the globe, so major ISPs already have connections in places where they can peer with Google.  The minimum qualifications are 5 Mbps of Google traffic and the ability to interconnect using Gigabit Ethernet at one of these 33 major Internet exchange points.

Google Global Cache

What's interesting is Google's caching strategy.  Just as Akamai puts servers in ISP's local facilities, Google is providing a distributed cache for their content.  This available to larger ISPs and allows them to serve Google content directly at the edge of their networks, thus reducing traffic on the ISP's backbone network.  Here's a representative rack that Google provides to the ISP.

YouTube's fine - Analysts don't understand Internet peering

As widely reported, Credit Suisse analysts have estimated Google's YouTube may lose $470M in 2009 and more in the future.  However, their estimates say Google will pay $360M for bandwidth in 2009.  I don't know how Google figures their cost of bandwidth, but anyone who understands anything about Internet transit/peering knows Credit is way off base.

Google does not pay for Internet Transit the way most tier 2/3 ISPs or most content providers must.  The economics are simple.  If you are a Tier 2 ISP, you have to purchase Internet Transit services from a Tier 1 network to handle that customer traffic which goes off your network and for which you cannot make other arrangements.  The most notable 'other arrangement' is peering.  If you have significant traffic to/from another specific network, you and the other network can both save Internet Transit costs by exchanging traffic locally, i.e. peering.  Of course an enormous amount of your traffic is directed to Google.  If you have a presence in any data center where Google has a presence, you would love to peer with Google, as that saves an enormous amount on your payments for upstream Internet Transit.

A similar effect plays out among Tier 1 providers.  If one tier 1 network cuts a special deal with Google, Google routes all their traffic through this provider and suddenly the other tier 1 networks have large asymmetries in their tier 1 peering arrangements.  Either they also cut deals with Google or they have to renegotiate their tier 1 peering arrangements to pay for the traffic asymmetry (something that's highly unlikely!).  Google is the one with leverage here!

I don't know what, if anything, Google pays for bandwidth, but it's not paying $360M for Internet transit.  Sorry Credit Suisse, you better go back to analyzing derivatives, credit swaps and other purely financial plays.

Google does have costs.  They have data centers in many parts of the world and they have a private fiber backbone that interconnects their sites and connects their private network to many, many potential peering points.  Operating their private backbone is a real cost to them and I haven't examined their financial reports to see if there is any way (from public data) of estimating their costs for this private network.

But until someone does this analysis, forget what you've read from Credit Suisse.

Best conference bet for 2009 ― eComm 2009, March 3-5, San Francisco

I attended a number of conferences in 2008, both interesting and not so interesting.  One conference stands out, for the range of interesting speakers and the variety of interesting people I met.  That was the first Emerging Communications Conference, eComm 2008, organized by Lee Dryburgh.  Many of talks from this conference are available on Slideshare and as podcasts on IT Conversations.

eComm 2009 is scheduled to take place at the San Fransico Airport Marriott, March 3-5, 2009.  I highly recommend you check it out.

This is not a trade show with vendors hawking today's products and multiple tracks full of vendor product pitches. 

Presenters have been chosen for the quality of their proposals:  is it new?  is it disruptive?  what will the audience learn?  (As an adviser, I've been in on those discussions).  Like last year, the format is one track spread over three days, with 15 minute presentations, 5 minute lightning presentations, panel discussions and social time.  It all adds up to a veritable fire hose of information.

There's a list of speakers here.  Major topics for 2009 (so far) include:

* Mobile Social Networking (MoSoSo)
* Open Handsets & the Open Ecosystem
* Both Voice and Video Evolution
* Convergence of Media with Personal Communications
* Open Spectrum
* Open Communication Platforms
* Leveraging Cloud Computing
* Social Computing
* Towards 4G Wireless
* P2P and Decentralization of Telecoms
* Communications enabling business processes, especially B2C
* New Forms of Contactability and Connectability
* Emerging Markets

And last, but by no means least, if you mention my name you get 20% off.  More specifically, if you enter the promo code "BroughTurner" (case-sensitive) at the appropriate point during registration, you'll get 20% off the registration fee.  This works now, while early bird rates are in effect, and I'm told it will also work right up to the last minute ("late", not on-site registration), although then it's 20% off the full conference rate, and only if the event is not sold out!

I hope to see you there.

A reason why Asia leads in mobile apps and is advancing with Internet apps

A picture is worth a thousand words.  Thanks to Ben Joffe for this picture, which is slide 21 of his presentation, The Digital Silk Road.

Of course, this is a simplification, as it misses China. 

China doesn't have 3G yet but they have more mobile phones than any other country and they are fast to adopt what can work on 2.5G.  For example, the Chinese adopted ringback tones very very rapidly after the application emerged in Korea -- years ahead of Europe or the US.  And, even at 2.5G speeds today, China supports a rich variety of mobile applications.

Likewise, while China's relative proportion of Internet connections above 5 Mbps is relatively small, there are more broadband Internet connections in China than in the US and China is adding new fiber Internet connections faster than any other country in the world.

The Mobile Web: Limited But Getting Better

I had an interesting discussion with Katrin Verclas of MobileActive.org as an event last month covering a range of topics including the prospects for mobile web access in emerging markets.  As a result, she and Patty Mechael pressed me to write up my thoughts on the mobile web today and over the next few years.

Very roughly, I'm depressed about today's still prevalent walled gardens, but very optimistic that open mobile access to the Internet will become available in coming years.  You can read my reasoning here.

Packet inspection renders Internet connection unusable for blogging

Here's one I hadn't run into before.

The typical hotel broadband access service checks the MAC address of your computer. When you first connect, it sees the new MAC address, interrupts whatever access you were about to make and instead delivers a sign-up screen.  After you've signed up, your packets are routed directly to and from the Internet until your sign up period expires.

Of course this means every packet is being inspected to verify the MAC address is for a computer that has paid for access.  This is very light packet inspection, nothing like the DPI that various ISPs have been called to task for.  It's also very fast so it doesn't introduce noticeable latency, at least until now.

Last night I was staying in the InterContinental Nehru Place in Delhi and I observed a strange effect on perhaps 1 out of 3 or 1 out of 4 page loads.  First I would see a very brief splash screen from the hotel's ISP, then my desired web page would load.  This was unusual but not a bother.  Next, while using Bloglines, every now and then the left screen list of unread posts would fail to refresh, instead displaying a message about access had timed out.  However the next refresh always worked, so again, I can live with that.

Finally I tried to create a blog post but I lost the text as I went to post it.  Ugh!  Apparently, Typepad's blog author's content posting page requires reasonably fast turn around time for update messages.

Clearly MAC address redirection was beginning and then being canceled.  I'd never run into anything like that before -- a case where even shallow packet inspection is a problem.

Visualizing African Internet Connectivity

Here is a great illustration of the not-so-fast or reliable connectivity at African Universities as seen from Trieste, Italy.  Only 46 seconds.

Thanks to Marco Zennaro at the Science Dissemination Unit (via TIER at UC Berkeley).

Similar motivations for mobile and Internet usage in rural Zambia

Over the years I've uncovered a number of serious studies of the social and economic impact of mobile phone adoption in emerging markets.  In fact, my list of links (embedded in various blog posts: 1, 2, 3, 4, 5, 6, 7, 8) has become a reference for several people, for example here. I've been somewhat less organized regarding the social and economic impact of Internet access, but I've resolved to track such references as well. 

Here's something I stumbled on this weekend -- a study by Paula van Hoorik and Fred Mweetwa of TNO Information and Communication Technology in the Netherlands, entitled "Use of internet in rural areas of Zambia."

What struck me in this study was the parallels between the social and economic benefits they found and those conventionally associated with the adoption of mobile phones.

Based on our study, the most important social benefits are:
• Internet enables people to keep their network and enlarge it by communicating with friends, family and others
• Internet enlarges the world of people in rural areas by giving access to information
• Internet brings knowledge and supports education

The most important economic benefits are:
• Reduction of commute time
• Saving money in a lot of different ways, such as only traveling to pick up something when you know it is actually there instead of having to return multiple times
• Bringing new opportunities and using them, such as learning new farming methods or opening an internet café to make a living.

Most studies of mobile phone adoption in emerging markets, especially amongst the poor, show that social reasons - communicating with friends and family - come first, economic advantages come second.  And among both social and economic advantages, the ability to eliminate journeys, i.e. save time by not having to walk to the next village to ask a question, is extremely important.

I suppose I shouldn't be surprised.  At root, they are both communications technologies and our use of such technologies is driven by basic human needs that are remarkably similar, for everyone, everywhere. 

How tiered Internet pricing could actually facilitate P2P

Time Warner Cable's planned experiment with tiered charging for Internet access has generated a flurry of coverage in the blogsphere, but no new insights (at least that I've seen).

The primary problem ISP's complain about is that 5% of their customers use 90% of the available bandwidth and when they examine this traffic, it's mostly peer-to-peer file sharing.  A reasonable question is how to allow as much of this traffic as possible without increasing an ISP's variable costs or slowing down their other users.

This may not be as difficult as it appears.   Indeed if Internet access was as competitive as mobile telephony, we might already have seen what I'm about to propose — a combination of bundled pricing equivalent to mobile's "free nights and weekends" and "free on-net calls" with a way to facilitate P2P traffic that leverages exactly these "free" periods.

An ISP's costs

ISPs have some costs which are relatively fixed and others that are tied to usage.  A network is a relatively fixed cost and when it's not full, the incremental cost of adding traffic is zero!  This is the reason mobile operators give away free nights and weekend.  They've built their mobile network for the peak daytime traffic, so it costs them nothing to run promotions that add incremental traffic at off hours.  Peak hours and off hours may be different for an ISP, but the concept is the same. When a data pipe is lightly loaded the ISP's cost of adding incremental traffic is zero.

On the other hand, some ISP costs are usage based, for example "IP Transit" or more properly, Internet Transit.  This is the ISP's upstream cost to send and receive traffic to/from the rest of the Internet.  However, even here, usage-based costs occur at heavy usage.  Light usage periods don't save money.  To understand what's happening, it's worth a digression on Internet Transit.

Internet Transit

Internet access is monopoly or duopoly or a heavily regulated industry.  The middle mile connections from the local network to the Internet backbone may or may not be competitive depending on where you are.  But the Internet backbone itself is extremely competitive.  If you can get to a major Internet Exchange Point in the US or Europe, there are many providers offering extremely competitive rates for Internet Transit.  Typically these services are priced on a megabit per second per month basis (Mbit/s/Month) with lower rates for higher volume commitments.  The other key idea is that charges are based on the 95th percentile of all the five minute data rate samples taken during the month.  So an ISP can have a few bursts above their typical rate, as long as they represent less than 5% of the sampled intervals.

But this also means there is no extra cost to run at or near the typical rate at all times.

Local traffic

Even more important, if file sharing is done with other computers on the same ISP's network, then there is no need to pay for Internet Transit at all.  The question is how to figure out which potential peers are "on-net" and which are "off-net."

Sending signals to P2P software

Most P2P file sharing software has relatively little knowledge of locality.  Some P2P software practices "prefix awareness," for example, Joost gives preference to peers in the same /24 IP address block when they are available.  But if a major operator provided an automatic way for P2P client software to determine whether a prospective peer's IP address was currently reachable "for free", it seems likely the file sharing community would leap on it, and if there's money to be saved, active file sharers would download the new clients immediately.

A standard way to present such information might be via an extension to the XML-based response codes in one of the whois information exchange proposals, e.g. from ICANN or from APNIC.  Also, while what I'm proposing might start as a pricing plan rather like a mobile operator's "free nights and weekends" and "free on-net calling," it's not hard to see extensions where an ISP could offer dynamic access to underused capacity to those programs that were prepared regularly interrogate an ISP's server and use just the advertised off-hours capacity.

In closing

People liked fixed price deals.  Unlimited is great, but there's plenty of experience with bundles of minutes and the idea of data bundles has already showed up in 3G mobile data plans.  The combination of several tiered data bundle prices with the availability of "free" connectivity for "on-net" peers and during off peak intervals is likely to appeal to file sharers and produce better results for both the sponsoring ISPs and file sharers alike.

The Moral Instinct, cyberjustice and Internet law

There's an excellent article in today's NY Times Magazine, The Moral Instinct, in which Steven Pinker summarizes recent neuroscience research on the human moral sense.  This bears directly on how we think about justice and morality in cyberspace.

He covers the enormous progress (at least since I was in college) in our understanding of where the human moral sense comes from, including what is universal, what is cultural and how we can be easily mislead by "an aura of sanctity, distracting us from a more objective reckoning of the actions that make people suffer or flourish."

Illusions are a favorite tool of perception scientists for exposing the workings of the five senses, and of philosophers for shaking people out of the naïve belief that our minds give us a transparent window onto the world (since if our eyes can be fooled by an illusion, why should we trust them at other times?). Today, a new field is using illusions to unmask a sixth sense, the moral sense. Moral intuitions are being drawn out of people in the lab, on Web sites and in brain scanners, and are being explained with tools from game theory, neuroscience and evolutionary biology.

While the article doesn't mention the Internet, a better understanding of our moral sense can only help temper our approach to law and culture in the new social relationships facilitated by the Internet.  Depending upon the cultural setting, subjects as diverse as ethnic humor, pornography and religious discussion can be completely acceptable or morally outrageous and yet, except for some language effect, the Internet cuts across cultural boundaries.  Then there are Internet-related tragedies like the death of Megan Meier which provoke discussion and calls for sweeping laws to regulate cyber-behavior.

Undoubtedly, our laws and culture will evolve in response to the Internet age, but hopefully we do this with care, taking advantage of what we can learn from history and from the new science of the moral sense.

Here are a few short quotes from Pinker that struck me:

There are many other issues for which we are too quick to hit the moralization button and look for villains rather than bug fixes.

Our habit of moralizing problems, merging them with intuitions of purity and contamination, and resting content when we feel the right feelings, can get in the way of doing the right thing.

Far from debunking morality, then, the science of the moral sense can advance it, by allowing us to see through the illusions that evolution and culture have saddled us with and to focus on goals we can share and defend. As Anton Chekhov wrote, “Man will become better when you show him what he is like.”

If you are at all interested in morality, neuroscience or evolutionary biology, I highly recommend the Pinker article.

edushi.com – Possibly the best city maps in the world?

After my last post (on sanyamap.com), Tariq Mufti pointed me at an even better source of cities maps, also in China.  It's Edushi.com (roughly E-City in Chinese) and they currently cover 24 cities. The title of this post is copied from Dr Andrew Hudson-Smith who's written up edushi.com in some detail.

Edushi.com is owned by and uses technology developed by Hangzhou Aladdin Information & Technology Company.  Their English language description of the technology includes this:

Aladdin has first put forward the new concept of Internet application: Edushi(or rather, E-City) which is a corresponding result of the local governments’ encouragement of building “digitalized cities”. As a platform for 3-D emulation online-interaction, E-City is based in the WEB GIS and virtual-real technology.

"Digitalized city" integrates E-map, E-yellow page (city telephone directory), E-business, virtual community etc. functions.

"Edushi" allocates various information of the city based on geo information of the real city.

"Edushi" does not only emulate the architectural shape, geological status of the real city, but also vividly reproduce the entire city as well as the social activity and economical activity on Internet.

They also mention support for mobile phones, in-car navigation, a community information platform and a digital TV terminal application.

While I don't read Chinese, if you float your cursor over the list of cities at the top of this page, you can see their URLs in standard ASCII.  Using this approach I was able to go to the city view for Xian and navigate to some tourist sites that I have visited.

All and all, a fascinating combination of 3D maps, virtual reality, directories and current city culture.

China's Internet, at least as interesting as anything in the west

A common misunderstanding in the west suggests Internet entrepreneurs in China just copying whatever works in the west.  I'm sure that's true in some cases, but I've written in the past about peer-to-peer TV where Chinese companies were several years ahead of Joost and others in the west.

Today, I noticed a fascinating post by a blogger operating under the name Sun Bin.  He points to a mapping/ virtual earth website for Sanya, a resort city in Hainan province in southern China.  If you are proficient in Chinese,  try the website yourself.  Otherwise, grok this:

Yes, this is just one city so it's hardly on the scale of Google Earth or Microsoft Virtual Earth, but the user interface looks very interesting as it combines satellite photography, street maps, SimCity graphics and actual photographs of buildings, all in one beautiful graphic.

My point is not that Chinese entrepreneurs are ahead or behind, but that there are more Internet users in China than in the US, and the entrepreneurs that cater to them are smart and are doing interesting things that we seldom hear about in the English language blogsphere.

I follow a few bloggers, like Sun Bin and Gang Lu, who cover China in English (at least part of the time) and I try to maintain contacts with individuals I know in China, but it hardly makes up for not being able to read Chinese.

Rate of growth of traffic at the Amsterdam Internet Exchange

Here's a great historical graph, from page 28 here, showing the growth in traffic at the Amsterdam Internet Exchange.

By my take off from this graph, there was 1000x growth in nine years.  That means traffic doubles every 11 months.  While this is only one data point, it's new to me and fairly up-to-date.  It's also roughly consistent with the 12 month doubling rate mentioned in my earlier post.

Connect 2007 Madrid opens with panel on the mobile industry

The session was entirely Q&A (no slides) which resulted in a great discussion – broad ranging and much better than talking heads reading slides!  Luca has already written up his reactions.

I'm writing this after the fact as, at the last minute, I was tagged to participate, filling in for Vincenz Wagner of Jamba who's arrival has been delayed.  The opening session at Connect 2007 in Madrid was entitled "Industry Overview" with Joel Hughes, VP & GM of our Mobile Applications business moderating.  In the end the panelists were:

• Philip Kelley, Director, Mobile TV Standardization, Alcatel-Lucent
• Kari Lahtinen, Business Development Manager, Elisa Corporation
• Peter Karney, Senior Technical Marketing Manager, NEC
• Brough Turner, i.e. yours truly :-)

Since I was participating, I have only a few interesting items (at least interesting to me) that I noted during this discussion:

Philip commented that the predominate use of 3G is to connect PCs to the Internet, i.e. dumb pipe mobile Internet access.  Philip also mentioned Triple Play, which seems old hat to me, as innovative in (parts of) Europe.  I argued that this was marketing innovation (bundling), not really a new service.  We agreed the innovation was in cost and convenience of the services.

Several panelists seem to think that innovation would come by porting Internet applications to the mobile space.  I argued that was currently true, but only because the Internet was open and mobile was still closed.  There are many characteristics of mobile (like mobility and intimacy) that will foster new applications but we don't yet have the open environment that allows zillions of developers to experiment.

At one point, I made a derogatory comment about most VoIP being just digital POTS.  The ensuing discussion brought out the parallel between Skype (which combines voice and IM) with what's happened in mobile telephony, i.e. the combination of voice and SMS to achieve the same objectives.  Either way, people want to determine the actual availability of the person they are about to call and people need a way to communicate when they can't talk.

The day the routers died (to music)

The RIPE 55 meeting in Amsterdam has ended and Raindeer just uploaded this summary to his blog Lunatic Thought:

"The RIPE 55 meeting has just concluded. There was much debate on what to do on the imminent depletion of the unallocated IPv4 pool in 2010. We could do nothing or we could create a market place and facilitate transfer of IP-adresses, but it's all a train wreck waiting to happen. This is best shown however by a beautiful song "The day the routers died" also available on Youtube written and performed by Gary Feldman. So please all upgrade to IPv6 soon, or else you will not get 40Gbit/s to your mother."

and points to this wonderfully funny video:

Internet Connectivity in Guilin, China — Great China Firewall, plus engineering & capacity issues

I'm in Guilin China for the Connect 2007 Asia conference.  I've got the Tor bundle (Tor & Privoxy & Vidalia) installed on my laptop so I can get around the Great China Firewall (GCF) and that works well.  Tor starts up automatically when my PC starts, but I only use it for browsing (by clicking the Tor button for Firefox) if I can't reach a site directly.  Yes, it's noticeably slower going through Tor, but it always works.  For example, just now, seeking the URLs to paste in about the GCF, I could get Rebecca MacKinnon's post directly, but timed out on this from Gigaom.  I clicked the Tor button, hit retry and the Gigaom post loaded.  So, the Tor bundle is a good setup which I highly recommend if you are traveling to China.

The bigger problem for me at the moment is basic throughput.  The Sheraton Hotel Guilin (where the conference is being held) has wired Internet connectivity in the rooms for 50 Yuan ($6.65) per day and free WiFi in the lobby, but only WiFi from China Mobile (CMCC) in the conference area (100 Yuan per day through the hotel business center for those of us without a China Mobile phone number).

Performance is roughly equivalent for all three connections.  Early in the morning or late at night (perhaps during the night, but I do have to sleep sometime), everything works well.  It's not the blazing speed I sometimes see in Beijing, but it works. Here are the results in my room at 7:37am against a server in Shenzhen (300 miles away):

and here's a test against a server in Los Angeles:

Unfortunately, by 10am things begin to slow down and by noon yesterday, I was seeing greater than 50% packet loss and it became impossible to blog the conference live.  I was using the China Mobile WiFi but going up to my room didn't help.

My guess:  There's a significant capacity problem between Guilin and the rest of China.  Not surprising I suppose, considering Guilin is a tourist city, not a business city.

Mashups: Web Meets Telco

Joel Hughes, General Manager for Mobile Applications at NMS, is moderating the second to last session at Connect 2007 entitled "Mashups:  Web Meets Telco" with panelists:

Matt Gross, Director of Product Management, WHERE by uLocate Communications -- a GPS-based platform for location aware services.  Other value they bring is that they have negotiated deals with several US operators (Alltel and Sprint) and they make these services available to 3rd parties.

Alan Quayle, Consultant, Business and Service Development -- talking his consulting practice and refers to British Telecom's Web21c as a sample application he's been involved with.

Kevin Nethercott, President and COO, LignUp -- talking about their web platform to program PBX functions and do enterprise mashups, specifically mentioning Mashup Camps and applications prototypes done in 15 minutes.

Sam Aparicio, CTO, Angel.com -- an IVR and call center solution that exposes a web API for control.  His example application is a mashup with Salesforce.com.

This real time blogging is new for me.  Perhaps the resulting notes will only be useful to me...  In any event, this is going to be more stream of consciousness...

First impression: there are several sources of telephony controlled by web interfaces, but they all cost money or are just sandboxes.  No one has mentioned APIs that let developers also participate in billing, or ways that web telephony can be available at or near free.  Surely one or the other is needed to see this industry take off.

For now, all the discussion is about enterprise developers.  Mashups are a new way for enterprises to develop their IVR and call center solutions.  So far, no examples of developers targeting consumers.  Alan has mentioned that BT's API has also been used for Salesforce.com integration.

Sam just alluded to a Verizon API.  I'm not sure what he's referring to.  I'll try and ask afterward...

Now Alan is discussing cost.  When location data cost 25 pence per inquiry, no one used it.  No one has a good suggestion of how these kind of services should be priced.  What about an application that needs two database dips, an international phone call and two international SMSs?  General agreement that this is an issue, but no specific suggestions or pointers to solutions.

The popular APIs to mashup with (besides Salesforce.com) are Google Maps, Flickr, Facebook and Amazon (for storage), at least today.

Now Joel has explicitly asked how things are priced.  Amazon is priced by the GByte per month and Salesforce.com is by user per month.  Enterprise telecom groups either buy the platform or pay per transaction until they buy the platform.  Their issues are reliability and minimizing costs.

Matt Gross describes the value uLocate brings to Sprint is to handle the long tail of developers that Sprint can't deal with directly.

Alan says BT's Web21c has helped expose issues and challenges.  They are still experimenting.  As far as location goes, the pricing is still too steep.  Alan expects that Google and others will note the GPS coordinates of every cell site, retrieve that from the handset and skip interfacing with the operator.  Operators will have to price their location info appropriately or they will be bypassed.  Consumers do seem willing to pay for safety-related location-based services.

Relation of mashups with IMS.  North American operators are adopting IMS to control VoIP telephony, but IMS is just the base for connection control.  The broader service delivery platform is less standard and yet that's what's required as the platform that supports web services.  SIP-based applications going a lot slower than web APIs.  In the end it's applications that matter.  No one on this panel cares about IMS except as an underlying layer that supports their web APIs.

A question from the audience: how do you see the handset interfaces w.r.t. mashups.  Kevin is focused on voice telephony.  He doesn't see the mobile handset as the application UI for now.  In the long term, he expects browser-based user interfaces, but for now, he's doing voice.  Alan seems to agree.  Handsets are so different that all you can rely on is basic voice and SMS.  Sam complains that even the so-called open API phones, you can't actually to deep features on the handset.  This crew is focused on voice.

Navizon is building the wireless covereage map I asked for

Last week I lamented the sketchiness of most wireless signal coverage maps and went on to suggest someone should "Use handset software to build mobile coverage maps."  Now Shai Berger has pointed out someone who's already doing just that.  It's Navizon from Mexens Technology.  Their focus is wireless positioning, i.e. determining your location, but their approach includes downloadable software for smartphones, PDAs and handheld PCs.

Mexens is the provider of the first positioning system that combines WiFi, Cellular and GPS information. The Wireless Positioning System triangulates signals broadcasted from Wi-Fi access points and Cellular towers to help users find their way in most major metropolitan areas worldwide.
                  
It is based on a collaborative network called Navizon. This is why we like to describe it as a peer-to-peer positioning system. Our users are sharing information about the position of WiFi access points and cellular towers, and by doing so, they keep our map of the Wireless landscape up-to-date.

And they offer such information in the form of a Google Maps mashup!  Here's their take on mobile cell sites in a western part of Newton, Massachusetts USA.

With software on the WiFi enabled mobile devices they can capture every WiFi signal any of their contributors pass.  Here's their map of WiFi signals on the northern edge of Natick Massachusetts.  Red are secure access points and green are unsecure, i.e. open, access points.

Clearly, Navizon participants have driven through the area, but no one has gone into the residential neighborhoods, as I'm confident this community is riddled with broadband users and residential access points.

While the website is in English, coverage is global.  Here are 200 of the more than 37K signals they've mapped in this part of London:

And 200 of more than 87K signals mapped in western Paris:

This is fascinating!  But I have other things to do now, so I'll print out their 20 page white paper to read at a later point.

Following more feeds in less time — BlogRovR

I'm a feed junkie.  Perhaps not as bad a some, but at one point my Bloglines account peaked at over 250 feeds.  Meanwhile, I keep stumbling on additional feeds I wish I could follow, but there's just no time.  Long ago I established a "Probation" folder for feeds I might want to follow and a "Too busy" folder for feeds like Slashdot where even scanning headlines is a problem.  Gradually more and more feeds ended up in "Too busy" or just unsubscribed. 

Now I've found an interesting solution — BlogRovr from Stickis.  They advertise themselves as:

the perfect companion for blog reading. Tell Rovr which blogs you like.  From then on, as you browse, Rovr fetches from them stories that link to the what you're browsing and shows them to you right on the page.

This happens via a tiny tray that briefly slides in from the right when you hit a new page (blog or otherwise) for which there is related content in any of your BlogRovR subscription. See the example on the right.

After two weeks of use, my BlogRovr subscriptions now include not only the feeds I subscribe to in Bloglines, but every other RSS feed I've stumbled on that looks even mildly interesting.  The net effect is that, if a mildly interesting blog covers a subject I otherwise happen read about, I find out about the related article(s).

I'm now removing overlapping coverage from my Bloglines subscription.  Who needs to see everything from all top 30 VoIP bloggers when most (certainly the top ten) present slightly different shades of opinion on the same set of stories?

I now follow the busiest sites in headlines-only mode and rely on BlogRovR to point out related articles.

So far, this appears to be a winning combination!

Use handset software to build mobile covereage maps

Over the past few weeks, while researching other subjects, I've stumbled on multiple websites providing mobile signal coverage maps.  Two days ago, I hit an excellent HSDPA vs EVDO coverage map for the US that I can't seem to locate right now.  :-)   Today, I stumbled on Signal Map, a very interesting US centric site that aggregates user experiences.

Earlier I had bookmarked Cell Phone Tower Search, a site based on the FCC's database of US towers, and Sitefinder, an official Ofcom site mapping UK basestations.

I know I have seen other sites that, like Signal Map, attempt to aggregate actual user experiences, for example, Dead Cell Zones.  But I haven't seen anyone who's taken the next logical step in acquiring accurate user data.

Why not offer mobile phone users some downloadable handset software that automatically captures as much information as possible and sends it to a common site like Signal Map?  Yes, that puts a cost on the subscriber who agrees to participate, but I know my mobile plan includes a bundle of SMS messages that I never exceed and a data plan that I don't typically exceed.  If I could configure how many SMS message were sent or how much data went out per month, I'd be happy to participate in such a scheme.

This approach may be problematic in areas where SMS and/or data is expensive, but with generous bundles and the (relatively) low cost of mobile service in the US, it seems likely this scheme could work.  And, one such phone might generate 20-50 reports per month, much more user created content that you are likely to get by asking users to type information into your website.

To the folks at Signal Map, what do you think?

Build it and they will come - Railroads in the 19th century

While I was in Asia my wife picked up an 1860 original leather bound copy of "The Story of the Life of George Stephenson" by Samuel Smiles.  If you've never heard of George Stephenson, he's known for The Stockton and Darlington Railway which opened on the 27th of September 1825, The Liverpool and Manchester Railway which opened on the 15th of September 1830 and, with his son Robert Stephenson, for the engine "The Rocket" which won the first major locomotive contest in October 1830, also setting a speed record (29 mile per hour).

The Stockton and Darlington was designed to carry coal to a port at Stockton-on-Tees.  To quote Smiles,

At first passengers were not thought of; and it was only while the works were in progress that the starting of a passenger coach was seriously contemplated.  The number of persons traveling between the two towns was small and it was not known whether these would risk their persons upon an iron road. <...>

No sooner did the coal and merchandise trains begin to run regularly upon the line than new business relations sprang up between Stockton and Darlington and there were many more persons who found occasion to travel between the two towns — merchandise and mineral traffic invariably stimulating, if not calling into existence, an entirely new traffic in passengers.

In 1829 the line was extended to new docks five miles down the river where a new town (Middlesborough) grew from nothing to more than 6000 residents within ten years.

For the much more ambitious Liverpool and Manchester, the promoters based their calculations on heavy merchandise traffic (coal, cotton and timber) but also thought that they might be able to capture half of the 400 people per day carried by coaches between the two cities.  Again quoting Smiles,

But the railway was scarcely open before it carried on an average about 1200 passengers per day.

These were heady results that lead in due course to waves of investment (peaking in 1836-37 and in 1846) producing bubbles not unlike our Internet bubble. But despite investment bubbles, and busts, traffic continued to grow throughout the 19th century, literally transforming society. The parallels with the Internet are interesting.  Andrew Odlyzko tells the investments part of the story in section 4 of this article.

What the quotes from Smiles should suggest is we're unlikely to guess what form future advances based on the Internet will take.  Broadband Internet access is important for it's option value, not for the applications we run today.

Put another way, while I love railroads, the Internet's long term benefit for humanity is more significant — likely to rank up there with the introductions of speech, writing and the printing press.

Power laws - 63 words account for half of everything said on TV

More specifically, an analysis of word frequencies done on 29,213,800 words, taken from TV and Movie scripts and transcripts that were available on the Internet in February 2006, shows that just 63 words account for 50% of the 29,213,800 words collected.

That's amusing.  What's more interesting is my source, the article A practical model for analyzing long tails by Kalevi Kilkki, Principal Scientist, Nokia Siemens Networks, in the May issue of First Monday.

I'm a fan of Chris Anderson, have read his article, skimmed his book and I have commented on the subject before. Kilkki's article is less anecdotal and more mathematical grounded than Chris Anderson (without being inaccessible).

Plus Kalevi Kilkki's got tons of interesting graphs.  :-)

Telephony Web Services

I first mentioned telephony web services in a post I wrote after last fall's Connect 2006 conferences.  That post generated some comments and private email discussions that lead to an article in my monthly column in the March issue of Internet Telephony magazine.

As planning for Connect 2007 is well underway, I'm looking at telephony web services again and I realize I forgot to point to my article.  So, for those of you who don't read the hard copy magazine and don't scan the TMC website (they don't provide RSS) searching for my monthly column  :-),  it's here and still very relevant.  Some excerpts:

This is a nascent field. Traditional fixed and mobile operators still run highly constrained voice networks. Yes, you can have your computer place and receive calls, but if you want to overlay a private numbering scheme for use between phones in your far-flung offices (a voice VPN) — such services are only available from your operator. Likewise, if you want to integrate your customer care software with call center functionality, you start with PBX technology, as public networks are not open. Even access to your own call history requires scraping data from itemized bills.

Does the new world of VoIP do any better? Not much. Major VoIP operators, like Vonage, use VoIP to reproduce traditional fixed-line services — basically “digital POTS” — no open APIs there. Skype offers PC-based APIs for the Skype client and you could argue that’s all that makes sense for a peer-to-peer service. There are no central services to expose through web service APIs. But the test is what can you do with their APIs and the answer is client extensions. Since Skype clients run as a single instance per PC, you can’t easily implement a multi-channel back-to-back user agent (B2BUA) and a VPN.

There's a lot more in the article including mention of specific companies, LignUp Corporation, Abbeynet S.p.A., Ubiquity Software and Angel.com.

We should have meaningful updates at this fall's Connect 2007 conferences in a session tentatively titled "Mashups:  Web Meets Telco." 

----------------------------------------------------
----------------------------------------------------

Connect 2007, hosted by NMS, is a series of global conferences that target key business and technical decision makers who are interested in learning about the vital drivers shaping the communications ecosystem — the business issues, the market drivers, the technology challenges, the compelling applications, and the end user demands. This year events take place in Boston, Madrid and Guilin, China.

Connect Americas	October 2–3	Boston, MA, USA
Connect Asia	October 17–18	Guilin, China
Connect EMEA	November 7–8	Madrid, Spain

Click here to sign up for future information updates on our 2007 Connect Conferences.

Dopplr - Looking for an invitation to the beta

Over the weekend, I noticed several people commenting on Dopplr, a new service (still in an invitation only beta) which:

... lets you share your future travel plans with a group of trusted fellow travellers whom you have chosen. It also reminds you of friends and colleagues who live in the cities you're planning to visit. You can use the service with your personal computer and mobile phone.

If you have an invitation to share, I'm interested.  rbt@nmss.com
Thanks.

F2C : Day 2 demos

Yuval Klein of Plymedia gave a cool demo.  Plymedia allows you to annotate any video on the net.  It works by providing an overlay, not by modifying the original video.  In that sense it's rather like the Chinese service Mojiti.  I assume there are others doing this as well, but I can't guess which will succeed.  In any event, the Plymedia demo is very cute. ),  It's based on Flash.

What's after YouTube? Annotations — Mashups with anyone's video

China Internet users have reached some kind of critical mass, as interesting Web 2.0 applications are emerging.  Gang Lu at Mobinode describes Mojiti, which makes it easy for anyone to annotate a video — any video!  Look at the list of services they support.  One wonders when the copyright police will come after them — and what, if anything, could be done about a Chinese site.

Quoting from Gang Lu's post,

With Mojiti, you can tell your own stories inside online videos. Users can add text, hyperlinks, shapes, thought and speech bubbles, images and more anywhere on the video screen to narrate their own videos, subtitle videos with any language, or just comment on any scene. Its technology works by adding an overlay on top of video from all major video sharing websites (such as YouTube, Metacafe, Google Video and more) and in all major video formats (Flash, Windows Media, QuickTime). Users annotate on this overlay and Mojiti synchronize their annotations to the underlying video.

Most important for me, Mojiti is catering to western audiences with English as well as Chinese sites.  YouTube scoped the market for first generation video sharing communities, but Mojiti is poised to scope the market for video mashups!

When you consider that China already has multiple competing peer-to-peer TV services, some running for several years now, while we have only Joost (formerly the Venice Project), still an invitation only beta, it's clear the Chinese Internet community is as close to the bleeding edge as any in the west.

Mobile phones vs. PCs to access the Internet - Crossover in 2007-08

As I catch up on back reading, I came on Tomi Ahonen's "Putting 2.7 billion in context: Mobile phone users".  It's a long but interesting article, worth printing out and reading in detail (something I can't do in the airport lounge) as it puts mobile phones in context:

...context. 800 million cars, 850 million personal computers, 1.3 B fixed landline phones, 1.4 billion credit cards, 1.5 billion TV sets. How many mobile phones in use today? In use today, yes, 2.7 billion (technically 2.7 billion in January, not December). They sold 950 million phones last year and the total worldwide mobile subscriber base grew from 2.1 billion to 2.7 billion. Three times as many mobile phones as automobiles or personal computers. About twice as many mobile phone owners as those of fixed landline phones or credit cards. And almost twice as many mobile phones in use as TV sets.

Yes, that 2.7 billion subscriptions includes duplicates.  There may be 2.4 to 2.5 billion discrete users, but the context is clear.  More importantly for the Web 2.0 crowd to think about:

During 2007 the first cross-over will happen, with more users accessing via phone than PC. Fascinating data coming on that usage as well, the Japanese regulator reports that those who access the web via mobile phone do so more frequently than those who access via a PC. Similar data now coming from several converged (broadband and mobile phone) web services like Flirtomatic in the UK. Its no wonder Google's new CEO Eric Schmidt, says the future of the internet is mobile.

So pay attention to Internet and mobile phone usage in Japan, Korea and China as there, mobile phones are already the dominate form of Internet access.

Under estimating future trends - like bandwidth requirements or mobile adoption

James Enck questions many of the estimates of future consumer bandwidth requirements, for example

BSkyB has publicly stated a working assumption that end-user bandwidth demand may double every five years.

and others, leading to:

I am uncomfortably reminded of other previous industry predictions which proved wide of the mark - "no one will ever need or want their own computer," "SMS has no potential as a consumer service," "mobile penetration will peak at 20%," "the public internet will never support acceptable quality for voice," etc.

I agree with James and would go further to suggest that doubling in five years is completely ludicrous — representing just 15% increase per year.  When I look at my personal, residential data connection, my capacity has grown from a symmetric 1200 bps Hayes modem in 1982, to an asymmetric 15 Mbps/ 2 Mbps Verizon FiOS service installed in the summer of 2005.  That's a 23 year history in which my download bandwidth grew 51% per year.  Even my lame 2 Mbps upstream service represents a 38% per year growth over the past 23 years.  Why should the next five year's growth drop to 15% per year?

Interestingly I was looking at a similar estimation error in a different field just a few hours ago — projected growth in the number of mobile subscribers for the next four years.  The particular report I was looking at was from one year ago.  I won't embarrass the firm, but they are globally recognized telecoms market analysts.  They estimated there were 2.1 billion mobile subscribers in the world as of December 2005 and they projected there would be 642 million new subscribers added by December 2009.  This is nuts!

According to their own estimates (confirmed by ITU data and other sources) there were 343 million new subscribers added in 2004 and 345 million added in 2005.  Why would the rate of additions suddenly drop to 160 million per year? or did they expect the rate of growth to tapper off to near zero in four years?  What were they thinking?

I don't know the answer for under-estimating bandwidth requirements, but I have a theory about under-estimating mobile phone adoption.  People don't believe Moore's law will continue.  If you expect mobile phone infrastructure will always cost what it cost in 2005, and people will never get wealthier (despite the positive economic benefits of mobile phones), then perhaps you could conclude there's a near term limit on how many people can afford mobile phone service. 

But Moore's law does apply!  Every year the capital cost to build mobile networks declines and every year the cost of low-end handsets goes down — in both cases, despite the ever increasing functionality of the mobile networks and handsets.  More than that, countries with growing mobile phone adoption also have growing economic prosperity, so more people can afford the every lower cost service.

Indeed, all indications are that we added more mobile subscribers in 2006 than in any previous year.  Even better for me personally, rumors are that Verizon is about to up the bandwidth limits on my current service from 15/2 Mbps to 20/5 Mbps.  Bring it on!

Asian submarine fiber route detail

James Seng has an excellent illustrated post showing submarine fiber route in Asia and providing an expert explanation of the impact of last Tuesday's magnitude 7.2 earthquake off the coast of Taiwan.

If you are at all interested in submarine fiber, the Internet backbone or the Internet in Asia, this is the article to read.  It beats anything else I've seen in the press.

Wonderful mobile Internet quotes show how far '3' has come

In a joint Yankee Group Trendsmedia webinar this morning, Caroline Gabriel (Research Director at Rethink Research) had some wonderful quotes:

"We see absolutely no demand for open Internet access on the mobile phone," Canning Fok, Group Managing Director, Hutchison Whampoa, 2004

"This is the Internet as it was meant to be and what people have been waiting for.  Mobile broadband is the natural next step for mobile services, extending the full power of the Internet to mobile handsets."  Canning Fok, Group Managing Director, Hutchison Whampoa, November 2006.

The second quote was part of Hutchison 3's recent launch of their X-Series flat rate open access mobile Internet service which has been widely discussed in the blogsphere.

Learning without teachers

Thanks to Aaditeshwar Seth pointing, on the TIER list at Berkeley, to this wonderful article about minimally invasive education as practiced by New Delhi physicist Sugata Mitra.

Sugata Mitra mounted a PC with a broadband Internet connection in a wall in a slum area and waited to see what would happen.  The best response was from getto kids ages 6-12, despite their rudimentary education and little knowledge of English.  Indeed, it appears by the time people at 16 or so, they come to believe they can only learn with teachers, while at earlier ages humans can learn anything!

The experiment wasn't without some tiny "teaching" but it certainly was minimally invasive as this excerpt from the interview relates:

The children create their own metaphors to do this. To give you an idea of what I mean, a journalist came up to one of these kids and asked him, "How do you know so much about computers?" The answer seemed very strange to her because the kid said, "What's a computer?" The terminology is not as important as the metaphor. If they've got the idea of how a mouse works and that the Internet is [like a wall they can paint on], who cares if they know that a computer is called a computer and a mouse is called a mouse? In most of our classes here at NIIT, we spend time teaching people the terminology and such. That seems irrelevant to me with these children.

But we also found that they would tend to plateau out. They would surf the Web -- Disney.com is very popular with them because they like games. And they would use [Microsoft] Paint. It's very, very popular with all of them.

Because these are deprived children who do not have easy access to paper and paint. Every child likes to paint, so they would do it with that program.  However, that's all they could do. So I intervened, and I played an MP3 [digital-music file] for them. They were astonished to hear music come out of the computer for the first time. They said, "Oh, does it work like a TV or radio?" I said, in keeping with my approach, "Well, I know how to get there but I don't know how it works." Then I [left].

... seven days later they could have taught me a few things about MP3. They had discovered what MP3 was, downloaded free players, and were playing their favorite songs. As usual, they didn't know what any of it was called. But they would say, "if you take this little box, and you drag this file into this box, it plays music." They had found out where all the Hindi music was on the Web and had pulled it out.

He also found adults asking these kids to do things for them on the computer.  And, remember, in only ten years these kids will be adults. All and all a very interesting article.  If you are at all interested in the spread of computer literacy, you should read this article.

Wikipedia - The social network is only part of the story

One of the wonderful phenomena of the past few years has been the emergence of Wikipedia as an incredible reference for just about anything.  If you've heard Wikipedia's founder Jimmy Wales speak (for example watch this 20 minute video from July 2005), you've also heard about Wikipedia's vibrant community, how 50% of the edits are done by just over 500 people and 75% of the edits are done by less than 1500 people.

It appears these statistics, while true, are very misleading!

Aaron Swartz has looked at who actually provides the most content and it's very different from who makes the edits.  First Aaron was just looking at specific articles, initially the article on Alan Alda.  But then he wrote a program to examine the Wikipedia archives and extract statistics.  He's written a very comprehensive article, but in summary:

If you just count edits, it appears the biggest contributors to the Alan Alda article (7 of the top 10) are registered users who (all but 2) have made thousands of edits to the site. Indeed, #4 has made over 7,000 edits while #7 has over 25,000. In other words, if you use Wales's methods, you get Wales's results: most of the content seems to be written by heavy editors.

But when you count letters, the picture dramatically changes: few of the contributors (2 out of the top 10) are even registered and most (6 out of the top 10) have made less than 25 edits to the entire site. In fact, #9 has made exactly one edit -- this one! With the more reasonable metric -- indeed, the one Wales himself said he planned to use in the next revision of his study -- the result completely reverses.
...

When you put it all together, the story become clear: an outsider makes one edit to add a chunk of information, then insiders make several edits tweaking and reformatting it. In addition, insiders rack up thousands of edits doing things like changing the name of a category across the entire site -- the kind of thing only insiders deeply care about. As a result, insiders account for the vast majority of the edits. But it's the outsiders who provide nearly all of the content.

Aaron's a smart guy.  As a teenager he co-authored RSS 1.0, worked on the W3C's RDF 1.0 Working Group and wrote RFC 3870.  Then he went to Stanford for a year before dropping out to become an entrepreneur.  I'm impressed he has the time to work on Wikipedia.  He's currently running for a seat on the Wikimedia Foundation Board.

Of course his article raises all sorts of governance issues for the Wikimedia Foundation.  Today, you have to have made over 400 edits in order to qualify to vote for members of the board of Wikimedia Foundation, i.e. only the editors can vote, not the folks who are actually contributing most of the content.  And not me whose contributions are very minor.  What's critical is that it remains easy and rewarding for stray individuals to add content.  I certainly hope the Wikipedia inner circle takes note of Aaron's data.

Internet-scale database - Google's Bigtable

There's a very interesting preprint of a paper from Google Research due to be presented at OSDI '06 in November.  It's Bigtable: A Distributed Storage System for Structured Data by Chang et al. (PDF here). 

Bigtable is a database that runs on top of the Google File System.  I first heard of it in the May '06 interview of Google's Jeff Dean by O'Reilly's Radar. Like the databases supporting other major Internet applications, it's not a straight relational database.  This paper gives the details for those of you interested in such things.

It also includes the following gee whiz items:

• The Google web crawl creates two tables totally 850 terabytes of data. 
• Google Analytics apparently keeps all clicks, forever.  It's currently using two tables totally 220 terabytes.
  Hmm..., if they really keep everything forever, this could end up larger than their web crawl.
• The maps for Google Earth occupy a mere 70 terabytes.

Optimism

This comment by Brian Thomas in Tom Evslin's Fractals of Change is a nice summary of my view of the Internet:

My personal belief here is that given equal exposure, the good generally wins over the bad, and better communication makes it easier for the exposure to be equal, so even though the Internet enables many new ways to degrade, defraud, and deceive people, it also provides new ways to expose evils.

There are many historical examples of this, from the invention of the printing press, then movable type, then telegraphy, telephony, radio, and finally the Internet. Gutenberg did more to break the power of medieval craft guilds and religious authorities, both of whom wanted to keep their knowledge secret.

Power law vs log-normal with really big bucks implications

Yesterday's post was rather rushed as I was about to get on an airplane.  Once on board, it occurred to me, there's another book I've read where the difference between log-normal and power law distributions is key.  That's The Misbehavior of Markets by Benoit Mandelbrot & Richard Hudson.  They argue modeling markets with normal distributions (in price changes, for example) is erroneous and leads economists to underestimate the likelihood of extreme events (like Black Monday in October 1987).

Put another way, small price changes occur very frequently and large price changes very infrequently.  If you plot these on a log-log scale you get a straight line out some distance. Conventional finance assumes this fits a log-normal curve which eventually falls off.  Mandelbrot, argues historical price data fits a fractal or scale-free model, i.e. follows a power law, meaning the likelihood of extreme events has been consistently underestimated.

Those of you in high tech in the US may have followed the accounting change which requires stock options be expensed by the company issuing them.  To do this one must be able to estimate the present value of the options are the time they are granted.  This is done using the Black–Scholes formula.  While the authors received the 1997 Nobel prize in economics (Scholes & Merton; Black had died in 1995) and the Black-Scholes formula is now written into accounting practice, Mandelbrot argues it consistently underestimates infrequent cases of extreme price volatility. 

As an interesting aside, Long Term Capital Management (LTCM), a hedge fund with Scholes & Merton on it's board, collapsed in 1998 due to a series of correlated repricing events that were extremely unlikely, but did occur in the late summer 1998.  LTCM lost $4.6B.

Power laws and log normal distributions appear in many disciplines, not just networking and markets.  In many cases it's hard to determine which applies.  Whether Mandelbrot is correct about power laws and finance, at least we can agree -- $4.6B is big bucks!

Scale free networks. long tails and big bucks - or maybe not

Some of my blog posts serve to organize a set of references on a topic that's important to me.  This is such an occasion -- hopefully one or two others may share this interest.

Back in 2002, I really enjoyed Albert-Laszlo Barabasi's book on networks, Linked: How Everything Is Connected to Everything Else and What It Means, but haven't paid much attention to scale-free networks since then.  Yesterday I stumbled on this interesting presentation by Chris Anderson at "SciFoo" last week, which prompted me to revisit the subject.  The notes in Anderson's presentation include two great references (Limpert et al., Mitzenmacher) for the mathematically inclined, and the current Wikipedia article on Scale-free networks has excellent content and a lot more references.

New to me, are the diverse models that generate networks which are scale-free but differ in other meaningful characteristics.  Barabasi discussed preferential attachment, a model which generates scale-free networks with highly connected nodes near the core.  These networks are robust even if a large fraction of the nodes are removed.  However the Internet, while a scale-free network, has it's highly connected nodes closer to the periphery, so the preferential attachment model doesn't apply.  And as Wikipedia currently puts it,

Indeed, many of the results about scale-free networks have been claimed to apply to the Internet, but are disputed by Internet researchers and engineers.

The "big bucks" part of my headline was prompted by Chris Anderson's The Long Tail: Why the Future of Business Is Selling Less of More.  Anderson's thesis, originally published in Wired magazine, is that the popularity of items in certain markets, like media, follows a power law -- the same law that applies to the connectivity of nodes in a scale-free network!  In such a market, a few products have enormous sales but there is a very long tail of products that have only a few sales each.  This long tail goes way out, so taken together, sales on the long tail represent big bucks.

Since Anderson's book was published there's been considerable argument about the long tail.  Nicholas Carr has good summary of the argument between Wall Street Journal columnist Lee Gomer and Chris Anderson.  But Andrew Odlyzko (on a private economist list, quoted here) has the best overall answer to the questions Lee Gomer raised.

As with the Internet, the key question is: does the phenomenon follow a power law (as scale free networks do)?  If however, it follows a log-normal distribution, there's a lot less money to be made selling products out on the long tail as illustrated by this slide in Anderson's presentation:

As Anderson puts it:

But here’s the problem. I’ve been assuming that the powerlaw is the “natural” shape of all these markets. But there are other distributions that start off straight on a log-log chart, then fall off the line not necessarily because of a bottleneck effect but because they’re simply not powerlaws. The best example of this is the “lognormal” distribution, which often stays straight for several orders of magnitude before sloping downward. How to tell whether a market that falls off the line is a natural powerlaw shape distorted by a removable bottleneck or a natural lognormal market that will look like that no matter what you do? In markets such as film or television archives, this could be a billion-dollar question.

Indeed!

Organizing the web - a 3rd approach

For at least the past seven years, Tim Berners-Lee and the W3C have been advocating and working on the semantic web, a set of organizing principals for the web.  Some parts, like XML, are in wide use but the project as a whole has made little progress.

Google has the opposite approach - look at the mess that individuals have created, and try and makes sense of it.  This has worked extremely well for text documents, but not so well for other kinds of information.  The ensuing dispute has cropped up repeatedly, e.g. Google vs the semantic web or  million dollar markup versus million dollar code.

Today I saw an interesting post from Aaron Swartz (the co-author of RSS 1.0) in which he proposes a third path. It's a rather obvious idea - now that I've read Aaron's article.  :-) 

Both the code and the markup positions make the assumption that users will be publishing their own work on their own websites and thus we'll need some way of reconciling it. But Wikipedia points to a different model, where all the users come to one website, where the interface for inputting data in the proper format is clear and unambiguous, and the users can work together to resolve any conflicts that may come up.

The Wikipedia model.  It's clear that all three approaches can work and all will be used, but I wonder if there aren't many other domains, besides an encyclopedia, where the Wikipedia approach would take off. 

If you are at all interested in the organization of information and the Web, Aaron's post is well worth reading.